Remote YouTube downloader Slack bot

In this short tutorial, we will see how easy it is to receive webhooks directly inside your applications using Webhook Relay Socket Server. Our application will be a Node.js daemon process which listens for the webhooks and downloads videos that the user (you) requests. While the daemon can be written in any language, I chose JavaScript for this one since it’s very easy to read and most of the people have encountered it in the wild.

For the webhook producer, we will use Slack and their slash commands. Advantages of Slack slash commands:

• No need for Slack client SDK
• Works through webhooks that can easily be processed in any language
• No authentication token required (just needs a webhook endpoint where to send requests)

Problem

Webhooks are awesome but your application needs to be exposed to the internet to receive them. It must also have a web server that can process those webhooks. For a simple application or that doesn’t have to be always online, configuring NAT/firewall might be an overkill.

Solution

Webhook Relay + WebSocket client can let your applications receive webhooks from 3rd party services without having a public IP/domain, configuring NAT, and even the web server becomes unnecessary. You can process webhooks right inside your application.

Prerequisites

1. Webhook Relay account
2. Slack account and workspace
3. npm and Node.JS on your machine

Create a webhook forwarding bucket

We will be to create a bucket (buckets are used to group inputs/outputs in Webhook Relay) to capture and relay Slack webhooks. Go to buckets page and create a new bucket called slash. It will get a default public endpoint which we will be using in the next step. Create an internal output too, the destination doesn’t matter but it will help us with debugging:

Creating Slack slash command app

First things first, we will need an easy way to send a webhook. I initially thought about trying out Airtable and Google Sheets but was quite disappointed with the lack of webhooks in their services. Zapier seems to be trying to help a bit there, but their webhooks can only work every 15 minutes and even then I didn’t receive any webhooks.. :) So, another obvious and easy choice would be Slack slash commands.

Slash Commands let users trigger an interaction with your app directly from the message box in Slack.

Creating a Slash command actually is a lot more straightforward than you would expect, have a look in the official docs. Come up with a name and select a workspace:

Now, from ‘Add features and functionality’ select Slash Commands and fill in some details:

You can get creative with the command prefix /dl as our application doesn’t consume it, only the text after the command will be used inside the code. In the Request URL add your Webhook Relay input endpoint that is https://my.webhookrelay.com/v1/webhooks/..... Once you are happy with the details, save and install it:

Now, whenever we type /dl https://www.youtube.com/watch?v=tPEE9ZwTmy0 it will send a webhook to Webhook Relay input and from there we can relay it to our app. Try it out, it will be captured.

Analysing Slack command payload

Once we have got the test payload, we need to check what’s inside it. Unfortunatelly I couldn’t find a way to make it send JSON and so we will have to work with Content-Type: [ "application/x-www-form-urlencoded" ]. Webhook Relay helpfully parses the message:

token: gp9AyhCMqffRI2pahREQrg2S
team_id: T3QT2DM0Q
team_domain: webhookrelay
channel_id: C3RLQ5C4C
channel_name: general
user_id: U3S9BEU6B
user_name: karolis
command: /dl
text: https://www.youtube.com/watch?v=tPEE9ZwTmy0
response_url: https://hooks.slack.com/commands/T3QT2DM0Q/495886160947/J0YHv46ot6nZNeHiGR4a1I12
trigger_id: 495384852801.126920463024.5438c17b6d97870e3a05f145c6d4bc70 

So we now know what fields are we going to take:

• text - this is our URL to download
• response_url - this is where we can reply to the user

Creating our application

Application code can be found on Github repo, feel free to clone it. To make our life easier, some main libraries:

• https://github.com/przemyslawpluta/node-youtube-dl - download the videos.
• https://github.com/ljharb/qs - query string parses (helps us parse Slack payload).
• https://github.com/request/request - simple HTTP client. Even though I would always prefer using a stdlib, in this case it just worked better for me.

For readers, here’s the whole application:

// app.js
const WebSocket = require('ws');
var fs = require('fs');
var youtubedl = require('youtube-dl');
var crypto = require('crypto');
var qs = require('qs');
var request = require('request');

var server = 'wss://my.webhookrelay.com/v1/socket';
var reconnectInterval = 1000 * 3;
var ws;

var apiKey = process.env.RELAY_KEY;
var apiSecret = process.env.RELAY_SECRET;

var connect = function () {
    ws = new WebSocket(server);
    ws.on('open', function () {
        console.log('Connected, sending authentication request');
        ws.send(JSON.stringify({ action: 'auth', key: apiKey, secret: apiSecret }));
    });

    ws.on('message', function incoming(data) {        
        var msg = JSON.parse(data);
        if (msg.type === 'status' && msg.status === 'authenticated') {
            console.log('Authenticated, subscribing to the bucket...')
            ws.send(JSON.stringify({ action: 'subscribe', buckets: ['slash'] }));
            return
        }

        if (msg.type === 'webhook') {
            processWebhook(qs.parse(msg.body))
        }
    });
    ws.on('error', function () {
        console.log('socket error');
    });
    ws.on('close', function () {
        console.log('socket closed, reconnecting');
        setTimeout(connect, reconnectInterval);
    });
};

var processWebhook = function (payload) {
    console.log('URL: ', payload.text)
    var tempFilename = 'tmp-' + crypto.randomBytes(4).readUInt32LE(0);
    var actualFilename = ''

    var video = youtubedl(payload.text,
        ['--format=18'],
        { cwd: __dirname });
    // Will be called when the download starts.
    video.on('info', function (info) {
        console.log('Download started');
        console.log('Filename: ' + info._filename);
        // saving filename for the later rename
        actualFilename = info._filename;
        console.log('Size: ' + info.size);
    });
    video.pipe(fs.createWriteStream(tempFilename));
    video.on('end', function () {
        console.log('Finished downloading!');
        // renaming file to the actual video name from our temp one
        if (actualFilename !== '') {
            fs.rename(tempFilename, actualFilename, function (err) {
                if (err) console.log('ERROR: ' + err);
            });
        }
        // sending response back to Slack
        respond(payload, {
            response_type: 'in_channel',
            text: `${actualFilename} finished downloading!`
        })
    });
}

var respond = function (payload, result) {
    request.post(
        payload.response_url,
        { json: result },
        function (error, response, body) {
            if (!error && response.statusCode == 200) {
                console.log(body)
                return
            }
            console.log(error)
        }
    );
}

connect();

Install dependencies from https://github.com/webhookrelay/slack-slash-downloader/blob/master/package.json by:

npm install

Running the app

To launch it, retrieve the tokens from tokens page and set them as an environment variables:

export RELAY_KEY=your-token-key
export RELAY_SECRET=your-token-secret

and then:

node app.js

Once it’s running, you can start downloading videos via Slack, just type:

/dl https://www.youtube.com/watch?v=H_4eRD8aegk

App logs should be similar to this:

URL:  https://www.youtube.com/watch?v=H_4eRD8aegk
Download started
Filename: justforfunc #43 - Migrating Go Modules to v2+-H_4eRD8aegk.mp4
Size: 61705357
Finished downloading!
ok

Your file should appear in the same directory as the application:

➜  slack-slash-downloader git:(master) ✗ ls -alh
total 97M
drwxrwxr-x  4 karolis karolis 4.0K Dec  5 23:37  .
drwxr-xr-x 12 karolis karolis 4.0K Dec  4 10:40  ..
-rw-rw-r--  1 karolis karolis 2.8K Dec  4 23:11  app.js
drwxrwxr-x  8 karolis karolis 4.0K Dec 12 19:39  .git
-rw-rw-r--  1 karolis karolis   27 Dec  4 23:11  .gitignore
-rw-rw-r--  1 karolis karolis  59M Dec  5 23:37 'justforfunc #43 - Migrating Go Modules to v2+-H_4eRD8aegk.mp4'
drwxrwxr-x 61 karolis karolis 4.0K Dec  4 12:07  node_modules
-rw-rw-r--  1 karolis karolis  38M Dec  4 11:29 'Overwatch Winter Wonderland Details Leaks, Skins and More!--b-DktHyWJk.mp4'
-rw-rw-r--  1 karolis karolis  386 Dec  4 12:07  package.json
-rw-rw-r--  1 karolis karolis  16K Dec  4 12:07  package-lock.json

You can modify the code to put it into your downloads directory or wherever you want to keep the files.

To Sum Up

In this technical demo we showed a practical use case of how a lightweight Node.js daemon can receive and process webhooks without having public IP or running a web server. This can greatly simplify the whole application, reduce attack surface and provide audit capabilities on top of it.