Production-ready Github and Jenkins setup behind a firewall

In this tutorial, we will configure Jenkins Blue Ocean to instantly receive webhooks from GitHub.com behind a firewall and without public IP/domain (which could be a corporate firewall, a network behind a NAT/CGNAT like you have at home). You can generalize this to other services too - such as BitBucket, GitLab, DockerHub, or anything that emits webhooks.

Main advantages of webhooks over polling are:

• No delay between polling requests
• Your Jenkins CI server doesn’t risk getting rate-limited by the GitHub API (exhausting API quota)

And the main advantages of Webhook Relay here are:

• A single agent can handle hundreds of Jenkins servers in your internal network.
• Any internal service can benefit from receiving webhooks without exposing it directly to the internet.
• Additional security layer as Jenkins is not exposed to the internet as webhooks by default are uni-directional, responses are not returned to the sender.
• Public endpoints can stay the same, but servers underneath can change (if you want to provision a new machine, a configuration can stay the same).

Support request: if you find any issues with the PR, please inform us at [email protected].

• Webhook Relay account, create one here.
• An Ubuntu VM. Other OS types should also be fine.
• Jenkins instance. We will not go into installing Jenkins itself as there are quite a few options and many articles on that. See Jenkins official docs for up-to-date instructions.
• GitHub account and a repository that you will want to use.

We will install an agent that will subscribe to webhooks and relay them to your Jenkins instance. An agent can be on the same machine or any other as long as it has network access to your Jenkins instance. A common setup is where one or several VMs have agents installed and are relaying webhooks to multiple machines inside your internal cluster.

Installation instructions for CLI can be found in the official CLI installation page. This tutorial is focusing on Linux x86-64 machines which is the most common type:

sudo wget -O \
  /usr/local/bin/relay \
  https://storage.googleapis.com/webhookrelay/downloads/relay-linux-amd64

Give it permissions to execute and update itself:

sudo chmod +wx /usr/local/bin/relay

Once the agent is downloaded, log in using the token key & secret pair generated from the tokens page. Just click on the “CREATE TOKEN“ button and either set environment variables in your shell or use relay login ... command.

Webhook Relay CLI has a shorthand command which can create the configuration and start forwarding webhooks, however since we are configuring for production, we will just create configuration but

relay forward --bucket github-jenkins http://localhost:8080/github-webhook/ --no-agent

It should display your public endpoint:

...
Forwarding configuration created: 
https://vlndyzsibcil98gdte7yp1.hooks.webhookrelay.com -> http://localhost:8080/github-webhook/

Take that *.hooks.webhookrelay.com URL and put it to your Github repository webhooks section. You can find it by going into Settings -> Webhooks. Add a new webhook configuration:

• Your unique generated endpoint (paid plans can choose subdomain name or use their controlled domains)
• Content type set to ‘application/json’
• Create a shared secret (you can use a password manager or just type openssl rand -base64 32 in your terminal)
• ‘Send me everything’

You can also always use your public endpoints and destinations in the buckets page.

In general it’s a good practice to set the shared secret.

We want the agent to reconnect if your machine restarts (after a power outage or more commonly after an update). Therefore we have two options:

• Docker - simple, when Docker is available in your system (recommended solution).
• Background service - a simple and powerful solution where agent installs itself as an OS background service.

In this case, we will let Docker always start & restart the Webhook Relay agent container. You can use the same access token key & secret that you have generated previously, otherwise, create a new pair here and export them as an environment variables:

export RELAY_KEY=XXXX
export RELAY_SECRET=YYYY
export BUCKETS=github-jenkins

Start the agent container:

docker run -d --restart always \
  --name webhookrelayd \
  --network host \
  --env RELAY_KEY=$RELAY_KEY \
  --env BUCKETS=$BUCKETS \
  --env RELAY_SECRET=$RELAY_SECRET webhookrelay/webhookrelayd

To view the logs:

docker logs webhookrelayd   
2020-06-25 21:36:24.354    INFO    using standard transport...
2020-06-25 21:36:24.474    INFO    webhook relay ready...    {"host": "my.webhookrelay.com:8080", "buckets": ["github-jenkins"]}

Additional documentation for the background service (logging, proxy, upgrade, removal) can be viewed here. We will just create a configuration file:

version: "v1"
key: XXXX         # will be encrypted on startup
secret: YYYY      # will be encrypted on startup
buckets:
  - github-jenkins

Upon startup, key and secret will be encrypted.

Now, in your machine let’s create a directory where we can put this configuration:

mkdir -p /opt/config/webhookrelay

You can use vim or any other text editor to open it and copy & paste the config:

vim /opt/config/webhookrelay/relay.yaml

Let’s start the agent:

relay service install -c /opt/config/webhookrelay/relay.yaml
relay service start

The secret from step 2 has to be added to Jenkins CI for it to recognize the webhooks. Go to Jenkins configuration and scroll down until you reach the GitHub section. Once there, click on “Advanced” and click “Add” next to Shared secrets. Set your secret here.

Click on create a new Jenkins CI CD pipeline, select “GitHub” as your code store:

Add access token and select repository. That’s it, if you push to the repository, you should see pipeline run:

If you open Webhook Relay logs dashboard you should see detailed logs. If they say:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 400 Signature was expected, but not provided</title>
</head>
<body><h2>HTTP ERROR 400 Signature was expected, but not provided</h2>
<table>
<tr><th>URI:</th><td>/github-webhook/</td></tr>
<tr><th>STATUS:</th><td>400</td></tr>
<tr><th>MESSAGE:</th><td>Signature was expected, but not provided</td></tr>
<tr><th>SERVLET:</th><td>Stapler</td></tr>
</table>
<hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.27.v20200227</a><hr/>

</body>
</html>

Then the secret in GitHub either is not set or doesn’t match.

This happens when the request is sent to a Jenkins API path that is not whitelisted for webhooks. It could be that GitHub is sending a request with some additional path (like /ghprb or anything else) which gets automatically amended to your destination resulting in a wrong final path.

To fix this, ensure that you are sending a request to a correct URL and that destination also doesn’t expect anything more.

In this case, it could be that you haven’t selected the correct repository when setting up a pipeline. When the webhook is received, based on received data Jenkins finds a repository to poll for changes. If repository from the webhook doesn’t match repository in the pipeline - it won’t do anything.