Production-ready Github webhooks on Jenkins behind a firewall

In this tutorial, we will configure Jenkins Blue Ocean to instantly receive webhooks from GitHub.com behind a firewall and without public IP/domain (which could be a corporate firewall, a network behind a NAT/CGNAT like you have at home). You can generalize this to other services too - such as BitBucket, GitLab, DockerHub, or anything that emits webhooks.

Main advantages of webhooks over polling are:

Github to Jenkins without public IP

And the main advantages of Webhook Relay here are:

Support request: if you find any issues with the PR, please inform us at [email protected].

Prerequisites

Configuration

We will install an agent that will subscribe to webhooks and relay them to your Jenkins instance. An agent can be on the same machine or any other as long as it has network access to your Jenkins instance. A common setup is where one or several VMs have agents installed and are relaying webhooks to multiple machines inside your internal cluster.

Step 1: Install and authenticate CLI

Installation instructions for CLI can be found in the official CLI installation page. This tutorial is focusing on Linux x86-64 machines which is the most common type:

sudo wget -O \
/usr/local/bin/relay \
https://storage.googleapis.com/webhookrelay/downloads/relay-linux-amd64

Give it permissions to execute and update itself:

sudo chmod +wx /usr/local/bin/relay

Once the agent is downloaded, log in using the token key & secret pair generated from the tokens page. Just click on the “CREATE TOKEN“ button and either set environment variables in your shell or use relay login ... command.

Step 2: Configure routing & GitHub

Webhook Relay CLI has a shorthand command which can create the configuration and start forwarding webhooks, however since we are configuring for production, we will just create configuration but

relay forward --bucket github-jenkins http://localhost:8080/github-webhook/ --no-agent

It should display your public endpoint:

...
Forwarding configuration created:
https://vlndyzsibcil98gdte7yp1.hooks.webhookrelay.com -> http://localhost:8080/github-webhook/

Take that .hooks.webhookrelay.com URL and put it to your Github repository webhooks section. You can find it by going into Settings -> *Webhooks. Add a new webhook configuration:

Github webhook configuration

You can also always use your public endpoints and destinations in the buckets page.

In general it’s a good practice to set the shared secret.

Step 3: Install agent as a service on your machine

We want the agent to reconnect if your machine restarts (after a power outage or more commonly after an update). Therefore we have two options:

Option 1: Installing agent as a Docker container

In this case, we will let Docker always start & restart the Webhook Relay agent container. You can use the same access token key & secret that you have generated previously, otherwise, create a new pair here and export them as an environment variables:

export RELAY_KEY=XXXX
export RELAY_SECRET=YYYY
export BUCKETS=github-jenkins

Start the agent container:

docker run -d --restart always \
--name webhookrelayd \
--network host \
--env RELAY_KEY=$RELAY_KEY \
--env BUCKETS=$BUCKETS \
--env RELAY_SECRET=$RELAY_SECRET webhookrelay/webhookrelayd

To view the logs:

docker logs webhookrelayd   
2020-06-25 21:36:24.354 INFO using standard transport...
2020-06-25 21:36:24.474 INFO webhook relay ready... {"host": "my.webhookrelay.com:8080", "buckets": ["github-jenkins"]}

Option 2: Installing the agent as a background service

Additional documentation for the background service (logging, proxy, upgrade, removal) can be viewed here. We will just create a configuration file:

version: "v1"
key: XXXX # will be encrypted on startup
secret: YYYY # will be encrypted on startup
buckets:
- github-jenkins

Upon startup, key and secret will be encrypted.

Now, in your machine let’s create a directory where we can put this configuration:

mkdir -p /opt/config/webhookrelay

You can use vim or any other text editor to open it and copy & paste the config:

vim /opt/config/webhookrelay/relay.yaml

Let’s start the agent:

relay service install -c /opt/config/webhookrelay/relay.yaml
relay service start

Step 4: Configure Jenkins webhook shared secret

The secret from step 2 has to be added to Jenkins for it to recognize the webhooks. Go to Jenkins configuration and scroll down until you reach the GitHub section. Once there, click on “Advanced” and click “Add” next to Shared secrets. Set your secret here.

Jenkins shared secret

Step 5: Configuring Jenkins pipeline

Click on create a new pipeline, select “GitHub” as your code store:

Repository configuration

Add access token and select repository. That’s it, if you push to the repository, you should see pipeline run:

Jenkins pipeline run

Troubleshooting

400 Bad Request from Jenkins

If you open Webhook Relay logs dashboard you should see detailed logs. If they say:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 400 Signature was expected, but not provided</title>
</head>
<body><h2>HTTP ERROR 400 Signature was expected, but not provided</h2>
<table>
<tr><th>URI:</th><td>/github-webhook/</td></tr>
<tr><th>STATUS:</th><td>400</td></tr>
<tr><th>MESSAGE:</th><td>Signature was expected, but not provided</td></tr>
<tr><th>SERVLET:</th><td>Stapler</td></tr>
</table>
<hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.27.v20200227</a><hr/>

</body>
</html>

Then the secret in GitHub either is not set or doesn’t match.

403 No valid crumb was included in the request

This happens when the request is sent to a Jenkins API path that is not whitelisted for webhooks. It could be that GitHub is sending a request with some additional path (like /ghprb or anything else) which gets automatically amended to your destination resulting in a wrong final path.

To fix this, ensure that you are sending a request to a correct URL and that destination also doesn’t expect anything more.

200 on Jenkins but nothing happens

In this case, it could be that you haven’t selected the correct repository when setting up a pipeline. When the webhook is received, based on received data Jenkins finds a repository to poll for changes. If repository from the webhook doesn’t match repository in the pipeline - it won’t do anything.