Setting up simple, self-hosted & fast CI/CD solution with Drone.io
Continuous Integration and Continuous Delivery have been very trendy topics in the DevOps world for the last several years. As more and more organizations strive to improve their test and release processes, we have observed many new tools appear in this ecosystem. All major cloud providers are offering their automation systems (some are very unsuccessful) so it’s getting quite hard to pick the right one that fits your use-case.
There are quite a lot of services (Travis, CircleCI, Google Cloud Builder) and self-hosted solutions to build, test and deploy your code, but actually, a few are free, open-source and self-hosted.
The most well-know CI/CD tools are Jenkins and GitLab CI (I have recently asked this on dev.to post). However, Jenkins has a huge memory footprint since it runs on Tomcat (Java) and the workflows are defined inside Jenkins itself, not in the repository that will be used to run those tests/build jobs.
As for GitLab CI, it’s very good but requires you to run your own GitLab (which is huge) or to be on gitlab.com (although now you can mirror your Github repos to Gitlab to start builds automatically, however, it adds complexity). You can run your own runner independently though, they seem to be quite reliable. One of the organizations that I am helping with backend development is using Gitlab CI to great success, but as I mentioned before, business pricing and server running costs could be significant.
Disclaimer: I have tried using Drone several years ago and I wasn’t happy with it. Not sure whether it was the actual product fault or just documentation but I couldn’t achieve what I wanted. This time seems very different. The product is stable, documentation is good and some very good examples are lying inside their own
.drone.yml
file: https://github.com/drone/drone/blob/master/.drone.yml. Builds are fast and reliable.
Enter Drone
I have looked into many alternatives and decided that at least for me, the best CI/CD solution right now is Drone. It’s an open source, very lightweight application written in Go that is available as a SaaS or as a self-hosted server. Drone consists of 2 components:
- Server is responsible for authentication, repository configuration, users, secrets and accepting webhooks.
- Agent are receiving build jobs and actually run your workflows.
Both server and agent are extremely lightweight services and only uses around ~10-15MB RAM. It basically means that you will not even feel that it’s running on your laptop, desktop or even a Raspberry Pi.
My setup
I am running Drone server on a Raspberry PI while the agent is running locally on my desktop which has lots of RAM and 16 CPU cores. This means that builds are a lot faster than any free public services could offer. What is more, since agents are connecting to the server over the Webhook Relay tunnels, I can deploy them on Kubernetes, spare laptops or other machines running anywhere:
Benefits of such configuration:
- No need to configure NAT, Drone server barely uses any resources so it can run on cheap hardware.
- Agents are decoupled and can be running on your home/office machines. By connecting to them over the tunnels, agent configuration remains the same, no matter where they are deployed (external cloud provider or internal office).
Future plans include transitioning from Raspberry Pi for Drone server to an Intel NUC (or some other low power alternative) and running some additional Drone agents on the same server.
Preparing configuration
Initial configuration is very straightforward. First, we create a tunnel on Webhook Relay to get our public subdomain. Then, we use that subdomain to create a Github OAuth app.
Once the server and tunneling daemon are running, you can connect/disconnect Drone agents without any hassle.
Setting up the tunnel for Github and remote access
I like dogfooding, therefore we are using our own tunnels throughout our stack (CI/CD, monitoring dashboards, various other automations). In this case, Webhook Relay tunnels provide the two most important things:
- Access to Drone server no matter where it is (on your laptop, desktop or on some Raspberry Pi), no need to configure a firewall or router to get remote access.
- Webhooks from Github will trigger the workflows.
Go to your tunnels page and create a new tunnel. If you are on a free tier, you will get a generated subdomain, it’s fine.
Here:
- name of the tunnel can be anything, but it has to match whatever you will set to the webhookrelayd container in the docker-compose.yaml. It acts as a filter.
- subdomain is your public subdomain under
*.webrelay.io
. If you are on a free plan, this will be auto-generated. - destination is the address of the drone-server container. Since docker-compose configures virtual networks, it has to match the service name from the docker-compose.yaml.
- crypto type is available for all paid plans, basically HTTP/TLS. I would recommend subscribing to get encryption! Otherwise, ping me a message and I can set up a free trial.
Configuring Github OAuth application
In this guide, we will be setting up Drone with Github. You can follow official docs. In short:
Create a GitHub OAuth application. The Consumer Key and Consumer Secret are used to authorize access to Github resources. The Authorization callback URL must match the below format and path, and must use your exact server scheme and host. So if your Webhook Relay tunnel address is my-drone-subdomain.webrelay.io
then the Authorization callback URL will be https://my-drone-subdomain.webrelay.io/login
.
Deploying Drone server & agent
I have initially started using Drone with both server and agent running on my desktop since it was only for my personal use, it was fine. However, later I detached server and moved it into a Raspberry Pi.
The all-in-one configuration looks like:
version: '3'
services:
drone-server:
container_name: drone-server
image: drone/drone:1.0.0-rc.5
ports:
- 8080:80
volumes:
- db-data:/var/lib/drone/
restart: always
environment:
# - DRONE_OPEN=false
- DRONE_SERVER_HOST=my-drone-subdomain.webrelay.io # tunnel hostname
- DRONE_GITHUB_SERVER=https://github.com
- DRONE_ADMIN=<your username>
- DRONE_GITHUB_CLIENT_ID=<github-client-id>
- DRONE_GITHUB_CLIENT_SECRET=<github-client-secret>
- DRONE_SERVER_PROTO=http # tunnel adds https on top
- DRONE_RPC_SECRET=<shared secret>
drone-agent:
container_name: drone-agent
image: drone/agent:1.0.0-rc.5
command: agent
restart: always
depends_on:
- drone-server
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- DRONE_RPC_SERVER=https://my-drone-subdomain.webrelay.io
- DRONE_RPC_SECRET=<shared secret>
- DRONE_RUNNER_CAPACITY=2
- DRONE_RUNNER_NAME="local"
webhookrelay:
container_name: drone-webhookrelay
image: webhookrelay/webhookrelayd:latest
command:
- --mode
- tunnel
- -t
- drone
restart: always
depends_on:
- drone-server
environment:
- KEY=<whr-key>
- SECRET=<whr-secret>
volumes:
db-data:
If you need another agent on a different machine, agent-only configuration:
version: '3'
services:
drone-agent:
container_name: drone-agent
image: drone/agent:1.0.0-rc.5
command: agent
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- DRONE_RPC_SERVER=https://my-drone-subdomain.webrelay.io
- DRONE_RPC_SECRET=<shared secret>
- DRONE_RUNNER_CAPACITY=2
- DRONE_RUNNER_NAME="local"
Once you have the config:
docker-compose up
It will ask you to authenticate with your Github credentials.
Working with Drone pipelines
One of the best things about Drone is their pipeline configuration. It seems like after the Drone showed the way, industry followed and now most of the CI/CD services implement the pipelines through the Docker containers. It means that your builds and tests are decoupled from physical hosts and the cleanup happens by design. An example pipeline, that
- builds
- unit tests
- runs end-to-end tests with a Postgres database
- sends a webhook to Slack
- builds and pushes an image to DockerHub
looks like this:
kind: pipeline
name: default
workspace:
base: /go
path: src/github.com/webhookrelay/app
steps:
- name: build
image: golang:1.11.4
commands:
- cd cmd/app && go build
- name: unit-test
image: golang:1.11.4
commands:
- go test -v `go list ./... `
- name: end-to-end
image: golang:1.11.4
environment:
POSTGRES_HOST: database
POSTGRES_USER: pguser
POSTGRES_PASSWORD: pgpass
POSTGRES_DB: pgdb
commands:
- make install-transponder
- make e2e
- name: slack
image: plugins/slack
settings:
webhook:
from_secret: slack_url
channel: general
username: drone
icon_url: https://i.pinimg.com/originals/51/29/a4/5129a48ddad9e8408d2757dd10eb836f.jpg
- name: docker
image: plugins/docker
settings:
repo: namespace/repo
auto_tag: true
username:
from_secret: docker_username
password:
from_secret: docker_password
services:
- name: database
image: postgres
environment:
POSTGRES_USER: pguser
POSTGRES_PASSWORD: pgpass
POSTGRES_DB: pgdb
By always using containers to run your pipelines, you will make your builds and tests truly portable. Whenever you push to Github, a webhook will hit Drone and it will start a build:
Tips and tricks
I would also like to share some very useful commands. To install drone
CLI, follow official docs.
Executing pipelines directly
You can also run pipelines directly with the Drone CLI:
drone exec --secret-file drone_secrets.yaml .drone.yml
A template for Drone secrets:
slack_url: https://hooks.slack.com/services/xxxxxxxxxxxx
Adding secrets through CLI
You can either add secrets through the web UI or use drone
directly from your terminal:
drone secret add -repository username/repository-name --name foo --data bar --allow-pull-request
Wrapping up
I have used extensively Jenkins, Gitlab runners, CircleCI and Google Cloud Builder. I guess I still like a lot Cloud Builder for GKE related things as it builds images fast and authenticates to GCR registry without any additional work. However, you do get stuck with their pipeline configuration files and can’t really move to any other vendor. As for Jenkins, it’s just hard to imagine running it efficiently and smaller servers. Drone, at least for me, is an ideal example of a modern service that focuses on efficiency and doing one thing right.