Setting up simple, self-hosted & fast CI/CD solution with Drone.io
Continuous Integration and Continuous Delivery have been very trendy topics in the DevOps world for the last several years. As more and more organizations strive to improve their test and release processes, we have observed many new tools appear in this ecosystem. All major cloud providers are offering their automation systems (some are very unsuccessful) so it’s getting quite hard to pick the right one that fits your use-case.
There are quite a lot of services (Travis, CircleCI, Google Cloud Builder) and self-hosted solutions to build, test and deploy your code, but actually, a few are free, open-source and self-hosted.
The most well-know CI/CD tools are Jenkins and GitLab CI (I have recently asked this on dev.to post). However, Jenkins has a huge memory footprint since it runs on Tomcat (Java) and the workflows are defined inside Jenkins itself, not in the repository that will be used to run those tests/build jobs.
As for GitLab CI, it’s very good but requires you to run your own GitLab (which is huge) or to be on gitlab.com (although now you can mirror your Github repos to Gitlab to start builds automatically, however, it adds complexity). You can run your own runner independently though, they seem to be quite reliable. One of the organizations that I am helping with backend development is using Gitlab CI to great success, but as I mentioned before, business pricing and server running costs could be significant.
Disclaimer: I have tried using Drone several years ago and I wasn’t happy with it. Not sure whether it was the actual product fault or just documentation but I couldn’t achieve what I wanted. This time seems very different. The product is stable, documentation is good and some very good examples are lying inside their own
.drone.ymlfile: https://github.com/drone/drone/blob/master/.drone.yml. Builds are fast and reliable.
I have looked into many alternatives and decided that at least for me, the best CI/CD solution right now is Drone. It’s an open source, very lightweight application written in Go that is available as a SaaS or as a self-hosted server. Drone consists of 2 components:
- Server is responsible for authentication, repository configuration, users, secrets and accepting webhooks.
- Agent are receiving build jobs and actually run your workflows.
Both server and agent are extremely lightweight services and only uses around ~10-15MB RAM. It basically means that you will not even feel that it’s running on your laptop, desktop or even a Raspberry Pi.
I am running Drone server on a Raspberry PI while the agent is running locally on my desktop which has lots of RAM and 16 CPU cores. This means that builds are a lot faster than any free public services could offer. What is more, since agents are connecting to the server over the Webhook Relay tunnels, I can deploy them on Kubernetes, spare laptops or other machines running anywhere:
Benefits of such configuration:
- No need to configure NAT, Drone server barely uses any resources so it can run on cheap hardware.
- Agents are decoupled and can be running on your home/office machines. By connecting to them over the tunnels, agent configuration remains the same, no matter where they are deployed (external cloud provider or internal office).
Future plans include transitioning from Raspberry Pi for Drone server to an Intel NUC (or some other low power alternative) and running some additional Drone agents on the same server.
Initial configuration is very straightforward. First, we create a tunnel on Webhook Relay to get our public subdomain. Then, we use that subdomain to create a Github OAuth app.
Once the server and tunneling daemon are running, you can connect/disconnect Drone agents without any hassle.
Setting up the tunnel for Github and remote access
I like dogfooding, therefore we are using our own tunnels throughout our stack (CI/CD, monitoring dashboards, various other automations). In this case, Webhook Relay tunnels provide the two most important things:
- Access to Drone server no matter where it is (on your laptop, desktop or on some Raspberry Pi), no need to configure a firewall or router to get remote access.
- Webhooks from Github will trigger the workflows.
Go to your tunnels page and create a new tunnel. If you are on a free tier, you will get a generated subdomain, it’s fine.
- name of the tunnel can be anything, but it has to match whatever you will set to the webhookrelayd container in the docker-compose.yaml. It acts as a filter.
- subdomain is your public subdomain under
*.webrelay.io. If you are on a free plan, this will be auto-generated.
- destination is the address of the drone-server container. Since docker-compose configures virtual networks, it has to match the service name from the docker-compose.yaml.
- crypto type is available for all paid plans, basically HTTP/TLS. I would recommend subscribing to get encryption! Otherwise, ping me a message and I can set up a free trial.
Configuring Github OAuth application
In this guide, we will be setting up Drone with Github. You can follow official docs. In short:
Create a GitHub OAuth application. The Consumer Key and Consumer Secret are used to authorize access to Github resources. The Authorization callback URL must match the below format and path, and must use your exact server scheme and host. So if your Webhook Relay tunnel address is
my-drone-subdomain.webrelay.io then the Authorization callback URL will be
Deploying Drone server & agent
I have initially started using Drone with both server and agent running on my desktop since it was only for my personal use, it was fine. However, later I detached server and moved it into a Raspberry Pi.
The all-in-one configuration looks like:
version: '3' services: drone-server: container_name: drone-server image: drone/drone:1.0.0-rc.5 ports: - 8080:80 volumes: - db-data:/var/lib/drone/ restart: always environment: # - DRONE_OPEN=false - DRONE_SERVER_HOST=my-drone-subdomain.webrelay.io # tunnel hostname - DRONE_GITHUB_SERVER=https://github.com - DRONE_ADMIN=<your username> - DRONE_GITHUB_CLIENT_ID=<github-client-id> - DRONE_GITHUB_CLIENT_SECRET=<github-client-secret> - DRONE_SERVER_PROTO=http # tunnel adds https on top - DRONE_RPC_SECRET=<shared secret> drone-agent: container_name: drone-agent image: drone/agent:1.0.0-rc.5 command: agent restart: always depends_on: - drone-server volumes: - /var/run/docker.sock:/var/run/docker.sock environment: - DRONE_RPC_SERVER=https://my-drone-subdomain.webrelay.io - DRONE_RPC_SECRET=<shared secret> - DRONE_RUNNER_CAPACITY=2 - DRONE_RUNNER_NAME="local" webhookrelay: container_name: drone-webhookrelay image: webhookrelay/webhookrelayd:latest command: - --mode - tunnel - -t - drone restart: always depends_on: - drone-server environment: - KEY=<whr-key> - SECRET=<whr-secret> volumes: db-data:
If you need another agent on a different machine, agent-only configuration:
version: '3' services: drone-agent: container_name: drone-agent image: drone/agent:1.0.0-rc.5 command: agent restart: always volumes: - /var/run/docker.sock:/var/run/docker.sock environment: - DRONE_RPC_SERVER=https://my-drone-subdomain.webrelay.io - DRONE_RPC_SECRET=<shared secret> - DRONE_RUNNER_CAPACITY=2 - DRONE_RUNNER_NAME="local"
Once you have the config:
It will ask you to authenticate with your Github credentials.
Working with Drone pipelines
One of the best things about Drone is their pipeline configuration. It seems like after the Drone showed the way, industry followed and now most of the CI/CD services implement the pipelines through the Docker containers. It means that your builds and tests are decoupled from physical hosts and the cleanup happens by design. An example pipeline, that
- unit tests
- runs end-to-end tests with a Postgres database
- sends a webhook to Slack
- builds and pushes an image to DockerHub
looks like this:
kind: pipeline name: default workspace: base: /go path: src/github.com/webhookrelay/app steps: - name: build image: golang:1.11.4 commands: - cd cmd/app && go build - name: unit-test image: golang:1.11.4 commands: - go test -v `go list ./... ` - name: end-to-end image: golang:1.11.4 environment: POSTGRES_HOST: database POSTGRES_USER: pguser POSTGRES_PASSWORD: pgpass POSTGRES_DB: pgdb commands: - make install-transponder - make e2e - name: slack image: plugins/slack settings: webhook: from_secret: slack_url channel: general username: drone icon_url: https://i.pinimg.com/originals/51/29/a4/5129a48ddad9e8408d2757dd10eb836f.jpg - name: docker image: plugins/docker settings: repo: namespace/repo auto_tag: true username: from_secret: docker_username password: from_secret: docker_password services: - name: database image: postgres environment: POSTGRES_USER: pguser POSTGRES_PASSWORD: pgpass POSTGRES_DB: pgdb
By always using containers to run your pipelines, you will make your builds and tests truly portable. Whenever you push to Github, a webhook will hit Drone and it will start a build:
Tips and tricks
I would also like to share some very useful commands. To install
drone CLI, follow official docs.
Executing pipelines directly
You can also run pipelines directly with the Drone CLI:
drone exec --secret-file drone_secrets.yaml .drone.yml
A template for Drone secrets:
Adding secrets through CLI
You can either add secrets through the web UI or use
drone directly from your terminal:
drone secret add -repository username/repository-name --name foo --data bar --allow-pull-request
I have used extensively Jenkins, Gitlab runners, CircleCI and Google Cloud Builder. I guess I still like a lot Cloud Builder for GKE related things as it builds images fast and authenticates to GCR registry without any additional work. However, you do get stuck with their pipeline configuration files and can’t really move to any other vendor. As for Jenkins, it’s just hard to imagine running it efficiently and smaller servers. Drone, at least for me, is an ideal example of a modern service that focuses on efficiency and doing one thing right.