- Use Case
- Log in →
- Docker Compose update on Github webhook
- Using Google Firestore for a Golang backend application
- Automated Jenkins builds on GitHub pull request
- Rules-based webhook filtering & routing
- Introducing Cloudflare support for Home Assistant remote access
- Setting up simple, self-hosted & fast CI/CD solution with Drone.io
- Controlling TV with Google Home, IFTTT and Node-RED
- Node-RED OwnTracks location tracking without public IP/MQTT
- Secure webhooks to Jenkins on Kubernetes
- Remote YouTube downloader Slack bot
Using Google Firestore for a Golang backend application
Aug 26, 2019, by Karolis Rusenas
Usually, when I need a database I just pick Postgres or embedded key-value stores such as the excellent boltdb, badger from dgraph or Redis (if I need a KV store but shared between several nodes). With flexibility comes the burden of maintenance and sometimes additional cost. In this article, we will explore a simple Golang backend service that will use Google Firestore as storage.
When I started working on a simple project called bin.webhookrelay.com, I picked Badger as a key-value store, attached a persistent disk to a Kubernetes pod and launched it.
bin.webhookrelay.com is a free service that allows you to capture webhook or API requests for testing purposes. It also lets you specify what response body and status code to return, as well as set an optional response delay.
The data model was (and still is) simple:
- Bin that has a certain configuration like what status code to return, response body, content-type header, and response delay.
- Request which is the actual captures webhook request with bin ID, request body, headers, and query.
Most of the time, KV stores such as boltdb or badger are great for such use cases. Problems arise when you want to either scale horizontally or have a rolling update strategy meaning that a number of instances of your application would have to surge during the update. While Kubernetes is great for running pretty much any workload, an update where it has to detach a persistent disk and reattach it to a new pod can lead to downtime and just generally slow updates. I always try to avoid such scenarios, however, webhook bin service was suffering from it.
This time, I decided to try out Cloud Firestore. You probably have already heard about Firestore (previously known as Firebase) and that it is very popular amongst mobile app developers who need to have a database for their Android and iOS apps. Apparently, it can also provide a really nice developer UX for backend applications! :)
For authentication, Golang Firestore client uses a standard mechanism that relies on a service account. Basically, you need to go to:
- GCP console
- Click on IAM & admin
- Go to service accounts (on the left navigation bar)
- Create a new service account with Firestore permissions
- Download the file, you can now use it for authentication
Docs can be found in the Google Cloud authentication section.
Application code is surprisingly simple. You get the client using Google application credentials, project ID and that’s it:
When creating a document, you can specify document ID and just pass in the whole golang struct without first marshaling it into JSON:
Note that we supply collection name:
Doc(b.GetId()) and set the struct fields
This really saves time! An alternative with a KV store would be something like:
To delete a bin in our case means deleting both the bin document and all associated webhook requests with it:
While it is very easy to store, retrieve and modify documents, some people will miss SQL type queries that can aggregate, count records and do other useful operations in the database. For example, to track document counts, you will have to implement a solution similar to one described here. My suggestion would be to spend more time planning data structure and what kind of operations are you planning to use before embarking on this journey :)
While being a bit skeptical at first, I quickly started liking Firestore. While running a managed Postgres would allow me to easier switch cloud providers, it would also make it more expensive to run. Keeping storage interface small means that you can implement Postgres (or any other database) driver in a matter of hours so then the most important things to look for are:
- How much maintenance the solution requires