SourceHut: builds.sr.ht Jobs on Ephemeral DigitalOcean Droplets

Let’s optimize costs and energy consumption of SourceHut-based Git infrastructures by integrating its CI service, builds.sr.ht, with a cloud infrastructure provider, namely DigitalOcean.

SourceHut: builds.sr.ht Jobs on Ephemeral DigitalOcean Droplets

A main reason for using SaaS services over home-brewed setups is cost. While there are plenty of great open-source services that can be self-hosted, the infrastructure required for them is usually more expensive than using a much cheaper or even free Software-as-a-Service offer. One area where this applies is Git infrastructure. With GitHub, GitLab and Atlassian offering free-of-charge solutions to safely store and publish code, it makes hosting GitLab CE, Gitea and others less attractive. But with growing concerns over SaaS offerings in general or strict non-disclosure agreements for code, it might be needed to have self-hosted infrastructure in place nevertheless. For my needs this infrastructure is built on Drew DeVault’s open-source (AGPLv3) platform SourceHut.

I’m a big fan of the pragmatic yet solid work Drew does in different areas and I share his opinions and views on many things.
SourceHut is well-designed from architecture as well as code perspectives and it is completely modular. Individual SourceHut services can be enabled/disabled as required, each services utilizes its own database and all services cross-communicate through Redis. Here is an architecture overview:

╔═══════════════════════════════╗             ╔═══════════════════════════════╗ 
║meta.sr.ht                     ║░            ║git.sr.ht                      ║░
║┌─────────────────────────────┐║░            ║┌─────────────────────────────┐║░
║│           service           │║░            ║│           service           │║░
║└─────────────────────────────┘║░            ║└─────────────────────────────┘║░
║┌─────────────┐ ┌─────────────┐║░            ║┌─────────────┐ ┌─────────────┐║░
║│     api     │ │  webhooks   │║◀─────┬─────▶║│     api     │ │  webhooks   │║░
║└─────────────┘ └─────────────┘║░     │      ║└─────────────┘ └─────────────┘║░
║┌────── ─────── ─────── ──────┐║░     │      ║┌────── ─────── ─────── ──────┐║░
║           database            ║░     │      ║           database            ║░
║└────── ─────── ─────── ──────┘║░     │      ║└────── ─────── ─────── ──────┘║░
╚═══════════════════════════════╝░     │      ╚═══════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     │       ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
                                       │                                        
╔═══════════════════════════════╗      │      ╔═══════════════════════════════╗ 
║lists.sr.ht                    ║░     │      ║todo.sr.ht                     ║░
║┌─────────────┐ ┌─────────────┐║░     │      ║┌─────────────┐ ┌─────────────┐║░
║│   service   │ │    lmtp     │║░     │      ║│   service   │ │    lmtp     │║░
║└─────────────┘ └─────────────┘║░     │      ║└─────────────┘ └─────────────┘║░
║┌─────────────┐ ┌─────────────┐║░     │      ║┌─────────────┐ ┌─────────────┐║░
║│   process   │ │  webhooks   │║◀─────┼─────▶║│     api     │ │  webhooks   │║░
║└─────────────┘ └─────────────┘║░     │      ║└─────────────┘ └─────────────┘║░
║┌────── ─────── ─────── ──────┐║░     │      ║┌────── ─────── ─────── ──────┐║░
║           database            ║░     │      ║           database            ║░
║└────── ─────── ─────── ──────┘║░     │      ║└────── ─────── ─────── ──────┘║░
╚═══════════════════════════════╝░     │      ╚═══════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     │       ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
                                       │                                        
╔═══════════════════════════════╗      │      ╔═══════════════════════════════╗ 
║hub.sr.ht                      ║░     │      ║builds.sr.ht                   ║░
║┌─────────────────────────────┐║░     │      ║┌─────────────────────────────┐║░
║│           service           │║░     │      ║│           service           │║░
║└─────────────────────────────┘║◀─────┼─────▶║└─────────────────────────────┘║░
║┌────── ─────── ─────── ──────┐║░     │      ║┌────── ─────── ─────── ──────┐║░
║           database            ║░     │      ║           database            ║░
║└────── ─────── ─────── ──────┘║░     │      ║└────── ─────── ─────── ──────┘║░
╚═══════════════════════════════╝░     │      ╚═══════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     │       ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
                                       │                                        
╔═══════════════════════════════╗      │      ╔═══════════════════════════════╗ 
║dispatch.sr.ht                 ║░     │      ║paste.sr.ht                    ║░
║┌─────────────────────────────┐║░     │      ║┌─────────────────────────────┐║░
║│           service           │║░     │      ║│           service           │║░
║└─────────────────────────────┘║◀─────┤      ║└─────────────────────────────┘║░
║┌────── ─────── ─────── ──────┐║░     │      ║┌────── ─────── ─────── ──────┐║░
║           database            ║░     │      ║           database            ║░
║└────── ─────── ─────── ──────┘║░     │      ║└────── ─────── ─────── ──────┘║░
╚═══════════════════════════════╝░     │      ╚═══════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     │       ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
                                       │                                        
╔═══════════════════════════════╗      │                                        
║builds.sr.ht-worker            ║░     │                                        
║┌─────────────────────────────┐║░     │                                        
║│           service           │║◀─────┤                                        
║└─────────────────────────────┘║░     │                                        
╚═══════════════════════════════╝░     │                                        
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     │                                        
                                       ▼                                        
┌─────────────────────────────────────────────────────────────────────────────┐ 
│                                                                             │░
│                                    Redis                                    │░
│                                                                             │░
└─────────────────────────────────────────────────────────────────────────────┘░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

Running a full-fledged SourceHut environment including builds.sr.ht requires at least two servers/VPCs:
One for hosting everything but the build.sr.ht-worker and one for solely the worker. When a new manifest is being submitted to the service, it adds a job to its queue, which in turn gets picked up by a worker. The worker then spins up a virtual environment for each individual build job.

While this setup can make sense for installations that handle a lot of traffic, it doesn’t do for smaller setups with as few as several hundred build jobs per month.

In order to optimize the infrastructural requirements, I began digging into the builds.sr.ht code and found out a couple of things:

  • It’s possible to consolidate the master and the worker into one single unit, by replacing shell=/usr/bin/master-shell with shell=/usr/bin/runner-shell in the config.ini
  • Provisioning of virtual environments is done through a simple script, which is configurable through the controlcmd=./images/control option in the config.ini
  • Each image has its own functions script that contains platform-dependent commands for adding repositories or installing packages
  • SSH connections are not only initiated through the controlcmd but also triggered from within the worker code, with little possibility to configure their parameters

I came to the conclusion that by altering the code here and there, it should be possible to remove the need for an individual worker instance, as well as KVM/Docker altogether. By connecting a cloud service provider, build.sr.ht would be able to provision virtual environments on individual VPCs on a per build job basis. Ultimately this would allow me to spare one instance (the worker) that would usually be running 24/7 even if it wouldn’t have any work to do. It would also save me from cramming all sr.ht services onto a single, likely more powerful and higher-priced instance, that would be able to run KVM/Docker and have enough resources left for running all other components.

To cut a long story short, I changed Drew’s worker code, altered the controlcmd script and implemented my own image for a DigitalOcean Debian 10 droplet. With these modifications in place, I was able to get build.sr.ht to spin up a new droplet for every build job, run the manifest and shut it back down afterwards. My setup is currently using the weakest (and cheapest) instance type on DigitalOcean. I will make this and a couple of other settings configurable in the future, though.

It’s possible to add other cloud providers using the same approach. The code for this can be found on my SourceHut instance or on sr.ht. Right now it’s a proof of concept and hence a little hacky – mainly caused by the lack of clean interfaces in SourceHut – but I’m working on integrating it more nicely.
Drew has already stated, that this approach is not supported (by him), therefore I will try to build it more like an extension for SourceHut which ideally wouldn’t require monkey-patching any of Drew’s code.

Write me in case you’re interested in testing this extension for yourself or if you’d like to contribute.

Happy building!

collections [ projects ] · tags [ infrastructure ]
published [ 2021-04-02 ] · updated [ 2021-08-26 ]