SourceHut: builds.sr.ht Jobs on Ephemeral DigitalOcean Droplets

Let's optimize costs and energy consumption of SourceHut-based Git infrastructures by integrating its CI service, builds.sr.ht, with a cloud infrastructure provider, namely DigitalOcean.

SourceHut: builds.sr.ht Jobs on Ephemeral DigitalOcean Droplets

A main reason for using SaaS services over home-brewed setups is cost. While there are plenty of great open-source services that can be self-hosted, the infrastructure required for them is usually more expensive than using a much cheaper or even free Software-as-a-Service offer. One area where this applies is Git infrastructure. With GitHub, GitLab and Atlassian offering free-of-charge solutions to safely store and publish code, it makes hosting GitLab CE, Gitea and others less attractive. But with growing concerns over SaaS offerings in general or strict non-disclosure agreements for code, it might be needed to have self-hosted infrastructure in place nevertheless. For my needs this infrastructure is built on Drew DeVault’s open-source (AGPLv3) platform SourceHut.

I’m a big fan of the pragmatic yet solid work Drew does in different areas and I share his opinions and views on many things.
SourceHut is well-designed from architecture as well as code perspectives and it is completely modular. Individual SourceHut services can be enabled/disabled as required, each services utilizes its own database and all services cross-communicate through Redis. Here is an architecture overview:

╔═══════════════════════════════╗             ╔═══════════════════════════════╗ 
║meta.sr.ht                     ║░            ║git.sr.ht                      ║░
║┌─────────────────────────────┐║░            ║┌─────────────────────────────┐║░
║│           service           │║░            ║│           service           │║░
║└─────────────────────────────┘║░            ║└─────────────────────────────┘║░
║┌─────────────┐ ┌─────────────┐║░            ║┌─────────────┐ ┌─────────────┐║░
║│     api     │ │  webhooks   │║◀─────┬─────▶║│     api     │ │  webhooks   │║░
║└─────────────┘ └─────────────┘║░     │      ║└─────────────┘ └─────────────┘║░
║┌────── ─────── ─────── ──────┐║░     │      ║┌────── ─────── ─────── ──────┐║░
║           database            ║░     │      ║           database            ║░
║└────── ─────── ─────── ──────┘║░     │      ║└────── ─────── ─────── ──────┘║░
╚═══════════════════════════════╝░     │      ╚═══════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     │       ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
                                       │                                        
╔═══════════════════════════════╗      │      ╔═══════════════════════════════╗ 
║lists.sr.ht                    ║░     │      ║todo.sr.ht                     ║░
║┌─────────────┐ ┌─────────────┐║░     │      ║┌─────────────┐ ┌─────────────┐║░
║│   service   │ │    lmtp     │║░     │      ║│   service   │ │    lmtp     │║░
║└─────────────┘ └─────────────┘║░     │      ║└─────────────┘ └─────────────┘║░
║┌─────────────┐ ┌─────────────┐║░     │      ║┌─────────────┐ ┌─────────────┐║░
║│   process   │ │  webhooks   │║◀─────┼─────▶║│     api     │ │  webhooks   │║░
║└─────────────┘ └─────────────┘║░     │      ║└─────────────┘ └─────────────┘║░
║┌────── ─────── ─────── ──────┐║░     │      ║┌────── ─────── ─────── ──────┐║░
║           database            ║░     │      ║           database            ║░
║└────── ─────── ─────── ──────┘║░     │      ║└────── ─────── ─────── ──────┘║░
╚═══════════════════════════════╝░     │      ╚═══════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     │       ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
                                       │                                        
╔═══════════════════════════════╗      │      ╔═══════════════════════════════╗ 
║hub.sr.ht                      ║░     │      ║builds.sr.ht                   ║░
║┌─────────────────────────────┐║░     │      ║┌─────────────────────────────┐║░
║│           service           │║░     │      ║│           service           │║░
║└─────────────────────────────┘║◀─────┼─────▶║└─────────────────────────────┘║░
║┌────── ─────── ─────── ──────┐║░     │      ║┌────── ─────── ─────── ──────┐║░
║           database            ║░     │      ║           database            ║░
║└────── ─────── ─────── ──────┘║░     │      ║└────── ─────── ─────── ──────┘║░
╚═══════════════════════════════╝░     │      ╚═══════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     │       ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
                                       │                                        
╔═══════════════════════════════╗      │      ╔═══════════════════════════════╗ 
║dispatch.sr.ht                 ║░     │      ║paste.sr.ht                    ║░
║┌─────────────────────────────┐║░     │      ║┌─────────────────────────────┐║░
║│           service           │║░     │      ║│           service           │║░
║└─────────────────────────────┘║◀─────┤      ║└─────────────────────────────┘║░
║┌────── ─────── ─────── ──────┐║░     │      ║┌────── ─────── ─────── ──────┐║░
║           database            ║░     │      ║           database            ║░
║└────── ─────── ─────── ──────┘║░     │      ║└────── ─────── ─────── ──────┘║░
╚═══════════════════════════════╝░     │      ╚═══════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     │       ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
                                       │                                        
╔═══════════════════════════════╗      │                                        
║builds.sr.ht-worker            ║░     │                                        
║┌─────────────────────────────┐║░     │                                        
║│           service           │║◀─────┤                                        
║└─────────────────────────────┘║░     │                                        
╚═══════════════════════════════╝░     │                                        
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░     │                                        
                                       ▼                                        
┌─────────────────────────────────────────────────────────────────────────────┐ 
│                                                                             │░
│                                    Redis                                    │░
│                                                                             │░
└─────────────────────────────────────────────────────────────────────────────┘░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

Running a full-fledged SourceHut environment including builds.sr.ht requires at least two servers/VPCs:
One for hosting everything but the build.sr.ht-worker and one for solely the worker. When a new manifest is being submitted to the service, it adds a job to its queue, which in turn gets picked up by a worker. The worker then spins up a virtual environment for each individual build job.

While this setup can make sense for installations that handle a lot of traffic, it doesn’t do for smaller setups with as few as several hundred build jobs per month.

In order to optimize the infrastructural requirements, I began digging into the builds.sr.ht code and found out a couple of things:

  • It’s possible to consolidate the master and the worker into one single unit, by replacing shell=/usr/bin/master-shell with shell=/usr/bin/runner-shell in the config.ini
  • Provisioning of virtual environments is done through a simple script, which is configurable through the controlcmd=./images/control option in the config.ini
  • Each image has its own functions script that contains platform-dependent commands for adding repositories or installing packages
  • SSH connections are not only initiated through the controlcmd but also triggered from within the worker code, with little possibility to configure their parameters

I came to the conclusion that by altering the code here and there, it should be possible to remove the need for an individual worker instance, as well as KVM/Docker altogether. By connecting a cloud service provider, build.sr.ht would be able to provision virtual environments on individual VPCs on a per build job basis. Ultimately this would allow me to spare one instance (the worker) that would usually be running 24/7 even if it wouldn’t have any work to do. It would also save me from cramming all sr.ht services onto a single, likely more powerful and higher-priced instance, that would be able to run KVM/Docker and have enough resources left for running all other components.

To cut a long story short, I changed Drew’s worker code, altered the controlcmd script and implemented my own image for a DigitalOcean Debian 10 droplet. With these modifications in place, I was able to get build.sr.ht to spin up a new droplet for every build job, run the manifest and shut it back down afterwards. My setup is currently using the weakest (and cheapest) instance type on DigitalOcean. I will make this and a couple of other settings configurable in the future, though. For more info check out this thread:

It’s possible to add other cloud providers using the same approach. The code for this can be found on my SourceHut instance or on sr.ht. Right now it’s a proof of concept and hence a little hacky – mainly caused by the lack of clean interfaces in SourceHut – but I’m working on integrating it more nicely.
Drew has already stated, that this approach is not supported (by him), therefore I will try to build it more like an extension for SourceHut which ideally wouldn’t require monkey-patching any of Drew’s code.

Write me in case you’re interested in testing this extension for yourself or if you’d like to contribute.

Happy building!

categories [projects] · published [2021-04-02] · updated [2021-04-02]