I just set up a step in my publication pipeline to automatically post on Mastodon when I publish a new blog post.

The Mastodon API for posting is pretty easy, per this guide. I grabbed a Mastodon token and put it into Vault to be accesible (as detailed in previous posts) to the CD pipeline.

Accessing Kubernetes secrets in Drone pipelines required installing the Kubernetes Secrets Extension¹, which was fairly easy². I already had Vault->Kubernetes Secret integration set up, so plumbing the secret in was also easy. I did run into a bit of confusion in that there’s no mention in Drone docs of how to specify a non-default namespace from which to fetch Kubernetes secrets - turns out there are two places that need to be set to that value, not just one.

With all the pieces assembled, I just needed to write some hacky Bash (is there any other kind!?) to check for commits that are creating a new file in the blog/content/posts/ directory (I don’t want to “announce” commits that are only updates, or that are changing the system rather than the content), and let ’er rip. As a next step, it would be nice to extract this logic to a standalone image³ for others to use as a Drone plugin - though my use-case is probably specific enough that this wouldn’t be valuable. If you’d like it, though, let me know!

(I’m still not 100% sure that my step to clear the cache on my main blog page is working as expected - if you see unexpected behaviour that could be due to old cached values, please let me know!)

My evolving views on CI/CD

During the process of implementing this change, I came to some realizations about the design and motivation of CI/CD systems.

I’m pretty strongly on-the-record as believing that Amazon’s internal Pipelines tool is one of the best things about developing there, and my opinion on that has only grown stronger with time (especially if you extend it to the dependency-version-management system, which almost entirely negates Dependency Hell). I was pretty surprised to get some pushback on that from an ex-colleague when I first stated that position - he was extremely dissatisfied with Pipelines, in ways that made no sense to me. Nearly two years later, I think I’ve reached some better understanding - and, unsurprisingly, the disagreement seems to come from valuing different criteria and trying to do different things.

Tasks to be completed by a pipeline

Amazon’s Pipelines system is designed to publish libraries and deploy web services, and is extremely good at those tasks. That’s all, though. If any post-build stages in your pipeline are doing anything other than “take a built-image and update a running version of a software system to use that image, then run any associated tests and block the pipeline if appropriate”, You Are Going To Have A Bad Time. For instance, if you want a stage to run arbitrary bash code, or to make an HTTP request, or upload a generated artifact to S3, or whatever, you can do those things (usually by hacking them in as functionality executed by a “Test” after a stage, or by setting up an external listener for SNS notifications), but you’ll be fighting friction along the way. Developers who wish to do those things may, therefore, conclude that Pipelines is a bad tool; but they’d be wrong to do so. Pipelines is a excellent tool for what it aims to do, which is purely and simply “publish a new version of a library” and/or “update the running versions of a web service”. It may be a bad tool for your use-case, but that doesn’t make it a bad tool⁴.

The first obvious question then becomes - how often do developers want to do something outside of those two tasks? And I think the answer is…almost never? I guess deployment of mobile apps would be one example (publishing an .apk probably looks different than building and publishing a Java/Python/etc. library, especially since you would need to run tests on deployed hardward before finalizing the publication), but I’m not really aware of any other examples. Please let me know if I’m missing something!

It is important to acknowledge, though, that Pipelines is only a good solution in a corporate culture where there is one-and-only-one tool for the various SDLC functions (build, image storage, deployment, testing, etc.). With that level of standardization, your CD system can essentially accept configuration rather than code - tell it the names of your packages, the number of stages you want, etc., and it can spin up a personalized instance of The One Solution from your configuration parameters and a template. In the OSS world where the freedom to plug in different providers (GitHub/Gitea/GitLab, CircleCI/Drone/Woodpecker/Jenkins, Argo/Flux, etc.) is important, that “templatization” is nigh-impossible, and your pipeline definition has to be logic, not just parameters. That is - while I do miss Pipelines, I recognize it couldn’t fly in the outside world.

Different conceptualizations of a pipeline in OSS vs. Amazon

Unsurprisingly, I came out of Amazon with an Amazonian view of the relationship between a pipeline and a service - that is:

1. They’re one-to-one - if you own and operate FooService, there is a single FooServicePipeline which manages build and deployment for the service (with pipeline stages corresponding to the various deployments of the service), rather than separate pipelines for build and for each deployment.
2. Multiple “waves” or executions can be progressing through a pipeline at a given time. The pipeline should take responsbility for ensuring that promotion into a stage does not start until the previous promotion has completed.
3. The version which is deployed to each stage is an emergent property of the executed logic of the pipeline, rather than being an input to the pipeline.

After nearly two years of experience with OSS offerings, my views are a bit more nuanced and broad.

One Pipeline per Service

I still tend to think that this is the most intuitive design - though, in fairness, you can squint and make “one pipeline with multiple stages” and “one pipeline which orchestrates multiple sub-pipelines (which are each responsible for a stage of the overall-pipeline)” look the same. This is really a question of definition - if you define “pipeline” as “the thing which handles build and deployment for all stages of the service”, then by-definition there’s only a single one. If you define “pipeline” as “a sequence of steps to be executed”, then it’s reasonable to say “there’s a pipeline for deploying to QA stage, there’s a pipeline for deploying to Prod stage, and there’s a pipeline for triggering those one-by-one”.

The important part, though, is the ability to set up dependencies and relations between the stages - “whatever was just deployed to QA stage (and passed tests) should now be promoted to Prod stage”. Thus, if you take the “one pipeline per stage” approach, there must be an “over-pipeline” to make a change from one to the other; even if it doesn’t exist as a standalone entity, and only exists implicitly in conditional logic determining whether completion of one step will trigger another. Which brings us to…

Promotion Isolation

Problem: One thing (of many!) that Amazon Pipelines did really well was to keep promotions isolated from one another. If a promotion is underway to Stage N and a promotion just succeeded to Stage N-1, the incoming promotion won’t trigger until the ongoing one has completed. This is naturally desirable - each promotion should acquire a “lock” on a stage that is not released until the change being promoted has been tested.

Solution: This is possible to replicate in some OSS systems by limiting max-concurrency of executions to 1. If using the single overall pipeline model, this is a somewhat-extreme option which reduces throughput - if two changes are submitted in quick succession, the second change cannot begin flowing through the pipeline until the first one has completed (or failed along the way), so of course there can be no race condition for a single stage - so, OSS systems probably work better with a model where each stage’s deployment, and each stage-to-stage promotion, is a standalone pipeline.

Problem: Depending on team preference, it was also common at Amazon to configure the pipeline so that promotion into any non-prod stage will require manual approval if the previous promotion failed. This is so that a failed deployment can be preserved for debugging, rather than being overwitten by the next change that comes down the pipeline, which will obfuscate the original error.

Solution: I still haven’t found a good way to implement this with OSS systems - this level of intra-step awareness seems to not be a common requirement of the systems. A deployment step(/pipeline) seems to only “know about” its own context, not the history of the “next” stage - indeed, the very notion of sequential stages does not seem to be “baked in” to these systems (see the earlier point that OSS pipeline systems are intentionally more abstract to support more supporting services, at the cost of having fewer high-level concepts available). One could of course implement this awareness, by extending the deployment step with API calls to the pipeline system which identify the next step and retrieve its history - but that’s home-rolled implementation of functionality that I’m used to having provided “for free”.

The pipeline is a software system which decides what should be deployed where, rather than a system which is told what to deploy

This is the big conceptual change that took me a long time to adjust to in OSS systems. In my opinion, the input parameter to a(n overall) deployment pipeline should be “the set of software packages to watch, build, and deploy”, not “which built-versions should be deployed where”. The pipeline itself is the software system responsible for triggering a deployment, running tests against the deployment, and determining whether the tests passed - and so, you should be asking the pipeline “is it correct to deploy image deadbeef to stage N+1?”, not telling it to do so. Directly “injecting” stage/version mappings into the pipeline (by writing to the Deployment Repo) breaks sequentiality - it becomes possible to deploy something to stage N+1 other than the version which was just deployed to stage N, thus obviating all the benefits of sequential deployment and testing⁵.

Having stewed on this a little, and with the benefits of writing this post to disambiguate the differing views of pipelines, I think I’ve reached a good solution, though:

• Structure the deployment logic (in whatever automation system runs your pipeline) as taking a version and a stage as parameters.
• Write deployment logic which updates the Deployment Repo with the appropriate version, and lock down permissions on the Deployment Repo so that only these automations can change it (though, see the last paragraph of this section)
• Either through an over-pipeline, or by adding a direct API call to the automation system at the end of a successful deployment, have the successful completion of “deploy version X to stage N” trigger a deployment of version X to stage N+1 (optionally, if stage N+1’s previous deployment was successful).

That way, the Deployment Repo becomes “internalized into” the pipeline - instead of the Repo being the interaction point at which the developers tell the pipeline what to deploy, it instead becomes a state-maintainance persistence store where the pipeline keeps track of what is currently deployed where (and to which external systems like Argo can couple/listen, to do their logic).

There’s still an issue here, because developers need to have write-access to the Deployment Repo of a service in order to be able to manage infrastructure of the service. Since this repo is where the mapping of “which images should be deployed to which stage” is found, developers have the ability to override the pipeline. That’s a solvable problem, though - automated checks during PRs can ensure that only those with appropriate permissions make changes in this mapping, while allowing any developer to manage general infrastructure. Again - something to roll for oneself, rather than having the functionality provided by the system, but that’s the trade-off you make for general-purpose functionality.

• CI/CD
• Homelab
• Meta
• Vault