GitOps Repository Structures and Patterns Part 4: Promotion Patterns

In this series of articles, I am introducing you to different structures and patterns that you can use to design your GitOps process. This part is about Promotion Patterns.

You can get an introduction to the topic of GitOps repository patterns and structures in the first part of this series, in the second part I introduce operator deployment patterns, in the third part repository patterns and in the fifth part wiring patterns. In the sixth part I show different implementations of the structures and patterns using example repositories.

The promotion patterns describe different ways to implement environments (also known as stages, for example "QA", "Staging" and "Production") with GitOps. We refer to this process as promotion. Sometimes the term promotion is preceded by a word such as release, application, environment, workload, or change. This article uses the terms environment and promotion in the following.

In GitOps, the promotion, i.e. the transition from one environment to the next, is realized by pull request.

Branch vs Folder per Environment

For the implementation, the central question is: How are environments realized, in folders or branches?

Experience shows that many teams (especially those with a background in software development) initially think of separating the environments by permanent branches. We are talking about the Branch per Environment (also Environment per Branch) pattern. The entire config, which is located on a branch, is always deployed to an environment. The promotion then takes place by a merge in Git. Interestingly, there are few success stories for this pattern. One example is Monitoring and Hardening the GitOps Delivery Pipeline with Flux by Florian Heubeck. The advantage cited here is that forcing pull requests on branches is easier than on folders. This is countered by many reports that advise against this pattern (for example Pinky Ravi and Scott Rigby in GitOps: Core Concepts & Ways of Structuring Your Repos) or even call it an anti-pattern (for example Stop Using Branches for Deploying to Different GitOps Environments by Kostis Kapelonis and Git best practices: Workflows for GitOps deployments by Christian Hernandez).

Instead, they recommend the Folder per Environment pattern (also "Environment per Folder" or with the term "directory" instead of "folder"). Here the environments are realized as folders on a branch (trunk-based). For the promotion, short-lived branches are created in which the changes are copied from one environment folder to the next. These short-lived branches contain only one feature at a time and are deleted again with the merge. Since environments differ only in a few areas, there is often a lot of duplication. However, with tools like Helm and Kustomize, this duplication can be prevented. Because these tools work at the file level and not the branch level, they can only do this with the Folder per Environment pattern. The Branch perEnvironment pattern therefore has more duplicated code. In addition, commits are built on top of each other. What if a feature is to be brought into the next environment that is not in the latest commit?

Cherry-picks are a solution here. However, these are not always trivial. With folders, you can simply copy files, which is much easier and less error-prone. There is also the danger of "drift" – branches can diverge because changes in a hotfix only end up in production, but the merges to the other environments do not take place. Here it can happen that unexpected (inconsistent) states arise during the Git merge – in production. A final argument: With branches, the complexity increases with the number of environments, with folders it remains constant.

Finally, regarding the discussion of the two patterns, it is worth mentioning that certain disadvantages can be prevented by settings in the SCM: Inconsistencies can be prevented for branches by allowing only fast-forward merges. Pull requests can be enforced by folder protection in some SCMs. Depending on the environment, some arguments may not apply, which can influence the decision between folders or branches.

Preview Environments

Another way to handle environments are Preview Environments (also "ephemeral/dynamic/pull request/test/temporary" environments). Here, when a pull request (PR, also merge request) is created, a deployment is triggered that is based on the config of the PR. After the PR has been merged or rejected, the deployment is automatically deleted again. This feature has been common for serverless Platforms as a Service such as Netlify or Vercel for some time. The terms Preview Deployments or Deploy Previews are used here. When deploying using a CI server, automatic deletion in particular is difficult to implement. With GitOps, specifically for example with Argo CD, this entire process can now be automated by means of its own controller and associated CRD ApplicationSet and its Pull Request Generator. This generates by means of templating per PR an application according to a template given in the ApplicationSet, for example myapp-{ {branch} }-{ {number} }.

With Flux (or at least Weave GitOps), this could soon look similar in the form of GitOps Sets. An exciting takeaway from this is that GitOps tools can provide features and benefits that go beyond the actual principles of GitOps. We can look forward to seeing what else will emerge.

Conclusion

Depending on the way your teams work, you can choose the best promotion pattern to deploy environments. Further possibilities are offered by the patterns presented in the other parts of this article series: Operator deployment patterns, repository patterns, wiring patterns and real-life examples.