GitOps Repository Structures and Patterns Part 1: Introduction and Overview

A key issue in the introduction of GitOps is the design of the GitOps process and associated repositories. This raises questions about:

• The structure of the repositories,
• implementation of stages/environments,
• use of branches,
• ratio of the number of GitOps operators to the number of clusters and/or namespaces and
• how everything should be wired in the end.

This series of articles addresses these issues, naming recurring elements as GitOps patterns. For an overview of the patterns described here, see the cloudogu/gitops-patterns repository on GitHub.

Mapping the real world and infrastructure – the GitOps Chasm

Let's imagine we are about to implement GitOps in our organization. A GitOps operator has already been selected. Often the choice here is Flux or Argo CD (for a comparison, see this article). Now it's time to design the GitOps process. This is where this article comes in.

An important question is about the use case. What is actually to be deployed via GitOps? Infrastructure or applications? This can influence the design. This article starts from probably the most common use case: deploying applications.

Also, the responsibility is an interesting issue: Who takes care of the GitOps process in an organization? Depending on the team topology and the degree of team autonomy, application teams take care of the GitOps operator themselves or delegate this. The complexity of customer requirements and product development on the one hand, to strongly operations-related topics like GitOps on the other hand, is very high. This is one of the reasons why platform engineering is increasingly gaining ground. Here, a dedicated platform or infra team develops and operates a so-called "internal developer platform" for the application teams. The focus here is on "developer experience": It should be easy for the application teams to deploy and operate their applications.

In such a scenario, it is obvious that the Platform team designs the GitOps process and creates abstractions that hide some of the overall complexity. The goal is ease of use and self-service for the application teams. In this discussion of teams and software products, thoughts of Conway's law are obvious. Conway recognized a regularity in the architecture of software systems: The communication structures in an organization directly influence the design. This is also true in the design of the GitOps process. In this respect, there is intentionally no standard for GitOps processes and for the structure of the associated repositories (Repos). Every organization is different, so GitOps processes will be different as well.

What actually needs to be done when designing these processes? The challenge is to map the "real world" onto the infrastructure (see figure below). This mapping can be pictured as a chasm that needs to be crossed – the GitOps Chasm. The GitOps Chasm is both a challenge and an opportunity. Overcoming it requires a concept of mapping the "real world" to infrastructure. For example: Are teams mapped as folders, branches or repositories? Will each team have its own cluster? Will each environment be mapped as its own cluster? This concept provides structure and thus maintainability. A start with GitOps thus has a clear advantage. Otherwise, without a concept, you would deploy from different places, repos, CI jobs or manually to the cluster. In the long run, no one can track who deployed what when and why.

Categories of GitOps patterns

While there is no one GitOps process that fits every organization, certain recurring elements have emerged from experience. So we can talk about patterns. More or less synonymous with patterns, the following terms appear in literature and in presentations: Strategies, Models, Approaches, Best Practices. Up to now, these patterns have neither been named nor categorized in a uniform way. Time to change this! Below, GitOps patterns are named and described in the following categories:

• Operator Deployment – Ratio of number of GitOps operators to Kubernetes clusters and namespaces.
• Repository – How many GitOps repos are used?
• Promotion – How are stages/environments modeled?
• Wiring – Bootstrapping the operator, connecting the repos, branches and folders, grouping resources.

Many of the names for the patterns are also used in other literature. An overview of the patterns and sources can be found in the repository cloudogu/gitops-patterns.

In the following parts of this article series, I will introduce the different GitOps patterns in more detail: operator deployment patterns, repository patterns, promotion patterns, wiring patterns and real-life examples.