Automation assistants: GitOps tools in comparison

The term GitOps is a combination of the name of the source code management system Git and the abbreviation Ops as in operations. The idea for adding this additional tool to the DevOps toolbox comes from the Kubernetes environment, and it promises a new level of IT automation. Like the continuous delivery approach, GitOps relies on maintaining all information in source code management. The difference, however, is that the deployment environment synchronizes its state continuously from Git, and the CI server is not responsible for the roll-out. The configuration must therefore be versioned in Git. Since this is treated like code, it is referred to as “Infrastructure as Code”.

It is no wonder then that there now is a growing number of GitOps tools to choose from. But what range of features do they offer? Is a single one sufficient, and can it automate “everything”? This article answers these and similar questions on the basis of specific examples. It lays out selection criteria and illustrates them by comparing the well-known GitOps tools ArgoCD and Flux v2.

GitOps tools: a confusing market

Compiling an overview of GitOps tools available on the market is not as trivial a task as it might sound. On the one hand, this is due to the fact that there is a certain amount of hype surrounding the term, and vendors like to add the term GitOps to their products for marketing purposes. On the other hand, it is difficult to clearly define the term GitOps, and it is used on different levels of the stack (from physical infrastructure to the applications that are running in the cloud) in varying levels of maturity. Here, the GitOps principles of the OpenGitOps project of the Cloud-native Computing Foundation (CNCF) will probably bring more clarity. The article “Hands off!” (published in iX 4/2021 - only available in German) delves into the topic in more detail.

Websites such as awesome-gitops, which was launched by Weaveworks, or gitops.tech, which was put together by INNOQ employees, provide an introductory overview of the available tools. When you take a closer look, you will see that the listed tools can be used to perform a wide variety of tasks related to implementing GitOps, and of course they also differ from one another in terms of their adoption, maturity, and how actively they are maintained. This article identifies three categories from the various use cases: Tools for Kubernetes, supplementary tools, and tools close to infrastructure. In addition, we compiled a table that summarizes the tools and their properties. The tables also contain various Git and GitHub-based metrics (current as of February 2021) that allow you to better assess their adoption, maturity, and how actively they are maintained.

Tools for Kubernetes

When it comes to GitOps tools, the first thing that usually comes up is the topic of operators for Kubernetes. In general, an operator (which is often also called a “custom controller”) is an application that runs in the Kubernetes cluster and automates operational tasks there (see Figure 1). This operator pattern is also used to implement GitOps. The GitOps operator is used to run the reconciliation loop, which synchronizes the target state declared in the Git repositories with the actual state of the cluster. In the event of differences (e.g., due to a new commit in Git), the operator takes care of convergence to the target state by applying Kubernetes resources to the API server. Experience has shown that additional features that go beyond the core feature set are required for efficient operation. These include observability and a command line interface (CLI) or a user interface (UI). We will learn more about this later in the “Criteria for selecting the right tool” sidebar (at the end of the article).

The blog post by Weaveworks, which coined the term GitOps in 2017, also names the first GitOps operator: Flux. In the meantime, this has been completely rewritten as Flux v2. In addition to Flux and Flux v2, the associated project “Flux” develops other components. Weaveworks has now handed the project over to the CNCF. By now, the project is in the third and therefore highest maturity level: graduated.

ArgoCD offers an alternative to Flux. It belongs to the Argo project, which is also based at the CNCF, and which is, just like Flux, in the third maturity level (graduated). A comprehensive comparison of the two GitOps operators can be found later in the article.

A newer competitor is Fleet, which is developed by Rancher. Its special ability is that it is able to manage not just one, but a fleet of clusters. PipeCD is similarly young and has an even broader focus. Like Fleet, it promises the ability to manage multiple Kubernetes clusters, and it also offers a UI. In addition, it can handle Terraform and some services from the major cloud providers.

GitOops combined with CI

Jenkins X also offers a broader focus, but in a different area. Contrary to what the name suggests, it differs greatly from the well-known Jenkins server. It’s not a monolithic tool, but rather it consists of different components, such as Tekton for running pipelines and Kaniko for building images. At the heart of Jenkins X is the CLI “jx”.

Overall, Jenkins X is more powerful than ArgoCD and Flux, and therefore it is more difficult to integrate into existing workflows. But it offers a significantly larger range of features. It is opinionated, i.e. it relieves the user of having to make many decisions, but it also reduces flexibility. Jenkins X is a complete continuous integration and continuous delivery (CI/CD) package. In contrast to this, the pure GitOps operators need an additional CI server for many use cases, which, for example, automates tests and builds images and makes them available in the registry.

Werf positions itself somewhere between a pure GitOps operator and a full CI/CD approach. The project was started under the name dapp, and then renamed werf in early 2019. Like an operator, it can apply Kubernetes resources from Git to a cluster. However, it runs outside of the cluster. This means that it does not utilize the pull principle, which is often associated with GitOps, in which the cluster itself pulls its target state from Git. Unlike ArgoCD and Flux, werf can also build images. An operator that runs in Kubernetes is planned (as of version v1.2).

GitOps loves operators

A central point of GitOps is the complete declarative description of the state in Git. During the process of converting from the imperative, classic CI/CD (now referred to as CI-Ops in some places to differentiate it from GitOps) certain questions will naturally arise. How do you continue to use templating tools such as Helm or Kustomize that were previously executed by the CI server?

The answer to this is usually: by using additional operators. These expand the Kubernetes API server using so-called Custom Resource Definitions (CRDs) and then listen for changes to the associated Custom Resources (CRs). These CRs allow the desired state to be described declaratively, which is a perfect match for GitOps.

Flux brings with it Helm and Kustomize operators, which allow Helm releases and Kustomizations to be described declaratively via CR. If such a CR is applied to the cluster (typically using the GitOps operator), the Helm or Kustomize operator takes over the templating or overlay. As an alternative to the operator, the result of templating can be written to Git (for example, via a CI pipeline). The advantages include reduced efforts for maintaining infrastructure and more transparency in the Git repository. However, these advantages are counterbalanced by the need to create comprehensive and difficult-to-maintain YAML descriptions in Git.

By storing the entire state in Git, this inevitably leads to the creation of secrets there, too. However, if they are left unencrypted, these present a large attack surface. The answer to the question of how encryption and decryption can be combined with GitOps is yet again: through additional operators.

Components for better security

One simple option that works well together with GitOps is Bitnami’s Sealed Secrets operator. It manages the key material in the cluster itself. There is a CLI for encryption that requires a connection to the cluster.

SOPS that was developed by Mozilla offers significantly more options, though at the expense of a more complex configuration. Here, the key material can come from the key management systems (KMS) of the major cloud providers, from your own HashiCorp Vault, or from configured PGP keys. SOPS itself does not contain an operator, but there are different ways to use it with GitOps. Flux v2 offers native support. There is also the helm-secrets plug-in, which can also be used in ArgoCD with the manual configuration. There is also a sops-secrets operator that has been developed by a third party.

Kamus may represent a compromise between Sealed Secrets and SOPS. It was created especially for the GitOps use case and includes an operator. It can either manage the key material itself or obtain it from the KMS of the cloud providers. Another special feature is that Kamus encrypts secrets directly for an application. They are then decrypted by the application itself or by an init container. This means that the unencrypted secret is never present on the API server and ideally also not in an environment variable with in the container.

If you are using an external KMS in any case, then there are other options, such as the kubernetes-external-secrets operator that was originally started by GoDaddy and the externalsecret-operator from Container Solutions. If you use HashiCorp Vault, you also have the option of using the Vault Secrets operator. This works similarly to the Sealed Secrets Operator, but instead of managing its own key material, it retrieves the secrets from Vault. The CNCF Technology Radar from January 2021 provides an overview of the types of tools that are available for secrets management.

Supplementary GitOps operators can also be used for deployment strategies, such as canary releases, A/B tests, and blue/green deployments, which have now been grouped under the term “progressive delivery”. The resources of most GitOps operators are not sufficient for this. One solution is Flagger. The tool that was launched by Weaveworks is now being developed as part of the Flux project. The Argo project also has an operator for this use case: Argo Rollouts. Both offer CRs for implementing progressive delivery strategies in interaction with various ingress controllers and service meshes.

Tools close to infrastructure

The term GitOps arose originally in the context of application deployments in Kubernetes. The tools for this use case are very mature. They are not limited to this use case, however. GitOps operators can also be used to roll out Kubernetes clusters. One scenario is to use the Kubernetes Cluster API, which was started as kube-deploy and renamed Cluster API (CAPI) in 2018. This can be implemented as follows: A GitOps operator runs in a management cluster and applies the CRs (defined by CAPI CRDs) stored in Git to the cluster. A infrastructure provider also running in the cluster reads these CRs and applies them to a target cluster.

As soon as this cluster is in place, the question arises of how applications can be rolled out there. Flux and ArgoCD can access the API server directly for this purpose. For other operators, such as Fleet and PipeCD, the architecture is specially designed for this case. They each offer one component for the management and target cluster: With Fleet, a manager operator connects to the agent operators, and with PipeCD, a control plane connects to daemons. So this does not require access to the API server from outside the cluster.

In addition to creating Kubernetes clusters, there is also an increasing number of opportunities to use various Infrastructure-as-Code (IaC) tools, such as Terraform, with GitOps. As was already mentioned, PipeCD offers support for Terraform. Terraform’s vendor, HashiCorp, now also offers an official Terraform Kubernetes operator. However, it needs access to HashiCorp’s Terraform Cloud. Alternatively, there are also third-party operators that can function without Terraform Cloud, such as the one developed by Rancher. However, it is still in alpha stage.

Another popular alternative for GitOps with Terraform is Atlantis: When generating a pull request, basing on the Terraform files that are found in the Git repository, it creates the Terraform plan and adds it to the pull request as a comment. After merging, it applies the Terraform plan. Atlantis is compatible with various Git providers. It can be flexibly hosted as either a binary, a Docker image, or as a Helm chart for Kubernetes. This makes it one of the few tools that will be of interest to those who want to implement GitOps without Kubernetes. Ansible Tower (and thus its open source upstream AWX) is also independent of Kubernetes. Red Hat considers the range of features to be comparable to a GitOps operator.

The example of Atlantis is, so to speak, symbolic of GitOps outside of Kubernetes. The pull principle of GitOps is more difficult to implement without a platform like Kubernetes that can run an operator. What remains often is the topic of “Operations by Pull Request”. However, with CI Ops the creation, modification, or merging of pull requests also triggers a CI/CD pipeline. This raises the question of whether the use of pull requests is sufficient for operations to be referred to as GitOps. It is more difficult to distinguish here. In fact, Atlantis does not refer to itself as a GitOps tool. But there are other examples, such as the Terraform alternative Pulumi. It works together with CI tools, such as GitHub Actions and GitLab CI, and it can be used to comment on pull requests on the associated platforms. Pulumi calls this GitOps.

Finally, there is one tool in particular that we should not leave out when creating a list of GitOps tools that are close to the infrastructure: Ignite, which was also launched by Weaveworks. It allows you to manage virtual machines (VMs) via GitOps. In order to do this, it runs a daemon on the physical host that can start and stop VMs in accordance with a description that is stored in a Git repository. Firecracker, which was originally launched by AWS, is used as virtualization technology.

So much and yet so little

Ironically, despite the abundance of tools, there are use cases that cannot yet be automated using GitOps. The selection of GitOps tools that are close to the infrastructure is significantly smaller than those for Kubernetes. The closer you get to the physical infrastructure, the less the tool support gets. Examples of tasks in operations that cannot yet be done via GitOps are starting physical machines, for example via the Preboot Execution Environment (PXE), or performing firmware upgrades in devices such as switches.

But even with the additional operators, not all use cases in operation can be fully automated with GitOps. One example is the Horizontal Pod Autoscaler in Kubernetes. To be compatible with GitOps, it would have to write the change in the number of replicas to a Git repository instead of sending it directly to the API server.

Another, even more complex example are the topics of persistence, backup, restore, and disaster recovery. Backup creation can be automated reliably using such operators as Velero. However, restoring a backup requires manual intervention. You can roll back to the state described in the GitOps repository, but this does not restore the state saved by the backup operator. Conversely, performing a manual restore using the backup operator does not reset the state in the GitOps repository.

Despite the abundant number of tools, there are still gaps in the feature set of the GitOps tool chain. Given the current dynamic of development, there is room for optimism that these gaps will be closed in the foreseeable future. In general, many of the available GitOps tools are based on Kubernetes. Even setting up the cluster or other infrastructure requires a Kubernetes cluster. Implementing GitOps without Kubernetes might requirea a bit of pioneering.

From the abundance of tools described above, it is now important to find the right one that satisfies your own requirements. The decision that you make will depend heavily on your use case. The “Criteria for selecting the right tool” sidebar (at the end of the article) will show you what to look out for.

Operators in comparison: ArgoCD vs. Flux v2

An example can better illustrate the criteria listed in the sidebar. So it makes sense to compare the two best-known GitOps operators, ArgoCD and Flux v2. As both projects continue to develop rapidly, so the comparison presented here can only be treated as a snapshot.

In order to gain an initial overview of similarities and differences, it is worth taking a look at the feature lists.

Both Flux v2 and ArgoCD have the following capabilities:

• RBAC
• Multi-tenancy
• Observability (health status, notifications, and metrics)
• CLI for automation and CI integration
• Start of synchronization via a webhook
• Calling webhooks using events (ArgoCD: hooks, Flux: notifications)
• Rollback/roll-anywhere for certain commits in the Git repository
• Support for Helm and Kustomize
• Multi-cluster support
• Execution of the container as an unprivileged user
• Security context is configurable

Features only in ArgoCD:

• Support for Ksonnet, Jsonnet, and others via the plug-in mechanism
• Web interface to administer and monitor applications in real time
• SSO integrations for UI and CLI

Features only in Flux v2:

• Definition of dependencies in Helm and Kustomize applications
• Support for SOPS
• Automatic updating of the image version in the Git repository
• Authentication of the CLI via Kubeconfig

Installation and configuration

ArgoCD can either be installed using simple Kubernetes resources and then patched or it can be rolled out using a configurable Helm Chart. All of the common settings are available, such as ingress and service accounts. Repositories, projects, applications, and various SSO integrations can also be preconfigured for deployment. The RBAC configuration is also extensive. You can define policies, roles, and groups yourself.

Another installation variant is the additional ArgoCD operator. This allows the actual ArgoCD components to be installed and configured via CRD. It is not documented how you can configure ArgoCD yourself via GitOps. This is conceivable, for example, using the ArgoCD operator. It remains to be determined whether this will work reliably and, above all, whether it supports continued operation via GitOps in the event of an error.

The Flux v2 operators are rolled out primarily via the CLI. However, it can also be used to generate all the necessary Kubernetes resources and to then deploy them to the cluster. There is currently no Helm Chart, which limits the usability of the common application mechanisms, however, in the meantime there is the possibility to install Fluxv2 via Terraform. During the initial installation, Flux sets up a Git repository. From this point forward, you can also configure the operator with GitOps. The CLI is extensive and allows you to control all aspects of the Flux v2 portfolio. You can use it to create, delete, read, start, and stop all Flux-v2-specific resources. This makes Flux v2 particularly suitable for scripting. This may seem contradictory for GitOps at first, but in practice there often is a gradual migration. The CLI can be used to create resources, which are then checked into a Git repository. This is useful for integratinh GitOps into existing CI/CD processes.

Application configuration

ArgoCD stores the target state internally through projects and applications. Repositories, clusters, and permissions can be defined in projects. In turn, any number of applications can be assigned to a project, where they will have access to the project’s clusters and repositories. ArgoCD saves these applications and projects as CRs in the cluster. The repository and the path to a deployment can then be found in an application. In addition, you can also describe configurations for Helm, Kustomize, etc. in declarative form there as well. ArgoCD then applies this deployment, which is defined via the application, to the cluster. However, ArgoCD does not save Helm releases as such in the cluster, but converts them into simple Kubernetes resources using helm template and applies them to the cluster. In contrast to Flux, the Helm releases cannot be queried via helm ls and there is no history of the releases in the cluster.

All relevant elements are defined using CRDs, which can be configured using GitOps. Specifically, this means that you can maintain the projects, applications, etc., which are defined as CRs in your own Git repository.

With Flux v2, the repositories form the central point. They are used both for the deployment of workloads and the configuration of Flux itself. Flux can roll out Kubernetes resources, Helm Release CRs, and Kustomize CRs as workloads. The Helm Release CRD and Kustomize CRD also offer all of the known features of these tools in descriptive form. Since the Git repositories are also defined as CR, they can also be managed via GitOps.

UI

ArgoCD comes with a web interface for administering and monitoring applications. Since GitOps is an asynchronous process, you do not receive immediate feedback when changes are made to deployments. The included notification mechanisms provide a remedy for this issue (see the Observability section). They provide feedback through various channels, such as chat or e-mail. In addition, ArgoCD offers the opportunity to monitor deployments in real time via the UI (see Figure 2). If parts of the deployment fail, you can identify them via the UI and view error messages and logs.

This allows the developer to analyze their deployments and correct errors all without having to access the cluster. For authentication , there are interfaces for common protocols, such as LDAP and OIDC. Via configurable roles and groups, users can granted access the projects and applications for which they are responsible. The developers of Flux v2 are currently working on a web interface. However, it is still in an experimental state.

CLI

All relevant features can be performed from the CLI for ArgoCD. Users can create and delete objects, such as applications or projects, as well as change states, such as rolling back applications or triggering synchronizations. This makes the CLI suitable for all types of automation and integration in CI pipelines. The CLI communicates with the ArgoCD server, which makes exposing the Kubernetes API server unnecessary.

The Flux v2 CLI can also be used to access all relevant features. It also offers the option of rolling out additional tenants, which can be advantageous when automating a multi-tenancy environment. The CLI communicates directly with the Kubernetes API server, which must therefore be accessible from outside.

Authorization

ArgoCD relies on an organizational structure based on projects and applications. Applications, Git repositories, clusters and permissions (RBAC), as well as allow and deny lists for resource types can be assigned to a project. Applications offer the same options for restricting permissions. This allows users to assign permissions very easily and very accurately, including at both the project as well as the application levels.

Flux v2 currently only supports restricting the operators, Helm releases, and Kustomization CRs via RBAC. Simple Kubernetes resources, such as ConfigMaps and services, can only be controlled via the operator’s permissions. At the project level, this might requires several instances of the Flux operator to assign project-specific permissions. Alternatively, access to the GitOps repository can also be implemented via a CI pipeline, which then only allows permitted resources to be pushed into the GitOps repository.

Observability

To send notifications about the synchronization status between Git and the cluster, ArgoCD utilizes the ArgoCD Notifications component, a notification system that is delivered with ArgoCD deployment. Events that arise when applications are rolled out can thus be forwarded to various channels. ArgoCD Notifications are currently supported on the SMTP, Slack, Opsgenie, Grafana, Telegram, and Webhooks channels.

As an alternative to ArgoCD Notifications, you could use Argo Kube Notifier and Kube Watch. However, these have to be operated in addition to ArgoCD. In any case, ArgoCD Notifications is best tailored to ArgoCD. It offers useful triggers and templates and can be installed and configured using the project’s own Helm Chart.

To monitor the components and managed deployments, ArgoCD exposes two sets of Prometheus metrics: “Application metrics” for monitoring the status of the synchronization and the health status of deployments as well as “API Server Metrics” for monitoring the requests and responses to the API server. In addition, there are ready-made Grafana dashboards that are based on these metrics. This makes it very easy to implement a monitoring cockpit for the entire system.

Flux v2 has a special controller for Observability, the notification controller. As is true for all other Flux components, alerts and notifications are configured via corresponding CRDs. They can be used to set up providers, i.e., channels, such as chats and webhooks, alert rules and recipients. The associated CRs can also be maintained in a Git repository and deployed via Flux v2. The Notification Controller is currently suitable for the Slack, Discord, Microsoft Teams, RocketChat, Google Chat, Webex, Sentry, Azure Event Hub and Generic Webhooks channels.

There is also the option of attaching the status to a Git commit. This can be done with the following providers: GitHub, GitLab, Bitbucket, and Azure DevOps. At the moment, Flux v2 cannot connect a provider via the SMTP protocol in order to send e-mails in this way.

There are a number of Prometheus metrics and Grafana dashboards that are available for monitoring the controllers and deployments (see Figure 3). On the one hand, this allows you to monitor the error-free execution of the controller, providing you with statistics on CPU and memory consumption, for example. On the other hand, you can monitor the state of all Flux v2 CRDs, such as Helm releases and Kustomizations. This also makes it possible to send alerts by e-mail again. The alert rules can be defined in either Grafana or Prometheus Alert Manager. Prometheus and Grafana can also be installed and configured using the Flux CLI. In a multi-tenant environment, it can be used to automate deployment using scripts.

Conclusion

Both Flux v2 and ArgoCD offer many features that also reflect years of practical experience with GitOps. Both focus strongly on the GitOps core features, which makes them easy to integrate into existing CI/CD infrastructure. However, if you start out with a greenfield approach, you have to build it up separately.

Overall, ArgoCD offers more options for configuration (for example, for authorization) and provides a graphical user interface. However, this UI has to be configured and operated, and it also offers a larger attack surface. It depends on the use case as to whether a UI is needed at all, and this thus justifies the higher effort. Users of OpenShift might encounter less effort, since ArgoCD is integrated into the platform as OpenShift GitOps.

Flux has certain little features that ArgoCD doesn’t have, such as support for SOPS and automatic updates for new image versions. Flux v2 is still in a 0.x version. However, the authors promise to keep the APIs stable and therefore consider the product ready for production. One possibility to see ArgoCD and Flux v2 in action and compare their features is the GitOps Playground project that was started by the authors.

Would you like some more?

There is a large number of tools that either bring themselves in connection with GitOps or that are often named in that context. The authors are available for discussions about the topic in the [GitOps discussion forum at heise online](https://www.heise.de/forum/iX/GitOps/forum-467865/ "GitOps topic on Heise online). To meet the often-asked question after market overviews “Why is program X not mentioned?” head on, here are a few reasons why this article does not consider some candidates. Left out are tools that

• Are only called imperatively or in the CI/CD process and therefore do not have a reconciliation loop (e.g. templating tools), even if they decorate themselves with the term GitOps;
• Are no longer developed actively;
• Are generally not recommended for use in production;
• Have a highly limited use case, for example tools that are tailored for a specific cloud provider;
• Are proprietary;
• Are still very new and therefore not yet widely used.

Details about the mentioned standardization of the term GitOps can be found in the article “Hands Off”, published in iX 4/2021.

Criteria for selecting the right tool Most of the GitOps tools are currently available in the Kubernetes environment. It is no wonder, because this is the oldest and most mature GitOps use case. This sidebar summarizes the important requirements based on the authors' own experience that you should consider when making a decision. However, many of these criteria apply not just to using Kubernetes. In this respect, they can also prove to be helpful for using GitOps in other environments.

Installation and configuration of the GitOps operator As is well known, there are many ways to roll out applications in a Kubernetes cluster. This also applies to GitOps operators. Common options are rolling out via Helm Chart, CLI, or in the form of simple Kubernetes resources. Another aspect is changing the configuration of the operator once it is deployed. There is an interesting point to note here: Can the operator itself be configured via GitOps?

In addition, it is interesting to note what parts can be adapted at runtime (for example, using CRs – Custom Resources) and which parts require the operator to be restarted. Does the tool permit a multi-tenancy solution? An operator capable of managing multiple cluster can be helpful for implementing a multi-tenancy solution. Operators can support this using their own components in the target cluster, or they can communicate directly with the API server. Depending on the infrastructure, direct access to the API server may or may not be desired.

Application configuration The Git repository must be defined and configured so that the GitOps operator can synchronize the cluster with a Git repository. It is important to map existing project structures with the operator used. It is also be interesting to find out whether the operator can handle multiple Git repositories and whether these can be added and configured without restart. In addition to the use of simple Kubernetes resources, there are other ways of rolling out applications in a Kubernetes cluster. These include Kustomize, Helm, Ksonnet, and Jsonnet.

The specific application types that an operator supports and whether they can be configured in accordance with the existing requirements will influence whether you decide to use it. Hooks can also play an additional role, i.e., they can react to events during the deployment in order to send messages, perform checks, or influence the deployment. Whether you need to map dependencies between resources may also be relevant. Examples of this include applying CRDs before associated CRs, or ensuring a database runs before the application is started. Occasionally, tools also offer additional opportunities for automation, such as, building images, writing new image versions to Git, or for executing entire pipelines.

UI A graphical user interface allows easy access to the operator configuration and the application resources. It enables error analysis and handling without accessing the cluster by monitoring cluster objects and manipulating manifests. However, it should be noted here that the changes are not synchronized in the Git repository. Only the associated CRs in the cluster are changed. A single sign-on (SSO) is advantageous, since an existing user management can be used to enable users to access the UI.

CLI A CLI can be used to develop scripts that automate GitOps processes or integrate them into CI pipelines. It is conceivable that new clusters will be rolled out, including operators, or the reconciliation loop will be triggered in existing workflows. The range of features that can be performed from the CLI will be a more important consideration depending on how extensively the CLI will be used.

Authorization The assignment of permissions is of central importance for GitOps, since in principle everything that ends up in the Git repository is also applied to the cluster. Critical or security-relevant objects could be changed, or the cluster could be compromised or otherwise rendered unusable. One countermeasure is to restrict the operator's access to certain types of resources. Role-based access control (RBAC) is suitable for making sure that the operator is not rolled out with the permissions of a cluster administrator. Additional restrictions that you impose using allow and deny lists on resources and resource types can further improve security.

Observability Due to the asynchronous nature of GitOps, feedback mechanisms play an important role. Because it is only after the operator has already deployed resources on the cluster that it is possible to know whether deployment was successful. Additional tools are required to receive these notifications (e.g., chat, e-mail, metrics, or commit status). Here you must consider which tools can be used and what effort integration into your own workflow is necessary.