GitOps and Kubernetes – Secure Handling of Secrets

Especially in well established companies, it may be the case that a certain way of handling secrets has become established, which is convenient, but not necessarily the safest solution. One example of this is the storage of secrets directly in the CI server. The use of a key management system (KMS) for storing secrets is a more secure solution, but it initially requires a lot of effort for the changeover. That is why KMS are not always used in the Kubernetes environment at present. GitOps with its declarative approach forces a rethink in the handling of secrets.

Kubernetes and Secrets

With Kubernetes, Secrets are basically stored unencrypted in the API server's underlying data store (etcd). This means that anyone who has access to the etcd can also obtain or modify secrets. Therefore, at least these measures should be taken:

• Enable encryption of Secrets using Encryption at Rest,
• implement a least-privilege approach for Secrets by configuring RBAC rules,
• restrict access to Secrets to specific containers by mounting Secrets only in containers where they are needed,
• finding a way to get Secrets into the cluster (for example, using the CI server), and
• evaluating the use of an external provider for storing Secrets (KMS).

This shows that even without GitOps, secure and appropriate handling of Secrets in Kubernetes is not trivial. This is reinforced by the declarative approach of GitOps, since Secrets must also be stored or at least referenced outside the cluster, namely in the source code management. This enforces a secure handling of Secrets. Most of the solutions described in the following are not only exclusively applicable with GitOps, but are basically applicable when using Kubernetes.

GitOps enforces different handling of secrets

With GitOps, the state of a project is described declaratively in the source code management and the deployment environment continuously synchronizes with Git. The CI server is only used for the build, the deployment is done by the GitOps operator. This approach allows to take automation to a new level, but also leads to the need to find a new way of handling secrets. By storing the entire state in Git, a way must inevitably be found to store secrets in a secure manner in Git, because stored directly and unencrypted in Git offers a large attack surface. For the secure handling of secrets there are basically two approaches:

• Encrypted storage of the Secrets in Git
• Storing the secrets in a Key Management System (KMS) and referencing them in Git.

Encrypted storage in Git

Sealed Secrets (Operator)

An option that easily works with GitOps is the Operator Sealed Secrets from Bitnami. Secrets encrypted with it can only be decrypted by operators running inside the cluster, not even by the original author. For encryption, there is a CLI (and a third-party web UI) that requires a connection to the cluster. The disadvantage of this is that the key material is stored in the cluster, the secrets are bound to the cluster and one has to take care of backups and operation.

Use of Key Management System (KMS)

When using a KMS, the first step is to choose a suitable tool. Since the major operators of public clouds such as Google, Microsoft, Amazon, etc. usually offer a KMS that is easy to use and Hashicorp Vault has established itself as a solution for on-premises operated clusters, the first step is quickly completed. The second step is then to integrate the KMS into the cluster. There are several ways to do this:

• Running a special operator
• Mounting secrets in the file system using Container Storage Interface (CSI)
• Injecting a side car into pods
• GitOps operator with either native support for KMS or via plugin

External Secrets (Operator)

External Secrets is an operator that integrates external KMS such as Hashicorp Vault or those of the major cloud providers. It reads secrets from the external APIs and injects them into Kubernetes secrets. The operator is a new implementation after the merge of similar projects from GoDaddy and ContainerSolutions.

Secrets Store CSI Driver

Mounting secrets into the file system of pods is another exciting way to go, as the CSI Driver is an official part of Kubernetes. It is developed by the Kubernetes Special Interest Group, making it potentially very durable. The CSI Driver provides providers for popular vendors, which in turn are developed by the vendors themselves.

Hashicorp Vault k8s (Sidecar Injector)

Hashicorp Vault k8s is an operator that modifies pods via a mutating webhook to connect between vault and pod via sidecars (additional containers) to provide secrets. This has the major advantage that no secret objects are created in Kubernetes here. The disadvantage is that this way only works with Vault.

SOPS

Mozilla Secret Ops (SOPS), already known from the time before Kubernetes, offers even more options – at the expense of a more complex configuration. Here, the key material can come from the key management systems (KMS) of the major cloud providers, an own HashiCorp Vault or from self-managed PGP keys. Since it is not Kubernetes-native, SOPS does not include an operator, but there are several different ways to use it with GitOps.

• Flux v2 provides native support.
• ArgoCD supports SOPS with the vault Plugin.
• There is also the helm secrets plugin, which can also be used in ArgoCD with manual configuration.
• There is also a third-party sops-secrets operator available.

Example: Hashicorp Vault with External Secrets Operator in the GitOps Playground

You can see first hand and try out the management of secrets with Hashicorp Vault and synchronization into the cluster with the External Secrets Operator in the GitOps Playground. The GitOps Playground is an open source project for trying out GitOps including a sample application. To the GitOps Playground

Conclusion

The declarative approach of GitOps leads to the need to rethink the handling of secrets. The curse and the blessing is that there are many options that enable secure handling of secrets. Thanks to the different approaches there are suitable operators for different requirements. We would be happy if you post questions, feedback or suggestions for other operators, the GitOps Playground or GitOps in general in our GitOps community.