Sealed Secrets for GitOps Secret Management

Einordnung ins Overtime-Projekt

Overtime wird per GitOps betrieben – sämtliche Cluster-Konfiguration liegt in Git. Damit Zugangsdaten (Datenbank, Object Storage, CI-Tokens) trotzdem sicher versioniert werden können, kommt Sealed Secrets zum Einsatz. Dieses ADR erläutert die Entscheidung und ihre Grenzen; eine Ablösung durch HashiCorp Vault ist als nächster Schritt geplant.

---

1. Kontext & Problemstellung (Deutsch)

Ein GitOps-Workflow speichert den gesamten gewünschten Cluster-Zustand in einem Git-Repository. Reguläre Kubernetes Secret-Objekte enthalten ihre Daten jedoch lediglich Base64-kodiert und nicht kryptografisch verschlüsselt. Das Committen dieser Objekte in ein Git-Repository stellt ein massives Sicherheitsrisiko dar.

Die Secret-Management-Lösung muss es ermöglichen, Zugangsdaten (wie Datenbank-Passwörter oder CI-Tokens) sicher in Git zu versionieren, während sie für jeden ohne direkten Cluster-Zugriff vollständig unlesbar bleiben. Zwar ist perspektivisch der Einsatz von HashiCorp Vault als zentrales Secret-Management geplant, für die aktuelle Phase wird jedoch eine leichtgewichtigere, Git-native Lösung benötigt.

---

2. Architecture Decision Record (English)

ADR-006: Sealed Secrets for GitOps Secret Management

Status

Initially accepted. To be superseded by HashiCorp Vault as soon as possible, as it will improve many areas (signing, centralized secret storage, etc.).

Context and Problem Statement

A GitOps workflow stores all cluster state in git. Kubernetes Secret objects contain base64-encoded (not encrypted) data — committing them in plaintext to a git repository is a critical security failure. The secret management solution must allow secrets to be version-controlled in git while remaining unreadable to anyone without cluster access.

Decision Drivers

• Encryption at Rest: Secrets must be encrypted at rest within the git repository.
• Cluster-Only Decryption: Decryption must only be possible strictly inside the target cluster (where the controller holds the private key).
• GitOps Compatibility: The solution must work natively with ArgoCD's pull-based GitOps model (no pushing secrets into the cluster directly from CI).

Considered Options

• Kubernetes External Secrets + Vault / AWS SSM — Secrets are stored outside git entirely. Requires running a Vault cluster or using a cloud secret manager. Introduces too much operational overhead for the current project phase.
• Sealed Secrets (Bitnami) — Asymmetric encryption (RSA-4096). The public key encrypts the data (can be done by anyone/CI), and the private key decrypts it (done by the cluster controller only). Uses a native Kubernetes CRD.

Decision Outcome

Chosen: Sealed Secrets.

Sealed Secrets fits the operational model perfectly: the kubeseal CLI tool encrypts a standard Secret manifest into a SealedSecret CRD using the cluster's public key. This SealedSecret is safe to commit to git. The Sealed Secrets controller running in the cluster decrypts it back to a standard Secret at apply time. ArgoCD syncs SealedSecret objects natively without requiring extra plugins and can distribute the decryption key onto respective clusters.

Key Management

• A shared TLS key is generated once and backed up offline.
• All clusters (management overtime-mgmt, test overtime-staging, production overtime-prod) currently use the same key, enabling the same sealed secret to be deployed to any cluster.
• Key rotation requires re-sealing all secrets, which is documented in overtime-infra/ansible-playbook/README.md.

Secrets currently managed

Secret	Namespace	Target Cluster
MinIO credentials	minio	overtime-staging, overtime-prod
MySQL credentials	mysql	overtime-staging, overtime-prod
MySQL backup S3 credentials	mysql-backup	overtime-staging, overtime-prod
Django DB URL	django-backend	overtime-staging, overtime-prod
GitLab CI deploy token	argocd	overtime-mgmt

Consequences & Trade-offs

Positive Consequences

• Version-Controlled Secrets: SealedSecret manifests are committed to overtime-gitops and version-controlled like any other Kubernetes manifest.
• Drift Detection: ArgoCD immediately detects configuration drift if a secret is modified in-cluster without updating the corresponding SealedSecret in git.
• Low Infrastructure Overhead: No additional secret store infrastructure (like a dedicated Vault cluster) is required to operate initially.

Negative Consequences / Trade-offs

• Shared Key Risk: Sharing a key across clusters means that a compromise of the controller's private key exposes all sealed secrets across all clusters. This is mitigated by restricting access to the management cluster and backing up the key to offline storage.
• Scope Constraints: Sealed secrets are cluster-scoped by namespace and name by default. A secret sealed for overtime-staging/mysql cannot be decrypted in overtime-prod without re-sealing. (This is a security feature, but can cause friction during environment cloning).
• Cumbersome Rotation: Key rotation requires all sealed secrets to be re-generated manually. This must be planned as a periodic operational task.

---

3. Implementation (English)

The implementation spans the cluster bootstrapping process and the GitOps manifests:

• overtime-infra/ansible-playbook/ — The gitops-setup.yml Ansible playbook bootstraps the Sealed Secrets controller into the cluster and injects the offline-backed-up private key.
• overtime-gitops/ — Contains the encrypted SealedSecret manifests per environment overlay, ready to be synced by ArgoCD.

This is what an example manifest that is committed to git looks like:

---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  annotations:
    sealedsecrets.bitnami.com/namespace-wide: "true"
  name: minio-credentials
  namespace: minio
spec:
  encryptedData:
    MINIO_ROOT_PASSWORD: AgBm9gMEB1xTMxV...=
    MINIO_ROOT_USER: AgDSx0Ok...77SHE
  template:
    metadata:
      annotations:
        sealedsecrets.bitnami.com/namespace-wide: "true"
      name: minio-credentials
      namespace: minio

---

4. Compliance Einordnung (Deutsch)

TBD