ADR-015: Supply-Chain Signing — Docker Images, SBOMs, and GitOps Commits

Einordnung ins Overtime-Projekt

Damit im Overtime-Cluster nachvollziehbar ist, dass ein laufendes Image tatsächlich aus der autorisierten Pipeline stammt und nicht manipuliert wurde, werden Images, SBOMs und GitOps-Commits kryptographisch signiert. Dieses ADR beschreibt das Zwei-Schlüssel-Modell für Dev- und Prod-Umgebungen sowie die GPG-Signierung der GitOps-Commits.

Status

Work in Progress to be tested and refined.

Context and Problem Statement

The CI pipeline builds Docker images, generates Software Bills of Materials (SBOMs), and commits image-tag updates to the GitOps repository. Without cryptographic signing, there is no way to prove that:

• an image running in the cluster was produced by the authorised pipeline and has not been tampered with,
• the SBOM attached to an image accurately describes what was built, or
• a kustomize overlay commit was written by the CI bot and not a third party with repository write access.

Decision Drivers

• No artefact metadata or key material may be written to public logs — the infrastructure is entirely private
• Every built image must be signed regardless of branch or MR — not only protected refs
• The production cluster must be able to enforce that only PROD-key-signed images are admitted; the dev/testing cluster uses a separate DEV key
• Private key material must never appear in job logs
• The solution must be straightforward to migrate to HashiCorp Vault with minimal pipeline changes

Considered Options — Image Signing

• Cosign keyless signing (Sigstore OIDC + Rekor) — no key management; GitLab OIDC token used to obtain a short-lived certificate from Fulcio; pipeline identity is written to the public Rekor transparency log. Ruled out: incompatible with private infrastructure (project path, pipeline URL, and commit SHA would be publicly visible).
• Cosign key-based signing — static ECDSA key pairs; private keys stored as masked CI variables; signatures stored in the OCI registry as artefact attachments with --tlog-upload=false. Chosen.
• Notation (Notary v2) — CNCF standard designed exclusively for private registries and enterprise KMS; no public log concept. Viable future migration target; ruled out now as unnecessarily complex for current scale.

Considered Options — GitOps Commit Signing

• Unsigned commits (status quo) — CI bot commits are indistinguishable from any other committer with write access to overtime-gitops. Ruled out.
• SSH commit signing — simpler key management but requires GitLab to recognise the key on the ci-bot account for the "Verified" badge; SSH signing of git commits has narrower ecosystem tooling support. Not chosen.
• GPG commit signing — mature, widely supported by Git, GitLab, and auditing tooling; GitLab displays a "Verified" badge on signed commits when the public key is registered on the signing account. Chosen.

Decision Outcome

Three complementary controls are implemented together:

1. Cosign image signing with two environment-scoped key pairs
2. Cosign SBOM attestation binding the CycloneDX SBOM to each signed image digest
3. GPG commit signing for every GitOps overlay commit made by the CI bot

---

Implementation

1. Key Generation (run once per key pair on a secure, offline-capable machine)

Cosign key pairs

# DEV key — for branch and MR pipelines (images deployed to dev/testing clusters)
cosign generate-key-pair
mv cosign.key cosign-dev.key
mv cosign.pub cosign-dev.pub

# PROD key — for tag pipelines only (images deployed to production clusters)
cosign generate-key-pair
mv cosign.key cosign-prod.key
mv cosign.pub cosign-prod.pub

Each command prompts for a passphrase that encrypts the private key at rest.

CI bot GPG key (no passphrase — see rationale below)

gpg --batch --gen-key <<EOF
Key-Type: EdDSA
Key-Curve: ed25519
Key-Usage: sign
Name-Real: Overtime CI Bot
Name-Email: ci-bot@overtime.local
Expire-Date: 2y
%no-protection
%commit
EOF

# Export private key (store in CI variable)
gpg --armor --export-secret-keys ci-bot@overtime.local > ci-bot-gpg.key

# Export public key (register on the ci-bot GitLab account and in overtime-gitops)
gpg --armor --export ci-bot@overtime.local > ci-bot-gpg.pub

The GPG key has no passphrase because it is generated solely for non-interactive CI use. The private key is protected by GitLab CI variable masking and the protected-variable mechanism; adding a passphrase would require gpg-agent socket management inside the container and provides no meaningful additional protection in this threat model.

---

2. CI Variable Setup

All variables are set in GitLab project Settings → CI/CD → Variables.

Variable	Type	Masked	Protected	Value
COSIGN_PRIVATE_KEY_DEV	Variable	✓	✗	Content of cosign-dev.key
COSIGN_PASSWORD_DEV	Variable	✓	✗	Dev key passphrase
COSIGN_PRIVATE_KEY_PROD	Variable	✓	✓	Content of cosign-prod.key
COSIGN_PASSWORD_PROD	Variable	✓	✓	Prod key passphrase
GITOPS_GPG_PRIVATE_KEY	Variable	✓	✗	Content of ci-bot-gpg.key
GITOPS_DEPLOY_TOKEN	Variable	✓	✗	GitLab token with write access to overtime-gitops (pre-existing)

COSIGN_PRIVATE_KEY_DEV and GITOPS_GPG_PRIVATE_KEY are not Protected because they are needed on feature branches and MRs. COSIGN_PRIVATE_KEY_PROD is Protected so it is only available on protected refs and tags.

---

3. Pipeline Integration

sign.yml — reusable Cosign templates

Two hidden job templates, both using gcr.io/projectsigstore/cosign:v3.1.1 (digest-pinned):

• .sign-image — calls cosign sign --key env://COSIGN_PRIVATE_KEY --tlog-upload=false on the canonical image tag. Signs the digest; other tags pointing to the same digest are implicitly covered.
• .attest-sbom — calls cosign attest --key env://COSIGN_PRIVATE_KEY --predicate bom.cdx.json --type cyclonedx --tlog-upload=false to bind the CycloneDX SBOM to the image as a verifiable in-toto attestation.

Both templates use the generic COSIGN_PRIVATE_KEY / COSIGN_PASSWORD variable names. Each concrete job in the consuming pipeline maps the right key pair via its variables block.

update-gitops.yml — GPG commit signing

gnupg is installed at job runtime (apk add --no-cache gnupg). The CI bot key is imported from GITOPS_GPG_PRIVATE_KEY, the key fingerprint is extracted, and git is configured with commit.gpgsign true. All subsequent git commit calls in the job are automatically GPG-signed. The keyring is removed in after_script.

Note: gnupg will be moved into the gitops-tools base image in a future image version bump to eliminate the runtime apk call.

Pipeline flow — branch / MR

test (parallel)
  └─ build-image
       └─ [scan stage - parallel]
            ├─ sbom ──────────────────┐
            ├─ trivy-vuln-scan        │
            ├─ trivy-license-scan     │
            └─ trivy-html-report      │
                 └─ [sign stage]      │
                      ├─ sign-image ──┼──► deploy-to-testing   (manual)
                      │   (DEV key)   │    deploy-to-production (manual, protected branch only)
                      └─ attest-sbom ─┘
                          (DEV key, needs sbom artifact)

Pipeline flow — tag (v*.*.*)

build-release-image
  └─ [scan stage - parallel]
       ├─ sbom-release ─────────────┐
       ├─ trivy-vuln-scan-release   │
       └─ ...                       │
            └─ [sign stage]         │
                 ├─ sign-release-image ──┼──► deploy-tag-to-testing    (manual)
                 │   (PROD key)          │    deploy-tag-to-production  (manual, protected tag)
                 └─ attest-sbom-release ─┘
                     (PROD key, needs sbom-release artifact)

Two-key model

Pipeline	Key	GitLab protection	Target cluster
Feature branch	DEV	Not Protected	dev/testing
MR	DEV	Not Protected	dev/testing
Tag (v*)	PROD	Protected	production (and testing for canary)
Templates repo — main	PROD	Protected	—
Templates repo — feature branch (manual)	DEV	Not Protected	—

The production cluster's admission controller is configured to accept only images signed with the PROD public key. This means a branch-pipeline image deployed to production via deploy-to-production (the protected-branch escape hatch) will be rejected at the cluster level — the gitops update goes through but the Pod never starts. This is intentional: cluster enforcement is the hard gate; the pipeline gate (deploy needs: [sign-image]) ensures the image is signed before the gitops commit is even attempted.

Deploy gating

All four deploy jobs in deploy-environments.yml declare needs: [sign-image] (branch pipelines) or needs: [sign-release-image] (tag pipelines). GitLab's DAG mode will not allow the deploy job to be triggered if the signing job did not complete successfully. The manual trigger button is visible but unavailable until signing passes.

---

4. Cluster-Side Verification

Cosign public keys — Sealed Secrets

Each cluster receives the public key of the key it should trust. The public key is not secret, but Sealed Secrets is used so the manifest can be versioned in overtime-gitops and applied via GitOps.

# Dev cluster — cosign-dev.pub
kubectl create secret generic cosign-pub-key \
  --namespace=security \
  --from-file=cosign.pub=./cosign-dev.pub \
  --dry-run=client -o yaml \
| kubeseal --format=yaml > apps/sealed-secrets/overlays/overtime-staging/cosign-pub-key.yaml

# Prod cluster — cosign-prod.pub
kubectl create secret generic cosign-pub-key \
  --namespace=security \
  --from-file=cosign.pub=./cosign-prod.pub \
  --dry-run=client -o yaml \
| kubeseal --format=yaml > apps/sealed-secrets/overlays/overtime-prod/cosign-pub-key.yaml

Commit and push to overtime-gitops; ArgoCD will sync the Sealed Secret, which the controller decrypts and applies as a regular Secret.

Admission enforcement (future)

A Kyverno ClusterPolicy referencing the cosign-pub-key Secret will verify every image before a Pod is admitted. The policy is a separate implementation step; the signing infrastructure is in place and ready.

GPG public key — GitLab account

Register ci-bot-gpg.pub under the ci-bot GitLab account (Preferences → GPG Keys). GitLab will then display a green "Verified" badge on every commit the bot pushes to overtime-gitops. Auditors can inspect any commit's signature without additional tooling.

---

5. Manual Verification (spot-check commands)

# Verify an image signature
cosign verify \
  --key cosign-dev.pub \
  registry.gitlab.com/alexb/overtime-backend:<sha>

# Verify and extract the attached SBOM
cosign verify-attestation \
  --key cosign-dev.pub \
  --type cyclonedx \
  registry.gitlab.com/alexb/overtime-backend:<sha> \
  | jq '.payload | @base64d | fromjson'

# Verify a GitOps commit signature
git -C /path/to/overtime-gitops log --show-signature -1

---

6. Future Migration to HashiCorp Vault

When Vault is introduced, the change to the pipeline is a single-line swap per sign job:

# Before (CI variable)
cosign sign --key env://COSIGN_PRIVATE_KEY ...

# After (Vault KMS)
cosign sign --key hashivault://cosign-dev ...   # or cosign-prod

The private key material moves into Vault and never leaves it — the pipeline receives only the resulting signature. This satisfies the BSI TR-02102 requirement that private key material be held in an HSM or KMS. The CI variables COSIGN_PRIVATE_KEY_* and COSIGN_PASSWORD_* are removed at that point.

---

Consequences

Positive

• Every image in the registry carries a verifiable signature cryptographically linked to the pipeline that built it
• SBOMs are bound to image digests as in-toto attestations — tampering is detectable
• Every GitOps overlay commit is GPG-signed — provenance of cluster state is auditable
• Production and dev clusters operate under independent trust roots — a compromised dev key cannot authorise production workloads
• No metadata leaves the private infrastructure at any point
• Renovate will auto-update the pinned gcr.io/projectsigstore/cosign image digest

Negative / Open Items

• Cosign key pairs are long-lived secrets. Rotation requires: generating a new pair, updating the CI variables, re-sealing and deploying the public-key Secret, and updating the Kyverno policy. This should be treated as a scheduled operational procedure.
• The GPG CI-bot key expires in 2 years; calendar a renewal before expiry.
• gnupg is installed at runtime in update-gitops.yml jobs until the gitops-tools image is rebuilt with it included.
• Admission enforcement (Kyverno policy) is not yet active — signatures exist but are not yet required for Pod admission.

Compliance Notes

Das detaillierte Mapping auf BSI IT-Grundschutz und CIS Benchmarks wird in später ergänzt.

Links

• overtime-ci-templates/sign.yml — reusable Cosign job templates
• overtime-ci-templates/update-gitops.yml — GitOps commit job with GPG signing
• overtime-ci-templates/deploy-environments.yml — deploy jobs gated on sign jobs
• ADR-006: Sealed Secrets for GitOps Secret Management
• Cosign key-based signing
• BSI TR-02102-1: Cryptographic Mechanisms