The Secured Safe — Homelab Secret Tier

Two Vaults, one credential chain, six hops from tfvars to Postgres · Author: George Ezejiofor

The Secured Safe — Homelab Secret Tier Two Vaults, one credential chain, six hops from tfvars to Postgres · Author: George Ezejiofor Rook-Ceph OSDs · KMS configured, never used · Architecture component · keys sit in the Ceph mon store Rook-Ceph OSDs KMS configured, never used keys sit in the Ceph mon store Tekton Chains · cosign x509 key via VSO · Architecture component · chains-controller SA only Tekton Chains cosign x509 key via VSO chains-controller SA only 1 · terraform apply · vault-config/environment/prod · Architecture component · .secrets.auto.tfvars · gitignored · Terraform 1 · terraform apply vault-config/environment/prod .secrets.auto.tfvars · gitignored 2 · Main Vault · 3 pods · one per machine · Raft · Architecture component · seal transit · recovery shamir 2 · Main Vault 3 pods · one per machine · Raft seal transit · recovery shamir 3 · Secrets Operator · vault-secrets-operator-system · Architecture component · 51 VSS · 36 VaultAuth · 1 Conn 3 · Secrets Operator vault-secrets-operator-system 51 VSS · 36 VaultAuth · 1 Conn 4 · K8s Secret · rendered from a Vault path · per-app namespace (x19) · refreshAfter 1h · overwrite true · Kubernetes 4 · K8s Secret rendered from a Vault path refreshAfter 1h · overwrite true 5 · Workload pod · envFrom / secretKeyRef · per-app namespace (x19) · never learns Vault exists · Kubernetes 5 · Workload pod envFrom / secretKeyRef never learns Vault exists 6 · CNPG Postgres · matching role + password · per-app namespace (x19) · same path, other end of the wire · PostgreSQL 6 · CNPG Postgres matching role + password same path, other end of the wire Transit Vault · 1 pod · holds ONE key · ns vault-transit · Shamir 1-of-1 · auto-unsealed Transit Vault 1 pod · holds ONE key Shamir 1-of-1 · auto-unsealed Installed, unused · agent-injector 2 · csi-provider 20 · Architecture component · 22 pods · no pod uses them Installed, unused agent-injector 2 · csi-provider 20 22 pods · no pod uses them writes KV-v2 transit unwrap scoped read renders envFrom connects never connected (https:// inside the mesh) cosign key unused ns vault-transit per-app namespace (x19) Legend Backend Database Security External

The two-Vault trick

  • • The main Vault is sealed at rest and the key that opens it is not inside it
  • • It lives in a one-pod Vault whose entire job is to hold that one key
  • • An in-cluster unsealer re-opens the transit Vault within ~15 s of any reseal
  • • A daily CronJob renews the auto-unseal token; without it, Vault was down for two months

Who holds what

  • • Terraform writes · VSO reads · Vault audits · Keycloak OIDC gates the human UI
  • • 51 VaultStaticSecrets across 19 namespaces behind exactly 1 VaultConnection
  • • Each app's policy grants read on its own paths only — no listing, no pivot
  • • Compromise one pod and you get that pod's own paths, and nothing else
  • • One Vault pod per machine (dc1, dc2, dc3), enforced by a zone spread rule

The gap this diagram will not hide

  • • Rook was pointed at https:// inside the mesh: TLS-in-TLS, it never connected
  • • The OSD dmcrypt keys are in the Ceph mon store; the rook/ KV mount is empty
  • • The real verification gap is insecureSkipVerify on the mesh DestinationRule, for every client
  • • Dashed arrows mean configured but not working, or installed and unused