How to secure Kubernetes secrets on bare metal?

How to secure Kubernetes secrets on bare metal?

(This question is based on a comment I made on How Vault agent solves Secret Zero challenge in Kubernetes?)

We are planning to run some of our software in a Kubernetes cluster running on bare metal (i.e. not using a cloud provider). We need to provide some secret data to our microservices (e.g. API keys, private keys), and want to ensure these secrets are stored and retrieved as securely as possible.

The Kubernetes Secrets documentation provides various recommendations:

• Enable encryption at rest

  • The documentation for this states:

    Caution: Storing the raw encryption key in the EncryptionConfig only moderately improves your security posture, compared to no encryption. Please use kms provider for additional security.

    ...

    Encrypting Secrets with a locally managed key [i.e. not kms] protects against an etcd compromise, but it fails to protect against a host compromise. Since the encryption keys are stored on the host in the EncryptionConfiguration YAML file, a skilled attacker can access that file and extract the encryption keys.

• Use an external secret store provider, e.g. Vault provider

  • This seems to be an alternative to using native Kubernetes secrets, and outsources secret management to an external component.

My thought process is:

1. Using the kms provider seems to require us to use a Cloud provider - which isn't an option - or develop our own - and I assume "rolling our own" isn't recommended for this sort of thing. So we have to use an external secret store provider, of which Vault is the only supported provider.

2. Vault requires the Pod to authenticate using its service account, which boils down to a JWT minted using a private key stored on a control plane node.

3. This private key is no more protected than a static secret defined in an EncryptionConfig file, assuming our control plane is separate from our worker nodes.

4. So, my question:

   What do we gain from using an external secret store, over using a static secret in an EncryptionConfig file?

   Is it just security theatre? Or is there a flaw in the reasoning above?

   Alternatively, is there a better way to manage Kubernetes secrets when running on bare metal?