boss | Run containers like a ross

When it comes down to storing credentials on a server or other runtime environment, you are somehow between the devil and the deep blue sea. There is no real good solution which is likewise usable.

Start asking yourself, what your threat model is.

• A: Secrets being persisted in version control, shared with others, or even worse, made public on GitHub etc.
• B: Secrets being exposed to unprivileged co-users of the runtime environment
• C: Secrets being exposed to privileged users of the runtime environment (including an attacker who compromised the system and was able to get privileged user rights).

Based on the threats defined, you can start assessing potential solutions to store and inject secrets. This will of course depend on your environment (e.g. OS, cloud provider, Kubernetes/Docker, etc.). In the following I will assume Linux as OS.

Pass in as parameter: Would mitigate threat A, but not B and C. Command line arguments can be revealed even by unprivileged users e.g. by ps -eo args

Store in config file: Would mitigate threat B, given that file permissions are set correctly. With regard to A, there is still a risk that the config file is unintendedly added to the version control. Does not mitigate threat C.

If you would use e.g. json format for the config file, this could be implemented easily with the Golang standard lib.

Store in environment variables: Would mitigate threats A and B, but not C. Privileged users can access the environment variables via /proc//environ. Also the question remains how you will set the environment variables in the runtime environment. If you are using a CI/CD pipeline to deploy your service, this pipeline could be used to inject the environment variables during deployment. Usually, the CI/CD engine come with some kind of variable store for secrets.

Drawback of this approach is that the environment variables will be ephemeral, so after a reboot of the runtime environment you would need to redeploy via the CI/CD pipeline or you need to ensure persistence of the secrets in the runtime environment, e.g. in a startup script.

Environment variables can be read easily with os.Getenv() or os.LookupEnv() from the standard lib.

Enter manually on start time: Would mitigate A and B, but privileged users would still be able to read the secrets from memory. Upon reboot of the runtime environment, the service will not be available until an operator enters the secrets manually. So this approach would probably be considered as impractical in many use cases.

Further considerations:

• Storing secrets in a database as suggested by brianmac shifts the question to "Where to store my db credentials?"

• Combining secret encryption with any of the solutions described above will require that the decryption key is made available to the service in the runtime environment. So you either need a TPM-based solution or you are faced with the question, where to store this key.

• "Secrets as a Service" solutions like Hashicorp Vault, Azure Key Vault, AWS Secrets Manager etc. will probably be oversized in your scenarion. They provide centralized storage and management of secrets. Applications/services can retrieve secrets from this solution via a defined API.

  This, however, requires authentication and authorization of the service requesting a secret. So we are back at the question how to store another secret for the service in there runtime environment.

  Cloud providers try to overcome this by assigning the runtime environment an identity and authorizing this identity to access other cloud resources including the "Secret as a Service" solution. Usually only the designated runtime environment will be able to retrieve the credentials of the identity. However, nothing can prevent an privileged user who has access the runtime environment from using the identity to access the secrets.

Bottom line is that it is hard to impossible to store secrets in a way that a privileged user or someone who compromised the system will not be able to get access.

If you accept this as the residual risk, storing the secrets in environment variables is a good approach as it can avoid persisting secrets. It is also platform agnostic and thus can be used with any runtime environment, cloud provider etc. It can also be supported by a variety of automation and deployment tools.