Managing multiple SSH private keys for a team
I have a medium-sized team (~10 people, but we expect to grow) that manages a set of EC2 servers on AWS (currently a few dozens, but this is also expected to grow).
As EC2 requires you to have an AWS generated private SSH key, without a passphrase (that's how AWS generates them) in order to access EC2 servers, as the number of servers and different environments and products my team manages (and as people are moving on and off the team), I'm less and less happy with our current solution for securing access to EC2 servers - namely having a single private key that is copied to each team members local machine.
I'm considering a few options, and I would appreciate it if you can suggest a better option and discuss why its better:
- Keep using a single SSH key for all our systems.
- Pros: simple to manage, relatively secure (assuming team trust)
- Cons: after a team member gets the key, there is no way to revoke access; a single leak compromises security of all systems; no passwords
- Have a single SSH key for each product/environment, distribute to all team members.
- Pros: Still not difficult to manage, relatively secure (assuming team trust)
- Cons: keys can't be revoked; a single leak compromises all systems (maybe not all, if a junior member that doesn't have all the keys was the source); no passwords; difficult to use as the user has to unload and reload keys when moving between environments
- Build a bastion server for each product/environment; create a single SSH key for each product/environment and install the private key in the bastion server's known user account; install each team members personal public key in the known user account.
- Pros: allows key revocation; compromise of a bastion compromises only 1 environment; if a leak of a user key is detected, compromise of untouched system can be easily prevented; allows use of passphrases to access bastions
- Cons: relatively complex to manage (creating additional servers and running non-trivial installation, adding team members, removing team members); costly (servers aren't free); complicates software tools used by team; complex key revocation
- Use a key storage service; create a single key for each environment/product and store in service; control access to the service using a password or a personal ssh key; operations start by identifying the environment/product accessed and obtaining the key into the SSH agent.
- Pros: easy to manage (assuming the service is available); keys can be revoked on a per user or per environment; single point to protect (that is not mission critical); Relies on OpenSSH agent to secure keys outside the service.
- Cons: Single point of failure; may complicate usage scenario
I'm currently leaning towards #4, but are there serious issues I'm missing? Is there a service like that that I can use or do I have to roll my own?
Note: we don't use an orchestration/configuration server a-la puppet/chef - our orchestration software is mostly home grown and installed on each team member's local system. It is basically just a set of recipes loaded from source control and used to execute various scenarios, mostly using AWS APIs. Each team member has a personal AWS API key and the orchestration software uses it to run the AWS API. In addition, some scenarios call for SSH access to system servers, and here is where I have the problem described above. The EC2 servers are accessed using the default AMI user (usually "ubuntu") and the software uses NOPASSWD sudo to execute local operations.
