Latest / Logosofia no Dia a Dia / SSH Key Strategy: Compartmentalization vs. Chaos
Transcript
- [Prompt] custom: Let's talk about best practices in SSH authentication today. It's common for developers to work acro | Hosts: corn, herman [Response] Corn: Daniel sent us this one — he's asking about SSH key strategy, and it's one of those questions where the security textbook answer and the thing developers actually do have been in a standoff for about twenty years. The specific tension: is it worth generating application-specific SSH keys for every service you manage, like a dedicated key just for your Hetzner VPS, versus reusing one key across everything? And with AI agents now in the mix, needing their own access, does that push us toward more compartmentalization or just more key chaos? Herman: This is the kind of topic where I get genuinely excited, because it sits right at the intersection of good cryptography and the messiness of actual human workflows. And I think the best way into it is to name the two camps clearly. On one side, you've got the one-key-per-device approach — you generate a single ed25519 key pair on your laptop, maybe another on your phone, and those keys authenticate you to everything. On the other side, application-specific keys — one key per service, per device, which means if you've got, say, three VPSs and two devices, you're managing six keys minimum. The security argument for compartmentalizing is straightforward: if one key leaks, the blast radius is limited to that single service. The argument against it is equally straightforward: nobody wants to spend their afternoon juggling a dozen key files. Corn: The blast radius argument is the one that sounds very adult and responsible in a security review, and then you actually try it and discover you've created a part-time job for yourself. Herman: Right, and that's exactly the tension Daniel's pointing at. Before we take sides, let's talk about what ssh-agent actually does under the hood, because understanding that changes how you think about the management burden. ssh-agent is a background process that holds your decrypted private keys in memory. When you attempt an SSH connection, the remote host sends a challenge, your SSH client asks the agent to sign it with the appropriate key, and the agent handles it without the client ever touching the raw key material. The key insight is that ssh-agent can hold multiple keys simultaneously — it's not a one-key-at-a-time situation. When you connect, the client tries keys in order until one works, or you can specify which key with the minus i flag. Corn: If you don't specify, it just runs through the keychain like someone trying every key on the ring until the lock turns. Herman: Which works fine with three keys and gets progressively more annoying — and potentially gets you locked out if the remote host has a MaxAuthTries limit. But here's where the dot ssh config Host block pattern changes everything. You can map specific keys to specific hosts declaratively, so the user never has to think about which key is which. A concrete example: you add a block that says Host hetzner dash vps dash one, then IdentityFile tilde slash dot ssh slash id underscore hetzner underscore vps underscore one. Now every time you SSH to that hostname, the correct key is selected automatically. You generate the key once, add the config block once, and from that point forward it's invisible. Corn: That's the part that makes application-specific keys viable, right? Without Host blocks, you're manually passing minus i flags every time, which means you'll stop doing it by day three. Herman: The config file is what turns key compartmentalization from a discipline you have to remember into infrastructure you set and forget. And I want to walk through the full key rotation lifecycle, because once you've got the config set up, rotation becomes a mechanical process rather than a panic. Best practice since OpenSSH six point five in twenty fourteen is ed25519 keys — they're about eighty characters, they're fast, they're secure. You generate a new one with ssh dash keygen dash t ed25519, add it to the agent with ssh dash add, append the public key to authorized underscore keys on the target host, verify you can connect with the new key, then remove the old public key from authorized underscore keys and delete the old private key. The whole thing takes maybe two minutes. Corn: Two minutes per key per service, and if you've got thirty services, you've just spent an hour rotating keys. Which is fine if you're doing it on a schedule, less fine if you're doing it because you just discovered a compromise at eleven PM. Herman: That's the operational reality, and it's why I think the conversation needs a dose of threat modeling before we prescribe anything. What are we actually optimizing against? If the threat is someone stealing your laptop and extracting keys from disk, full disk encryption is your primary defense and key compartmentalization is a secondary layer. If the threat is a compromised remote host that captures your key during authentication — well, SSH is designed so the private key never leaves your machine, so that's not the vector. The real threat where compartmentalization shines is when a key file leaks through some side channel — a misconfigured backup, a dotfile repo accidentally made public, a CI pipeline that logs environment variables. Corn: The CI pipeline one is real. I've seen a developer commit a private key to a repo because they were tired and it was in the wrong directory, and suddenly their key to everything is sitting in GitHub history. With app-specific keys, that mistake costs you one service instead of all of them. Herman: That's not a hypothetical. There was a well-known incident a few years back where a single developer key committed to a public repo gave attackers access to multiple production services because the key was reused everywhere. With application-specific keys, the blast radius would have been a single staging environment. The cleanup would have been rotating one key instead of thirty. Corn: The security case is clear. The question is whether the operational friction is low enough that people will actually do it, and that's where the tooling matters. You mentioned dot ssh config Host blocks. What else makes this tolerable? Herman: The password manager as SSH agent backend is the big one. Tools like onePassword and Bitwarden can act as SSH agents — you store your keys in the vault, and the agent loads them on demand. This solves the multi-device problem. You generate a key once, store it in the vault, and it's available on your laptop, your phone, your tablet, wherever you're authenticated to the password manager. You're no longer manually copying key files between machines. The vault is the source of truth, and the agent handles the rest. Corn: That's the bridge between the security ideal and the practical workflow. Without it, you're generating separate keys per device per service, and the combinatorial explosion is real — three devices times ten services is thirty keys, and you're manually syncing them. Herman: That's where people give up and go back to one key. But with a password manager SSH agent, you generate the key once, it's available everywhere, and the Host block in your dot ssh config handles key selection. The management overhead collapses to the initial setup plus periodic rotation. Corn: Let's talk about the AI agent complication, because that's where this gets interesting and where Daniel's question about what agents are pushing users toward really lands. An AI coding agent like Claude Code or a specialized CLI needs to SSH into a server to deploy something. It needs a key. But it's stateless and ephemeral — it doesn't have a password manager vault, it can't type a passphrase, and you definitely don't want to hand it your primary key that has access to everything. Herman: This is where I get animated, because the agentic workflow exposes a limitation in how most developers think about SSH authentication. The instinct is to generate a key pair for the agent, store the private key somewhere the agent can reach it, and move on. But that key now lives in the agent's environment — which might be a container that gets rebuilt, a CI runner that logs everything, or a cloud VM you don't fully control. The key is static and long-lived, and if that environment is compromised, the key walks out the door. Corn: The agent-specific key is better than giving the agent your main key, but it's still a static credential sitting in a potentially leaky environment. Herman: And this is where SSH certificates enter the conversation as a fundamentally better pattern. Instead of generating key pairs, you set up a certificate authority — something like HashiCorp Vault or even a small custom CA using OpenSSH's built-in CA support. The CA has a signing key. When an agent needs access, it authenticates to the CA — using whatever identity system you already have, like OIDC or a service account token — and the CA issues a short-lived SSH certificate. That certificate is valid for, say, one hour. The agent uses it to connect, does its work, and the certificate expires. There's no long-term key to manage, no key to rotate, no key to leak in a meaningful way because it's already useless by the time an attacker might find it. Corn: That's elegant. But it's also infrastructure. Setting up an SSH CA is not a five-minute task for someone managing three personal VPSs. Herman: It's not, and I want to be honest about that. Vault is powerful but it's a whole service to run and maintain. For a solo developer with a handful of servers, the operational overhead of running a CA probably exceeds the security benefit. But there's a middle ground that's worth exploring, and it connects back to the authorized underscore keys restrictions Daniel's prompt implicitly touches on. You can generate a dedicated key for an agent — no passphrase, since the agent can't type one — and then restrict that key on the target host using the command equals and from equals directives in authorized underscore keys. Corn: Walk me through what those restrictions actually enforce. Herman: The command equals directive limits the key to executing a single specified command, regardless of what the connecting client asks for. So if your agent only needs to run a deployment script, you set command equals slash opt slash deploy dot sh, and that key cannot be used for an interactive shell, cannot be used to run arbitrary commands, cannot be used for port forwarding. The from equals directive restricts the key to connecting from a specific IP address or range. So even if the key leaks, it's only usable from your CI runner's IP, and only to run that one script. Combined, you've turned a potentially catastrophic key leak into a very narrow, very boring problem. Corn: That's the kind of pragmatic middle ground I can get behind. You're not setting up a CA, but you're also not handing an agent unrestricted shell access with a key that never expires. Herman: It addresses a real failure mode. I've seen teams give an AI agent a key with full shell access because it was the path of least resistance, and then the agent ran a command that was correct in intent but destructive in execution — something like rm dash rf in the wrong directory because of a path variable that wasn't set. The command equals restriction is a safety rail as much as a security control. Corn: The agent can't rm dash rf your server if its key only permits running deploy dot sh. Herman: And that's a good segue into something I think is under-discussed in the SSH key conversation: key provenance. When you're responding to an incident, you need to know which key was used for what, and when. If you've got one key on every device connecting to every service, and you see a suspicious login from an unfamiliar IP, you have no idea which service was the entry point. Was the key stolen from your laptop? From a backup? From a CI pipeline? With application-specific keys, the key itself tells you the blast radius — if key hetzner vps one is used from an unknown IP, you know exactly which service to lock down and exactly which key to revoke. Corn: That's the forensics argument. It turns every key into a label that narrows the investigation. Herman: It connects to something Daniel's prompt gestures at — the idea that AI agents are pushing users toward application-specific credentials. I think that's right, but not for the reason most people assume. It's not that agents inherently need per-service keys. It's that agents are going to be generating and using credentials at a scale that makes provenance essential. If you've got three different agents touching six different services, and something goes wrong, you need to know which agent, which service, which key. A single shared key makes that question unanswerable. Corn: Let's address the question Daniel asked head-on: is it really reckless to use the same SSH key for multiple services? I think the answer is — it depends on what those services are, and I don't think that's a cop-out. Herman: It's not a cop-out at all. For low-value personal infrastructure — a VPS running a hobby project, a Raspberry Pi on your home network, a dev server with no customer data — the operational cost of managing separate keys probably exceeds the security benefit. If your threat model is opportunistic attackers scanning for weak passwords, and you're using key-based auth at all, you're already ahead of the curve. The marginal gain from compartmentalizing your three hobby VPS keys is real but small. Corn: The calculus flips when you're dealing with production infrastructure, customer data, or anything where a compromise has legal or financial consequences. At that point, compartmentalization isn't optional — it's table stakes. Herman: The threshold where automation becomes mandatory is lower than most people think. Managing three keys manually is fine. Managing ten is annoying but doable with dot ssh config. Managing thirty across multiple devices without a password manager or a CA is where you start making mistakes — forgetting to rotate a key, leaving an old key on a decommissioned server, losing track of which key goes where. Corn: That's the point where the "just one key" temptation becomes strongest, because the management burden has grown but the tooling hasn't kept up. Herman: And that's why I think the practical recommendation needs to be a graduated approach, not a binary one. But before we get to recommendations, let me address a misconception that comes up constantly in these discussions: SSH agent forwarding. A lot of developers use it to avoid managing multiple keys — they forward their agent from their laptop to a jump host, then from the jump host to the target. The problem is that agent forwarding exposes your agent socket to the remote host. If that jump host is compromised, an attacker can use your forwarded agent to authenticate to anything your keys have access to. Corn: Agent forwarding is the SSH equivalent of leaving your key ring on the counter at a party and hoping nobody picks it up. Herman: That's exactly the image. And yet it's everywhere because it's convenient. The safer alternative is SSH's ProxyJump directive, which handles the multi-hop connection without exposing your agent socket. It's one line in your dot ssh config — ProxyJump jump-host — and it tunnels the connection through the jump host without forwarding the agent. Corn: I want to circle back to something you mentioned earlier about the OpenSSH release this year, because it changes the key generation conversation in a way that's relevant to Daniel's question about what's current best practice. Herman: OpenSSH nine point eight, released in April, added post-quantum hybrid key exchange — specifically ML-KEM combined with ECDH. The practical implication is that the key exchange during connection setup is now resistant to future quantum attacks, even if you're still using ed25519 keys for authentication. This doesn't change what kind of keys you should generate — ed25519 remains the recommendation — but it does mean the transport layer is future-proofed in a way it wasn't before. If you're setting up new infrastructure today, making sure you're on OpenSSH nine point eight or later is a meaningful security upgrade. Corn: The key generation advice is stable — ed25519, eighty characters, best practice since twenty fourteen — but the transport security has quietly improved under the hood. Herman: That's the kind of thing that matters if you're thinking about key lifecycle over years rather than months. A key you generate today might still be in use in twenty twenty-nine, and you want the connection that key establishes to be resistant to attacks that might become practical by then. Corn: Let's talk about the knock-on effect of adopting application-specific keys, because I think there's something counterintuitive here. When you have one key that works everywhere, key rotation feels like a big scary operation — if you mess it up, you lock yourself out of everything. So you don't rotate. The key sits there for three years. With app-specific keys, rotation is a small, routine operation — you rotate one key, test it, move on. The compartmentalization actually makes rotation less intimidating, which means you're more likely to do it regularly. Herman: The psychological barrier to rotating your one master key is enormous because the stakes feel enormous. Rotating the key for hetzner vps three is a Tuesday task. And regular rotation matters — not because keys cryptographically degrade, but because the longer a key exists, the more opportunities it's had to leak through some channel you've forgotten about. Corn: The key that's been on your laptop for four years has been through four years of backups, dotfile syncs, accidental copies, and who knows what else. You can't audit its exposure history. Herman: And this connects to what we were saying about the CI pipeline compromise. If you rotate keys every ninety days, a key that leaked six months ago is already useless. The rotation cadence is your safety net for the leaks you never detected. Corn: Alright, so where does this leave us? Let's distill this down to something someone can actually use tomorrow morning. Herman: I'd frame it as an incremental path. Stage one: start with one key per device. Generate a fresh ed25519 key on each machine you use, add it to your services, and retire whatever ancient RSA key you've been dragging around since twenty eighteen. That alone puts you ahead of most developers. Stage two: graduate to one key per service category. A personal key for hobby VPSs, a work key for employer infrastructure, a CI key for pipelines. This limits cross-contamination between contexts without exploding the key count. Corn: Stage three is where you go application-specific, but only for high-value targets. Production servers, anything with customer data, anything where a compromise would ruin your week. For those, generate dedicated keys, use Host blocks in dot ssh config, and set a rotation reminder. Herman: For AI agents specifically, the recommendation is different from human users. Generate a dedicated key with no passphrase — since agents can't type one — and lock it down with command equals and from equals restrictions in authorized underscore keys. The agent gets exactly the access it needs and nothing more. If the agent's environment changes — a major update, a new CI runner, a different cloud provider — rotate that key immediately. Corn: What about rotation cadence? I've seen everything from "rotate every thirty days" to "rotate when you remember. Herman: The pragmatic answer: personal keys, rotate yearly. Production keys, every ninety days. Agent keys, after every major agent update or when the agent's environment changes. And any key, regardless of schedule, rotate immediately if you suspect it's been exposed — even slightly. The cost of rotation is low enough that erring on the side of caution is almost free. Corn: The password manager SSH agent integration is the piece that makes all of this sustainable across multiple devices. Without it, you're back to manually syncing key files, which is where the whole system breaks down. Herman: It's worth naming explicitly: if you're using onePassword or Bitwarden as your SSH agent, your dot ssh config maps hosts to keys stored in the vault, and the agent loads them on demand. You set it up once per device, point it at your vault, and from then on key management is just vault management. Authenticate to the vault, and all your keys are there. It's the closest thing to a seamless multi-device SSH experience that exists today. Corn: I want to touch on the SSH CA approach one more time, because I think it's where the trajectory is pointing even if it's not practical for everyone today. The idea that you don't manage keys at all — you issue short-lived certificates per session, and they expire before they can be a problem — that's the end state that makes the whole "how many keys should I have" question irrelevant. Herman: And for teams, it's increasingly the standard. A team using Vault as an SSH CA has agents request a signed certificate valid for one hour. No key files to manage, no rotation schedule to maintain, no question of which key goes where. The certificate is issued on demand, scoped to a specific host, and expires automatically. If a certificate leaks, the attacker has at most an hour to use it, and the audit log shows exactly which certificate was issued to whom and when. Corn: That's the dream. But for the solo developer managing three VPSs, the practical path is the graduated approach you outlined — start with per-device keys, add Host blocks, integrate a password manager agent, and compartmentalize selectively where it matters most. Herman: The best SSH key strategy is the one you'll actually maintain. A perfect compartmentalization scheme with thirty keys that you abandon after a week because it's too much friction is worse than a single key you rotate religiously every year. The goal isn't theoretical purity — it's reducing the probability that a key compromise cascades into a catastrophe, while keeping the daily workflow light enough that you don't hate your life. Corn: That's the pragmatist's security principle right there. Don't let the perfect be the enemy of the good enough that you'll actually do. Herman: I think that's where Daniel's instinct is right — application-specific keys are good security practice, but they need tooling support to be sustainable. The dot ssh config Host block pattern and the password manager SSH agent are the two pieces that make the difference between a security ritual you perform for a week and a system you maintain for years. Corn: Before we wrap, there's an open question I want to leave listeners with. As AI agents become more autonomous, are we going to see SSH keys replaced entirely by OIDC-based authentication or WebAuthn? The SSH protocol is evolving — the post-quantum key exchange in OpenSSH nine point eight is one signal — but the authentication layer might look very different in five years. Herman: I think we're heading toward a world where long-lived SSH keys are the exception rather than the rule, especially for non-human access. Short-lived certificates issued by an identity-aware CA, tied to a workload identity rather than a key file — that pattern solves too many problems to not become the default. The question is how quickly the tooling trickles down from large teams to individual developers. Corn: Until it does, the ed25519 key with a Host block and a rotation reminder is your friend. Herman: Now: Hilbert's daily fun fact. Hilbert: In the nineteen thirties, British Somaliland issued a series of postage stamps overprinted with the portrait of King George the Fifth — but a printing error on one batch caused the king's face to appear upside down relative to the frame. The colonial administration, fearing the error would be interpreted as a deliberate insult by Somali nationalists, ordered the entire batch destroyed. A single sheet survived because a postal clerk in Berbera used it for routine mail before the recall notice arrived. That sheet is now valued at over two hundred thousand pounds. Corn: A postal clerk in Berbera accidentally created a six-figure collectible because he was too efficient to wait for the memo. Herman: That's the most expensive "didn't check his email" in philatelic history. Corn: This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop. If you enjoyed this episode, tell someone who still uses an RSA key from twenty seventeen that it's time to rotate. Find us at my weird prompts dot com. I'm Corn. Herman: I'm Herman Poppleberry. Rotate your keys.