Latest / Logosofia no Dia a Dia / KVM Over IP: Remote Server Access When SSH Dies
Transcript
- [Prompt] custom: let's talk about kvm over IP devices which can be used for out of band access to severs with a broke | Hosts: corn, herman [Response] Corn: Here's the nightmare. It's two in the morning, you're in a hotel room two thousand miles from your rack. Your server's networking stack has eaten itself — maybe a bad kernel update, maybe a misconfigured interface, doesn't matter. SSH is dead. IPMI's not responding. The machine is on, fans spinning, doing absolutely nothing useful, and you have no way to reach it. What do you actually do? Herman: You book a flight. Or you call someone with a key and try to talk them through BIOS screens over the phone, which is its own special circle of hell. Corn: The function key. Yes, the one that also does volume. No, don't hold the volume — okay, we missed the window, reboot it again. Herman: I've lived that call. It's how you lose friends. Corn: Daniel sent us this one, and it's exactly about escaping that scenario. He's asking about KVM over IP devices — the hardware boxes that give you keyboard, video, and mouse access to a server as if you're standing right in front of it, even when the operating system is a smoking crater. And the real question he's digging into is architectural: once you have one of these things, how do you connect it to the outside world securely? Do you pair it with a separate bastion host running something like Tailscale or Cloudflare Tunnel, or are we seeing the market converge toward all-in-one devices that bundle the KVM and the secure access layer into a single box? Herman: This is one of those topics where the question sounds narrow — it's just remote access, right? — but the answer pulls in hardware design, network architecture, trust boundaries, and a bunch of assumptions most people don't realize they're making until their server is down and they're staring at a login screen they can't reach. Corn: Where do we even start with this? Because I think most people hear "KVM over IP" and picture something completely wrong. Herman: They picture a software KVM switch — the little box that lets you toggle one keyboard and monitor between three computers on your desk. That's not what we're talking about. A KVM over IP device is a dedicated piece of hardware that physically captures the HDMI or DVI output from your server, emulates a USB keyboard and mouse on the other end, encodes that video stream, and serves it over the network. It operates entirely below the operating system. When your server's network stack is fried, when the kernel panicked and the machine is sitting at a recovery prompt, when you need to change a BIOS setting — this thing still works. Software remote desktop can't touch that. Corn: Because software remote desktop is a guest in the operating system's house, and when the house burns down, the guest burns with it. Herman: SSH, RDP, VNC — they're all services running on the host OS. If the host OS can't bring up a network interface, those services never start. The KVM over IP device is outside that failure domain. It's watching the server's video output directly and injecting keystrokes at the USB level. The server doesn't even know it's not a real keyboard. Corn: Which sounds like magic until you realize you've now put a tiny computer with full physical access to your server onto your network, and that tiny computer has a web interface and default credentials and probably a few unpatched CVEs. Herman: It's the whole game, really. The KVM part is a solved problem — we've had VGA capture chips and USB gadget mode for years. The interesting question is the network architecture around it. And that's where the market splits in a way that's genuinely useful to map out. On one side you've got the consumer and prosumer devices — PiKVM, TinyPilot — that are essentially Raspberry Pis with HDMI capture hats and some very clever software. On the other side you've got the enterprise rack-mount units from Raritan, Lantronix, Vertiv — redundant power supplies, dual network interfaces, LDAP integration, four concurrent users, and price tags that look like used car listings. Corn: The architectural question Daniel's asking sits right in the middle of that split. Do you treat the KVM device as a dumb peripheral that lives on an isolated management VLAN behind a separate bastion host, or do you let the KVM device itself be the secure entry point by running Tailscale or a Cloudflare Tunnel directly on it? Herman: Let's start by looking at what's actually on the market, because the hardware capabilities determine what architectures are even possible. And then we can get into the mechanism that makes these things work when your server is completely dead — which is the part that surprised me when I first dug into it. Herman: It's a dedicated hardware appliance that physically plugs into your server's HDMI or DisplayPort output, captures that video signal, encodes it as a stream, and serves it over the network. On the input side, it presents itself to the server as a USB keyboard and mouse — the server thinks it's talking to real peripherals. None of this depends on the host operating system being alive. Corn: Which is the part that separates it from everything else. If I SSH into a server and the network stack is toast, I get nothing. If I try RDP or VNC, same story — those are services running on the OS, and when the OS can't bring up an interface, they never start. Herman: The KVM over IP device sits outside that entire failure domain. You can watch the BIOS post, enter the boot menu, select a recovery kernel, type a LUKS passphrase — all things that happen before the operating system even loads. Software remote access tools are guests in the OS's house. This thing is standing outside looking through the window with its own key to the back door. Corn: Which sounds like the kind of power you don't want to hand out casually. And that brings us to IPMI, because that's what usually gets thrown around as the alternative. iDRAC, iLO, all the baseboard management controllers. Herman: IPMI does solve some of the same problems — it gives you out-of-band access, remote console, power control. But it's proprietary, it's expensive, and in homelab or small business environments it's often disabled, misconfigured, or tied to a dedicated network port that nobody bothered to cable up. Enterprise servers ship with it, but the licensing alone can cost more than an entire PiKVM. And if you're running consumer hardware, IPMI simply doesn't exist. There's no iDRAC for a used Optiplex you turned into a Proxmox node. Corn: The real question Daniel's getting at isn't just "what's a KVM over IP device" — it's how you architect the thing once you have it. You've got this little box with god-mode access to your server and a web interface. Do you bolt it onto an isolated management VLAN and stick a separate bastion host in front of it, or do you let the KVM device itself run Tailscale or a Cloudflare tunnel and become its own secure entry point? Herman: That's the fork in the road. And it's not just a networking preference — it's a trust boundary decision. If the KVM device is your secure endpoint, you're betting that its software stack is hardened enough to face the internet, even through a VPN. If you put a bastion in front of it, you're adding complexity and another thing to maintain, but you get logging, intrusion detection, and a clean separation of concerns. Corn: One box that does everything, or two boxes where each does one thing well. The entire episode is really about which of those you pick and why. Herman: Let's map the actual hardware, because the market splits in a way that dictates what architectures are even possible. On the consumer side, the clear winner in terms of mindshare is PiKVM. Version four is based on the Raspberry Pi Compute Module 4, runs about three to four hundred dollars depending on whether you get the pre-assembled kit or piece it together yourself. It's got HDMI capture, USB gadget mode for keyboard and mouse emulation, and the whole software stack is open source. Corn: By "open source" you mean "willing to spend a weekend compiling kernel modules and questioning your life choices. Herman: Some people find that therapeutic. The other big player in the prosumer space is TinyPilot — their Voyager 2 is pre-assembled, around four to five hundred dollars, also Pi-based, but the whole pitch is that you plug it in and it just works. Browser-based UI, no command line required. Both support ATX power control through GPIO pins, so you can physically power-cycle the server by toggling the motherboard's power switch header. That's the feature that saves you from the "fans spinning but CPU halted" state. Corn: The consumer side is basically Raspberry Pis wearing different hats. What happens when you cross into enterprise territory? Herman: The price goes up by a factor of ten and suddenly everything has redundant power supplies. Take the Raritan Dominion KX IV — rack-mount, dual power inputs, dual network interfaces, supports four concurrent users watching the same server simultaneously. That's a two to five thousand dollar box. The Lantronix Spider Duo is more compact, single-port, around eight to twelve hundred, aimed at the "one critical server in a remote closet" use case. And Vertiv's Avocent MPU8 is modular, eight ports, starts around three thousand. The key differences aren't just build quality — it's LDAP and RADIUS authentication, SNMP monitoring so your NOC gets alerts if the KVM itself goes down, and support for dedicated out-of-band management networks that are physically separate from your production traffic. Corn: Enterprise is paying for multi-user, audit trails, and the assurance that a single failed power supply doesn't blind you. But underneath all of that, the fundamental mechanism that makes any of these work when the server is dead — that's the same across both markets, right? Herman: It is, and this is the part that surprised me when I first dug into it. The killer feature is something called USB Ethernet gadget mode. Here's the scenario: your server's PCIe network card is dead. Not misconfigured — dead. Or the driver failed to load, or the kernel networking stack is so corrupted that the interface won't come up. Normally you're done. But the PiKVM is plugged into the server's USB port, and it can present itself to the server as a USB Ethernet adapter. The server sees a new network interface appear on the USB bus. The Pi gets a direct Layer 2 link to the server through a path that has nothing to do with the PCIe NICs, nothing to do with the kernel's networking configuration — it's a separate physical path through the USB controller. Corn: The server thinks someone just hot-plugged a USB-to-Ethernet dongle, and suddenly the Pi has network access to a machine whose own network stack was a brick two seconds ago. Herman: And that link works even if the operating system is in a crashed state, because the USB controller is handled at a lower level — the BIOS or the kernel's USB subsystem enumerates the gadget as a network device before the full networking stack even initializes. It survives OS panics, driver failures, misconfigured bridges, all of it. Software remote access tools can't replicate this because they live inside the networking stack that just died. This is a hardware bypass. Corn: Which is both brilliant and terrifying, because you've now given a Raspberry Pi a direct network pipe into a server that may be in a vulnerable state — no firewall, no intrusion detection, just raw Layer 2 adjacency. If someone compromises the PiKVM, they're not just on your LAN. They're inside the server at the BIOS level. Herman: That's the pivot. The security challenge isn't theoretical. PiKVM ships with default credentials — root, root. If you plug it into your network and forget to change those, you've just installed a backdoor with physical access to your server and a convenient web UI for anyone who finds it. Shodan searches for these things. Even with changed passwords, the Pi is running a full Linux distribution with a web server, a video streaming service, and a bunch of dependencies. Every unpatched vulnerability in that stack is a potential entry point to the one device on your network that can see your server's BIOS screen and type arbitrary keystrokes. Corn: It's the ultimate "who watches the watchmen" problem. The KVM device watches the server, but who's watching the KVM device? And if the answer is nobody, you've just concentrated your risk into a three-hundred-dollar box running Debian. Herman: Which is exactly why the bastion architecture question matters. You either isolate this thing on a management VLAN where nothing can reach it except a hardened jump host, or you make the KVM device itself the hardened endpoint by putting Tailscale or a Cloudflare tunnel directly on it. Those are the two paths, and they have very different failure modes. Herman: Let's walk through both patterns, because the tradeoffs aren't obvious until you've actually tried to recover a dead server at two in the morning and discovered your bastion host decided to apply kernel updates and reboot itself. Corn: The universe has a sense of humor and it's exclusively timing-based. Herman: Pattern A is the traditional enterprise approach. You put the KVM device on an isolated management VLAN — nothing can talk to it except a dedicated bastion host. That bastion runs Tailscale or WireGuard, it's the single ingress point from the outside world, and it's configured to log every session, run intrusion detection, and act as a jump box for other management tools. The KVM device never touches the internet directly. It doesn't even have a default gateway. Corn: The bastion is the bouncer and the KVM is the VIP room. Nobody gets near the server without going through the bouncer first. Herman: The bouncer keeps a guest list. That's the compliance angle. If you're subject to PCI-DSS or SOC 2, you need audit trails showing who accessed what and when. A dedicated bastion can log every SSH connection, record every KVM session, and ship those logs to a SIEM. The KVM device itself — especially a PiKVM — isn't built for that kind of logging. It'll show you who's connected right now, but it's not generating the kind of tamper-evident audit trail an auditor wants to see. Corn: The downside being you now have two devices to maintain, patch, and troubleshoot. And when your server is down, you're praying the bastion isn't the thing that failed. Herman: That's the real failure pattern. The bastion adds a dependency. If the bastion's power supply dies, or its SD card corrupts, or you pushed a bad iptables rule, you've lost access to the KVM even though the KVM itself is perfectly fine. Enterprises solve this with redundant bastions and out-of-band access to the out-of-band access — at which point you're building a Russian doll of management infrastructure. Corn: Which is how a homelab becomes a second job. Herman: Which brings us to Pattern B — the all-in-one approach. You install Tailscale directly on the PiKVM. The Tailscale docs make this almost insultingly simple: sudo apt install tailscale, sudo tailscale up. That's it. The PiKVM gets a Tailscale IP, and now it's reachable from any device in your tailnet — your laptop, your phone, wherever you are. No bastion, no jump host, no management VLAN. The KVM device is its own secure endpoint. Corn: The Tailscale blog has a full guide on this — they specifically call out the PiKVM integration as a supported use case. One command and the device is on your mesh network with end-to-end WireGuard encryption and device-level authentication. No open ports, no public IP. Herman: That's the key security argument for Pattern B. People hear "no bastion" and think it's less secure, but the alternative to Tailscale-on-device isn't "behind a bastion" — for most homelabbers, the alternative is port forwarding the PiKVM's web interface through their router with a strong password and hoping for the best. Tailscale on the device is dramatically more secure than that. You're not exposing anything to the internet. The device only accepts connections from other nodes in your tailnet, and those nodes have to authenticate with Tailscale's coordination server first. Corn: The risk shifts from "someone found my open port" to "someone compromised the PiKVM's Tailscale node." And if they've done that, they probably have bigger ambitions than watching me mistype a LUKS passphrase three times. Herman: The real con of Pattern B is concentration of risk. The PiKVM becomes a single point of compromise. If an attacker owns it — through a vulnerability in the web UI, the video streaming service, or the underlying OS — they don't just have network access. They have physical-level access to your server. They can see the screen, type commands, mount ISOs, reboot the machine. It's the keys to the kingdom in a three-hundred-dollar box. Corn: Which is why the third option — the hybrid — is where things get interesting. And I don't think enough people talk about this. Herman: The hybrid is elegant. You run Tailscale directly on the PiKVM, so you get the simplicity of all-in-one — no bastion to maintain, one command setup. But you also deploy a lightweight bastion, and I mean lightweight — a Raspberry Pi Zero 2 W running Tailscale as a subnet router, sitting on the same management VLAN. That thirty-five-dollar board advertises the management subnet to your tailnet, so you can reach the PiKVM through it as an alternative path. Corn: You have two ways in. Direct to the PiKVM for everyday use — fast, simple, no extra hops. And through the subnet router as a fallback if the PiKVM's Tailscale daemon explodes, or as an audit point if you want to log sessions for a particular server. Herman: The subnet router costs thirty-five dollars and draws maybe two watts. It's not a full bastion — it's not running intrusion detection or session recording. But it gives you network diversity and a second authentication path. If you're really paranoid, you put the subnet router on a separate VLAN with different firewall rules, so an attacker who compromises the PiKVM doesn't automatically own the path through the router. Corn: There's another dimension to this that Daniel's question hints at but doesn't name directly, which is Cloudflare Tunnel versus Tailscale. They solve different problems and the choice changes your security posture. Herman: Cloudflare Tunnel is a completely different model. Instead of a mesh VPN where every device gets its own identity and IP, you run the cloudflared daemon on the PiKVM — either as a native binary or in a Docker container — and it creates an outbound-only tunnel to Cloudflare's edge. No inbound ports, no public IP needed, not even a VPN client on your end. The PiKVM's web UI becomes accessible through a Cloudflare-managed domain, and you layer Cloudflare Access on top for authentication — email OTP, device posture checks, whatever policies you configure. Corn: The upside is you don't need to install anything on the client side. You open a browser, hit the domain, pass the Access challenge, and you're looking at your server's console. The downside is you're completely dependent on Cloudflare as your auth proxy and your network path. If Cloudflare has an outage, or your PiKVM can't reach Cloudflare's edge, you have no access. Herman: Critically — no LAN-only fallback. Tailscale works over local network addresses when you're on the same subnet, no internet required. Cloudflare Tunnel needs a round trip to Cloudflare's edge even if you're sitting in the same room as the server. That's fine until your ISP goes down and you need to fix a server that's also on the same local network. Corn: Tailscale for "I want mesh networking and don't mind installing a client," Cloudflare Tunnel for "I want browser-only access and I'm willing to trust Cloudflare as my auth layer." They're not competitors so much as different threat models. Herman: You can stack them. PiKVM running Tailscale for your primary access path, cloudflared in a Docker container as a backup for when you're on a machine where you can't install a Tailscale client. It's a few extra lines of config and it gives you redundancy at the access layer. Corn: Which brings us to the "mission-critical" question. If this server really matters — if downtime means lost revenue or someone's medical records are on it or your entire family is yelling at you because Plex is down — what does a truly resilient setup look like? Herman: It's layers. The KVM device itself needs redundant power — either a PoE hat with a UPS-backed switch, or a separate USB-C power supply on a different circuit from the server it's managing. You don't want the same PDU failure that took down the server to also kill the KVM. Then redundant network paths: the PiKVM's Ethernet for primary connectivity, plus a USB LTE dongle for cellular failover. PiKVM supports these natively — plug in a compatible modem, configure the APN, and you've got a backup path that doesn't depend on your local ISP or your router or anything in your rack. Corn: The server could be in a data center with a total network outage, and you're still reaching the KVM over a cell tower. Herman: The enterprise devices bake this in. The Raritan Dominion KX IV has a built-in modem slot — you're not dangling a USB dongle off the front, it's integrated into the chassis with an external antenna port. But for a homelab, a PiKVM with a USB LTE modem gets you the same capability for a fraction of the cost. Corn: We've mapped the hardware, we've walked through the architectures, we've talked about the failure pattern. Let's boil this down to what you should actually do, because I think people come to this topic wanting a shopping list and they leave with an existential crisis about trust boundaries. Herman: The decision framework is straightforward once you stop overthinking it. If you have one or two servers and you want the simplest possible setup — and I mean the one where you're not going to forget to patch the bastion host for six months — get a PiKVM or a TinyPilot and run Tailscale directly on it. One command, you're done. The device is reachable from your phone, your laptop, anywhere in your tailnet. No bastion, no management VLAN, no extra hardware. Corn: For most homelabbers and small businesses, that's the right answer. The threat model where someone compromises your PiKVM's Tailscale node and then uses it to attack your server is real but it's also a pretty sophisticated attacker who's targeting you specifically. Herman: If you have a rack of servers, or you're subject to compliance requirements, or you just sleep better knowing there's a bouncer at the door — put the KVM devices on an isolated VLAN and stick a dedicated bastion host in front of them. A fifty-dollar thin client running Linux and Tailscale is all you need. It logs sessions, it gives you a single choke point to monitor, and if a KVM device gets compromised, the attacker still has to pivot through the bastion to reach anything else. Corn: Now the checklist, because this is the stuff people forget until they're locked out at two in the morning and the bastion won't answer. Number one: change the default passwords immediately. PiKVM ships with root root. I don't care if it's on an isolated VLAN, change the password before it touches your network. Herman: Number two: disable direct internet access. The device should only be reachable through your VPN or tunnel. No port forwards, no DMZ, no "just this once." If you need a web UI from outside, it goes through Tailscale or Cloudflare Tunnel or it doesn't go at all. Corn: Number three: enable session recording. PiKVM can record video of every KVM session to an external drive. It's the closest thing you'll get to an audit trail without a full bastion setup, and when something goes wrong at three in the morning and you can't remember what you typed, you'll want that recording. Herman: Number four — and this is the one nobody does — actually test the dead server scenario. Pull the network cable. Corrupt the OS networking config. Simulate the exact failure you bought the device for, and verify you can still reach the KVM and recover the server. Don't wait until the real outage to discover your USB gadget mode doesn't work with that particular motherboard's USB controller. Corn: That's the one that separates "I bought a KVM" from "I have out-of-band management. Herman: As for where the market is heading — it's converging toward all-in-one, but slowly. PiKVM's open-source stack means it's adding features faster than the enterprise vendors can ship firmware updates. If you're building a new setup today, buy a PiKVM v4 and run Tailscale on it. You get about ninety percent of enterprise functionality for ten percent of the cost. The remaining ten percent — multi-user concurrent access, LDAP authentication, SNMP monitoring — only matters when you hit a scale where you're already paying someone to manage your management infrastructure. Corn: At that scale, you're not listening to a podcast for architecture advice. You're writing a purchase order to Raritan. Herman: Here's the open question I keep coming back to. PiKVM v4 now supports USB mass storage emulation — you can mount an ISO remotely and boot the server from it, as if you'd walked into the data center with a USB stick. That's a feature that used to require enterprise gear. So as these prosumer devices keep absorbing capabilities that were exclusively Raritan's domain five years ago, what's the endgame for the enterprise vendors? Do they open-source their stacks to compete on features, or do they cede the low end entirely and become pure compliance appliances? Corn: I think they become compliance appliances. Nobody buys a Raritan because it captures HDMI better — they buy it because the auditor wants to see LDAP integration and SNMP traps and session logs that survive a subpoena. The hardware is a commodity now. The audit trail is the product. Herman: Which means the rest of us get to ride the open-source wave and pocket the difference. Corn: That's really the closing thought here. The best out-of-band management setup is the one you configure before you need it. A three-hundred-dollar PiKVM and a Tailscale subnet router can save you a cross-country flight, a ruined vacation, or a very awkward phone call with someone who doesn't know what a BIOS looks like. That's a weird prompt worth answering. Herman: Now: Hilbert's daily fun fact. Hilbert: In the high medieval period, Eton fives was played with a hand-protection glove weighing approximately one-twelfth of a stone — which converts to roughly one point two modern pounds, or the weight of a small loaf of bread in a Yamal Peninsula trading post circa twelve hundred AD. Corn:...right. Corn: This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop. If you enjoyed this, leave us a review wherever you get your podcasts — it helps. We'll be back next week with whatever Daniel sends us. Herman: Probably something about antenna design. I can feel it.