Latest / Logosofia no Dia a Dia / When Your Server Won't Even Boot a Live USB
Transcript
- [Prompt] custom: Sometimes when an Ubuntu server isn't booting, it's very difficult to try to get low-level diagnosti | Hosts: corn, herman [Response] Corn: Daniel sent us this one — and if you've ever stared at a black screen at two in the morning with a dead server and a sinking feeling in your stomach, this is your episode. He's got an Ubuntu server that won't boot. Not just the OS — he can't even get a live USB like SystemRescue to come up. And he's asking whether there are diagnostic tools, software or hardware, that can tell him if it's the motherboard, the RAM, the SSD, or something else entirely, when every normal recovery path is just... Herman: This is the nightmare. This is the moment where you realize the gap between "my kernel panicked" and "my silicon gave up" is about a mile wide, and most of us spend hours in that gap just swapping parts and praying. Corn: The live USB thing is what really gets me. You grab SystemRescue — version eleven oh two, Arch-based, beautiful piece of work — and you think, this is my escape hatch. I'll boot from USB, run fsck, check the drives, figure out what's corrupted. But you plug it in, hit power, and... Same black void. That's when the cold sweat starts. Herman: Because a live USB is not magic. It still needs the host's memory controller to initialize. It still needs the PCIe bus to enumerate. It still needs the CPU to execute its microcode. If any of those are compromised — if the memory controller is stuck in a training loop, if the PCIe root complex isn't coming up — then your SystemRescue USB is just a fancy piece of plastic with an LED that never blinks. Corn: It's the hardware equivalent of calling for backup and getting a busy signal. Herman: And that's the threshold we're talking about here. Daniel's server has crossed from "software problem" into "hardware doesn't know it's a computer anymore." And most admins, even experienced ones, hit this wall and immediately start doing the wrong thing — they replace the motherboard, then the RAM, then the power supply, and three hundred dollars later they find out it was a dusty DIMM slot. Corn: The parts cannon. I've seen it. I've done it. There's a certain kind of shame in realizing you RMA'd a perfectly good CPU because you didn't check the twelve-volt rail. Herman: That's the cost of misdiagnosis, right? It's not just time — it's money, it's downtime, and it's the slow erosion of your own confidence in your troubleshooting skills. Every time you guess wrong, you wonder if you actually know what you're doing. Corn: Let's define what we're actually up against when the server gives you absolutely nothing. Herman: There are really three distinct failure signatures, and they point to entirely different parts of the hardware. The first is a POST failure — the machine powers on, fans spin up, but you never even reach the bootloader. No beep, no display, nothing. That's the firmware telling you, in its own mute way, that something fundamental can't initialize. Corn: POST failures are chipset-specific, right? The codes mean different things depending on whether you're on an Intel C six twenty-one or an AMD B five fifty. Herman: Intel's C six twenty-one uses port zero x eighty for debug output, AMD's B five fifty uses port zero x eighty-four. Same hex code, completely different meaning across platforms. A zero x fifty-eight on Intel is a memory initialization error. On AMD it could be something totally unrelated. So if you're just staring at a two-digit display on the motherboard with no reference table, you're decoding hieroglyphics without a Rosetta Stone. Corn: Which is why you print out the POST code table for your specific chipset and tape it to the server rack before you need it. Herman: The second failure signature is the kernel panic before initramfs — you see GRUB, you select the kernel, and then it explodes before it ever mounts a root filesystem. That's often a CPU or memory controller issue, because the kernel is trying to initialize page tables and the memory subsystem is giving it garbage. The third one, and honestly the most demoralizing, is the black screen of nothing — power LED on, fans spinning, no video output at all. That can be the GPU, the motherboard, the PSU, or even a short in a USB port dragging down the five-volt rail. Corn: That last one is miserable because it gives you exactly zero information. You're just... in the void. Herman: That's where the live USB assumption collapses. SystemRescue version eleven oh two is genuinely fantastic — it's Arch-based, it ships with fsck, ddrescue, btrfs-restore, the whole toolkit. But it's still just software. It loads into the same physical memory, talks to the same memory controller, traverses the same PCIe bus. If the memory controller is stuck in a training loop on DIMM slot A two, the live USB never even gets to execute its first instruction. Corn: The live USB didn't fail because it's a bad tool. It failed because the hardware never gave it a chance to be a tool. Herman: That's the distinction. And it's why Daniel's question is actually two questions. One, how do I diagnose hardware when software can't run? And two, how do I know which layer of hardware is broken before I start buying replacement parts? Corn: Which brings us to the thesis of this whole thing. When software-level diagnostics are unreachable — when you can't boot an OS, can't run a memory test, can't query SMART data from a running kernel — you have to drop down to hardware-level interrogation. IPMI, serial consoles, component-level isolation. Tools that don't need the host to be a functioning computer. Herman: The key insight is that some of these tools are already in your server. The BMC — the Baseboard Management Controller — is a completely independent computer inside your computer. It has its own ARM processor, its own memory, its own network stack. It doesn't care if the main CPU is dead. It doesn't care if the RAM is corrupted. It's watching the whole time, logging sensor data, tracking POST codes, and it can give you a serial console over the network even when the mainboard is a paperweight. Corn: It's the server's black box recorder, and most people never check it. Herman: Most people don't even know it's there. Or they think it's only for enterprise gear — but that's a misconception. ASRock Rack boards, Supermicro boards, even some high-end ASUS workstation boards ship with BMCs. IPMI two point zero has been standard since two thousand four. The Serial Over LAN feature that lets you see POST output remotely? That's been in the spec for twenty-two years. Corn: Before you reach for a screwdriver, before you start pulling DIMMs, before you order a new motherboard in a panic at three in the morning — you check the BMC. You fire up ipmitool, you read the sensor logs, you activate the serial console, and you let the server tell you what it was trying to do when it died. Herman: This is where it gets beautifully concrete. Let's say your server is headless — no monitor, no keyboard, sitting in a rack or a closet. You can't even see if it's POSTing. But if the BMC is alive and on the network, you open a terminal on your laptop and type `ipmitool -H [BMC IP] -U admin -P [password] sol activate`. That's it. Suddenly you're watching the UEFI boot sequence scroll by, line by line, over the network — memory training, PCIe enumeration, option ROM loading — all of it, even though there's no physical display attached to the machine. Corn: The server could be sitting there with no video output at all — dead GPU, dead display controller, whatever — and you're reading its POST messages on your laptop like a remote serial terminal from nineteen eighty-five. Herman: And here's what makes this powerful for Daniel's scenario. If the server is hanging at "DRAM Initialization" and never progressing, SOL shows you that. The kernel panic log can't tell you anything about DRAM initialization because the kernel never loaded. The OS never loaded. The bootloader never loaded. But the BMC was watching the whole time, and SOL is just streaming what the BMC sees. Corn: I've actually seen this exact case. Server with no video output, fans spinning, no beep codes. Hooked up SOL, and the console was stuck in a loop — "Memory training failed on DIMM A2, retrying." Over and over. Swapped the stick in slot A two, and it POSTed immediately. Never booted an OS to diagnose that. Herman: That's the case study that proves the point. Without SOL, you're pulling every DIMM, reseating everything, guessing. With SOL, you know it's slot A two in thirty seconds. Now, the BMC gives you more than just a serial console. `ipmitool sensor list` pulls every temperature sensor, every voltage rail, every fan speed reading the BMC has been tracking. If your twelve-volt rail is sitting at eleven point two volts, you've got a PSU problem, not a RAM problem. `ipmitool sel list` dumps the System Event Log — that's where the BMC records ECC correctable errors, uncorrectable errors, thermal throttling events, all timestamped. Corn: You can see that DIMM slot B one logged forty-seven correctable ECC errors in the last hour before the server went dark. That's not a maybe. That's your diagnosis. Herman: `ipmitool chassis power cycle` gives you a hard reset that doesn't require physically pulling the power cord — useful when the server is in a colo facility an hour away. But here's the question Daniel's scenario raises: what if the BMC itself is unresponsive? What if the BMC firmware is corrupted, or the dedicated NIC PHY is dead, and you can't even ping the management interface? Corn: That's the moment you go from "I'll fix this remotely" to "I need to physically touch this machine." And the tool you bring is a POST card. Herman: A thirty-dollar PCIe POST card — or an LPC bus analyzer if the board has the header — that reads the hex codes the firmware is writing to port zero x eighty or zero x eighty-four during initialization. Those codes are the last thing the board is trying to tell you before it gives up. A zero x fifty-eight on an Intel C six twenty-one platform is "memory initialization error." A zero x fifty-five is "memory not installed." A zero x A two is "IDE detect" — which means the board got past memory and CPU init and is now stuck on storage enumeration. That tells you the CPU and RAM are probably fine. Corn: The POST card is giving you a progress bar for the boot process, expressed as hex codes, at a level below anything an OS could ever report. Herman: It's the fallback when the BMC is dead, which does happen. Corrupted BMC firmware, failed flash chip, network isolation — if you can't reach the management controller, the POST card is your next best window into what the silicon is actually doing. Between those two — IPMI SOL and a POST card — you can diagnose almost any no-boot scenario without ever loading a kernel. Herman: What if the BMC itself is dead? Now we're in pure hardware forensics territory. No network console, no sensor logs, no remote power cycling. You're standing in front of a metal box that gives you nothing but fan noise. Corn: This is where the minimum configuration method saves you from yourself. Strip the system to one stick of RAM in slot A one, no drives, no PCIe cards, no nothing. Just the CPU, one DIMM, and the motherboard. If it POSTs, you add components back one at a time until it doesn't. If it still won't POST, swap that single DIMM for another stick and try again. Herman: The order matters. Slot A one is almost always the primary channel on consumer and server boards alike — the memory controller trains that slot first. If the board can't POST with a known-good stick in A one, you've ruled out RAM configuration and you're looking at the CPU, the motherboard, or the PSU. Corn: That's the systematic elimination part. You're not guessing. You're building the computer back up from its bare minimum and watching exactly where it breaks. Herman: Let's say it does POST with one stick. The natural next step is to run Memtest86 plus version seven point zero — the open-source fork Martin Whitaker and the community have been maintaining. It supports DDR five, up to two hundred fifty-six cores, and here's the clever bit: it loads its own code entirely into the CPU cache. It doesn't depend on the host memory controller being fully functional for its own execution. Corn: Which is why it can sometimes run even when the memory subsystem is partially broken. But that's also the trap. Herman: If the memory controller itself is failing — not the DIMMs, but the silicon on the CPU that manages them — Memtest can crash, or worse, report false positives that send you chasing ghosts. I've seen a server that passed Memtest for twelve straight hours, no errors. But it was still kernel panicking randomly under load. Swapped the CPU into a known-good board, and the panics followed the CPU. The memory controller was degrading under thermal load, and Memtest's access patterns never hit the failing transistors. Corn: Memtest passing is not a clean bill of health. It's a specific test under specific conditions, and intermittent faults — especially thermal ones — can slip right through. Herman: If Memtest won't even run? If it crashes on launch or the system reboots the moment it tries to load? That's almost certainly the CPU memory controller or the motherboard's VRM. At that point you're swapping the CPU into a test board or probing voltages with a multimeter. Corn: Let's talk about the PSU before we get to the motherboard, because this is where I've personally wasted the most time. A failing power supply can mimic RAM failure, CPU failure, motherboard failure — it can look like literally anything. Herman: The mechanism is voltage ripple. If the twelve-volt rail has enough AC ripple on it — even a few hundred millivolts — it corrupts memory operations without ever dropping low enough to trigger a fault LED. The PSU looks fine. The voltages read normal on a basic multimeter. But under load, those ripples are flipping bits in RAM, and the kernel panics look exactly like bad DIMMs. Corn: What's the fastest way to rule out the PSU without a spare unit sitting on the shelf? Herman: A PSU tester. Twenty to fifty dollars, plugs into the twenty-four-pin ATX connector, the EPS twelve-volt, the PCIe power cables — and it loads each rail and checks for ripple. Not just voltage presence, which a cheap tester will do, but actual ripple under load. If you don't have one, the next best thing is to swap in a known-good PSU. But that assumes you have a spare. The tester is cheaper than a spare PSU and it fits in your toolkit. Corn: It's faster than the alternative, which is replacing the motherboard, the RAM, and the CPU before you finally try a different power supply and everything magically works. Herman: The cost of misdiagnosis there is brutal. A two-dollar capacitor on the VRM fails, you replace a three-hundred-dollar motherboard. A DIMM slot gets dusty, you RMA perfectly good RAM. I've seen a server where the only problem was a single bulging capacitor on the CPU Vcore regulator — visible if you just looked at the board — and the admin had already ordered a new CPU. Corn: Visual inspection feels almost too simple to mention, but bulging or leaking capacitors are surprisingly common on boards that have been running for five-plus years. You don't need a schematic. You just need eyes. Herman: For the SSD side of Daniel's question — NVMe drives have a trick that most people don't know about. Even when the PCIe link is degraded and the drive won't show up in the OS, the NVMe Management Interface still works. It's a separate out-of-band channel defined in the NVMe spec, and `nvme-cli` version two point x can query SMART data through it — reallocated sectors, wear level, power-on hours, everything. Corn: If the PCIe link fails to train entirely, the drive is invisible no matter what. Herman: And that's where a USB-to-NVMe adapter becomes worth its weight in gold. Pull the drive, drop it in the adapter, plug it into another machine, and suddenly you can read the SMART data. I saw this exact case — an M dot two drive that wasn't detected on the host's slot, zero signs of life. Put it in a fifteen-dollar USB adapter, ran `nvme smart-log`, and there it was: fifty reallocated sectors and a warning threshold tripped. The drive was dying, and the host's PCIe controller had stopped trying to train the link as a protective measure. Corn: The adapter bypasses the host's PCIe bus entirely and lets you talk to the drive over USB. That's a diagnosis you'd never get from inside the server. Herman: It tells you definitively whether the problem is the drive or the motherboard's M dot two slot. If the drive works fine in the adapter, your slot is dead. If the drive is full of reallocated sectors, your slot is fine and the drive is toast. Herman: Let's distill all of this into a practical checklist you can tape to your server rack. Daniel's scenario — dead server, no live USB, no idea what's broken — has a workflow, and most of it fits in a shoebox. Corn: A dead server kit. I love this. What's in it? Herman: A thirty-dollar PCIe POST card — the kind that reads port zero x eighty or zero x eighty-four hex codes. A PSU tester, twenty to fifty bucks, the kind that checks ripple under load, not just voltage presence. One known-good stick of DDR four or DDR five, whatever your servers use. A USB-to-NVMe adapter — fifteen dollars, fits in your pocket — for pulling drives and reading SMART data on another machine. And a USB drive pre-loaded with Memtest86 plus version seven point zero and SystemRescue version eleven oh two. That whole kit costs less than a single unnecessary motherboard replacement. Corn: The known-good RAM stick is the unsung hero there. It's your control variable. If the board POSTs with your test stick but not with the installed DIMMs, you've isolated the fault in thirty seconds. Herman: The other thing you do before the server ever dies — and this is the one people always skip — is configure your BMC to log to a remote syslog server. IPMI SOL is incredible when the machine is down, but it only shows you what's happening right now. If you've been streaming the BMC's system event log to a remote syslog for the past six months, you have historical POST data. You can see that ECC errors started spiking on DIMM B one three weeks ago and ramped up until the server finally failed last night. That's forensic evidence, not guesswork. Corn: When the server goes completely dark, you're not starting from zero. You're starting from a timeline. Herman: And here's the eighty-twenty rule that saves more time than any tool: eighty percent of no-boot failures are the PSU, RAM seating, or a single bad DIMM. Check those first. Reseat the RAM. Swap the PSU or test it. Try one stick at a time. Do not touch the motherboard or the CPU until you've ruled out those three. I've watched people tear down an entire server because they didn't reseat a DIMM that had walked out of its slot from thermal cycling. Corn: The boring stuff is usually the culprit. And the final thing — and I know this sounds like office-supply advice — print out the POST code table for your specific chipset. Intel C six twenty-one uses port zero x eighty. AMD B five fifty uses port zero x eighty-four. The same hex code means completely different things on each. If you're standing in front of a dead server with a POST card showing zero x fifty-eight and no reference table, you're still guessing. Herman: Tape it to the inside of the rack door. Laminate it if you're feeling fancy. The moment that POST card gives you a code, you want the answer in seconds, not after a frantic phone search while your heart rate climbs. Herman: Where does this leave us as hardware gets more complex? Because the diagnostic workflow we just walked through — POST cards, IPMI SOL, minimum configuration boot — that was designed for a world where one server is one box with one set of DIMMs and one CPU socket. That world is already dissolving. Corn: CXL memory pooling. You're not diagnosing a DIMM in slot A two anymore — you're diagnosing a memory pool that's shared across four nodes and connected through a CXL switch that has its own firmware, its own error registers, its own failure modes. Herman: The diagnostic tools for that barely exist. When a CXL-attached memory pool throws errors, which server's BMC is responsible for logging it? The memory appliance? The switch in between? Right now the answer is "it depends on the vendor," which is the least comforting phrase in hardware diagnostics. Corn: The POST card of the future might need to be a CXL protocol analyzer — something that sits on the link and decodes cache-coherent traffic in real time. That's not a thirty-dollar tool. That's a five-figure piece of lab equipment. Herman: There's another failure pattern coming that I think is underappreciated. Firmware TPM and measured boot. More and more servers are sealing encryption keys to specific PCR values — measurements of the firmware, the bootloader, the kernel. If your motherboard fails and you replace it, those PCR values change. The TPM sees different hardware and says "I don't recognize this machine" and refuses to unseal the keys. Corn: A hardware failure doesn't just take your server offline. It can cryptographically lock you out of your own data. Herman: And most people don't back up their TPM recovery keys because "the TPM is on the motherboard, the motherboard is reliable." Until it isn't. A blown VRM doesn't care about your encryption policy. Corn: That's the kind of failure pattern that turns a bad day into a career-defining disaster. And it's not theoretical — we're going to see this in homelabs and small businesses within the next few years as measured boot becomes the default. Herman: Which brings us back to Daniel's original question. The tools we've described — the POST card, the PSU tester, the NVMe adapter — those work today. They're cheap, they're proven, and they'll save you from the parts cannon. But the next generation of diagnostic tools needs to be built for a world where the "server" is a logical construct spread across a rack, and where a hardware failure can lock you out of everything you own. Corn: The kit we described is your starting point. But the open question — the one I want listeners to sit with — is what that kit looks like in five years. If you're building homelab infrastructure now, what diagnostic hooks are you designing in before you need them? Herman: Check the show notes — we'll have links to the POST code reference PDF for Intel and AMD platforms, plus the Memtest86 plus version seven point zero download page. Build your dead server kit this week, not the week your server dies. Corn: Thanks as always to our producer Hilbert Flumingtop. This has been My Weird Prompts. If you've got a diagnostic war story or a tool we missed, email the show at show at my weird prompts dot com — we read every one. Herman: Now: Hilbert's daily fun fact. Hilbert: In Ethiopia, the Christmas game of genna is a form of field hockey played with curved wooden sticks and a wooden ball, and it descends from a tradition that according to local accounts originated when shepherds celebrated the birth of Christ by spontaneously inventing the sport — but the earliest written descriptions of the game don't appear until the early medieval period, roughly a thousand years later, leaving a gap in the historical record wide enough to drive a hockey stick through. Corn:...hockey stick through a historical gap. Alright then. Herman: We'll be back next week.