The used HEDT server: the silicon nobody talks about

The Substrate Series · 03 — The hardware arbitrage

Decommissioned datacenter platforms give you server-class PCIe lanes and bandwidth at scrap prices — if you know where to look, and are willing to do the work.

The middle tier was deleted on purpose

There used to be a tier of hardware that sat between consumer PCs and enterprise servers — a class of platform aimed at enthusiasts, content creators, and small workstations. It had multi-GPU as a first-class feature, generous PCIe lane counts, ECC support, and room for two CPUs and ten DIMM slots. It was called HEDT — High-End Desktop — and Intel and AMD systematically retired it across the last several product cycles. The X99 platform from 2014 was one of the last full-strength examples. Threadripper non-Pro is the closest current equivalent on the consumer side; Threadripper Pro and Xeon W-3500 are the workstation-tier replacements, and they are priced as workstations, not as enthusiast parts.

The retirement of HEDT was not an accident. It was a deliberate move to widen the gap between consumer and workstation/server pricing and to push prosumer multi-GPU workloads up into the higher-margin tier. The mainstream consumer platforms that replaced HEDT — AM5 on the AMD side, LGA 1851 on the Intel side — were designed in a way that actively prevents serious multi-GPU work, in ways most builders don't realize until they try.

This post is about what that gap looks like in practice, what's still available on the used market that bridges it, and the war story of one specific build — the GA-X99-UD3 that I got for free, fought for two months, and learned the entire platform from before graduating to the board I needed all along.

The lane-theft mechanism nobody warns you about

Modern consumer motherboards advertise PCIe Gen 5 x16 for the GPU and PCIe Gen 5 x4 for one or two M.2 slots, and they market the configurability as a feature. Read the fine print on any AM5 or LGA 1851 board manual and you find what's actually happening: most of the high-speed lanes are shared. Populating an M.2 slot drops your GPU slot from x16 to x8. Populating a second M.2 drops it again. Adding a second GPU drops both to x4. The board has roughly 24 to 28 usable high-speed lanes from the CPU, and any configuration that puts more than one bandwidth-hungry device on it forces a triage.

For a single-GPU gaming or single-NVMe workstation build, this triage is invisible because the workload never actually saturates the bus. For ML work — where you're moving model weights through PCIe between cards or between host RAM and GPU during training — the triage shows up as a measurable performance ceiling. And there is no path to get more lanes. The platform was designed without them.

HEDT didn't have this problem. The X99 platform's high-end CPUs (Xeon E5 v3/v4, Core i7 Extreme) brought 40 PCIe Gen 3 lanes from the CPU itself, plus another ~8 from the chipset. That's enough to run two GPUs at x16, an NVMe SSD at x4, and a 10 GbE NIC at x8 without sharing or compromise. Three-way and four-way GPU configurations were not just supported but actively marketed. PCIe Gen 3 is slower per lane than Gen 5, but at 40 lanes vs 24 the total useful bandwidth on an X99 board is competitive with or exceeds anything in the current consumer tier for multi-GPU work.

This is the arbitrage. The silicon that solves multi-GPU correctly was built in 2014, has been sitting in decommissioned servers for a decade, and is available on the secondary market for prices that don't track its actual capability.

The free board that was the teacher

In March 2026 I inherited a GA-X99-UD3 from my mother's old Photoshop PC. The board is a budget X99 from 2014: Xeon E5-2690 v4 (14-core, 28-thread, 40 PCIe lanes), four DIMM slots populated with DDR4-2133 RDIMMs, and a stock BIOS dated 06/15/2018 with version F23c — the last firmware Gigabyte ever shipped for the board.

The plan was simple. Drop a used Tesla V100 16 GB into the primary x16 slot, repaste with Arctic MX7, replace the thermal pads with Kryonaut Extreme, mount a pair of Arctic S4028-6K 40 mm server fans onto the passive heatsink, and stand up an inference and small-scale training node. The thermal prep was straightforward. The boot was not.

What followed was two months of intermittent warfare.

The first problem was that the BIOS F23c — the latest firmware ever shipped for this board, eight years after its design — has no Above 4G Decoding option. That option is the BIOS toggle that tells the platform it's allowed to map device BARs (Base Address Registers) above the 32-bit memory ceiling. The Tesla V100 demands a 16 GB BAR window for its memory aperture, which is above the 32-bit limit by definition. Without Above 4G Decoding the platform tries to fit the V100's BAR into a window that doesn't exist, the enumeration fails, and the board refuses to POST.

The second problem was the board's lane allocation. Most X99 boards use PCIe lane switches — PEX 8747s and similar — that present x16 to every populated slot and multiplex demand back to the CPU's 40-lane budget. The GA-X99-UD3 doesn't. It is switchless. The lanes are hard-wired to specific slots in specific patterns: x16/x0/x16/x8 with two cards, or x8/x8/x16/x8 with three. There is no path to get two GPUs both at x16 unless you populate the specific slot pair the board was designed for, and the secondary display card has to go in a slot that takes lanes from the primary.

The third problem was Volta enumeration on 2014 consumer firmware. The Tesla V100 is a datacenter card with the management features and large-BAR behavior that the contemporaneous consumer BIOS team never anticipated. Two cards — a V100 in slot 1 and a GTX 970 in slot 4 for display — would intermittently boot, intermittently hang on POST, and intermittently enumerate one card but not the other. The failure mode would change between cold boot and warm boot.

The fourth problem was the one that actually taught me the platform: figuring out which of the above was responsible for which symptom. There is no way to learn that without doing the work.

That work took the form of what I now call IFR archaeology — pulling the Intermediate Firmware Representation out of the BIOS image, identifying the hidden setup variables for Above 4G handling and MMIOH window placement, and forcing them through NVRAM edits because the board's setup UI didn't expose them. It took the form of lane-allocation diagnostics from lspci -vt trees and setpci register reads to confirm whether the GPU had actually negotiated x16 or had silently dropped to x8 because of a slot population conflict I hadn't anticipated. It took the form of BAR-window debugging when the V100 enumerated but reported zero usable memory because the BAR had been mapped to an address space that wasn't backed by physical RAM mapping.

Two months. Intermittent boot loops. Frustrating intervals where the system would work for a week and then refuse to come up. The kind of debugging where you take notes on which specific cold-boot sequence does and doesn't work, and you start to suspect the PSU rail, and you eliminate the PSU rail, and you suspect the slot, and you eliminate the slot, and you slowly build a model of the platform's failure modes that is far more detailed than anything in any document.

What I had at the end of those two months was a single-GPU MirrorForge node running stably, a dual-V100 build that worked when it worked and didn't when it didn't, and the kind of working knowledge of the X99 platform that nobody acquires from reading.

The free board didn't owe me anything. It taught me the platform. The phrase I keep coming back to is the teacher. The next board I bought, I bought because the free board made me earn enough knowledge to know exactly what was missing.

The graduation

On May 27, 2026 I ordered an ASUS X99-WS/IPMI. It arrives next week.

This is the board the GA-X99-UD3 was a budget approximation of. Same socket — LGA 2011-3 — so the Xeon E5-2690 v4 transfers. Same memory standard — DDR4 RDIMM — so the existing DIMMs transfer. Same boot drive, freshly cloned over Thunderbolt 4 to a Samsung 990 2 TB and verified bit-perfect with ddrescue and cmp (35m45s sequential read at 477 MB/s, then 35m37s byte-for-byte compare — zero errors). The V100s transfer. The migration is supposed to be invisible from the OS up.

What changes is everything underneath.

The X99-WS/IPMI is a workstation board, built in the era that expected datacenter GPUs as a normal workload. Above 4G Decoding is a first-class BIOS option. MMIOH base and size are exposed as adjustable parameters. The board has five PCIe 3.0 x16 slots with a native 40/28-lane distribution path: with a 40-lane CPU, dual-x16 is the documented configuration, not a slot-population trick. The V100s land in their genuine x16/x16 lanes, consuming 32 of the 40 CPU lanes; a GT1030 display card sits in a remaining slot at x8 or x4, which is irrelevant to a display card that pushes a thousand bytes per second.

(Correction noted: an earlier version of my own notes claimed this board had PLX 8747 switches. It doesn't — that's the X99-E WS, a different board in the same family. The WS/IPMI is a no-switch, native-lane-distribution board. The dual-x16 path comes from the CPU's lane budget directly, which is exactly what 40-lane Xeons are built for.)

The other thing the WS/IPMI brings is in the name: IPMI. The board has an ASMB8-iKVM management controller with a dedicated management LAN port, remote KVM over IP — BIOS and POST visible remotely — remote power control on a dead or off machine, OS-independent sensors, and virtual media that lets you mount an ISO over the network. This is the out-of-band management layer that post 06 will cover in depth. The short version: I paid $319.63 for this board, shipped and taxed, on the used market — which is cheap enough that the on-board IPMI alone justifies the price compared to anything in the consumer tier. The equivalent capability via a Pi-based gateway requires more parts, gives you fewer hardware-level controls, and as I'll explain in post 06 the two layers complement each other rather than replace each other.

I'll keep the Pi gateway on this node, and I'll have ASMB8 IPMI also. Hardware KVM, true power control, and virtual media from the BMC. Arbitrary scripting, mesh integration, and software orchestration from the Pi. Defense in depth on the management plane is one of the things the X99 war taught me to want.

The full inventory, as of May 2026

Since this post is the one where the parts list matters, here's the actual current state of the lab. Five compute nodes, one switch, one router, one UPS, and a small pile of storage. Every line is something the rest of the series will refer back to. Skim this section or skip it on a first read; later posts cite specific line items.

Legion (primary orchestrator, dev workstation, main inference for newest models) - Lenovo Legion Pro 7i - Intel Core Ultra 9 275HX (Arrow Lake-HX, 24 cores) - 64 GB DDR5 - RTX 5090 Laptop 24 GB - Storage: 2 TB PCIe Gen 5 NVMe (root), 4 TB WD SN850X (data), plus an Orico 40 Gbps Thunderbolt enclosure with a Samsung 990 EVO Plus 2 TB, and an SN770 1 TB 2230 in a USB-C dock case - This is the dev box and the highest-bandwidth inference node

MirrorForge (training worker, dual-V100 inference node — the box this post is mostly about) - ASUS X99-WS/IPMI arriving next week, $319.63 all-in (currently GA-X99-UD3, free hand-me-down) - Xeon E5-2690 v4 (14 cores, 28 threads, 40 PCIe lanes), $40 used - 32 GB DDR4 RDIMM (transferring across the board upgrade) - 2× Tesla V100 16 GB PCIe (HBM2, 900 GB/s each, 32 GB aggregate) — $299.32 and $304.67 all-in - Seasonic Vertex GX-1200 PSU (new, Facebook Marketplace), $150 - Storage: 2 TB Gen 4 NVMe (boot, freshly cloned to a Samsung 990 2 TB) + 250 GB SATA SSD (scratch) - Subtotal of the named line items above: $1,113.62. Cooling (Arctic S4028-6K fans, MX7 paste, Kryonaut Extreme pads, laptop brick, buck converter, dual PCIe→EPS Y-adapters) and the boot NVMe add roughly another $250. The full MirrorForge node sits well under $1,500 marginal.

ASUS F15 (visual perception, image-gen experiments, ASUS-side mesh node) - ASUS TUF F15 (FX507VI) - i7-13620H - 32 GB DDR5 - RTX 4070 Laptop 8 GB - Storage: 1 TB Gen 4 NVMe + 2 TB WD SN850X

Orin (Jetson Orin Nano Super — Nexus backend host, edge inference) - NVIDIA Jetson Orin Nano Super (67 TOPS AI) - 8 GB - 1 TB Gen 4 NVMe

Pi5 (Hailo-8 memory engine, camera capture, voice pipeline) - Raspberry Pi 5, 8 GB - Hailo-8 NPU (26 TOPS edge inference) - 1 TB Gen 4 NVMe

picontrol (Pi Zero 2 WH — out-of-band gateway, mesh monitor, Tailscale subnet router, planned fan/LED controller) - Raspberry Pi Zero 2 WH - USB gadget piggyback to MirrorForge as primary uplink (g_ether at 10.9.0.2/24, MirrorForge NATs to the LAN); WiFi (CenturyLink5927) as fallback only - Tailscale (100.101.17.42) advertising 192.168.50.0/24 to the tailnet — entire LAN reachable from anywhere I'm logged into Tailscale - Cockpit web UI for remote management, ntfy push notifications on the mirrorforge-alerts topic - mesh_monitor.py systemd service: pings all four mesh nodes every 30 s, polls V100 + 5090 GPU temps every 60 s, disk + crash detection every 5 min, GPU warn 80 °C / crit 90 °C - Dedicated 5 V PSU — USB-port-only power causes brownouts under load, dual-power crashes (a small but illustrative power-domain story for post 04)

Cold storage and shared - 16 TB WD Elements (cold archive) - 2× 1 TB WD portable drives (working backups) - USB switch between Legion and ASUS (rarely used, mostly KVM-style legacy)

Network - Symmetric gigabit fiber WAN: measured 987 Mbps down, 948 Mbps up - Netgear GS316 16-port gigabit switch - 12-inch Cat 6 runs between the rack-adjacent nodes - GL-MT1800 private router as VPN/WiFi backup (mesh-routable secondary path) - APC Back-UPS Pro 1500VA (BX1500M) covering the rack-side nodes and the switch

Display and console - Acer 5-in-1-out HDMI switch fed by Pi5, Orin, and MirrorForge — headless boxes share one display when one is actually needed, which is rarely - Everything else is tmux + SSH + Tailscale

The line items above are the entire substrate this series is built on. Every later post takes one or more of them and walks through the reasoning for the choice and the alternatives that didn't get picked. The interesting move is not any single component — it's the way the components compose into a single workspace where any node can reach any other and any GPU can serve any node.

The labor-not-money tax

The arbitrage on used HEDT is real. The cost is paid in hours, not in dollars, and I want to be specific about what those hours look like, because the appeal of the cheap build collapses if I oversell it.

You will read board manuals from companies that no longer exist, looking for jumper diagrams and lane-mapping tables that were never properly published.

You will dump BIOS images and walk through Intermediate Firmware Representations looking for hidden setup variables, because the front-end BIOS UI on a 2014 consumer board does not expose the controls a Tesla V100 needs.

You will sit at a workbench with a cold-boot reset cycle of three minutes per attempt while you binary-search across firmware variables, slot populations, and power states for the configuration that survives.

You will buy adapters. The two PCIe-to-EPS12V Y-cables I'm using to feed both V100s their full 12 V come from a niche market that exists because Tesla cards don't use the same power connectors as gaming cards. You will read forum posts in three languages and dismiss most of them.

You will, at some point, decide the work was worth it, because the system you have at the end is not a hobby project. It's a piece of working infrastructure that runs production workloads, the silicon was decommissioned by people who paid ten times what you paid for it, and the knowledge you acquired is the kind that doesn't expire when the silicon does.

The auto-mechanic model: skills compound, tools amortize, entry money is bounded, and the work is the moat. There is no shortcut, and there isn't supposed to be.

New-equivalent pricing

To make the arbitrage concrete, here is the same compute profile in new dollars vs the actual used-build receipt. Prices are May 2026 spot checks from eBay (used) and Newegg/Amazon (new). Used-build column is what I actually paid, after shipping and tax.

The arbitrage is real and it's roughly 6× on this build. The HBM2 bandwidth math from post 02 still applies — two V100s in genuine x16/x16 deliver 1.8 TB/s of aggregate memory bandwidth, which is in the same neighborhood as a single 4090 and meaningfully ahead of a single 3090, at a fraction of the cost. The compute capability ceiling is lower than the new-equivalent build's, but only on workloads that are FLOPS-bound rather than bandwidth-bound, and as covered in post 02, most ordinary inference work isn't FLOPS-bound.

A note on the alternative comparison. If your workload genuinely fits in 32 GB of VRAM and you don't care about future expansion, a single RTX 5090 build on a consumer AM5 board avoids the multi-GPU architecture entirely. At real street prices in May 2026 — AIB cards $3,500–5,000+, FE at $1,999 MSRP effectively unobtainable, all of it driven by the GDDR7 memory shortage — plus a modest mainstream board and CPU, that build runs $5,000–6,500 all-in. Still 3–4× the used dual-V100 build, and it forfeits the headroom to grow into a third card without rebuilding the whole platform. The arbitrage shrinks if you concede the multi-GPU requirement; it stays roughly 6–10× if you don't.

The contemporary version of this arbitrage

X99 + V100 is the arbitrage that worked for me this build cycle. The next one is already visible on the secondary and grey markets.

The current sweet spot for used HEDT is the EPYC Milan generation — 7003-series CPUs on SP3-socket motherboards. Single-socket Milan parts bring up to 64 cores and 128 PCIe Gen 4 lanes. That's enough to put three or four full-bandwidth GPUs at native x16 with no lane-sharing tricks, and PCIe Gen 4 doubles per-lane bandwidth compared to X99's Gen 3. Used EPYC 7443 (24-core, $400–600) and 7543 (32-core, $700–1,000) Milan CPUs trade actively on eBay; Chinese SP3 boards from JGINYUE and similar vendors sell on AliExpress and elsewhere in roughly the same price band as the X99-WS/IPMI used and have the same firmware-quirk-archaeology profile that the GA-X99-UD3 had — lower cost than a Threadripper Pro WRX80 platform, higher labor tax, with the same payoff shape if you're willing to do the work.

This is the platform tier that the next MirrorEthic build cycle will land on. The skill that the X99 war taught me is the skill that makes the EPYC build tractable. The free board genuinely was the teacher, and the auto-mechanic model isn't a metaphor — it's literally how the skills compound across build cycles.

What this post is really about

The hardware arbitrage is real, and the receipts back it up. The deeper point of this post is the relationship between knowledge and capability.

I could not have walked in cold and bought the right ASUS X99-WS/IPMI board. I would not have known to value the on-board IPMI, or to care about MMIOH window sizing, or to check for native dual-x16 vs PLX-multiplexed dual-x16. The cheap GA-X99-UD3 fought me for two months and taught me what to want. The expensive purchase that follows the cheap one isn't a fall from grace — it's the graduation that the cheap one was always preparing me for.

If you take only one thing from this post, take this: the cheap build, done seriously, makes the next build cheaper and better. Not because the parts compound, but because the operator does.

The patterns the next posts cover — power separation, networking, out-of-band management, mesh routing, substrate-independent roles — are each instances of that same dynamic. Every one of them has a teacher version that costs nothing and a graduation version that costs less than the convenience-tax tier and works better.

The middle tier was deleted on purpose. It still exists if you know where to look, and the secondary market is where the deletion never quite landed.

Next: post 04, "Separation of concerns: power and thermal" — server fans on a laptop brick, the static-pressure argument, and why your cooling shouldn't share rails with your GPUs.