The fog over AI hardware

The Substrate Series · 02 — Myth-bust

"You need an H100 for AI" is commercial framing, not technical reality — and here's who benefits from you believing it.

Five interests, one direction

If you've spent any time in AI hardware discourse, you've absorbed an assumption: serious work requires datacenter-class silicon. It is so consistent that it reads as technical fact. It isn't. It's the result of five distinct commercial interests that happen to all point the same direction.

NVIDIA's product segmentation. Every NVIDIA generation since Volta has carefully placed a fence between what consumer cards can do and what datacenter cards can do. NVLink got stripped out of consumer cards. Multi-instance GPU is datacenter-only. Driver licensing for virtualization is datacenter-only. The vRAM ceilings on consumer parts are set exactly low enough to be insufficient for serious multi-card work, and exactly high enough to feel generous for gaming. The segmentation is the product strategy.

The cloud rental model. Cloud GPU pricing depends on the assumption that the alternative — running your own — is infeasible for most people. AWS p4d.24xlarge is $21.96/hr on-demand as of May 2026 — that's $2.75 per A100-hour, eight A100 40 GB SXM4s in the node. A used V100 PCIe 16 GB is $285–380 forever (price floor on China-direct, US-shipped tax-included closer to $300–380). A used A100 80 GB is more complicated: SXM4 modules — which need a carrier board and datacenter cooling to actually use — trade at $5,500–9,200; PCIe drop-in cards at $18,500–21,000. The gap between those two A100 prices is the integration labor this series is about. Cloud providers compete with each other and with no one else; they do not compete with the secondary market, because the secondary market is invisible in the discourse they help shape.

Frontier-lab capital narratives. "Compute is the moat" is the story that justifies billion-dollar fundraises. It is partially true — at the frontier — and entirely useful as a brand. The labs do not benefit from a public conversation in which mid-scale research is shown to run perfectly well on $300 hardware.

The content-sponsorship economy. Hardware YouTube is sponsored by NVIDIA partners, AIB makers, prebuilt-system vendors, and cloud GPU providers. None of them are sponsored by the eBay reseller you'd buy a Tesla V100 from. The bias isn't malicious — it's structural. The economic gravity points away from "use what's already in the secondary market."

Benchmark marketing. The benchmarks that get quoted are the ones that favor the latest silicon. Raw FP32 TFLOPS, FP8 tensor throughput, MLPerf training scores. The metrics that actually dominate most real workloads — memory bandwidth, dollars per GB/s, watts per useful token, time-to-first-token on bog-standard models — are systematically de-emphasized because they make older silicon look much better than the marketing wants to admit.

None of these interests are doing anything wrong. They're each rationally optimizing for their own position. The collective effect is a fog over the entire field, and the fog hides the fact that the workload most independent researchers and small AI shops are actually doing is not the workload the marketing is priced for.

Three workloads, not one

The reframe is the single move that collapses most of the discourse. AI workloads divide into three regimes that have almost nothing to do with each other from a hardware-requirements standpoint.

Frontier pretraining is training a foundation model from scratch — billions of parameters, trillions of tokens, weeks of wall-clock. This genuinely requires the gear the marketing implies: clusters of H100s or B200s, high-bandwidth interconnects, professional power and cooling. If you are doing this, you already know it. You have the capital, the team, and the rented or owned datacenter footprint. This work is roughly 1% of what gets called "AI" in practice and roughly 100% of what the discourse implicitly assumes everyone is doing.

Research training is training small to mid-scale models — say, sub-100M to a few B parameters — for experimental work. Ablation studies, new architectures, fine-tunes, exploratory runs. This is the actual day-to-day of academic and independent research, and the requirements look entirely different. A single RTX 5090 trains a 350M-parameter model overnight. A V100 trains a 150M model overnight. A 4070 with 8 GB of VRAM trains smaller models faster than people expect. None of this is gated behind H100s. It is gated behind knowing where to look for cards that cost what they should cost.

Inference is running already-trained models in production. This is the largest workload by a long way in real-world AI use. Image generation, language model serving, embedding generation, voice synthesis, classification. And it is dominated by memory bandwidth, not FLOPS — because at inference time you are streaming model weights through the GPU once per token or per denoising step, and how fast you can stream those weights is set by the memory bus, not the matmul units.

The discourse never quotes memory bandwidth because the marketing-favored metrics are FLOPS. So let's quote it.

The table the marketing doesn't lead with

Pulled from active eBay listings and Best Value GPU's price tracker on the morning of May 28, 2026. Memory bandwidth from manufacturer datasheets. Treat the consumer-card rows as a dated observation of a volatile market — they will look slightly different next month. The V100 row will look the same. That asymmetry is itself a result the marketing doesn't lead with.

The 16 GB ladder — same capacity, just bandwidth and price

Here is the cleanest possible version of the argument. All four cards below ship with 16 GB of VRAM, so capacity is controlled out of the comparison. The only axes that differ are memory bandwidth and street price.

Read the bandwidth column against the price column. The cheapest card has the highest bandwidth. Three framings of the same fact:

• A $300 card from 2017 has the same memory bandwidth as a $1,000 card from 2025. V100 900 GB/s vs 5070 Ti 896 GB/s — closest match, biggest price gap, and the version that fits in a single tweet.
• More than double the bandwidth of a 5060 Ti 16 GB for two-thirds the price. V100 900 vs 5060 Ti 448, same capacity, knockout.
• The 4070 Ti Super is around $1,000 and still has 25% less bandwidth than a V100 that costs a third as much. V100 900 vs 4070 Ti Super 672 — the prosumer card most people would name as a serious inference build sits below the used datacenter card on the metric that actually matters.

Dual-V100 kicker. Two used V100s in a single chassis is 1,800 GB/s aggregate of HBM2 bandwidth and 32 GB total VRAM, for about $600 in cards. A single new 5070 Ti at ~$1,000 is 896 GB/s and 16 GB. Two used V100s deliver double the bandwidth and double the VRAM at sixty percent of the price.

The required caveat (and why scoping makes the table unimpeachable)

The consumer cards above win decisively on almost everything else: FP8 and FP4 native tensor cores, DLSS, raw compute density, power efficiency (the 5070 Ti's 300 W does dramatically more FLOPS than the V100's 250 W), modern quantized model formats, and they are meaningfully better training cards. The 5070 Ti also had a documented ROP defect on under 0.5% of early units — a QA issue, unrelated to bandwidth, but worth knowing if "5070 Ti problems" comes up.

The table above is specifically bandwidth-per-dollar — the axis that dominates inference throughput. Scope to that axis and the argument is untouchable. Drift it to "the V100 is just better" and any informed reader buries you. The fog moves cards on prestige; the table moves them on the metric inference actually cares about.

The wider landscape

For reference, here is the full bandwidth-per-dollar context across the broader market, including consumer cards above 16 GB and the datacenter tier above 32 GB:

Read this carefully, and read it in ratios. Absolute prices age; ratios don't.

The V100 moves bytes per second at roughly $0.32/GB-s. That's:

• ~3.3× cheaper per GB-s than a used 3090 at the low end of its current range.
• ~7.4× cheaper per GB-s than a used 4090.
• ~6× cheaper per GB-s than even an entry-AIB 5090.
• ~8× cheaper per GB-s than a used A100 80 GB SXM4 — and ~30× cheaper per GB-s than an A100 80 GB PCIe.
• ~23× cheaper per GB-s than an H100 80 GB SXM5.

Notice the A100 row split. The SXM4 module and the PCIe card are the same silicon and ship the same model number; the PCIe card is roughly 3× the price because it solves the integration problem — you can drop it into a normal PCIe slot rather than needing a carrier board and datacenter-class cooling. That gap is exactly the labor this series is about. The PCIe card has the labor priced in; the SXM4 module has the labor as your problem to solve. Both are reasonable buys for different operators.

There's a quietly ironic twist in the 2026 numbers. The 5090 is the most-marketed card on this list, and its real street price — well above its $1,999 MSRP — is being driven by a global GDDR7 memory shortage that NVIDIA has acknowledged is unlikely to clear before 2028. Memory bandwidth is the metric that actually dominates inference; the new consumer cards are scarce precisely because the memory the marketing rarely highlights is scarce. The used datacenter cards from 2017 are unaffected by the GDDR7 crunch because they use HBM2 from a fully amortized prior generation. The arbitrage is, in the most literal possible sense, what the marketing doesn't lead with — and a reader checking the same eBay listings next month will find consumer prices have moved again and the V100 hasn't, which confirms the thesis.

This is the receipt the discourse doesn't quote. Every inference workload — image generation, LLM serving, embeddings, voice — is gated by how fast you can move weights through the chip. On that axis, the V100 in 2026 is still the right buy for any independent operator who is willing to do the work to bring one home.

What this looks like in practice

I run a FLUX-dev image generation service on a single Tesla V100 16 GB. It's wrapped as an MCP tool that any of my mesh nodes can call. The model runs as a GGUF Q4_K_S quantization through stable-diffusion.cpp; it generates a 1024×1024 image in roughly forty seconds at twenty steps.

That is not the throughput of a $20,000 H100 sitting in a colocated rack. It is exactly the throughput a small operator needs for actual personal or small-business use — image generation for marketing assets, design work, internal tooling, asset prototyping. At forty seconds per image and roughly 250 W under load, each image costs less than a tenth of a cent in electricity. Add generous overhead for cooling and idle power and you're still inside one cent per image. The commercial FLUX-dev API charges roughly five cents per image. The unit economics differ by an order of magnitude, on the same model, with the same outputs, and the card paid for itself in the first ten thousand images.

The SDXL pipeline on the same card is ~7 seconds per image at 25 steps. The same math applies. The same V100, used by the same MCP server, switches between models because the workload — generative image inference — is memory-bandwidth-bound and the V100 has plenty.

This is not exotic. This is the boring, ordinary case that the discourse insists requires enterprise gear and doesn't.

The honest caveats

This is the section where I refuse to oversell.

There are cases where you genuinely do need the expensive gear, and pretending otherwise would be the same kind of dishonest framing the fog produces in the other direction.

Frontier-scale training. If you are pretraining a 70B-parameter model from scratch, the V100 stack is not going to get you there in any reasonable wall-clock. The interconnect bandwidth, the per-card VRAM, the fp8 native tensor units — these matter at frontier scale and they matter a lot. You will need H100s, and you will need a lot of them, and the marketing for that case is approximately correct.

Sub-second latency LLM serving at scale. If you are serving an LLM to thousands of concurrent users and you need every request answered in under 200 ms, the raw FLOPS and the fp8 paths on the H100/B200 generation are doing real work. The V100 will serve a 7B model fine for an internal tool. It will not serve a 70B model to a thousand simultaneous users with low first-token latency. The right card depends on the load profile.

fp8 and the very newest model formats. V100 is FP16 native, with TF32 not supported. Models that ship in fp8 (Mistral's NeMo formats, some recent FLUX variants, several LLM checkpoints) need to be requantized to fp16 or int8 to run on Volta. This is usually a one-line model load option, but it is a real friction point that the new cards don't have.

Workloads that genuinely need more than 24 GB in one card. Some recent video-generation models and the larger uncompressed LLM checkpoints push past 24 GB. The 5090 has 32 GB, which helps. The A100 80 GB and H100 80 GB are dominant here. There are usually quantization-based workarounds, but if your workload genuinely needs the headroom, that's the case where the price gap is the price of admission.

Anything where the time-to-result of a one-shot project matters more than the cost. This is the convenience tax I'll keep coming back to throughout the series. There are real situations where renting an H100 for two hours is the correct choice — a short-horizon project, a deadline, a moment where your time is the constrained resource rather than your money. That's a legitimate trade. This series is for people who'd rather pay in hours than dollars; the people who'd rather pay in dollars than hours are making a different rational trade and I'm not going to dismiss it.

The pipeline inversion — the premium path is the narrower path

There's one more move to add to the honest caveats, because it cuts in the opposite direction from the ones above and the asymmetry matters more than the fog admits.

The hosted-API path — gpt-image-1, Imagen 4, the various Claude and ChatGPT image endpoints — is marketed as the safe default. Just hit the endpoint, pay per call, no hardware, no maintenance. That framing presents the API as a broader option than building your own substrate: more accessible, more turnkey, fewer headaches.

The framing is wrong.

A hosted image-generation API is the first ten percent of an image-production pipeline at a premium. It gives you generate. It does not give you img2img. It does not give you inpaint. It does not give you outpaint. It does not give you upscale. Each one of those is a real, routine step in any production workflow, and on the API path each one is either unavailable, charged per attempt at the same rate as the original generate, or routed to a different vendor entirely. You can buy the first step at frontier quality. You cannot buy the pipeline.

A local stack — even a humble one running on a used V100 — has all of it. Generate → img2img → inpaint → outpaint → upscale is a single workflow with no per-call billing, total control over every parameter, and the ability to chain steps without ever leaving the substrate. Real-ESRGAN x4 running on that same V100 will, in fact, take whatever the hosted API produced and finish it — the $300 used datacenter card is the finishing stage that makes the $0.19-per-call frontier output actually usable for production.

This is the inversion that the fog buries. The premium path is the narrower path. Less capability than the cheap local substrate you could have built. Marketed as the broader path because the premium-narrowness is invisible until you try to iterate.

The honest version of the API caveat is: one-shot-and-done with no GPU is a genuine but narrow niche. Hosted APIs win on text rendering inside an image, on instruction-following for unusual compositions, on certain kinds of prompt handling where current open-weights image models still lag. They do not win on iterability, on total cost over many calls, or on the ability to produce a finished asset rather than a single attempt.

I hit this directly this week. MirrorForge was offline for the board swap; I needed the hero images for this series; the local mirage MCP was unreachable. I fell back to gpt-image-1 and Imagen 4 — the same hosted APIs the convenience-tax framing in post 10 is about. Both worked. Both produced exactly the artifact they were paid to produce: a single image per call, no iteration available, no inpaint, no outpaint, no upscale. Once MirrorForge is back, the local pipeline will take those images as raw material and finish them — and from that point onward, the same prompts will run free, end-to-end, with all of the iterability the API path doesn't offer.

The fog said the API was the broader path. The receipt says it's the narrower one.

What survives the fog

Strip the five commercial interests out of the discourse and what you're left with is a much simpler picture.

• Frontier pretraining: H100-class, and the marketing is roughly correct.
• Research training at sub-3B parameter scale: a single 5090 or a pair of V100s does the job.
• Inference, by far the largest category of actual AI work: memory-bandwidth-bound, and the used datacenter cards still dominate on dollars-per-byte-per-second.

The fog tells you all three need the same hardware. The receipts say they don't.

The next post walks through the build that makes the inference and research cases concrete — the used-HEDT server, the GA-X99-UD3 that fought me for two months, and the ASUS X99-WS/IPMI graduation. It's where the platform-level story gets specific, and where the case for used HEDT stops being a $/GB-s argument and starts being a parts-list and a war story.

Next: post 03, "The used HEDT server: the silicon nobody talks about."