Substrate routing: workload placement across the mesh

The Substrate Series · 09 — Advanced pattern

Route each workload to the substrate that runs it best, gate on health, and the whole mesh feels like one seamless machine you can forget about.

The motivating problem

In May 2026 I wired up an image-generation MCP server — mirage-imagegen — fronting the FLUX family of models. The first deployment ran everything on MirrorForge's V100s. Standard FLUX-dev image generation ran at a respectable ~40 seconds per 1024×1024 image. Then I tried FLUX-fill, the inpainting model in the same family, and the same V100 took three times as long.

The reason traces to model format and silicon-generation interaction. FLUX-dev ships in a mature Q4_K_S GGUF quantization that runs efficiently on Volta-class silicon through stable-diffusion.cpp. FLUX-fill quantizations exist but were less mature at the time I set up the pipeline; I deployed it on the full fp16 weights, which the V100 supports but doesn't accelerate well on its first-generation tensor cores for this specific operator mix. The Blackwell 5090 in Legion runs the same workload roughly 3× faster. The measured gap is consistent with the tensor-core-generation hypothesis, but I want to phrase it as an observed result with a plausible mechanism, not a definitive causal claim — a careful reader who's run quantized FLUX-fill in ComfyUI will know that some of the gap is fixable by changing the pipeline rather than the substrate.

The naive solutions:

• Run everything on Legion. But Legion is the dev workstation, the primary inference node for newer models, and the host for all the foreground work I'm actually doing. Pinning long image-generation requests to Legion means user-facing latency spikes during interactive work.
• Run everything on MirrorForge. But FLUX-fill is slow there, slow enough that the user-experience cost is real.
• Spawn a Legion process per FLUX-fill call. Possible, but it leaks resource contention into the dev workspace and there's no good fallback if Legion is in the middle of a training run.

What's actually needed is a router that knows about both substrates, knows about their current health, and can make a per-request decision. FLUX-dev requests stay on MirrorForge, FLUX-fill requests route to Legion when Legion is healthy, and if Legion is busy or down the requests fall back to MirrorForge with a degraded latency expectation but no failure.

That's substrate routing. Same idea as cloud load balancing, scaled to a 2–5 node mesh, written in about 80 lines of Python.

Two tiers of routing — the generic primitive, and the specialized routers on top

Before the FastAPI service that runs mirage-imagegen, there's a thing further down the stack that needs naming: routing across the substrate pool is itself a first-class primitive of the mesh's orchestration layer. The mirage router isn't the general mechanism — it's a specialized router that sits on top of the general mechanism. Getting that distinction right is the difference between "I built one image-gen service" and "the mesh has substrate routing as a property."

The generic routing primitive is exposed as a small set of tools on the orchestration layer's MCP surface:

• knowledge_gap(query, gap_description) — accepts a query and a description of what's missing; infers required_capabilities (CUDA, large-BAR, etc.) and vram_required_gb from the description (or accepts them explicitly), then dispatches to the best-matching node in the substrate pool, including Thundercompute as a fallback when nothing local fits. This is the agent dispatch path: the agent expresses intent (need); the primitive places by capability and availability.

• instance_route_recommend(required_capabilities, vram_required_gb, task_duration_estimate_sec) — on the Instance-Delegate MCP, returns a ranked candidate list with reasons and (for virtual nodes) the MCP URL to dispatch to. Capability-and-availability matched against the registry + live pool_instances state.

• mesh_send(message, routing: {required_codebase, required_capabilities, preferred_node}, action_type: invoke|delegate) — carries explicit routing intent alongside payload, so a sender can describe the substrate they need without naming a specific node.

• pool_instances — enumerates live instances with capabilities, hardware, current tasks. The health-and-capability input to placement.

Two properties to call out.

The contract is the routing surface. An agent (or a human; same path) doesn't name a destination, it names a need. "CUDA + 24 GB VRAM + this codebase + a path that can reach this MCP server" is a contract; the primitive matches that contract against the substrate pool and dispatches. This is what makes the mesh agent-operable in the sense post 07 introduced: the agent describes intent, the orchestration layer places it, the placement is bounded by the role scope of the destination (post 08). Intent-stated routing IS the agent dispatch mechanism, and it's the same mechanism for human dispatch.

This is also where the security model lands mechanically. The same required_capabilities contract that places the work also bounds what the agent can ask for: a request for capabilities the agent's identity isn't authorized to dispatch fails at the routing call, before any substrate sees it. The three-layer containment model from post 07 isn't aspirational; one of its three layers is exactly this contract.

The generic primitive handles most routing across the mesh without bespoke code. Agent-dispatch, embedding generation, ad-hoc inference, broadcast queries, capability-matched delegation — all go through the orchestration layer's tools and out to whichever substrate the registry matches best.

The specialized router would be an optimization on top of that — for a workload whose placement is too subtle for a generic capability-match. Image generation is the obvious candidate: you might want per-request health gating, sticky-fallback hysteresis, latency-aware queueing, and a router that knows FLUX-dev's optimal substrate is different from FLUX-fill's. None of that fits in a generic capability-match.

Here's the honest part, and it's the most useful thing in this post. I designed that specialized router for image generation and then didn't build it. What actually runs today is deliberately simpler: mirage-imagegen is a single ComfyUI + FLUX service (mirage-mcp, public at mirage.example.com) pinned to MirrorForge's V100s, and the handful of requests that would genuinely benefit from Blackwell get dispatched through the generic primitive or, honestly, by hand. Pinning-plus-generic beat building-a-bespoke-router, and it wasn't close.

That's the YAGNI call this whole series keeps making, applied to my own infrastructure: the specialized router is a real pattern you'll reach for when a workload's traffic and placement-subtlety actually demand it, but most workloads — including this one, so far — are served fine by the generic primitive plus a sensible default substrate. I'm going to walk through the full specialized-router design anyway, because the patterns inside it (circuit breaker, sticky fallback, health-gated fallback, scoped binding) are exactly what you'll need when you cross that threshold — and because the most important design skill is recognizing the threshold you haven't crossed yet. Read what follows as "the design I scoped, the one you'd build at the next tier of traffic," not "a service humming away in my rack right now." At the end I'll come back to why the line between those two is the point.

What the specialized router would do

The router sits in front of the substrates that can serve a given workload. For image generation that means a small FastAPI service on the management plane — the natural place for routing decisions, because it's reachable from any client and has SSH-key access to both compute nodes. In this lab that's a watcher-class node (the always-on watcher box from post 07, not a thrashing compute box); the point is that the router lives on the stable management tier, wherever that happens to be, never on the substrates it's routing between.

The service would have three responsibilities:

Receive the request. A client calls the MCP endpoint with a workload and any parameters — a FLUX-dev prompt, a FLUX-fill prompt plus an input image, a SDXL prompt.

Pick a substrate. Look at the workload type, look at the current health of the candidate substrates, pick the right one.

Forward the request. Send the request to the chosen substrate's worker process. Stream the response back to the client. Log the routing decision and the latency.

The routing logic — the interesting part — is a small priority-ordered table. For mirage-imagegen:

workload: flux-dev
  primary: MirrorForge (Volta works well on Q4_K_S GGUF)
  fallback: Legion (works fine but uses prime dev silicon)

workload: flux-fill
  primary: Legion (Blackwell accelerates fp16 fill ops)
  fallback: MirrorForge (slow but functional)

workload: sdxl
  primary: MirrorForge (~7s at 25 steps, V100 is fine here)
  fallback: Legion

That's the entire policy. Three workloads, two substrates each, primary-and-fallback for each. The "interesting" part of substrate routing is mostly correctly characterizing the workloads. The mechanics are a few if statements.

Health-aware routing

The router can't statically assign workloads to substrates because the substrates aren't always healthy. Legion is the dev workstation; it can be in the middle of a heavy editor session, a benchmark, a training run. Asking it to handle a slow inference request mid-training is the wrong call. So primary doesn't mean always-primary — it means primary when healthy, falls back when not.

The health check is a /status endpoint on each substrate's worker process. The endpoint reports:

• gpu_util_pct (from nvidia-smi --query-gpu=utilization.gpu)
• vram_used_gb and vram_total_gb
• gpu_temp_c
• running_jobs (a count of currently-active inference processes)
• last_request_ts (when the worker last accepted a request)

The router polls /status on each substrate every 5 seconds, caches the result, and uses it in routing decisions. A substrate counts as healthy for a workload when:

• gpu_util_pct < 70 (substrate isn't already saturated)
• vram_free_gb >= workload_vram_estimate (there's room)
• gpu_temp_c < 80 (substrate isn't thermally stressed)
• running_jobs < max_concurrent_jobs_for_substrate (the queue isn't full)

If the primary substrate isn't healthy, the request routes to the fallback. If neither is healthy, the request queues for a short interval before failing with a clear "all substrates busy" error rather than silently waiting forever.

This is what most people call a circuit breaker in cloud-native terminology. The router protects the primary from being overwhelmed during heavy load by routing around it, and protects clients from hanging indefinitely by failing fast when nothing's available.

The fallback state machine

The state of the fallback decision is small enough to draw.

[primary healthy] ─request→ primary
[primary degraded, fallback healthy] ─request→ fallback (log: "fell back")
[both degraded, queue depth < N] ─request→ queue (retry primary in 5s)
[both degraded, queue depth ≥ N] ─request→ 503 "all substrates busy"
[primary recovers] ─next request→ primary (sticky preference)

Two notes on the design choices.

Sticky to primary on recovery. When the primary comes back to healthy, the router immediately resumes routing to it for new requests. This is a small but important property — without it, the router would oscillate between primary and fallback as the primary's load fluctuates around the threshold, and the cache effects of switching substrates (each one warming up the model in VRAM) would cost real latency.

Queue depth cap, not unbounded queueing. Unbounded queueing is the cloud-native antipattern that causes incidents. The router's queue caps at N (currently 4) requests; beyond that it fails fast. Clients see a clear "busy" error and can retry with backoff, rather than waiting for an arbitrary length of time on a request that may never complete.

The state machine is a 30-line Python file. The whole router, including the FastAPI scaffolding and the systemd unit, is roughly 80 lines plus configuration.

Network scoping: bind to the dedicated link only

The router has to be reachable by the clients that call it, but it should not be reachable from the home LAN or the public internet. Two different scopes:

The management plane's tailnet identity is the right reachability for control requests — me reaching the router from a phone or a laptop, listing what's available, checking status, kicking a substrate. This is the user-facing scope.

The point-to-point 10 GbE subnet between MirrorForge and Legion is the right scope for data requests — the bulky inference traffic that benefits from the bandwidth. The router service would bind to both network interfaces, but the worker processes bind only to the 10 GbE subnet, so inference data never flows over the regular LAN.

The firewall configuration enforces this: the management-plane router accepts HTTP only on its overlay interface; the MirrorForge and Legion worker processes accept HTTP only on their 10.40.10.0/30 10 GbE point-to-point interfaces.

The consequence: data flows over fast dedicated wiring, control flows over the management overlay, and the regular LAN is never involved in either. This is the same separation-of-concerns logic from post 05 applied to the services that ride on the network, not just the network itself.

What this implements, in cloud-pattern vocabulary

A reader from a cloud-native background can recognize each piece:

• Circuit breaker. The fallback state machine. When the primary is unhealthy, requests route around it for a defined recovery window.
• Health-aware load balancing. The /status polling. Requests route to whichever substrate is healthiest, not just whichever is "primary" by default.
• Sticky priority routing. The "sticky to primary on recovery" property. New requests prefer the primary when it's healthy, with hysteresis to prevent oscillation.
• Scoped service discovery. The dual-network binding. Control surfaces are reachable from one scope, data surfaces from another, and the scopes don't bleed into each other.

These are production patterns. They're patterns that companies pay six-figure sums for in service-mesh products (Istio, Linkerd, Envoy-based custom routers). The hobbyist-scale version is 80 lines of Python, runs on a $40-class management board, and serves the same functional role — when you need it. The reason I keep hedging that clause is the honest center of this post: I have the design, I have the patterns, and I still haven't had a workload push me across the line where writing those 80 lines beats pinning a substrate. That's a feature of the scale, not a failure of nerve.

The reason the hobbyist version is so small is that it has a closed world. The router would know about exactly two substrates and three workloads. There's no service discovery beyond reading the local registry; there's no autoscaling because there's nothing to scale; there's no complex routing policy because three workloads and two substrates is a small policy space. The production complexity in real service meshes comes from handling thousands of services and unknown traffic patterns. A handful-of-nodes lab doesn't have either of those problems — which, as it turns out, is also exactly why the bespoke router stayed on the drawing board.

The "forget about it" test

The way to know whether substrate routing is working is whether you can forget about it.

The naive setup before any consolidation: every time I called image generation from my laptop, I had to think about which workload I was running and which substrate was best. FLUX-fill calls went to one endpoint, FLUX-dev calls to another, and if a node was busy I felt it.

What I have now: a single mirage-mcp endpoint pinned to the V100s. Every call goes to one place; I don't think about it, I don't remember which model lives where. The "forget about it" property is already achieved — and notice it was achieved by pinning to a well-characterized default substrate, not by building the dynamic router. That's the whole lesson. The router's job is to make the substrate choice disappear from your conscious workflow; for image generation, a sensible pin made the choice disappear without a single line of routing code. The dynamic router earns its keep only when one pinned substrate can no longer serve the traffic well — when FLUX-fill volume grows enough that eating the V100's slower fill path actually hurts, or when the V100 is contended enough that falling back to Blackwell stops being an edge case. Then you build the state machine above. Not before.

That's the property to optimize for, and the discipline that comes with it: the architecture should fade out of the conscious workflow once it's working, and you should reach for the cheapest mechanism that makes it fade. If a pin does it, pin. If you're still routing in your head, then the router earns its complexity.

The corollary, which is also the operational reality, is that the router is the right thing to test and monitor. If a substrate is silently degrading, the router's logs will show fallbacks happening more frequently. If a workload's characterization is wrong, the router's latency distribution will skew. The router is the choke point through which all the policy flows, which makes it the natural place to observe whether the policy is right.

The real multi-workload routing map

For full transparency, here is the actual routing layout across the mesh as it stands, split by which mechanism handles each workload:

Image generation had the most operational pain (the V100/Blackwell asymmetry on FLUX-fill), so it's the workload I scoped a specialized router for — and then didn't ship, because pinning to the V100 plus the generic primitive turned out to cover it. LLM inference and distributed training will likely get specialized routers as their traffic grows; for now, the generic primitive plus the operator's head is enough. Notice how many rows resolve to "pinned" or "in my head": that's not a gap, it's the system telling you those workloads haven't earned a router yet.

Two things to take from the table. First, all of the routing that is actually automated across the mesh runs on the generic primitive — there is currently zero bespoke routing code in production, and the mesh works fine. Second, the path from "in my head" to "in code" is the same for every workload, and it has an honest first step most write-ups skip: identify the pain, check whether a pin or the generic primitive already kills it, and only if not, write the workload-aware policy, wrap the generic primitive in a small FastAPI service, declare the policy in a contract, and watch the latency distribution. Image generation is the worked example of the design; the fact that it still doesn't run that design in production is the worked example of the discipline.

What this buys you

Mesh routing is a first-class capability, not network plumbing.

Once it's in place:

• The mesh behaves like one machine from the client's perspective. Clients call one endpoint; the right substrate serves the request.
• Substrate-specific quirks (which model format runs well on which silicon) get absorbed into the routing policy and stop leaking into client code.
• Adding capacity is a config change. New substrate appears in the routing table; new workloads route to it where appropriate. No client code changes.
• Substrate failures become routing events, not workload failures. Clients keep working at degraded performance instead of hitting hard errors.

The generic primitive delivers most of that today: capability-matched placement, Thundercompute as an automatic overflow when local VRAM runs out, and a single endpoint per workload so clients never name a node. The richer property cloud operators call graceful degradation — full performance when healthy, automatic health-gated fallback when not — is what the specialized router adds on top, for the one or two workloads that eventually earn it.

The whole story is about a generic routing primitive you already have, a handful of sensible substrate pins, a roughly-80-line FastAPI router you build only when a pin stops being enough, and a discipline about which substrates are best for which workloads. For most of the mesh, the first two are enough — and knowing that is the actual skill.

Next: post 10, "Velocity of capability acquisition" — the capstone. Why the real moat is how fast new capability slots into a system that already knows how to use it.