Substrate-independent roles

The Substrate Series · 08 — The architecture-level move

Stop thinking in machines, start thinking in roles. Nodes become interchangeable substrates for declared roles, and adding capability stops requiring a rebuild.

The confusion most builds make

Most home labs — and most small production setups — bundle two things that should be separate. They conflate the machine I bought with the role it serves. The machine has a name, a parts list, a personality, a place on the shelf. Workloads accrete onto the machine over time because it's the machine I have, and the workloads end up shaped by what the machine can do rather than the other way around.

The cost of this is invisible until you add a node. Then it shows up everywhere. The new machine needs every script that referenced the old machine's hostname updated. The old machine's services need to be split — some staying, some moving — because some of them really belong on the new hardware and some don't. The decisions you made about what runs where get re-litigated. The whole system has to be partly rebuilt to absorb the new substrate.

The fix is to invert the relationship. Roles are declared first. Substrates are interchangeable. The role doesn't know which machine is currently serving it; the role just exists, and a machine happens to be the substrate that currently runs it. When a new machine arrives, it joins the substrate pool. When an old machine retires, the role moves to a different substrate. The role is the durable abstraction; the machine is the disposable instance.

This is the architecture-level move the series has been building toward. Posts 04 through 07 covered concrete patterns — cooling, networking, management, mesh. Each of those is an instance of separation-of-concerns at the hardware level. This post is the same principle at the system level, applied to roles and substrates.

The cloud-native parallel

The pattern isn't new. Kubernetes invented an industrial-scale version of it more than a decade ago. The Kubernetes vocabulary maps directly:

• Pod = the declared workload. Stateless (mostly), reproducible, scoped to a specific job.
• Node = the substrate. A piece of hardware that has resources to spend.
• Scheduler = the thing that decides which node a pod runs on.
• Deployment = the declaration that some pod of a given type should always be running, without specifying where.

A Kubernetes operator never says "this workload runs on this physical server." They say "this Deployment should have three Pods running of this kind." The scheduler figures out where. If a node dies, the scheduler reschedules the pods onto other nodes. If you add a node, the scheduler starts using it. The operator's mental model is roles and capacity, not machines and assignments.

Operator-scale labs don't need the full Kubernetes machinery. The container orchestration overhead is wildly out of proportion to a roughly seven-node mesh. But the pattern — declare the role, let the substrate be interchangeable, let new substrates slot in without rebuilding — is exactly what makes a mesh generative rather than rigid.

What this post describes is that pattern at hobbyist scale. No Kubernetes, no Ansible, no Terraform. A handful of TOML role contracts, a directory tree, a few systemd units, and a discipline about what knows about what.

The role-template pattern

The skeleton looks like this:

~/openclaw-cognitive/nodes/        # source of truth — one TOML contract per node
  legion.toml
  mirrorforge.toml
  serve-01.toml      # asus, rescoped to the serving tier
  orin.toml
  pi5.toml
  picontrol.toml
  fab.toml
  thundercompute.toml
        │
        │  scripts/project_nodes.py
        ▼
~/.claude/registry/nodes/*.yaml    # projection — what routing + agents read

There are two layers here, and the split is deliberate. The source of truth is a directory of TOML contracts — one per node — that I hand-edit. A projection script flattens those into the YAML registry that the routing layer and agents actually read at runtime. You edit the contract, never the projection; the projection is regenerated. (This is the same author-here / serve-there seam the rest of the series keeps using — you'll see it again as the SERVE tier below.)

Each contract is a declaration about what role(s) and tier the node serves, what hardware it has, what always-on services it's responsible for, and what context an instance should load when it comes up there. A concrete example, lightly trimmed:

tier      = ["CAPTAIN", "DEV"]   # see the tier table below
node      = "mirrorforge"
lan       = "192.168.50.105"
tailscale = "100.88.204.19"

[hardware]
cpu     = "Xeon E5-2690 v4 (14c/28t)"
ram_gb  = 32
gpu     = ["Tesla V100 16GB PCIe", "Tesla V100 16GB PCIe"]
vram_gb = 32
capabilities = ["cuda", "hbm2", "fp16", "fp32", "large-bar"]

# A "contract" service: something always-up that the watchdog must keep alive.
# health exits 0 when the service is well; restart is what the reconciler runs
# when it isn't; critical decides whether a failure pages a human (post 07).
[[services]]
name       = "mirage-mcp"            # ComfyUI + FLUX image generation, V100-pinned
artifact   = "imagegen-mcp@comfyui-flux"
port       = 8190
health     = "curl -fsS http://localhost:8790/health"
public_url = "https://mirage.example.com"
critical   = true

The contract is a declaration on two axes at once. What the node is: MirrorForge is a CAPTAIN/DEV substrate with two V100s and these capabilities. What the node is responsible for keeping up: the mirage-mcp service, with a health check and a restart command that the post-07 reconciler reads directly. The health/restart/critical triple is the exact seam that turns "self-diagnosing" into "self-healing" — the role declaration and the repair contract are the same file.

The corresponding contract for picontrol looks completely different:

tier      = "GATEWAY"
node      = "picontrol"
tailscale = "100.101.17.42"

[hardware]
cpu    = "Raspberry Pi Zero 2 W"
ram_gb = 1

[gateway]
advertise_route = "192.168.50.0/24"   # standby; pi5 is primary
role            = "redundant subnet router + ntfy hub + pi5 dead-man"

[[node_services]]
name     = "ntfy"                     # the fallback alert channel — keep it alive
health   = "systemctl is-active --quiet ntfy"
restart  = "sudo systemctl restart ntfy"
critical = true

Different tier, different hardware, different responsibility, but the same shape of file. A reader of either contract can tell, at a glance, what the node is for, what it must keep running, and how to reach it. (Note [[node_services]] vs [[services]]: the first is always-up local infra the watchdog keeps alive; the second is a public frozen artifact on the serving tier. Same fields, different blast radius.)

The convention extends naturally to virtual nodes — the Thundercompute node I'll discuss below — which have a contract in the same directory and follow the same schema, with a node_type = "virtual-mcp" field that flags them as on-demand rather than always-on.

This is the entire infrastructure-as-code surface, scaled to operator-level needs. There is no Kubernetes. There is no Ansible. There is a directory of TOML contracts that describe what each node is, should be doing, and should keep alive — and a projection script that flattens them into the registry tools and agents read. That's it.

The tier model: roles, graded by how much they're allowed to change

Once you have roles, a second classification falls out that turns out to matter more than the role names themselves: how stable is the thing that lives here supposed to be? A research box and a public API endpoint are both "nodes with roles," but they have opposite relationships to change — you want to thrash on one and you must never thrash on the other. Encoding that as a tier on each node is what keeps the two from contaminating each other.

Two rules carry almost all the weight. Always-on infrastructure — monitoring, storage, routing, notifications, canonical state — never lives on a DEV or compute node, because those reboot and thrash. Dev work never runs on a SERVE node, because the whole point of SERVE is that it doesn't move under you. A node can hold more than one tier (Legion is CAPTAIN and DEV; pi5 is WATCHDOG and GATEWAY), but the tiers it holds tell you immediately what you're allowed to do to it. The tier is the role concept applied to mutability instead of capability — and it's the thing that lets a single small fleet host both a research playground and a public product without the first taking down the second.

Role scope as agent security boundary

The same per-node role declaration that the contract above shows is also the security boundary for an agent operating on that node. Post 07 introduced the three-layer containment model for the agent-operable mesh; this is where the second layer — per-node role scope, enforced by the routing layer — lives in practice. The context: and capabilities: fields don't just describe what the node can do; they bound what an agent dispatched onto the node is permitted to do, because the same routing layer that loads context against those fields also blocks reads and writes outside them. Scope is intrinsic to the routing path, not a separate guard at the perimeter. A rogue or prompt-injected agent inherits the node's declared scope and cannot reach beyond it — bounded blast radius, not "can't happen," but bounded by a layer that was already there for routing reasons. The security property falls out of the architecture; you don't have to bolt it on.

Why this beats coordinator-worker agent frameworks

A common alternative people reach for is a coordinator-worker agent framework — one central process that maintains state, delegates tasks to workers, and aggregates results. Frameworks like AutoGen, CrewAI, and LangGraph all have this shape at their core.

The substrate-independent roles pattern is not that. It is a peer architecture, not a hub architecture. There is no central coordinator. Each node knows its own role, knows the shape of the registry, and can independently decide whether a piece of work is for it or should be routed elsewhere.

The differences that matter in practice:

No single point of failure. A coordinator-worker framework has a coordinator. If the coordinator dies, the workers don't know what to do. The mesh has no coordinator. If any node dies, the remaining nodes still know who they are and what their roles are.

Peer autonomy. Each node is fully autonomous. Legion can decide to take on a piece of work without asking permission. Picontrol can issue an alert without coordinating with anyone. The interactions between nodes are publish-subscribe (ntfy, registry updates, mesh-monitor heartbeats) rather than request-response through a central process.

Cross-node delegation, on equal footing. When MirrorForge has a workload that would run better on Legion, it doesn't ask a coordinator to schedule it. It talks directly to Legion. The mesh's cross-node push primitive — exposed in the orchestration layer — handles this: any node can push a delegated task to any other node, and the receiving node's mesh-watcher daemon spawns a fresh Claude Code session to handle it. The delegation is peer-to-peer.

Easier to reason about. A coordinator-worker setup has emergent behavior — the queue depth, the coordinator's scheduling decisions, the worker pool state. A peer mesh has each peer's local state. Debugging is a matter of asking "what does this specific node think is going on?" rather than "what does the global state of the coordinator look like?"

The catch: peer architectures require every node to know the shape of the registry. That's what the YAML files solve. Every node reads the same registry directory; every node has the same view of who exists and what they do. No central coordinator, but a shared, replicated, declarative description of the substrate pool.

The MirrorEthic role inventory

Here's the actual role map for the lab as it stands, with role declarations on the left, the tier the work runs under in parentheses, and the substrates currently serving each role on the right. It has shifted noticeably from where it sat a couple of months ago — which is the point of the whole post: the roles moved, the architecture above them didn't.

A few things to notice.

Most roles have a fallback declared. When the primary is down or busy, the fallback can serve the role with degraded performance. The fallback isn't a hot-standby in the failover sense; it's an explicit acknowledgement of where the work can run if it has to.

Most substrates serve multiple roles, but always within a tier discipline. Legion is the CAPTAIN (the four MCP servers) and a DEV box (dev workstation, training-primary, inference-primary-for-new-models). MirrorForge is training-worker, Volta-compatible inference, and the image-generation substrate. pi5 is watchdog, storage, and gateway. Roles share substrates without conflict because the role concept is declared scope, not machine reservation — and the tier on each role keeps the thrashy work and the always-on work from colliding even when they share silicon.

Some roles have no fallback, and naming that honestly is the point. The fab Pi is the only box wired to the printers and the QA arm; the Orin is the only Jetson; the four CAPTAIN MCP servers live only on Legion. Those are real single points of failure to track, not things to paper over with a fake fallback. (The one I did fix recently: the watchdog and the subnet router used to be single-homed; they now have picontrol as a genuine independent backup, which is why they show a fallback where most infra roles wouldn't.)

The map moved out from under several rows since the last revision. ASUS used to serve visual-perception; that role was retired and ASUS was rescoped to the SERVE tier (below). pi5 used to be a Pi Zero doing voice and camera work with a Hailo NPU; it's now a Raspberry Pi 5 doing watchdog/storage/gateway, and the Hailo came out. A whole fabrication role appeared. None of those changes touched the layer above the registry — they were edits to which substrate serves which role, which is exactly the property the architecture is supposed to buy.

The substrate pool is heterogeneous on purpose. Different substrates have different capability profiles, and the role declarations let the heterogeneity show through. The 5090 isn't trying to be a V100; the V100 isn't trying to be a 5090. Each substrate is in the role its capabilities make it best for, and the role declarations are explicit about which.

The rescope test: when a role leaves a substrate

The flashy version of the architecture is adding hardware. The quieter, more telling version is what happens when you retire a role and repurpose the box underneath it — because that's the operation a machine-centric setup handles worst.

ASUS is the worked example. It started life in the role inventory as the visual-perception substrate — an RTX 4070 laptop running a gRPC vision service on :50051. That role stopped earning its keep, so I did two things that a bundled setup makes hard and a roles-and-substrates setup makes trivial: I retired the role (stopped and disabled the gRPC service, dropped it from the registry projection), and I rescoped the substrate to a completely different tier. ASUS is now serve-01, a SERVE node: it hosts the frozen, public t3atlas model endpoint behind api.t3atlas.dev, and nothing dev runs on it anymore.

That rescope is where the SERVE tier's discipline earns itself. A SERVE node is never edited live — when the served artifact needs a change, I fix it on a DEV box and re-push it through a versioned deploy with a health-gated rollback, so a bad deploy rolls back instead of taking the public endpoint down. The cloudflared tunnel that fronts the endpoint moved onto ASUS itself (copied creds, no browser login), so api.t3atlas.dev no longer transits Legion and survives a Legion reboot. The box didn't change; its tier changed, and the rules that apply to it changed with the tier. Same machine, retired from one role, reborn in another, with a different contract about what I'm allowed to do to it.

The machine-centric version of this would have been a small project: untangle the vision service's dependencies, decide what the laptop is "for" now, rewire the scripts that pointed at it. The roles-and-substrates version was a contract edit and a service swap. The substrate was always disposable; only the role was ever real.

The "new hardware just slots in" test

The honest test of whether a roles-vs-substrates architecture is real is what happens when new hardware arrives.

A bundled, machine-centric setup absorbs new hardware with friction. You have to decide what's moving to it, update scripts that referenced the old machine, possibly migrate services, and figure out how the new machine fits into your operational habits. The first week of a new machine is a busy week.

A roles-and-substrates setup absorbs new hardware as a config change. The new machine gets a YAML file declaring its hardware and capabilities. The existing roles get reviewed for whether the new substrate serves any of them better than the current one. The mesh registry replicates the new node's metadata. Done.

The X99-WS/IPMI motherboard swap on MirrorForge was the most recent test case, and it passed exactly as the model predicts. The board swap was mechanical — same socket, same CPU, same RAM, same boot drive, same V100s. From the role-and-substrate perspective, nothing changed. MirrorForge came back serving the same roles, with the same capabilities, on the same network endpoints. The mesh registry didn't even need to update. The only differences a careful reader would notice are a couple of new capability flags ("native-x16-x16", "ipmi-asmb8-ikvm") and some updated context-loading hints about the IPMI surface. The box was down for the swap — and because always-on infra had already been moved off MirrorForge onto pi5 (the watchdog/storage/gateway split from post 07), the rest of the mesh didn't notice the outage. The board was replaced underneath; the role above didn't move.

Consider what happens when a DGX Spark arrives on the secondary market. NVIDIA's $3,999 desktop AI workstation, shipped October 2025, built around a single GB10 Grace Blackwell Superchip — one 20-core Arm CPU and one Blackwell GPU multi-die — with 128 GB of unified LPDDR5X memory and a memory bandwidth of ~273 GB/s, FP4-native tensor cores, 1 PFLOP of sparse compute, packaged as a small desk box. The marketing position is "your next AI workstation"; the implicit invitation is to slot it into the training-primary role and call it an upgrade.

The role layer flatly refuses to do that. Look at the capability profile and the answer falls out.

A DGX Spark has 128 GB of unified memory — enormous for the price — but its memory bandwidth (~273 GB/s) is lower than the $285 used V100's 900 GB/s, and a third of what the laptop RTX 5090 in Legion delivers (24 GB of GDDR7 at roughly 900 GB/s on its 256-bit bus — the mobile part, not the 1,792 GB/s desktop card it shares a name with). The bandwidth-dominates-inference argument from post 02 says, unambiguously, do not put this box in training-primary or fast-inference roles. The Spark's actual niche, expressed as a YAML declaration:

node = "dgx-spark"
role = ["capacity-host"]

[hardware]
cpu                   = "GB10 Grace Blackwell Superchip (20c Arm + Blackwell GPU)"
unified_memory_gb     = 128
memory_bandwidth_gb_s = 273
capabilities          = ["fp4-native", "large-unified-memory", "low-latency-not-required"]

[routing_policy]
prefer_when = [
  "model_size_gb > 32",
  "model_fits_nowhere_else_in_mesh",
  "latency_not_critical",
]

That role is narrow on purpose. It says: route work here when a model is too big to fit on any other substrate in the mesh, and when you can tolerate slow per-token throughput because the alternative is "doesn't fit at all." That's a real role — there are 200B-parameter FP4 checkpoints right now that don't fit anywhere else in this lab — but it's a narrow conditional role, not the training-primary slot the marketing implies.

This is the architecture protecting you from a bad purchase instinct. The naive read of "NVIDIA flagship, just released, slots into training-primary" is wrong, and the role-profiling discipline is what catches it. The whole series is about seeing past marketing; having the role layer correctly file an over-marketed box into a narrow conditional niche instead of the prestige role its branding implies is the thesis eating its own dog food.

It also reframes the buy/no-buy decision. The Spark at $3,999 is paying NVIDIA for integration and a small power envelope, much the way the A100 80 GB PCIe at $20,000 charges you 3× over the SXM4 module for solving the integration problem (post 02). That's the convenience tax — the same tax the capstone (post 10) closes on. If the capacity role is the one you actually need, building a comparable capacity-tier from used parts — high-VRAM accumulator boxes with sufficient unified memory — costs meaningfully less for the same role. The role layer makes that comparison visible, and the marketing tier loses on it.

The property the role layer buys you isn't that the Spark, or any new substrate, automatically becomes training-primary. It's that the substrate pool generates the right role for any new substrate based on its capability profile, rather than letting marketing assign roles by prestige.

The virtual-substrate proof

The Thundercompute integration is the cleanest single proof that the architecture is real.

Thundercompute provides on-demand H100, A100, L40S, RTX 4090, and T4 instances via an MCP server endpoint. It's not a physical box in my home. It has no LAN address. It has no Tailscale identity. It has no power supply I'm responsible for.

It is, nevertheless, a fully-fledged node in the MirrorEthic mesh. The contract:

node      = "thundercompute"
node_type = "virtual-mcp"
role      = ["burst-compute", "training-primary-when-mesh-saturated"]

[hardware]
gpu_pool     = ["H100 80 GB", "A100 80 GB", "L40S", "RTX 4090", "T4"]
capabilities = ["cuda", "hbm3", "fp8", "tensor-cores-4th-gen"]

mcp_url = "https://api.thundercompute.com:8443/mcp"

[routing_policy]
prefer_when = ["vram_required_gb > 24", "local_mesh_busy"]

The instance-delegate MCP server reads this declaration the same way it reads any other node's. The knowledge_gap routing logic considers Thundercompute as a candidate substrate alongside the local nodes. When a workload's VRAM requirement exceeds anything in the local mesh, Thundercompute gets routed to. When the local mesh has capacity, it doesn't.

The Thundercompute instance is a real piece of hardware running real silicon — it's just somebody else's hardware, somewhere else, with a different billing model. From the role-and-substrate perspective, that's the entire difference. The role declaration doesn't care whether the substrate is on my shelf or in a datacenter on the other side of the country. The role says "I need a substrate with these capabilities and a path to reach it." Thundercompute fits. The mesh uses it.

This is the move that turns a local lab into a hybrid mesh that's as many nodes as the work needs, without changing anything structural. The local substrates are persistent. The remote substrates are on-demand. The role abstraction makes them interchangeable from the perspective of the workload.

What this buys you

When roles are first-class declarations and substrates are interchangeable instances, three properties emerge.

Adding capability is a config change, not a rebuild. A new node is a new YAML file. The existing roles get reviewed for whether the new substrate serves them better; if so, the role moves; if not, the new node enters the pool with a less-busy schedule. There is no migration project.

The substrate pool is heterogeneous on purpose. Different substrates have different capability profiles. The role declarations let you exploit that — assigning each role to the substrate it's best on — instead of papering over the differences with a one-size-fits-all assumption.

Workloads route to the right substrate automatically. When you've declared the roles and capabilities, the routing decision is mostly mechanical. Post 09 covers what that looks like in code.

The cost of all of this is a directory of YAML files and the discipline to keep them current. The benefit is a mesh that grows under work — that gets more capable every time you add a substrate, without ever needing to rebuild the layer above.

That's the architectural difference between a homelab and a system. A homelab is a collection of machines you happen to own. A system is a substrate pool serving declared roles, and the substrates are the disposable thing.

Next: post 09, "Substrate routing: workload placement across the mesh" — the generic routing primitive that places most work automatically, the circuit-breaker pattern at hobbyist scale, and the YAGNI call that kept the bespoke image-gen router on the drawing board.