MirrorEthic

MirrorEthic / Writing / The Substrate Series

The mesh that diagnoses itself

Garret Sutherland · The Substrate Series

A quiet corner of the lab under one warm lamp — a tower, a small switch, single-board machines, each with a status light, none of them special: any node can check on any other.

The Substrate Series · 07 — Concrete pattern

Build so any node can diagnose any other node, and your system survives failures that take down hub-and-spoke setups.

The hub-and-spoke problem nobody warns you about

Most home labs — and a lot of small-team production setups — are hub-and-spoke. There's a primary machine you SSH into, and from there you reach the others. The primary machine runs your tmux sessions, hosts your code, and is the thing your phone or your laptop talks to when you want to do something. The other machines are spokes; they exist to serve work the hub dispatches.

This works fine until the hub is what fails. When the hub goes down, you lose:

The "obvious" fixes don't really solve it. A second hub introduces coordination problems. A management VPS gives you one more box to maintain. An IPMI on the hub helps but only at the hardware level. The structural issue is that the coupling between you-the-operator and your-system goes through one chokepoint, and the chokepoint has the same failure rate as any other component.

A mesh dissolves that problem. Any node can reach any other node, any node can be the entry point, any node can observe any other node's state. The operator's identity travels with them, not with a specific box. When a node fails, the others notice and the operator notices through whichever of the surviving nodes they happen to talk to first.

This post is about how to build that mesh out of the substrate the rest of the series has already covered.

Tailscale as the membership layer

The single biggest leverage point in mesh-style management is identity. The question "is this connection allowed to do something privileged?" needs a stable answer that doesn't depend on what IP address things have, what network they're on, or what NAT they're behind.

Tailscale solves this. Under the hood it's WireGuard with a control plane on top that handles key exchange, NAT traversal, and routing. To you it's "every machine in my account can talk to every other machine in my account, by name, with the same identity, from anywhere." The machines I log into are members of the same tailnet:

The operational consequence: I can SSH to mirrorforge by name from my laptop in a coffee shop, my phone with Termius, a borrowed machine at a co-working space, or — and this matters more than it should — from any other node in the mesh. There is no central VPN concentrator. There is no firewall to punch through. There is no DDNS to keep current. The mesh exists as long as any two endpoints have any path to the Tailscale control plane.

The mesh takes this one step further by advertising a subnet route (192.168.50.0/24) into the tailnet — and advertising it from two nodes (pi5 primary, picontrol standby) so the route itself has no single point of failure. The entire home LAN becomes reachable through whichever router is up, so even legacy devices that aren't running Tailscale themselves — the GS316 switch, the GL-MT1800 router, the UPS network management card — are reachable as long as either subnet router is on the LAN. (One sharp edge worth the warning: a node that is physically on the advertised LAN must never --accept-routes, or it starts routing its own LAN neighbours through the tunnel and makes itself unreachable. Cost me an afternoon to diagnose the first time.)

The cost: $0 for personal Tailscale use — six users, unlimited user devices on the personal plan as of the April 2026 pricing overhaul (the old 100-device cap is gone; tagged infrastructure resources are capped separately around 50).

The two-layer security model

The fact that the mesh exists doesn't mean everything is wide open. Security on the mesh is a two-layer model:

Layer 1: the LAN perimeter. The home network is behind NAT and a stock consumer firewall. Nothing from the public internet routes directly to any mesh node. The traditional security boundary still applies: if your home network's perimeter is compromised, the LAN is compromised, just like any other home network.

Layer 2: Tailscale identity. Every connection over Tailscale carries a WireGuard identity tied to a specific device. SSH on every node is configured to accept connections only from Tailscale-routed source addresses (or from the LAN, depending on the node). Even on the LAN, SSH uses key auth, not passwords. A LAN-side attacker who got onto the home network couldn't impersonate a Tailscale identity without the corresponding private key on a Tailscale-enrolled device.

These two layers fail independently. Compromising the LAN doesn't compromise Tailscale identity (the WireGuard keys aren't on the LAN-accessible boxes, they're on the Tailscale-enrolled devices). Compromising a Tailscale-enrolled device (say, my phone gets stolen) doesn't compromise the LAN's perimeter — and Tailscale's admin console lets me revoke the stolen device's keys instantly, before the thief unlocks the screen.

This is the same separation-of-concerns logic the series keeps coming back to. Two independent security domains is more robust than one tightly-coupled one, and easier to reason about.

tmux as a location-independent workspace

The other half of the dissolve-the-hub trick is making the operator's sessions live on the substrate, not on the client.

The pattern: every node runs persistent tmux sessions. Long-running work — training runs, watching a log tail, an editor open on a project file — happens inside a tmux session. The client (my laptop, my phone, a coffee-shop terminal) attaches to the tmux session on whichever node owns the work. Disconnect the client and the work keeps running. Connect from a different client and pick up exactly where you left off.

There is no notion of "my computer is doing the work." The work is being done by a specific node. I happen to be looking at it from wherever I happen to be sitting.

Concretely:

The cost: roughly zero. Tmux is preinstalled on every Linux distribution. The setup is fifteen lines of bashrc and a handful of systemd units to start the sessions at boot.

The result: my "workspace" isn't tied to any client device. It's a property of the mesh. I can step away from my desk, walk to a coffee shop, open a borrowed laptop, attach to a tmux session on MirrorForge or Legion, and pick up exactly where I left off — the work was never on the client; it was in tmux on the server, accessible through Tailscale from anything that could open a terminal.

The reachability graph

A useful exercise: draw the diagram of which nodes can reach which other nodes by which paths. For MirrorEthic in May 2026:

There is no node whose failure isolates any other node from being reachable. pi5 going down doesn't isolate MirrorForge from Legion (the 10 GbE link is direct) and doesn't sever off-LAN access (picontrol's redundant subnet route covers it). MirrorForge going down doesn't isolate anything. Legion going down doesn't take the mesh with it (the other nodes can still reach each other via Tailscale and via LAN). The whole thing is resilient by topology, not by any single load-bearing component — and the two subnet routers exist specifically so the gateway role has no single point of failure either.

The corollary: there isn't a "central" management box. watcher is the most-frequently-used entry point for off-LAN access, but it isn't necessary — if it dies, picontrol's route plus MirrorForge or Legion's own Tailscale identity still reach everything from anywhere.

The mesh as agent-operable, not just human-operable

One more property is worth naming explicitly, because it's the part of the architecture that's furthest ahead of what most independent operators have built.

The mesh's coordination layer doesn't distinguish "operator at a terminal" from "operator via MCP." Both are identities that can dispatch through the orchestration layer; both can retrieve docs, telemetry, and routed context; both can hand work to specific nodes. An MCP-capable model — Claude or any other MCP client — is a first-class operator in the mesh the same way I am, scoped by the same rules. Posts 08 and 09 cover the mechanics; the relevant point here is that this is a capability the mesh has, not a future plan.

The moment that capability is named, a security-literate reader has a question: what is the threat model when an external agent can dispatch tasks to your nodes and pull your docs? Answering honestly requires three independent containment boundaries — defense in depth, the same architectural shape this series has been building at every other layer:

  1. The MCP contract. An agent doesn't get a shell. It gets a structured contract — a defined schema for dispatch and retrieval, with typed inputs and typed outputs. The surface area an agent can reach is the contract, not the box. The contract is the equivalent of the API gateway between the orchestration layer and the substrate.

  2. Per-node role scope, enforced by the routing layer. The same scoped context-routing that runs on every human prompt also governs what a given node's role can see and do (this is hologram-Pro's job; post 08 covers it). An agent dispatched onto a node inherits the node's role scope. It cannot reach beyond what the role is provisioned for because scope enforcement is the routing mechanism, not a bolt-on guard. Scope is intrinsic to the path, not a guard at the perimeter.

  3. Claude Code permission layer. Per-action permissions configured at the agent level — even within role scope, sensitive actions require explicit permission or human-in-the-loop confirmation. This is the per-call equivalent of the per-role scope above.

Three independent boundaries. A prompt-injected or otherwise rogue agent is bounded — not made impossible, bounded — by all three. The honest framing is "bounded blast radius across three layers," not "can't happen." A security-literate reader trusts the bounded framing more than the impossible one, and the framing is also true.

The same separation-of-concerns discipline that isolates a V100 fan failure from the compute box also isolates a misbehaving agent from the things it shouldn't touch. The pattern is the same all the way up the stack.

From self-diagnosing to self-healing

When I first built this, the observer only noticed. It would ping, check, and push a notification — diagnosis, then a human (me) to fix it. That's where the title of this post comes from, and it's still the floor of what the mesh guarantees. But the observer has since grown a second job: for the class of failures that have a known fix, it applies the fix and only escalates the ones that need judgment. The mesh moved from self-diagnosing to self-healing, and the two layers are worth separating.

The observer node. The always-on observer is a dedicated Raspberry Pi 5 8 GB — logical node pi5, hostname watcher — sitting on its own. It is deliberately the dumbest, most stable box in the fleet: no training, no research thrash, nothing that reboots. It holds three roles that all share the same property of needing to be up when everything else is down — observer (the watchdog), storage (a dedicated 1 TB HDD for liveness logs and a cortex backup), and gateway (the primary Tailscale subnet router for the LAN). The design rule, which post 08 makes explicit, is that always-on infrastructure lives on the node that depends on nothing it watches.

The reconcile loop. watcher runs a reconciler that walks a set of service contracts — every always-up service in the fleet declares a health check and a restart command (post 08 covers the contract files). Every 30 seconds the reconciler:

Alerts flow out through one path only: the watchdog → ntfy, on a local topic (mesh-alerts) with a public fallback (ntfy.sh/gsmesh-alert-4jq8zx — a stand-in here; the real one stays private, because on ntfy.sh the topic name is the auth) so the page reaches my phone whether or not I'm on the tailnet. Within roughly 30 seconds of an unrecoverable event, my phone is buzzing — but most events never get that far, because the restart already fixed them.

The dead-man behind the observer. The obvious question: who watches watcher? A second, even smaller box — picontrol, a 1 GB Pi Zero 2 W — runs a dead-man's switch that watches pi5 and holds the fallback ntfy hub. It is independent of pi5 by design, so it can raise an alert precisely when the observer itself goes silent. picontrol also stands by as the redundant subnet router. (In the earliest version of this lab, picontrol was the observer; the role graduated to the more capable Pi 5 when the reconcile-and-heal job outgrew a Pi Zero, and picontrol kept the dead-man and notification duties it was already good at.)

The crucial property survived the upgrade: the observer is on a separate node from the things it observes, and the observer-of-the-observer is separate again. If MirrorForge crashes and stops emitting metrics, watcher notices because the health check fails. If watcher itself crashes, picontrol notices and pages. If both somehow went down, the compute boxes were never depending on either of them to do their actual work — the observability layer is strictly additive, never load-bearing for compute.

What divides the two layers is whether the fix needs judgment. A dead ntfy daemon, a dropped tmux session, a tunnel that fell over on reboot — those have a known restart, so the reconciler just does it. A training run whose loss diverged and ate the disk does not have a safe automatic fix; restarting it blindly would be wrong. So that one still pages a human. The next section is exactly that case.

The ambient layer: health you notice instead of read

There's a third rendering of the same watchdog state, and it doesn't live on a screen at all.

The push notification is the right channel when I'm away — it reaches into my pocket from anywhere on Earth. But when I'm in the room with the machines, a phone buzz is the wrong modality: it makes me stop, unlock, and read to learn something my peripheral vision could have told me for free. So the same health state the reconciler already computes gets rendered as light, on the power box that feeds the edge tier (RailCore, from post 04). The lid is a 54-LED ring behind a translucent diffuser: green and slowly breathing means the whole tier is healthy; a red arc pulsing means a node is down, and which arc tells me which one. You don't read it. You notice it — the ring going red registers pre-attentively, before you've consciously looked, the way motion at the edge of vision does.

The architectural discipline is that the ring is a read-only renderer. It runs no health checks of its own; it polls the watchdog's existing state over WiFi and draws it. This matters for the same reason the memory layer has one canonical writer: duplicating authority is how two systems start disagreeing about reality. If the ring ran its own probes, it could show green while the watchdog shows red, and now you have two truths and no way to know which is real. One authority, many renderers — a phone push, a dashboard, a glowing ring — all drawing the same state.

And it degrades honestly, which is the whole test of a status light. A display that can't tell "everything is fine" from "I've gone blind" is worse than no display, because it lies green. So the failure modes are made visually distinct: if the health payload stops updating, the ring desaturates to a grey-blue sweep — the colours are no longer trusted, and it says so. If the watchdog node itself dies, a second independent path drives a teal comet around the ring — watcher-down, confirmed from outside the watcher (the same dead-man principle as picontrol, rendered as light). If the ring loses the network entirely, it falls back to a lone white heartbeat: alive, but isolated. Each broken layer looks different, so a glance tells you not just that something is wrong but which part went blind. The one rule the display obeys above all: beauty yields to truth. It's allowed to run a slow rainbow aurora when everything is calm and a print is finishing — and the instant something needs attention, the flourishes stop and the alert grammar takes over. (The power and fabrication side of that object is post 04's and post 11's story; here it's just the newest surface the self-diagnosing mesh renders itself onto.)

A phone on a stand next to a coffee cup, terminal session open — a full recovery run from a borrowed table.

A real recovery story

A few weeks ago a training run on MirrorForge went sideways at 2 AM local time. The model's loss diverged, ate the available NVMe scratch space, and the run process started spinning on disk-write errors. The notifications fired in this order:

  1. ntfy push to my phone: "MirrorForge disk warn: /var at 92%." 2:17 AM.
  2. Two minutes later, ntfy push: "MirrorForge disk warn: /var at 94%."
  3. Three minutes after that, ntfy push: "MirrorForge process t3_train_v36 running 0s — likely crashed."

The phone woke me up. I unlocked it, opened Termius (an SSH client), connected over Tailscale to MirrorForge, ran tmux attach, saw the spinning process, killed the run, freed the disk space, and went back to sleep. Total time from waking to fixed: about 4 minutes. The phone never left the bedroom.

This is the judgment case from the previous section: a diverged run that ate the disk has no safe automatic fix, so the reconciler didn't try to restart anything — it paged me and let me decide. That whole sequence was possible because:

None of the pieces are exotic. Each one is a standard tool. The architecture is the thing that compounds them into a system that's operable from a phone at 2 AM rather than requiring a desk and a wired connection.

The Yellowstone week — the extended-duration version

The 2 AM story is a four-minute recovery. The same property at week-long scale is the proof I'd actually lead with if there were only room for one.

A few weeks ago I left for a family wedding plus a national-park vacation (Yellowstone and the Montana side; previous trips have been Bryce, Arches, Zion — it's a standing pattern). Full week away. Before leaving I scoped and kicked off two agentic Claude Code instances in tmux on two different nodes so nothing needed babysitting:

The agentic Claude Code instances made iteration decisions, adjusted ablation variants, and handled the routine of a research week without me at a desk. Seven days of mesh-resident research happening while I was on a trail. (Before leaving I also snapshotted both projects' code and docs to a DockCase 2230 enclosure with a WD SN770M 1 TB — small status OLED on the drive, useful in a hotel room — in case I needed source 500 miles from home.)

The mobile operator rig was minimal:

Roughly $150 of phone accessories supervising roughly $1,800 of lab running multi-day training. The ratio is the thesis. The control surface is trivial because the architecture lives in the mesh, not the client.

There was real friction. The AULA over Bluetooth is annoying in ways nobody who hasn't tried it knows about — Bluetooth keyboard latency at the OS level on a phone host is noticeable. The AR glasses are a luxury, not a workhorse. National-park cell reception is mid. None of that broke the architecture; the work happened anyway.

The honest calibration on what this proves. This proves unattended-run reliability and remote agentic operability. It does not prove the V100 beats newer cards for training — that's a different claim, the bf16 and fp8 caveats from post 02 still stand. On the reliability and operability axes, the week-long story is what most operators don't realize is achievable on this class of hardware: reliable enough to leave running unsupervised for a week and operable enough to supervise from a phone in Yellowstone.

The "Mesh" phone folder

The supervision wasn't just SSH-in-tmux. On the Fold's home screen, a folder labeled Mesh contains one-tap PWAs into every node and service: Tailscale, ntfy, Cockpit on MirrorForge, code-server on Legion, the Nexus telemetry dashboard, the Hologram interface, ttyd-backed terminal links, per-project shortcuts for the projects I happened to be running. That folder is the "operator's workspace is a property of the mesh" thesis made literal — the control surface is a phone folder.

Permanent auto-restarting tmux-over-Tailscale on every node

Every node runs a permanent tmux session that survives reboots and is reachable via Tailscale from anywhere I'm logged in. Reachable-from-anywhere is the default state, not aspirational. That invariant is what makes a phone folder into an operator's workspace.

Nexus as the synthesis + triage layer

When I open the Nexus dashboard from a hotel bed, I'm not reading tail logs. Nexus ingests training telemetry and serves live loss / PPL / learning rate / iterations-per-second / ETA, per-node system vitals (one screenshot during the trip showed Legion's 5090 at 94% GPU utilization, 23–24 GB VRAM, 66 °C under sustained load — a real hardware-under-load receipt), the T³ ecology panel, and benchmark deltas. On top of the telemetry sit two tiers of AI synthesis: a local model doing five-minute micro-digests, and Grok 4.1 Fast (2 M context) doing deeper analysis that says things like "this is SGD noise, not structural collapse" and surfaces medium / high / low priority recommendations.

Honest framing on the synthesis layer: Nexus reads the runs and tells me whether to worry. I decide. Synthesis + triage, not autonomous research decision-making. The cognitive layer reads the runs and pages me with interpreted findings; the AI does not run my research. Most of the time, nothing needs me — which is why a week in Yellowstone is a workable mode of operation rather than a research-pause.

(That observability and cognitive stack is its own architectural story — a peripheral cognitive layer over research telemetry — and I'll write it up as a follow-on series rather than cramming it into the substrate-infrastructure thread. What you've just read is enough to know it exists.)

A dark rack with status LEDs quietly lit and an empty chair in front of it — the mesh watching itself so nobody has to sit here.

What this buys you

Most systems require you to be present and the management plane to be up. A mesh requires neither.

You can be in a coffee shop, on a phone, on a friend's laptop, in a different country. The mesh doesn't care. Tailscale gives you the identity, tmux gives you the workspace, watcher gives you the observability and the auto-repair, picontrol gives you the dead-man behind the observer, and the reachability graph gives you the redundancy. The mesh is a place to work that exists independently of any single piece of hardware you happen to be touching.

The cost of all of this, in dollar terms, is roughly $80 of Pi-class parts (the Pi 5 observer plus the Pi Zero dead-man) plus zero for the software stack. The cost in operator skill is one afternoon to configure Tailscale on every node, an evening to set up the tmux session pattern, and a few hours to write the reconcile loop — the health-check-and-restart logic is a couple hundred lines of Python, most of it the contract parsing.

The cost of not having this is the experience most operators have when something goes wrong away from their desk: a panic, a drive home, a missed event, a lost training run. The architecture above replaces all of that with "the phone buzzed, I fixed it from bed."

The mesh heals what it can, diagnoses what it can't heal, and notifies you only for the failures that actually need you. It stays reachable from anywhere you have a network. That's the difference between a hobby setup and a system you can forget about.

Next: post 08, "Substrate-independent roles" — stop thinking in machines, start thinking in roles. The role-template pattern, why it beats coordinator-worker agent frameworks, and the "new hardware just slots in" test as the proof the architecture is real.