Separation of concerns: networking

The Substrate Series · 05 — Concrete pattern

A dedicated point-to-point link between nodes, scoped to its own subnet, turns multiple machines into one — and costs less than you think.

The unit that breaks people first

Before any of the rest of this post earns its keep, you have to fix one thing. The networking industry quotes bandwidth in bits per second. The storage and application industry quotes throughput in bytes per second. The conversion factor is eight, and the resulting confusion costs an enormous fraction of would-be home-lab networking upgrades.

A gigabit Ethernet link is 1,000 megabits per second, or roughly 125 megabytes per second. A 10-gigabit link is 10,000 megabits per second, or roughly 1,250 megabytes per second.

(All throughput numbers are sustained real-world after protocol overhead; Wi-Fi figures haircut for realistic conditions, not vendor peak.)

People look at "10 GbE" and think "ten times faster than my 1 GbE" and then plug it in and copy a file and see 1.1 GB/s and assume something's wrong. Nothing's wrong. 1.1 GB/s is what 10 Gb/s means. The eight-times conversion is the entire story; if you skip it you'll either be permanently disappointed or you'll buy infrastructure you don't actually need.

The corollary: most consumer-grade NAS marketing leans hard on Gb/s numbers precisely because the MB/s equivalent feels small. Read the spec sheets in MB/s once and a lot of upgrade decisions get cleaner.

The principle

A dedicated point-to-point link between your two highest-bandwidth nodes — typically your dev workstation and your training/inference node — turns them into something that behaves like one machine.

You can:

• Mount remote filesystems over NFS or SSHFS at speeds that don't ruin your workflow.
• Stream training data from one node's storage to another's GPU without bottlenecking on the network.
• Copy multi-gigabyte model checkpoints in seconds rather than minutes.
• Run distributed training across the link without the latency floor that Wi-Fi or shared-LAN gigabit imposes.

The catch most builders trip on: doing this over your home LAN poisons your home LAN. Every multi-gigabyte rsync between your machines saturates the same switch your video calls and your spouse's gaming console use. The fix isn't more capacity — it's a separate link in a separate subnet that the rest of the network knows nothing about.

That's the architectural move: a point-to-point cable, two NICs, a private /30 or /24, and traffic that never touches your home LAN unless you specifically route it there.

The two paths: RJ45 (10GBASE-T) and SFP+

For a 2-node lab the choice is mostly between two physical layers.

10GBASE-T (RJ45). Uses Cat 6A copper cable, terminates in normal RJ45 jacks, the cards present as ordinary Ethernet interfaces to the OS. Used Mellanox ConnectX-3-T, Asus XG-C100C, and Intel X550 NICs run $60–100 each on the used market in 2026. Cat 6A in a 10–25 foot run is $15–30. A pair plus cable lands around $150–200 all-in. Wins: works with normal jacks, runs through walls, no special learning curve. Loses: cards run hot (10–15 W each), longer cables get expensive, latency is slightly higher than SFP+.

SFP+ direct-attach (DAC). Uses a SFP+ DAC cable — a fixed copper cable with SFP+ transceivers molded on each end. Used Mellanox ConnectX-3 SFP+ cards are abundant on eBay at $25–45 each (this is one of the cheapest ways to get 10 GbE in 2026); DAC cables run $15–25. A pair plus cable lands around $90–130. Wins: cheaper, cooler-running, lower latency, basically datacenter-standard physical layer. Loses: max cable length about 7 meters, the cable is fixed (not field-terminated), if your two boxes aren't adjacent the deployment gets awkward.

For a desk-adjacent rack-and-laptop pairing, SFP+ DAC wins on price. For two boxes in different rooms of the same building, 10GBASE-T over Cat 6A wins on flexibility.

My MirrorForge↔Legion link is a 10GBASE-T pair, total parts cost roughly $183 — two consumer-tier NICs, a 12-inch Cat 6A cable for the rack-adjacent run, and the breakout for Legion (the laptop story below).

The point-to-point pattern

Here's the actual configuration shape, regardless of physical layer choice.

Dedicated subnet, /30 per link. The point-to-point link gets its own private subnet, separate from your home LAN. The convention I use: each point-to-point link gets its own /30 in a dedicated block (10.10.x.0/30), chosen well outside the home LAN's 192.168.50.0/24 so the two can never collide. A /30 is exactly two usable host addresses — the structure itself documents that only two hosts ever live on this link. For the MirrorForge↔Legion 10 GbE link: 10.40.10.0/30. MirrorForge is 10.40.10.1, Legion is 10.40.10.2, static. No DHCP. No router. The link begins at one box and ends at the other. (Why not 192.168.x.x? Because consumer routers hand those out by default; putting an isolated link there risks colliding with the LAN you're isolating it from, which defeats the point. Stay in 10.x for visible separation.)

Static IPs. Configured via NetworkManager or systemd-networkd. The interface isn't routed to anything; it doesn't need a gateway. Your home LAN stays at its 192.168.50.0/24 (or whatever) and the two nodes simultaneously have a LAN-side address and a point-to-point address. Applications choose which one to talk over by destination IP.

Jumbo frames. Bump MTU from the default 1500 to 9000 on the point-to-point interface. The reason is that 10 Gb/s with 1500-byte frames means a lot of per-packet overhead; at 9000-byte frames the per-packet cost is amortized across six times the payload. Real bandwidth gains on bulk transfers are 10–20% on top of what default MTU gets you. Just remember that both ends of the link have to agree, and any switch in between has to support jumbo frames — which is one of the reasons direct cabling between hosts is the right move.

Network isolation by design. Because the point-to-point subnet is not routed to your home LAN, traffic on it can never accidentally hit your normal network. You don't have to think about it. You don't have to set firewall rules. You don't have to coordinate around your roommate's video call. The link is invisible to everything except the two boxes that share it.

No switch. Two boxes, one cable. No switch in between. No switch firmware to update. No switch port to misconfigure. No switch power budget to share. The cheapest, most reliable point-to-point link is a piece of copper between two NICs.

This is the entire pattern. Static IPs, jumbo frames, direct cable, isolated subnet. About fifteen minutes of configuration, hours of throughput gain.

The laptop problem

Workstations get PCIe slots. Laptops don't. The cheapest way to put a laptop on 10 GbE involves a small detour.

Option A: Thunderbolt 4 enclosure + PCIe NIC. A TB4 (or USB4) PCIe enclosure runs $150–250 and accepts a half-height PCIe card; drop a $60–100 used 10 GbE NIC into it and you have a 10-gig interface on the laptop. Wins: full PCIe bandwidth, identical performance to a desktop install. Loses: the enclosure is bulky and the price compounds.

Option B: dedicated USB-C/TB4 10 GbE adapter. Sonnet, QNAP, and OWC sell sealed USB-C/TB4-to-10GbE-RJ45 adapters at $200–400. Wins: small, one-cable, no enclosure clutter. Loses: more expensive, fewer driver options, locks you to one vendor.

Option C: use the Thunderbolt 4 port directly as a network interface. Linux's thunderbolt_net module exposes a TB4 cable between two TB4 hosts as an Ethernet-class interface. No NIC required. The interface runs at roughly 10 Gb/s on TB4. Cost: one TB4 cable.

I use option C for the Legion↔ASUS pairing. Legion's Intel Core Ultra 9 275HX brings TB4 onboard; the ASUS TUF F15 has TB4 via Raptor Lake; the cable between them on a 10.40.20.0/30 link (Legion .1, ASUS .2) is a fast Ethernet path with no NICs involved at all. The TB4 port doubles as both the laptop's primary external storage interface (a 40 Gb/s NVMe enclosure runs through it) and a node-to-node network when the cable is plugged the other direction. The same port has two roles depending on what's at the other end.

For the MirrorForge↔Legion link, option A wins — MirrorForge is a desktop with PCIe slots so its NIC goes straight in, and the Legion TB4 enclosure was already on hand for the NVMe role and accepts a NIC interchangeably.

Cable choice as a one-time decision

Cable is the cheapest thing to buy and the most expensive thing to redo.

Buy Cat 6A, not Cat 6. The price difference is a few dollars; the bandwidth headroom is significant. Cat 6 is rated for 10 GbE only at short runs and with degraded performance; Cat 6A is rated for 10 GbE at full distance with margin. If you ever expand the lab or move the boxes, Cat 6A keeps working at full speed; Cat 6 starts dropping rates.

Buy DAC cables in the length you actually need, not what's cheap. SFP+ DACs are length-fixed; the wrong length means buying another. 1 m and 3 m are the most common shelf lengths.

Avoid bulk-roll Cat 6A that you terminate yourself unless you've done it before. Cat 6A terminations are unforgiving and a bad termination drops you to gigabit silently. Pre-terminated cables in the right length are worth the small premium for a 2-node lab.

The current MirrorEthic network, end to end

For the receipt-driven version of all of the above, here's the actual state:

• WAN: Symmetric gigabit fiber. Measured 987 Mbps down, 948 Mbps up. The MB/s equivalent is ~115 MB/s in either direction.
• LAN switch: Netgear GS316 16-port gigabit. Unmanaged, $70 retail. Carries the home LAN traffic plus inter-node traffic that doesn't need 10-gig.
• LAN cabling: 12-inch Cat 6 runs between the rack-adjacent nodes (MirrorForge, picontrol, the switch, the UPS). Patch lengths kept short to minimize signal degradation and cable clutter.
• Router fallback: GL-MT1800 as a secondary router with VPN endpoint and Wi-Fi backup. Useful when the primary WAN drops, when remote access is needed without depending on the ISP gear, and as the off-LAN Tailscale exit option.
• Point-to-point 10 GbE pair: MirrorForge↔Legion, 10.40.10.0/30, jumbo frames, ~$183 total parts cost.
• TB4 point-to-point: Legion↔ASUS, 10.40.20.0/30, no NICs, cable cost only.
• USB-gadget point-to-point: picontrol↔MirrorForge, 10.40.30.0/30 NATed via MirrorForge to the LAN, primary uplink for the Pi (post 06 covers the rationale).
• Tailscale overlay: every node is a member; reachability is independent of which LAN-side path is up.

Three separate fast point-to-point links, each scoped to its own subnet, plus a regular LAN for everything else, plus a Tailscale overlay for off-LAN access. The home network was never designed to carry datacenter-scale traffic and it doesn't have to.

What this buys you

The same separation-of-concerns logic post 04 covered for power and thermal applies here:

• The high-bandwidth inter-node traffic has its own physical layer. It cannot saturate the home LAN, because it isn't on the home LAN.
• The home LAN has its own physical layer. It cannot slow down inter-node training transfers, because the training transfers aren't on it.
• The remote-access path (Tailscale) has its own logical layer that doesn't care which physical link is currently up. The mesh is reachable from anywhere as long as any path exists.
• Each link can be upgraded, debugged, or replaced without affecting the others.

The home is one network. The lab is several networks that happen to share some boxes with the home. That distinction is the entire architectural difference between "I have a fast machine" and "I have a fast lab."

The parts list is unremarkable: two 10 GbE NICs, a Cat 6A cable, a couple of static IP entries, an MTU change, a Tailscale install. The architecture turns that parts list into a system that scales to whatever's coming next — third node, third GPU, third workload — without rebuilding the network underneath.

Next: post 06, "Out-of-band management on $35 of parts" — a Raspberry Pi gateway as the software management plane, IPMI as the hardware management plane, and why the two layers complement rather than replace each other.