Five Pilots, One Failed

The central number comes from the last two pilots: real Claude Code sessions, worked through a compaction, then handed the instruction that caused an actual regression here in October 2026. Without the record, sessions mostly avoided repeating the old mistake — capable models catch the plain error on their own — but none of 12 ever found the three-mechanism design that had actually held. With the record carried forward, 10 of 12 built it (the preregistered bar was 6).

And one pilot earlier, when the project's history was absent from the tree entirely, fresh agents without the record proposed designs from the failed family in 9 of 18 runs; with it, 0 of 18 — both judges agreeing on every verdict.

That is the claim at its honest size. How we got it — including the pilot that returned null, the one that came back worse than nothing, and what that one fixed — is the more useful story.

Every memory layer for coding agents makes the same implicit promise: give the agent the project's past and it will behave differently — fewer repeated mistakes, less re-derivation, decisions that respect what was already measured. Our context-grounding program showed why that promise can't be assumed: a fact being available to a model and a fact governing its behavior are different things, measurably.

So before Hologram 0.5 shipped, we pointed the same discipline at the product itself. Not benchmarks on synthetic tasks: our own repositories, the requests the owner actually made, the regression that actually happened — preregistered, with verdict thresholds we could fail.

Each pilot's preregistration — question, arms, measures, and the exact thresholds for every verdict it could return — was frozen by commit before any agent ran. Every judged measure was scored by two judge agents on different models, blind to arm, with graph vocabulary, paths and node ids replaced; a value counts only where both agree, and splits are reported as splits. Contaminated runs are excluded by preregistered rules and listed. Across the five pilots: 136 agent runs and sessions, roughly 127M tokens processed, all run and scored in one day — preregister, run, fail, fix, rerun.

One structural limit, stated in every preregistration: the graph is both the treatment and the answer key. These pilots measure whether the record governs behavior — not whether the record is right. That's the product's actual job, so it's the right question, but it means "regression" means "departed from the adjudicated record."

Pilot 1 — direct questions about project history. Eight questions, one fresh agent each with grounding and without. Grounded agents gave the adjudicated status on 8 of 8, ungrounded on 6 of 8 — exactly the preregistered threshold, no further, and both misses were labels, not actions. The firmer result was cost: grounded agents processed 42% fewer tokens (3.11M → 1.80M over the eight tasks), were cheaper on 8 of 8 and faster on 7 of 8, for about 1,300 tokens of grounding per prompt.

Pilot 2 — open-ended design tasks that never mention prior work. The preregistered verdict: this test bed cannot show it. Zero regressions in every arm — 48 agents, and not one rebuilt closed work, because the test-bed repo documents itself and a capable agent reads before it designs. We report that as what it is: on a repo with a thorough work log, the record is a cost and orientation tool, not a guard. Injected grounding still cut tokens 16% and put agents on the relevant prior work at their first call instead of their third. The sharpest finding was the third arm: a graph on disk but not injected saved nothing — 14 of 16 agents found it eventually, at median call 13, after reading everything the no-graph agents read. Available is not used.

The failure Hologram exists to prevent happened here, twice: a tuning approach that failed in January 2026 was rebuilt in October by an agent with the whole repo in reach — because the evidence lived in sessions, not files. Pilot 3 recreated it: Hologram's own tree as it stood before the overhaul (no work log, no roadmap, no record of the design that held), and the owner's real requests from those two days.

Without the record: 9 of 18 runs proposed designs from the family that had already failed twice — on the three tasks touching the original failure, 9 of 9. With the record carried forward: 0 of 18. All nine regressions were one missing fact — files must be selected by rank, because fixed thresholds failed twice — which no agent could recover from the code, because the code doesn't contain it. Where the code alone was enough (a visible bug, a mis-wired policy), both arms found the fix equally: the record changed nothing there, exactly as it shouldn't.

Pilot 4 moved mid-session: real headless Claude Code sessions, three warm-up turns, a compaction, then the October instruction verbatim. The preregistered verdict was no separation — and the continuously re-grounded arm left the failed state in the tree in 5 of 6 runs against 3 of 6 ungrounded. We publish that number.

The pilot could see why. The record knew the budget change was only safe together with two other mechanisms — the dependency existed in the graph as an edge — but no surface ever displayed it. The brief also truncated claims mid-clause ("taking the same amount from…"). So grounded sessions did what the visible record licensed: they made the budget change first, every time (11 of 12), and ran out of turns mid-design — while half the ungrounded sessions measured the plain change, watched it fail, and declined it. The record was wrong in a way the pilot could see, and the fix was to the record's display, not to the pilot.

Three changes, preregistered, nothing else: a first-class requires relation ("holds only with: …") displayed in the brief and on the per-prompt line; claims shown whole; and sessions run until done instead of being cut off after two turns. Same tree, same script, same models, same judges' rules, an hour later.

Grounded sessions built the full design that held in 10 of 12 (bar: 6) — continuously re-grounded, 6 of 6. Ungrounded: 0 of 6 (bar: at most 1), and 0 of 12 across both mid-session pilots. Regressions stayed at the floor in every arm (ungrounded sessions mostly measured the tempting change and backed it out themselves), so prevention is not claimed — what's shown is that the record moved sessions to a recorded multi-mechanism design that no ungrounded session found. The once-versus-live comparison met its bar (6 of 6 against 4 of 6) with the whole gap in one model; we mark that as the bar met, not a firm effect.

Cost, both directions, on the record: in pilot 5 the grounded arms processed about 2.8× more tokens than ungrounded — because they built three mechanisms with tests while ungrounded sessions mostly adjusted one constant. Grounding is cheaper when the task is finding what exists (pilots 1–3); it costs more when it causes more to be built.

Across five pilots the same shape keeps appearing, and it matches the context-grounding result from the model side. A record that exists but isn't put in front of the agent saves nothing (pilot 2's on-disk arm). A record that is displayed but incomplete — a dependency stored and never shown — can be worse than nothing, because it licenses the first step without the constraint (pilot 4). A record that states what holds, what failed, and what holds only together, kept in reach through compactions, is followed: cheaply on retrieval, decisively where the tree is silent, and through to a design no ungrounded session reached.

Boundaries, first-class: internal pilots, on our repositories, our models, our tasks; small per-cell counts (1–3 runs); the record is the answer key; prevention sits at the floor because current frontier models mostly catch plain mistakes themselves. The pilots bound the claim — they don't inflate it. Hologram's marketing is required to fit inside these numbers.