In Helmsman, a language model scores evidence and deterministic code turns those scores into a decision. Model scoring is the only variable cost in that loop; the arithmetic after it is almost free. So when we began comparing engine variants, the goal was to repeat the expensive step only when an experiment’s question required it.

Two kinds of change

A module is one area of evidence the model scores. Its rubric is the written instruction the model follows. An engine is a named, versioned bundle of configuration and rubrics.

A calculation change, such as a different weight, reuses recorded scores under a different deterministic rule. A rubric change alters what the model is asked, so the affected module needs a new score. A July 19 decision record made this the rule: scores live once in a shared store no engine owns, and a rubric change re-scores only the affected module, within a planned model budget.

What a stored score knows about itself

  • The engine registry row holds an ID of the form <family>@<scope>, a family version, the formula it uses and two content fingerprints. config_sha256 hashes the asset’s configuration files and the whole strategy directory. rubric_pack_sha256 hashes the sorted per-module rubric versions.
  • Rubric pins give each module a rubric_version: the first 12 hex characters of the SHA-256 of its rubric file.
  • The score row is keyed by content: asset, module, rubric version, run date, run kind, data vintage (which version of the inputs was scored; live in production) and attempt. No engine or strategy appears in the key. The payload is the model’s full output plus status, backend, model and cost.
  • Run rows record the code revision, config fingerprint and engine ID. Backtest results record a spec hash, a dataset hash, the code revision and a validity label.

To make this concrete, I ran a synthetic trace through the real registry, score-store and dataset-hash code from the reviewed revision, against a throwaway local SQLite ledger. The asset, modules, rubric text, weights and prices are invented; the hashes and hit/miss results are what the code produced on September 22, 2026. It is not a production record.

SYNTHETIC identity record: real field names, invented inputs

engine            demo_family@asset_x   version v1   status shadow
formula_id        synthetic_formula
config_sha256     4d87ed26…5fe1239a
rubric_pack       58068cce…13c80d30
rubric pins       alpha → 2603e14d75db   beta → 53f519a75ab1

score row key     asset_x · alpha · 2603e14d75db · 2026-01-05 · daily · live · attempt 1
score payload     full output JSON · status ok · backend · model · cost

run row           engine_id · config_sha256 · git_sha
backtest result   spec_hash · dataset_hash · code_git_sha · "[A/exploratory]"

A score row does not know which engine used it. Engines that pin the same rubric version for a module read the same row.

Which change invalidates what

Change (synthetic) Config fingerprint Rubric-pack fingerprint Score lookups Work to redo
Same scores written again Unchanged Unchanged 0 new rows None
Variant engine, one weight different Changed Unchanged Both hit Deterministic math only
Variant engine, different beta rubric Changed Changed alpha hit, beta miss Score beta once, then math
Base engine edited to that same beta rubric Changed Changed Both hit None
Engine family version v1 → v2 Unchanged Unchanged Both hit Re-register identity only
Different day or data vintage n/a n/a Both miss New scoring

Only beta missed for the rubric variant. After one new row was written, the base engine still read its original beta score and the variant read the new one; score rows are append-only, so both coexist. The fourth row follows from content addressing: the variant had already paid for that exact rubric text, so the base engine’s edit cost nothing.

One caveat on the fingerprint columns: in the trace, each variant lived in a separately named copy of the strategy directory, and the configuration fingerprint also covers file paths, so the variant rows would change on the name alone. A separate step edited the base engine in place to isolate content: a weight edit changed only the configuration fingerprint, and a rubric edit changed both.

Evaluation has its own identity. Changing a backtest’s exit rule, the condition that closes a simulated trade (“sell if the price falls from $100 to $97, or after five days”), changed its spec hash without touching a score. Revising one synthetic price from $100.00 to $101.00 changed the dataset hash, so no result can silently mix both versions of the data.

A fingerprint that means less than its label

The registry schema calls config_sha256 the calculation-change fingerprint. But it hashes the whole strategy directory, and the rubrics live inside it. In the trace, editing only a rubric moved both fingerprints.

So the pair must be read together: config changed with the rubric pack unchanged means a calculation change; a changed rubric pack means a rubric change. Lookups are unaffected because they use per-module pins, but a report that classified experiments by the config hash alone would call every rubric change a calculation change. I record this as a characteristic of the reviewed revision.

Why identity mattered in practice

Stored runs were not one series. A July 19 read recorded in the design spec found 93 distinct configuration fingerprints across 285 stored runs. Treating those scores as one history would silently compare different engines, so the spec requires every dataset slice to group by module, rubric version and configuration fingerprint.

The older reuse path was all or nothing. Before the shared store, one strategy could reuse another’s outputs only if their whole rubric directories hashed identically and the upstream run had already written its files. Otherwise it fell through to a paid model call, which the code now logs loudly so a switch from free reuse to paid work cannot pass silently.

A variant that is never graded produces no evidence. The first variant engine was designed to run “dark”: it records a decision each day, such as “lean long at $100” (expect the price to rise), that nobody publishes or trades. It differed from its base only in configuration and reused its scores, so it added no model cost. But grading then covered only the first strategy per asset: a second engine would decide daily, look active and never be graded. We made per-strategy grading a prerequisite, and it landed on July 19 before the variant was registered.

Written everywhere, read carefully

The live runner writes every scored module into the shared store on every run. The read path, which rebuilds the decision function’s exact inputs from stored rows, is opt-in, so adding the store did not change live decisions. Its gate is a parity test: store captured real runs’ module outputs, rebuild them, rerun the decision function and require identical decisions. A missing row reads back as unavailable, never as an inferred score. I ran that test file at the reviewed revision on September 22: 11 passed, covering nine captured days and two store-contract checks.

Replay does not recover the past

A backtest replays a trading rule over past data: if the rule would have bought at $100 and sold at $104, it records a simulated $4 gain per unit. The hard question is what the past data knew at the time. We use four modes:

  • Stored-score replay. Scores were written before outcomes existed, so they contain no hindsight. Calculation experiments run here, but the window starts at live operation in early July 2026, so it supports hypotheses and negative results, not a claim that a variant makes money.
  • Forward observation. A decision is recorded before its outcome: “lean long at $100” written on Monday, graded when Friday’s price is known. Only this mode can support a money claim, and it is not exempt from luck: a hundred variants run forward still produce lucky winners, so sample-size and chance checks apply.
  • Price-only checks. Decades of prices cannot validate a model-scored engine, but they can test the simulator’s exits, costs and calendar handling.
  • Historical re-scoring, which we chose not to build. A model scoring 2023’s evidence today may have seen 2023’s outcomes in training: judging a decision at $100 after effectively seeing the move to $110 does not reproduce the uncertainty at $100. Evidence must also replay as first published: a price reported as $2.10 and later corrected to $2.15 must replay as $2.10. Helmsman never captured pre-launch evidence that way, and it cannot be collected again.

One narrow form remains designed for later: an older model whose training ended before the window, re-scoring evidence stored after launch. feed_vintage is in the key so such a re-score could never collide with the live row, but a slot for historical scores does not make that history valid. Every result carries a label such as [A/exploratory], and in the reviewed code only a forward-registered result that meets the sample floors supports a live change or a money claim.

Why the replay kernel stayed small

We considered a larger event-driven engine for sweeps. Those engines simulate orders one at a time at minute resolution and cannot accept a precomputed decision, while a sweep is array math over stored daily rows. We built a thin pandas and NumPy kernel that loads one immutable snapshot and runs many experiment specs over it. That was a choice about workload shape; other parts of the platform can still need richer execution models.

Evidence and limits

  • Implementation: registry, score store, read path, runner writes, dataset and spec hashes and validity labels, built in July 2026 with edits through August.
  • Decision records: the July 19 records for score-once, backtest validity, forward-first validation and the kernel choice.
  • Synthetic trace: one invented asset, two modules and eight steps against a throwaway local ledger on September 22, 2026. It shows identity and cache behaviour, not cost savings, run volume or any engine’s results.
  • Historical figure: the fingerprint count is from the July 19 design spec and was not re-measured.
  • Not covered: formulas, weights, thresholds, rubric content and any engine’s results.

Scoring once works only if every stored score can say exactly what produced it. With that identity in place, most experiments are reruns of deterministic math, and new model work is limited to changes that alter what the model reads.