Helmsman started with a question I keep coming back to: how do you make an AI-assisted conclusion inspectable?

A model can write a plausible explanation. That does not tell you whether the inputs were complete, whether a calculation was correct, or whether the same rules were applied on the previous run. In Helmsman, those responsibilities have different homes. This article follows the line between them inside the research engine: what a model may hand to code, what the code checks, and what happens when the handoff fails.

A rule with no expiry date

The engine’s first hard rule is short. Models produce component scores, flags from a fixed list and short evidence strings. Every sum, weight, cap (a ceiling on how strongly a result may be stated), gate (a pass/fail check), probability and difference is deterministic code. The explanation step may describe the result but never changes a number. In July 2026 we recorded this as a permanent platform decision; its condition for being revisited is “never”.

The reason is practical. If code computes every number in a brief, the formula can be displayed, reviewed as data and swapped between engine variants without running the models again, and every figure can be traced to a function and a configuration revision.

Captured evidence
    ↓
Independent model assessments   (bounded scores + cited evidence)
    ↓
Validation at the boundary      (schema + per-module rules)
    ↓
Deterministic aggregation and checks
    ↓
A brief explaining the result
    ↓
Later outcome evaluation

What a module may say

Each analysis module is one model call, with its own rubric and its own slice of the captured data. Its reply must satisfy this JSON Schema (descriptions trimmed):

{
  "type": "object",
  "required": ["module", "components", "direction_hint", "confidence",
               "flags", "delta_vs_previous", "data_complaints", "summary"],
  "additionalProperties": false,
  "properties": {
    "module": { "type": "string", "pattern": "^[a-z][a-z0-9_]*$" },
    "components": {
      "type": "array", "minItems": 1,
      "items": {
        "type": "object",
        "required": ["id", "score", "evidence", "data_ref"],
        "additionalProperties": false,
        "properties": {
          "id":       { "type": "string", "pattern": "^[a-z][a-z0-9_]*$" },
          "score":    { "type": "number", "minimum": -2, "maximum": 2, "multipleOf": 0.5 },
          "evidence": { "type": "string", "maxLength": 200 },
          "data_ref": { "type": "string" }
        }
      }
    },
    "direction_hint":    { "enum": ["bullish", "bearish", "mixed", "neutral"] },
    "confidence":        { "enum": ["high", "medium", "low"] },
    "flags":             { "type": "array", "uniqueItems": true,
                           "items": { "type": "string", "pattern": "^[A-Z][A-Z0-9_]*$" } },
    "delta_vs_previous": { "enum": ["better", "worse", "unchanged", "no_prior"] },
    "data_complaints":   { "type": "array", "items": { "type": "string", "maxLength": 200 } },
    "summary":           { "type": "string", "maxLength": 280 }
  }
}

The shape is deliberately narrow. A score is a half-step from −2 to +2, never a price or a probability. For a direction component, +2 means the evidence strongly favours a higher price. A bullish reading on an asset trading at $100 says the evidence favours it trading above $100 later; bearish is the opposite. confidence describes the quality of the evidence, not the model’s conviction. Each score carries a short evidence string and a data_ref naming the feed or field it came from, rather than a remembered market fact.

Three rules sit outside the schema because they depend on configuration. The component ids must match the module’s configured list exactly, with no extras and no omissions. Flags must come from that module’s own list. The module field must name the module that was asked.

What the model never sees matters as much. The weights that combine component scores stay in configuration and never enter the prompt. A module sees its own inputs, a shared market-context block and its own previous output, never another module’s answer. That removes one direct path for an early answer to influence the rest; it does not prove the models’ biases are independent.

One synthetic reply, four outcomes

I passed synthetic replies through the engine’s real validator, retry loop and brief renderer from the reviewed revision. The module, its component ids, flag, input rows and rubric are invented; no real configuration or rubric was loaded, and a scripted stand-in replaced the model endpoint. The results below are what the code returned on September 22, 2026.

# Synthetic situation Model calls Result
0 The module’s input slice is empty 0 insufficient_input
1 The reply matches the contract 1 ok
2 The reply breaks the contract, then a valid retry 2 ok_retried
3 Prose, then JSON with no components 2 unavailable

Step 0 happens before any model call. A module can declare what enough input means, such as at least one row in a report. The trace called that check directly; in a full run, a failed check records insufficient_input and spends no call.

Step 2 is the interesting one. The first synthetic reply was:

{
  "module": "example_supply",
  "components": [
    { "id": "inventory_trend", "score": 2.7,
      "evidence": "inventories fell sharply", "data_ref": "example_report.rows" },
    { "id": "price_target", "score": 1.3,
      "evidence": "from general market knowledge", "data_ref": "memory" }
  ],
  "direction_hint": "bullish",
  "confidence": "very high",
  "flags": ["BUY_NOW"],
  "delta_vs_previous": "no_prior",
  "data_complaints": [],
  "summary": "SYNTHETIC: strong setup.",
  "entry_price": 101.5
}

The validator returned eight errors:

: Additional properties are not allowed ('entry_price' was unexpected)
components/0/score: 2.7 is greater than the maximum of 2
components/0/score: 2.7 is not a multiple of 0.5
components/1/score: 1.3 is not a multiple of 0.5
confidence: 'very high' is not one of ['high', 'medium', 'low']
unknown component ids (ids are law): ['price_target']
missing component ids: ['import_pace']
flags outside the example_supply enum: ['BUY_NOW']

The reply tried to hand code a number off the scale, a component nobody asked for and a price. The engine never produces trading instructions: no entry (the price at which you would buy, say $100), no stop (a preset exit that caps the loss; selling at $95 limits it to $5 per unit), no target (where you would take profit, say $110) and no sizing (how much to buy, say $5,000 worth). None of these has a field in the contract, so a model cannot slip one through as an extra key.

The runner does not repair a rejected reply. It sends the same request once more, with the validator’s errors appended. The synthetic retry was valid, so the module was accepted as ok_retried.

Step 3 shows the other ending. The first reply was prose. The second was wrapped in a Markdown code fence, which the parser strips because it is formatting; the content itself is validated unchanged. The empty component list still failed, and the module became unavailable.

Every call, including each retry, produced an audit row. The engine stores each row’s raw prompt and completion with a prompt version derived from the rubric file’s hash, so outputs from different rubric versions are not mixed.

What the code does with a valid reply

From here, code does the arithmetic. It combines component scores into module-level results using weights from configuration, combines those, runs constraint rules and stores every intermediate value as a metric row. The final decision is validated against its own schema before it is written. If aggregation fails, the run exits with an error: the design treats a wrong decision as worse than no decision.

Missing input stays visible. The contract tells a module to score a component with no usable input as 0 and say so in data_complaints, never to guess. When a whole module is unavailable, its result is recorded as missing. The arithmetic treats it as neutral so totals stay defined, while its status travels separately to the constraint rules, which can apply a cap. A module with insufficient input binds a cap just as an unavailable one does. A missing component keeps its weight, so absence cannot make a result look stronger.

There is one wrinkle in the contract: a score of 0 means both “mixed evidence” and “no evidence”. The complaint list and the module status carry the difference. The score alone does not.

Why the contract lives in code

The first live scoring run, on July 7, 2026, failed at this boundary. The rubrics were market-judgment documents that told the model to follow the schema without showing it. The live models omitted data_ref on every component and invented keys of their own, so both modules in that run were marked unavailable.

The fix was not to put JSON into the rubrics. The runner now appends a code-generated output contract to every rubric when it assembles the prompt, built from configuration: the exact shape, the module’s component ids and its allowed flags. Rubric authors edit judgment; code owns the format. The same release began printing validator errors to the run log, so an unavailable module shows why.

The next day, an audit closed a subtler gap. A module can declare extra output fields, but a declaration that redefined a base property such as components would have removed the score bounds. That redefinition is now refused when configuration loads, and again when the schema is merged.

The retry is part of normal operation. On July 8, the first scheduled morning with every module running, every module was accepted; one needed its single retry.

The explanation comes last

The code calls the explanation step the judge. By the time it runs, the decision and its metrics are already stored. It receives the decision and each module’s short summary and writes a few prose sections. A disagreement with the decision goes to a separate notes file and changes nothing. If the judge is unavailable or its reply does not parse, the brief goes out with the code-rendered table alone, marked judge-unavailable.

The judge’s reply has a weaker contract than a module’s. It must parse as a JSON object, and the renderer uses only the prose sections it knows; in the trace, an invented extra section did not appear. Its field types are not validated, though. When I passed a list where a prose section should be a string, the renderer raised an AttributeError. In the reviewed revision, that happens after the decision is stored, and the run’s error handler would mark the run failed instead of degraded. I reproduced the renderer exception on synthetic input, not the full pipeline. Nothing mechanically checks that numbers in the prose match the table either; the code-rendered table is the authority.

Replay follows the same split. Model calls run at temperature zero, but repeating a prompt is still not replaying a computation. Replays of saved days feed the frozen inputs and recorded module outputs back through the aggregator, with no model or judge call, and compare the result bit for bit with the stored decision, apart from the two fields recording which code and configuration ran.

Evidence and limits

  • Implementation: the module-output schema, validator, retry loop, aggregator and brief renderer in the engine revision reviewed on September 22, 2026, built July–August 2026. The rule became a recorded platform decision on July 18, 2026.
  • Replay: synthetic replies through the real validator, retry loop and renderer, with a scripted stand-in for the model. It demonstrates the contract, not model quality, live rejection rates or latency.
  • History: the July 7 failure and fix, the July 8 safeguard and the July 8 retry come from the project’s commits, session log and status record.
  • Not covered: rubrics, prompts, weights, cap and gate rules, thresholds, real module names and any decision the engine produced.

A model is good at reading messy evidence and a poor place to keep arithmetic. Helmsman lets the model classify and cite, lets code count, and keeps enough records to tell which one was wrong.