# Building a real-time news parsing pipeline

> One synthetic headline traced through a partial observation, an upgrade, duplicate deliveries, an edit and a retraction, using the ingestion writer's real storage contract.

Published: 2026-09-22  
Updated: 2026-09-23  
Work period: June 2026  
Product: [Helmsman](https://adityadalmia.com/projects/helmsman/)  
Canonical HTML: https://adityadalmia.com/blog/helmsman-realtime-news-parsing/

The news pipeline in Helmsman receives headlines in real time, normalizes them, keeps their history, and makes collection failures visible. This article covers only that ingestion layer: how observations become durable records. What happens to a headline after it is stored is outside its scope.

The hard part is that a headline is not one event.

## One headline, eight observations

A story can arrive first as a thin observation, then with richer metadata, then edited, then retracted. Several collectors run at once for redundancy, and a collector that reconnects backfills items it may already have seen. The storage contract has to turn that stream into one correct current record plus an honest history.

To show the contract concretely, I replayed one **synthetic** headline through the pipeline's real single-event writer, from the June 2026 implementation, against a disposable local Postgres 16 database. The headline, identifiers, category and timings are invented. Steps 1 and 6 came from collector A's fallback observer; the primary-stream deliveries came from collectors A and B. The row states and results below are what the writer actually produced on September 22, 2026. They are not a production capture and say nothing about live volume or latency.

The collectors had three kinds of input:

- **Primary stream**: the authoritative structured feed.
- **Fallback observer**: an independent reader of the rendered page, kept so a protocol change cannot silently blind collection. It sees less: no category, no breaking flag, no related instruments.
- **Backfill**: a history read after reconnecting, to fill any gap.

| # | Observation (synthetic) | Writer result | Current row after | Audit rows |
| --- | --- | --- | --- | --- |
| 1 | Fallback observer sees the headline first | Inserted | Title only, no category; version 1 | 1 |
| 2 | Primary stream, 28 ms later, full metadata | Upgraded | Empty fields filled; first-seen fields kept; version 2 | 2 |
| 3 | Second collector, same payload | No change | Version 2 | 3 |
| 4 | Backfill after reconnect | No change | Version 2 | 4 |
| 5 | Edited title | Updated | New title; version 3 | 5 |
| 6 | Late fallback observation with old, thin data | No change | Edited row kept; version 3 | 6 |
| 7 | Retraction | Tombstoned | Row kept, `deleted_at` set | 7 |
| 8 | Same retraction again | No change | Tombstone unchanged | 8 |

Two tables do different jobs. The `news` table holds one row per source item: the best current knowledge of it. The `news_events` table is append-only and receives every observation, including every duplicate, with its source, collector and observation time. After eight deliveries there was exactly one current row and eight audit rows.

## Why step 2 needed a new contract

The original writer used insert-or-ignore: the first write for an item won, and later writes for the same identifier were discarded. That handles duplicates, but it makes arrival order decide data quality.

In production, the fallback observer could sometimes win the race by milliseconds. Its thin row then occupied the identifier, and the richer primary-stream observation that arrived next was ignored. The stored item stayed without its category, which could make an important item look unimportant. It was a silent data-integrity failure: nothing crashed and no duplicate appeared.

The audit table made the failure recoverable. The rich observation had never been lost; it had only failed to reach the current row. A one-time repair rebuilt affected historical rows from their best authoritative observation in `news_events`, deliberately without waking downstream processing for old news.

The June 12 correction replaced insert-or-ignore with a guarded monotonic upsert. "Monotonic" means a field may move only toward more information:

- A fallback observation may create a row, but may never modify an existing one.
- An authoritative observation may modify an existing row only if it adds something: the row is still marked as fallback-derived, an empty category can be filled, or the breaking flag can move from false to true.
- Empty fields can be filled. Populated fields are never replaced with empty or thinner values.
- The first-seen time, first source and first collector are preserved.
- If nothing improves, the database update does not run. There is no version increment and no downstream wake-up, so repeated deliveries do not cause flapping.

Steps 3, 4 and 6 show the useful kind of idempotency. Idempotency means that repeating an operation has the same effect as doing it once. Here it is more specific than "ignore duplicates": duplicate deliveries are recorded as evidence, but only an observation that improves the item changes the current state.

Edits and retractions use separate paths. An edit changes the displayed fields and increments the version. If an edit arrives for an item never seen before, the writer creates the item through the same guarded upsert. A retraction sets a tombstone rather than deleting the row, so a reader can distinguish "retracted" from "never received".

## What the trace also showed

The writer sends a database notification to wake downstream processing when it creates a row, upgrades one, or applies a material edit. In this run, steps 1, 2 and 5 sent that notification, and duplicates did not.

The retraction in step 7 set the tombstone but did not send a notification. That is visible in the writer code as well as in the replay. A consumer that relied only on notifications would learn about a retraction when it next read the current row, not immediately. I record it here as a characteristic of the reviewed revision, not as a claim about whether any live consumer was affected.

## Quiet does not mean broken

A period with no new stories is different from a disconnected collector.

Each collector reports connection activity separately from content arrival. Protocol keepalives advance the activity time even when no headline arrives. A stalled connection and a quiet news period therefore produce different health signals, rather than both appearing as "no headline for five minutes".

## Evidence and limits

- **Implementation**: the single-event writer, schema and upgrade rules from the June 2026 ingestion service, including the June 12 upsert change.
- **Replay**: one synthetic item, eight observations, run locally on September 22, 2026. It demonstrates the storage contract, not throughput, production latency or collector uptime.
- **Not covered**: the source's access details, and any logic that consumes stored headlines.

Once repeated, partial, corrected and retracted observations are represented explicitly, real-time ingestion becomes something to inspect rather than a stream we hope is complete.