A scheduler migration is not finished when the new manifests look correct. It is finished when the new scheduler owns the work, the old scheduler no longer owns it, and the monitoring describes the system that actually runs.

Helmsman’s research engine moved its scheduled work from Kestra to native Kubernetes CronJobs in July 2026 and removed the last of the old path in August. Kestra stayed up for other workloads on the platform. The reason was specific: in our single-instance setup, registering many job definitions in one burst could crash the Kestra process, which blocked adding more work. A CronJob is an ordinary Kubernetes object, so there is no shared scheduler process to overload.

The work is organized per asset: one market the research follows, such as a traded instrument whose price is quoted in dollars (say, $100). Each asset has a small group of jobs: data pulls, a daily research job that writes and delivers a brief, a watchdog that checks the expected work happened, and a few follow-ups. We moved those groups one asset at a time.

The invariant: exactly one owner

Every job has exactly one declared owner. Two owners run the work twice; for the daily job, that means a second brief. Zero owners is the quieter failure: the work stops and nothing crashes.

We made ownership a line of configuration. One file maps each asset to a scheduler, and the generator that writes both the Kestra definitions and the CronJob manifests reads it:

# Illustrative; asset names are invented
default: kestra
assets:
  asset-a: kestra
  asset-b: cronjob

To check the mechanism, I ran the real generator from the July 12, 2026 revision against three synthetic assets, each with six invented jobs, on September 22, 2026 (UTC). That revision also has the third owner value explained below. The declared owner fully determined the output:

Declared owner Kestra definitions emitted CronJobs emitted CronJobs active
kestra 6 6 0 (all suspended)
cronjob 0 6 6
parked 0 6 0 (all suspended)

In all three runs, no asset had both a Kestra definition and an active CronJob. This demonstrates the generator’s contract, not what a cluster was running.

The tests enforce the same partition over the real committed output. Their first version proved only “at most one owner”; review found that an asset whose CronJob manifest went missing would still pass. Before any real cutover, the check was tightened so every source job appears exactly once.

A suspended CronJob exists but creates no runs. Every generated CronJob shipped suspended, with a bounded starting deadline: without one, Kubernetes immediately schedules the runs missed while suspended. The daily job’s deadline is about two minutes, because a long deadline under its no-overlap policy can turn a skipped run into a surprise late one. It also gets zero automatic retries. It had no delivery-level deduplication, so a retried run that had already delivered before being evicted would deliver again. A failed run alerts, and a person re-runs it.

One asset’s cutover

Configuration and cluster are different places, and the cutover had to be ordered across both. The GitOps controller that applies the CronJob manifests was deliberately left on manual sync. Automatic sync would have applied a newly active CronJob within minutes of the commit, before the asset’s definitions had been removed from Kestra.

This is the trace for the first asset, a group of 11 jobs, on July 10 and 11, 2026 (UTC):

Step What changed Active owner after How a double run is prevented How it was verified
0. Stage CronJobs generated beside the Kestra definitions and applied, all suspended Kestra Suspended objects create no runs 90 of 90 CronJobs suspended; no Jobs created; Kestra restart count unchanged
1. Flip One line in the ownership file; the asset’s 11 Kestra definitions leave the repository and its 11 CronJobs become active, in one commit Kestra (cluster not yet changed) Manual sync: nothing is applied yet. With the definitions gone from Git, a restarted push loop cannot re-register them Tests passing in the flipped state (the first attempt failed; see below)
2. Delete and drain Each definition deleted through Kestra’s API, spaced out; in-flight runs checked Neither, deliberately The old push loop never deleted anything, so removal had to be explicit Each definition present, deleted, then absent; Kestra restart count flat; no run in flight
3. Pinned sync CronJobs applied at the exact flip revision, not the branch head CronJobs A commit landing during the drain cannot be applied by accident Sync result named the flip revision; exactly 11 CronJobs active and 79 suspended; a manually started job wrote to the database; the watchdog fired on schedule
4. First daily run Nothing CronJobs Cutovers were timed between an asset’s daily runs The next daily job ran on CronJobs about two and a half hours later and delivered its brief

The zero-owner state in step 2 is intentional. It lasts minutes, falls between daily runs, and a missed tick of a frequent check is covered by the next. The dangerous state, two owners, never opens. Configuration holds exactly one owner; during a cutover, execution is held to at most one scheduler firing.

The flip commit first failed two tests. One was expected: a count test written before any cutover asserted that no CronJob was active. The other was new: older fixtures reused a real asset’s name and silently read the production ownership file. Both were fixed before merging, and the merge revision, not the failing commit, became the sync target.

Five more assets followed the same order over the next five hours. Two drains found runs that Kestra had reported as running for 27 hours with no live process behind them; they were stopped before the switch. Assets enabled later were CronJob-owned from the start.

Rollback when a revert is not enough

The planned rollback runs in reverse: suspend the CronJobs and confirm nothing is running before re-enabling Kestra.

The real test was the final asset. Its flip was committed, then the cutover was cancelled before the cluster steps ran. The asset stayed on Kestra, but the next planned sync would have applied a revision in which its CronJobs were active.

A revert was unsafe: restoring the old definitions to the repository would have made each later release push them to Kestra again, the burst pattern that crashed it. A plain suspend was impossible, because the generator equated “suspended” with “owned by Kestra”. We added a third owner value, parked: no Kestra definition in the repository, and a suspended CronJob. Kestra kept running the definitions already registered on its server.

The repository and the running system now disagreed on purpose. The repository said “parked”; execution was on Kestra, with frozen definitions referencing an older image and configuration. The migration record stated that divergence explicitly.

The cancellation was reversed later that day and the flip back to CronJobs committed. A session summary called the finish executed. A cluster check four days later found the CronJobs still suspended and the Kestra definitions live. Nothing had double-run, because sync was manual and the last applied revision predated the flip. But the branch head now contained the arming commit, so an unpinned sync would have created the double run. The check also recorded a trap: the controller’s sync-revision field is its comparison target; the revision actually applied is in the last operation’s result. The cutover completed that evening: delete, drain, pinned sync.

Monitoring had to move too

The first asset moved on the evening of July 10 (UTC). The alert rules moved in a separate repository about seven hours later.

The old “a run crashed” rule matched pods by a name pattern no CronJob pod used, so for cut-over assets it matched nothing. The other watched Kestra’s replicas, which mean nothing once Kestra is not the scheduler. Six CronJob-aware rules replaced them: a failed job, a missed daily schedule, a missed watchdog schedule, a pull with no success for days, a failed batch data sweep, and the metrics source going blind.

Pulls are excluded from the failed-job alert, because a pull that exits reporting unavailable data is honest, not broken; a naive rule would have fired on five such pulls that day. Missed-schedule rules ignore suspended CronJobs, so parked work pages no one. Before the commit, the rules were checked against live metrics and fired on nothing.

So the trace does not show the old scheduler disabled, the new one enabled and monitoring updated at one revision. Ownership switched at one revision per asset; monitoring followed hours later, in another repository.

Removing the old path

On August 10, the engine’s last definition in Kestra, already disabled, was deleted, leaving the engine’s inventory there empty. The push loop was removed by deleting its manifests from the GitOps repository, so the controller tore it down; killing its process could have started a replacement that re-pushed whatever was still committed. Kestra’s restart count stayed flat. The generator’s Kestra output was deleted without changing a byte of any generated manifest. The month-old plan needed four corrections first, including a wrong identifier for the last definition and a leftover permission grant it had missed.

Ownership then became two states, cronjob and parked, with parked as the default. An enabled asset missing from the ownership file is a build error, so enabling an asset and arming it remain separate decisions.

One planned step has not happened: switching the CronJob application to automatic sync once no second scheduler remained. The estate repository’s revision on September 22 (UTC) still declares manual sync. That is an observation of configuration, not of the cluster.

Evidence and limits

  • Design: the migration runbook begun on July 11, 2026: ownership rule, cutover order and rollback order.
  • Historical receipts: commit history and session records for the July 10 to 16 cutovers and the August 10 removal, and the platform change log for July 11 and August 10. Cluster states quoted here are what those records observed. I did not query the cluster, Kestra or the monitoring system for this article.
  • Replay: the July 12 generator, run locally on September 22 (UTC) against three invented assets. It demonstrates the ownership contract only.
  • Not covered: schedules, job contents, data sources, and anything the jobs compute.

Write intent down in one place, change the cluster only from a named revision, and check what actually runs before calling the move done.