Skip to content

fix(config): reload a global config written before the watch started (flaky watcher test) - #662

Merged
atlas-from-plumb merged 4 commits into
mainfrom
atlas/fix-544-config-watcher-startup
Oct 14, 2026
Merged

atlas-from-plumb merged 4 commits into
mainfrom
atlas/fix-544-config-watcher-startup

Conversation

@atlas-from-plumb

Copy link
Copy Markdown
Collaborator

What

TestGlobalConfigWatcher_DeleteKeepsLoadedConfig failed on main (verify, macOS) and on unrelated PRs, after 10 s with "the watcher never reloaded strict=true". It isn't test slack: the test lands in a real window in the global config watcher.

The window. The daemon loads config.toml, then starts globalConfigWatcher.Run. A write in between raises its events before any watch exists. The one-second stat tick then compares against the inode the first attach just recorded, which is the same one, so it sees no change. The daemon keeps the older config until the next edit.

The test. It saves right after go gw.Run(ctx). On a loaded runner the save beats the first attach, so nothing ever reloads. Sibling tests of the same shape share the race.

The fix. When Run's first attach finds the file, it arms one debounced reload. This is the same "reload, we may have missed something" step onError already takes after recreating a lost watcher. There is no retry and no wider wait.

Also in this PR

A second commit carries two follow-ups from the review of #651:

  • its comments cited PLAN-544 for the census cost, a card number since given to this flake; they now say PLAN-542;
  • the cost test logs its measured time beside its 1.5 s bound.

Tests

  • New: TestGlobalConfigWatcher_ReloadsAFileWrittenBeforeTheWatch. It writes the file before Run, which makes the window certain. It is red on the old code (never reloaded within two ticks) and green with the fix.
  • Existing: the global watcher tests pass with and without -tags integration.
  • Mutation (plumb mutation_test):
    • "never arm at start" is killed;
    • "the helper ignores armed" is killed;
    • "always arm at start" survives, and is equivalent: with no file, the settle's attach fails and the reload is skipped, so behaviour is identical.

Verification

  • make verify-full-container: result in a comment below.
  • Independent review: requested from a peer agent.

🤖 Generated with Claude Code

@atlas-from-plumb

Copy link
Copy Markdown
Collaborator Author

Verification: make verify-full-container on head 84d4861 exited 0.

  • make verify-full exited 0.
  • At most one plumb daemon ran at a time (859 samples).
  • No daemon was left after the suite.
  • The container is gone.

atlas-from-plumb and others added 4 commits October 14, 2026 17:37
…(PLAN-544)

The daemon loads config.toml, then starts the watcher. A write in between
raised its events before any watch existed, and the stat tick compares
against the inode it attached to, so nothing reloaded until the next edit.
The same window is why TestGlobalConfigWatcher_DeleteKeepsLoadedConfig (and
siblings that Save right after `go Run`) timed out at 10 s on loaded CI
runners: the Save beat the first attach.

Run now arms one debounced reload when its first attach finds the file, as
onError already does after a blind recreate. The new test writes the file
before Run, which makes the window certain; it fails on the old code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Plumb-Session: scarlet-badger
…ed time

#651's comments named PLAN-544, a number since given to an unrelated card.
The cost test now logs the census time beside its 1.5 s bound, so a CI
slowdown reads as a shrinking margin before it reddens (review NIT on #651).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Plumb-Session: scarlet-badger
…ubscribes

Review findings on the start reload:
- The watcher started before store.Subscribe(topoPool.Reconcile); a start
  reload landing before that subscription updated the store but never the
  pools built from the first load. It now starts after the subscriber.
- Arm the debounce timer unconditionally: with no file the settle's attach
  fails and nothing reloads, so the armed flag was unobservable.
- The new test fails with Run's own error if Run ends early, and waits 10 s
  rather than 2.5 s so a loaded runner cannot fail a correct watcher.
- The census Logf comment and the CHANGELOG wording now claim only what holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Plumb-Session: scarlet-badger
…44 entry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Plumb-Session: scarlet-badger
@golimpio
golimpio force-pushed the atlas/fix-544-config-watcher-startup branch from 319337a to 5a84dd2 Compare October 14, 2026 06:37

@golimpio golimpio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved under the delegated merge bar, on head 5a84dd2:

  • an independent agent review returned PASS twice: first on the fix, then on the commit that folded in its findings;
  • CI is 9/9 green on this head;
  • make verify-full-container exited 0 on this head;
  • plumb mutation_test: the start-reload mutant is killed by the new test.

@atlas-from-plumb
atlas-from-plumb merged commit fd3d991 into main Oct 14, 2026
9 checks passed
@atlas-from-plumb
atlas-from-plumb deleted the atlas/fix-544-config-watcher-startup branch October 14, 2026 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants