You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
config reload swaps the config and reconciles nothing: sources are not added, extractors and ingest ignore it, and specs are never resolved #134
Found while fixing #119. A reload today is scalo's reloader loading the file, validating it, and writing it into SharedConfig with a version bump (crates/fetcher/src/main.rs:334-373 and :705-717). Intervals, jitter, filters and routes follow it because the tasks re-read them per tick. Nothing else does. #119 lands the REMOVE half for fetch sources; this is the rest of the same gap.
A source ADDED by a reload never starts. builtin_instances() has one production caller, at startup (main.rs:516). Today a second source reaches a running pod by the pod roll the chart's config checksum triggers, which is why b3b49f0 could prove the idle case with tests alone. The reconciler from a source removed by a config reload keeps ticking its provider until the process restarts #119 warns by name when it sees a new id; making it spawn needs resolve_config_specs (crates/fetcher/src/config/resolve.rs) run on the reloaded config first, and the reloader callback is sync.
The reload path never resolves env: or vault: specs. resolve_config_specs runs once at startup (main.rs:262), so every reload writes a config with those fields still literal into SharedConfig. No live effect today because drivers bake identity at construction. It is the landmine under the ADD half and under anything else that starts reading identity from the shared config.
A source MUTATED under an unchanged id keeps its baked auth, region and endpoints. Only filter, routes, topic suffix and interval follow the config (crates/fetcher/src/driver.rs:310-327, crates/fetcher/src/scheduler/mod.rs:119-122). Change sources.aws.region on a live pod and it keeps polling the old region, without a warning.
Container extractors bake their config at spawn and exit only on the global token (crates/fetcher/src/extractor/container/mod.rs:343-352, spawned at main.rs:636-653), so an extractor removed by a reload keeps running its image on its interval.
The ingest listener reads enabled once (crates/fetcher/src/ingest/mod.rs:193-196) from a config cloned at main.rs:617, so ingest.enabled: false in a reload leaves the port bound and accepting until restart.
PipelineState::reload_config (crates/fetcher/src/pipeline/mod.rs:496-502) is dead outside its own test. It reads as if the fetcher has a reload hook it does not have. Delete it or make it the entry point.
Done when a reload that adds, removes or changes a source, an extractor or the ingest listener has the same effect as a restart on that config, pinned by a test per lifecycle.
#286 (16ab848) covers the fetch sources. A reload starts an added source, stops a removed one and rebuilds a changed one. Each source has one generation at a time.
Changes to the container extractors or ingest listener still need a restart, as docs/internals.md says. The extractors are spawned once in main.rs (extractor/container/mod.rs:343-352). The listener reads its config once (ingest/mod.rs:193-196).
Activity
fix(scheduler): stop a source's fetch task when a reload removes it
github-actions commentedon Sep 24, 2026
Released in v1.4.27 — https://github.com/hyperi-io/dfe-fetcher/releases/tag/v1.4.27
fix(reload): start, stop and rebuild sources on a config reload
fix(reload): start, stop and rebuild sources on a config reload
fix(reload): start, stop and rebuild sources on a config reload
catinspace-au commentedon Nov 12, 2026
#286 (16ab848) covers the fetch sources. A reload starts an added source, stops a removed one and rebuilds a changed one. Each source has one generation at a time.
Changes to the container extractors or ingest listener still need a restart, as docs/internals.md says. The extractors are spawned once in main.rs (extractor/container/mod.rs:343-352). The listener reads its config once (ingest/mod.rs:193-196).
github-actions commentedon Nov 12, 2026
Released in v1.4.54 -- https://github.com/hyperi-io/dfe-fetcher/releases/tag/v1.4.54