perf: adaptive file-source reading (stat-before-read, hot/cold, many directories)

[allamiro]

Why

FileSource treats every discovered path identically every cycle: discover_sources globs the configured paths, and run_cycle reads each one in turn. A directory with 500 files gets 500 sequential reads per cycle, whether or not any of them changed.

That is wasteful in a way the Kafka side is not, because a file that has not been appended to is cheap to detect — a stat is orders of magnitude cheaper than a poll. Nothing currently uses that.

Real deployments make this worse: log directories are mostly cold. A few files are being actively written; the rest are rotated history that will never change again.

What "adaptive" should mean here

Cheap signals first, in order of cost:

1. mtime/size stat before opening. If neither changed since the last read, skip the file entirely — no open, no seek, no read. This alone should eliminate most of the work in a directory of rotated logs.
2. Hot/cold classification. A file that produced records recently is hot: read it every cycle. A file untouched for N cycles is cold: stat it every cycle (cheap) but only read on change.
3. Rotation-aware. A rotated file (inode changed, size shrank) must be re-read from 0, not resumed at the old byte offset. This interacts with security: file rotation race can delete the replacement file #65 — a known rotation race — so the two should be looked at together.
4. Bounded per-cycle work. With thousands of files, a cycle should not be unbounded. Prefer hot files, round-robin the cold ones, and record what was deferred so the deferral is visible rather than silent.

Explicitly NOT a fixed backoff timer

A naive "back off sources that returned nothing" heuristic was implemented for Kafka and reverted in #99: shrinking the poll window made the source less able to return data, which grew the streak further — a starvation trap with no recovery path, measured at a 48x regression.

The file case is safer because a stat is a reliable emptiness check, not a probabilistic one — skipping on unchanged mtime cannot starve a file the way a shortened poll window starves a Kafka topic. But any design here must state why it cannot trap, and a test should pin that a file which starts changing again is picked up promptly.

Multiple directories / locations

Config takes a list of glob paths. Worth handling explicitly:

• globs over different filesystems (one slow NFS mount should not stall local reads — this is where B2 concurrency matters)
• very large directories where the glob itself is expensive per cycle
• overlapping globs producing the same file twice

Measure first

Same discipline as #99, which is what finally located the Kafka bottleneck: instrument before optimising. The read_cycle_profile log already emits productive_read_ms / empty_read_ms / failed_read_ms per source type, so the file-source split is already visible. Read it before changing anything — in the current lab, file sources are a rounding error (cycle=2ms), so this issue is speculative until a workload shows otherwise.

Three optimisations in this repo were aimed at stages that were not the bottleneck (#94, #97, and the reverted attempt in #99). Do not add a fourth. Build a directory of many files, measure, then decide.

Related

• perf: source reads are serial for ALL adapters (kafka, file, syslog) — shared read-path design #96 B2 (file/syslog concurrent reads) — complementary; that is about overlap, this is about not doing the work at all
• security: file rotation race can delete the replacement file #65 file rotation race
• perf: source reads are serial for ALL adapters (kafka, file, syslog) — shared read-path design #96 B1 — drop the single-slot active field

[allamiro]

Status audit — 2026-07-25

PARTIAL — keep open; measurement-gated.

Completed prerequisites:

• the single active slot was removed (refactor(adapters): drop the single-slot 'active' field from all adapters (#96 B1) #107 / perf: source reads are serial for ALL adapters (kafka, file, syslog) — shared read-path design #96 B1);
• file reads now overlap with bounded concurrency (perf(adapters): read file and syslog-spool sources concurrently (#96 B2) #108 / perf: source reads are serial for ALL adapters (kafka, file, syslog) — shared read-path design #96 B2);
• inode/descriptor-based rotation safety was strengthened (security: file rotation race can delete the replacement file #65/fix(adapters): derive file marker identity from the open descriptor #124).

Still missing:

• stat-before-open/read skip when inode/size/mtime are unchanged;
• hot/cold classification;
• bounded per-cycle scheduling with visible deferral;
• overlapping-glob deduplication and large-directory/distributed-filesystem measurements.

The current read_all_batch_candidates still creates a read job for every discovered file and opens it; metadata is used for identity/rotation safety, not as an unchanged-file skip signal.

Waiting on: a representative many-file workload and read_cycle_profile result. If it proves the bottleneck, implement the adaptive policy and prompt-resume tests.