Why
SyslogSpoolSource has the same shape as FileSource: cursors: HashMap<path, SpoolCursor> plus a single active: Option<String> slot, one spool file read per call, all files read sequentially each cycle.
Spool files have a property that makes adaptivity especially worthwhile: rsyslog appends to a small number of active spool files while older ones sit complete and unchanging. Re-reading the cold ones every cycle is pure waste, and the same stat-before-read signal applies (see #100).
Spool-specific considerations
Beyond what #100 covers for plain files:
- rsyslog
impstats / queue files have their own naming and rotation conventions; a spool file being actively written by rsyslog can be appended to while the engine reads it, so partial-line handling at EOF matters (FileSource already ignores a partial trailing line — the spool source should be checked for the same behaviour, and a test should pin it).
- Spool directories can be large and churn quickly. Discovery cost per cycle is a real factor, not just read cost.
- A spool file that rsyslog has finished with may be deleted underneath the engine. Adaptive skipping must not mask a file disappearing between stat and read.
Same trap warning as #100
A fixed backoff on "returned nothing" was implemented for Kafka and reverted in #99 — shrinking the window made the source less able to return data, growing the streak further. Measured 48x regression. Stat-based skipping does not have that failure mode because a stat is a reliable emptiness check rather than a probabilistic one, but any design must say why it cannot trap, and a test must pin that a spool file which resumes being written is picked up promptly.
Measure first
read_cycle_profile (merged in #99) already reports productive_read_ms / empty_read_ms / failed_read_ms per source type, so the syslog split is observable today. In the current lab it is negligible, so this issue is speculative until a real spool workload shows otherwise.
Three optimisations in this repo were aimed at non-bottlenecks (#94, #97, reverted attempt in #99). Build a representative spool directory, read the profile, then decide.
Related
Why
SyslogSpoolSourcehas the same shape asFileSource:cursors: HashMap<path, SpoolCursor>plus a singleactive: Option<String>slot, one spool file read per call, all files read sequentially each cycle.Spool files have a property that makes adaptivity especially worthwhile: rsyslog appends to a small number of active spool files while older ones sit complete and unchanging. Re-reading the cold ones every cycle is pure waste, and the same stat-before-read signal applies (see #100).
Spool-specific considerations
Beyond what #100 covers for plain files:
impstats/ queue files have their own naming and rotation conventions; a spool file being actively written by rsyslog can be appended to while the engine reads it, so partial-line handling at EOF matters (FileSourcealready ignores a partial trailing line — the spool source should be checked for the same behaviour, and a test should pin it).Same trap warning as #100
A fixed backoff on "returned nothing" was implemented for Kafka and reverted in #99 — shrinking the window made the source less able to return data, growing the streak further. Measured 48x regression. Stat-based skipping does not have that failure mode because a stat is a reliable emptiness check rather than a probabilistic one, but any design must say why it cannot trap, and a test must pin that a spool file which resumes being written is picked up promptly.
Measure first
read_cycle_profile(merged in #99) already reportsproductive_read_ms/empty_read_ms/failed_read_msper source type, so the syslog split is observable today. In the current lab it is negligible, so this issue is speculative until a real spool workload shows otherwise.Three optimisations in this repo were aimed at non-bottlenecks (#94, #97, reverted attempt in #99). Build a representative spool directory, read the profile, then decide.
Related
activefield