sys-snapd is a high-performance system observability tool built in Rust. It captures high-resolution snapshots of your system's state, providing crucial data for debugging performance issues and analyzing crashes.
This tool addresses a critical gap between traditional monitoring solutions (like top or Prometheus), which typically operate at a granularity of seconds, and low-level kernel debugging. sys-snapd can record system state at a millisecond frequency, allowing you to see the micro-spikes and rapid changes that are invisible to other tools.
sys-snapd operates in two primary modes via subcommands:
- Daemon Mode (
sys-snapd daemon): Runs continuously, capturing system snapshots at a high frequency (configurablerate_ms). It maintains an in-memory ring buffer of the most recent snapshots. If the system exceeds a configured threshold (like CPU or memory usage), it flushes this buffer to disk, creating a "black box" recording of the events leading up to the incident. - Snapshot Mode (
sys-snapd snapshot): Captures a single, detailed snapshot of the system's current state. This snapshot can be printed to standard output or saved to a specified file. This is useful for immediate health checks or debugging.
- Rust and Cargo
To build the project, run the following command:
cargo build --releaseFor optimal performance, it is recommended to build with Link Time Optimization (LTO) and target the native CPU:
RUSTFLAGS='-C target-cpu=native' cargo build --releaseAfter a successful build, the binary will be located at target/release/sys-snapd.
Copy the binary to a location in your PATH:
sudo cp target/release/sys-snapd /usr/local/bin/sys-snapd can be used directly from the command line or integrated with system services like systemd.
To capture a single snapshot and print it to standard output:
sys-snapd snapshotTo save a snapshot to a file:
sys-snapd snapshot --output /path/to/snapshot.jsonTo run the flight recorder in daemon mode. This will continuously capture system states and flush to a crash dump file if anomalies are detected based on the configuration.
sys-snapd daemonFor systemd integration, you would typically configure a service file to run sys-snapd daemon in the background.
sys-snapd can be configured via a TOML file. The application will first try to load from /etc/sys-snapd.toml. If not found, it will look for sys-snapd.toml in the current working directory. If neither is found, a default configuration is used.
A sample configuration file (sys-snapd.toml):
# sys-snapd.toml
# The duration of the in-memory buffer, in seconds.
buffer_seconds = 600 # 10 minutes
# The interval at which to capture snapshots, in milliseconds.
rate_ms = 5000 # 5 seconds
[thresholds]
# The CPU usage percentage that will trigger a flush of the buffer.
cpu_percent = 80.0
# The memory usage in bytes that will trigger a flush of the buffer.
mem_bytes = 8000000000 # 8GBContributions are welcome! Please feel free to open an issue or submit a pull request.
This project is licensed under the terms of the MIT license.