Skip to content

fix: correct ANOMALY_PROBABILITY env var misapplication, #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yahiaongh merged 1 commit into from
Jul 23, 2026
Merged

fix: correct ANOMALY_PROBABILITY env var misapplication, #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/MILESTONES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Status and scope for each phase of SentinelOps. Dates are approximate; this is a
## Known gaps (tracked, not blocking)
- No Grafana/Prometheus scrape config yet — metrics are exposed but not yet centrally collected
- Fixed: log producer was assigning `level` independently of `status_code` (a 200 response could randomly be logged as ERROR) — root cause of inflated error-rate metrics. Fixed by deriving level from status_code.
- Still open: even after that fix, ANOMALY_PROBABILITY (tuned 0.02 → 0.0015 based on burst-duration math) has not brought error rates down to the expected ~5% in live testing — actual observed rates remain 25-37%. The duty-cycle calculation may be missing a factor (e.g. multiple concurrent bursts per service, or burst state not resetting as assumed). Needs a fresh look with the producer's actual runtime behavior, not just the math.
- Fixed: the ANOMALY_PROBABILITY tuning (0.02 → 0.0015) from the previous fix never actually took effect — a copy/paste error had applied the new value to `redpanda-console`'s environment block (which ignores it) instead of `log-producer`'s. Corrected and verified live: error rates dropped from ~30-50% to ~0-15%, matching expected burst-driven anomaly frequency (confirmed against actual burst log timestamps in a sample window).

## Recently closed
- `anomaly-rust`: added `detect()` end-to-end tests, including a regression guard for the directional-severity bug
Expand Down
5 changes: 1 addition & 4 deletions infra/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,6 @@ services:
depends_on: [redpanda]
environment:
KAFKA_BROKERS: redpanda:9092
LOG_TOPIC: service-logs
EMIT_INTERVAL_SECONDS: "0.2"
ANOMALY_PROBABILITY: "0.0015"
ports:
- "8080:8080"
networks: [sentinelops-net]
Expand Down Expand Up @@ -76,7 +73,7 @@ services:
KAFKA_BROKERS: redpanda:9092
LOG_TOPIC: service-logs
EMIT_INTERVAL_SECONDS: "0.2"
ANOMALY_PROBABILITY: "0.02"
ANOMALY_PROBABILITY: "0.0015"
restart: unless-stopped
networks: [sentinelops-net]

Expand Down
Loading