feat: 1.20 detector cluster — transition speed (D15), time-of-day habit (D12), value correlation (D13) - #378
Merged
Merged
Conversation
…heir learned floor (D15) A thirteenth statistical detector, transition_time, adapted from AMiner's MinimalTransitionTimeDetector. A transition is one step a → b between consecutive events of one stream whose series-field values differ; the stream is the source, split by a new partition_field (the identifier whose moves are timed), and rows without it are left out rather than pooled. Built on _ngram_inner_sql with n = 2, which gains an optional partition_col and now emits pkey and the arriving event's id. Two frames from the start: min-transition learns each pair's fastest baseline transition (over at least stat_transition_min_transitions of them) and flags a suspect transition that undercuts it by stat_transition_min_ratio; self-min-transition takes the pair's next-fastest transition anywhere in the scope as the leave-one-out floor. A zero floor is skipped and counted in a warning. Score = 1 − observed / reference. Gate entry, params model, /anomalies and agent knobs, persisted-run snapshot (partition_field, min_transitions), three settings with registry specs, the method card with a Stream knob, TransitionTimeFinding, a two-bar evidence figure labelled by reference_kind, and ANOMALY_DETECTION.md §15. CACHE_VERSION moves to 5 for the shared subquery's new columns. The demo case gains an administrator's routine jump-host hop (JUMP-01, then FILE-01 20–90 s later) as the floor and the contractor landing on FILE-01 two seconds after each wmic call as the signal, asserted in both frames. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…p (D12) A fourteenth statistical detector, time_of_day, adapted from AMiner's PathValueTimeIntervalDetector. Per (field, value) the day is cut into bucket_minutes-wide wall-clock buckets read in an explicit IANA timezone; the value's habit is the buckets holding at least stat_habit_min_bucket_count reference occurrences (for values with at least stat_habit_min_baseline of them), and an occurrence in any other bucket is reported, scored by the circular distance in hours to the nearest habitual bucket. Cadence measures the gap between arrivals; this reads the hour on the wall. Both frames: habit learns from the baseline window and scores each suspect window; self-habit takes the value's busy buckets across the scope and scores its thin ones. The zone is a server setting (stat_habit_timezone, UTC by default) overridable per run, validated against zoneinfo and a strict token pattern before it is inlined, and snapshotted with bucket_minutes into the persisted run and every finding — the same instant is a different hour elsewhere. Gate entry (always offered in the self frame, needs_setup in the baseline frame without a baseline), _TimeOfDayParams, /anomalies and agent knobs, five settings with registry specs, the method card with a bucket choice and a zone box, TimeOfDayFinding, a day-strip evidence figure, and ANOMALY_DETECTION.md §16. The agent tool's docstring is rewritten compact to keep the tool schema under its budget (43,832 over 35 tools). The demo case gains a one-off manual afternoon backup run (the self-frame signal) and moves the contractor's lateral movement onto the jump host at 03:00, a host whose baseline logons are an administrator's office hours; the nightly backup's move to 03:40 is the benign hit. Asserted in both frames. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…(D13) A fifteenth statistical detector, value_correlation, adapted from AMiner's VariableCorrelationDetector and intra-record: for a field pair it mines implication rules A = x ⇒ B = y — an antecedent value with at least stat_correlation_min_support reference events whose dominant consequent accounts for at least stat_correlation_rule_confidence of them, both directions — and reports a window in which the rule's violation rate rises: a 2×2 G-test of conforming against violating events between the reference and the window, one Benjamini–Hochberg pool per run, an effect floor of stat_correlation_min_ratio on the violation-rate ratio. Only rises are reported; a rule that appears is a proportion shift. Both frames: rule-g-test mines from the baseline window and tests each suspect window; self-rule-g-test mines from the scope and tests each leave-one-out slice against the rest. Pairs come from an explicit field list or the recommender's top stat_correlation_auto_fields categorical fields, capped at stat_correlation_max_pairs with a warning; each pair is one GROUP BY a, b scan capped at stat_correlation_max_rows_per_pair rows. Findings carry the rule as mined, both sides' counts and violation rates, and the consequent value that most often took the rule's place; the allowlist key is the combo one. Gate entry (two categorical fields; a sliceable span in the self frame; needs_setup in the baseline frame without a baseline), params model, /anomalies and agent knobs (rule_confidence), run snapshot, seven settings with registry specs, the method card, ValueCorrelationFinding, a two-bar evidence figure, and ANOMALY_DETECTION.md §17. The agent tool docstring is trimmed again to keep the schema under budget (43,845 over 35 tools). The demo case needed no new signal: the contractor's user ⇒ home workstation rule holds over three weeks and breaks on every host the intrusion visits, asserted in both frames. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ain int GET /anomalies?bucket_minutes=30 answered 422 "Input should be 15, 30, 60, 120, 180 or 240": a query string cannot carry an int literal, so the Literal-typed Query param rejected every value. Both the query param and the tag request body take an int now and the runner's own membership check answers 422 for anything that does not divide the day. Found by the end-to-end verification of the 1.20 detectors; the storable-params test gains the three new methods' out-of-bounds cases. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Tagging transition_time / time_of_day / value_correlation findings no longer 500s: tag_anomalies writes a reason per new finding type instead of falling through to the frequency-spike branch, whose fields they lack. - The Bucket knob works for every option: the form sends numeric choices as numbers, and the params model also accepts "15" for 15, since a stored detector keeps the client's shape. - transition_time judges a zero-length observation by its source's timestamp resolution. In a source with sub-second timestamps it stays instant; in a whole-second source it means "under a second" and is judged and scored at that one-second bound, so it no longer undercuts any floor at score 1.0. Held-back pairs are disclosed in a warning; the finding records timestamp_resolution and observed_upper_bound_seconds. - Settings refuse detector defaults the detectors refuse: habit bucket widths that do not divide the day, unknown IANA zones, a correlation min_ratio <= 1 or an fdr_q outside (0, 1]. The bucket list and zone check live in core/time_of_day.py, shared by the settings and the detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner
Author
|
Fixed all four review findings in b2ef552:
Local full suite: 3541 passed with the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The three cheap AMiner analogs left on the roadmap, one commit each, every one with both frames from day one:
transition_time— a value pair reached faster than its learned floor, per source and perpartition_fieldstream (e.g. an account across hosts), built on the shared n-gram assembly. Baseline frame: the baseline's fastest transition of the pair; self frame: the pair's next-fastest transition anywhere in scope (leave-one-out). Zero floors are skipped and disclosed.time_of_day— per value, the wall-clock buckets it habitually occurs in, read in an explicit IANAtimezone(server defaultstat_habit_timezone, per-run knob) that is validated, inlined, snapshotted into the run and carried on every finding. Score = circular distance in hours to the nearest habitual bucket. Distinct from interval cadence: a job that moves keeps its cadence and breaks its habit.value_correlation— implication rulesA = x ⇒ B = ymined per field pair in both directions (support and confidence floors), tested per window with the proportion-shift G-test, one BH pool, an effect floor on the violation-rate ratio. Only breaks are reported. Pair explosion is bounded by three disclosed caps (auto fields, pairs, rows per pair).Each commit carries its gate entry, params model,
/anomaliesand agent knobs, persisted-run snapshot, settings with registry specs, method card, finding type, evidence figure,ANOMALY_DETECTION.mdsection (§15–§17) and a demo-case signal asserted in both frames.CACHE_VERSIONmoves to 5 (the shared subquery emits two more columns). The agent tool's docstring was compacted twice to keep the tool schema under its 44,000-char budget (43,845 over 35 tools, recorded inAGENT.md). Roadmap entries removed, README/CLAUDE counts updated, CHANGELOG has an Unreleased block, PROGRESS has sessions 239–241.Demo changes: an administrator's routine jump-host hop (the D15 floor), the contractor landing on FILE-01 two seconds after each wmic call (the D15 signal), a one-off manual afternoon backup run (the D12 self-frame signal), and lateral movement moved onto the jump host at 03:00 (the D12 baseline signal). D13 needed nothing:
m.okonkwo ⇒ WKS-004breaks on every host the intrusion visits.A last commit fixes a bug the end-to-end run found: the legacy
/anomaliesquery endpoint rejectedbucket_minutes=30, since a query string cannot carry an int literal.Not yet a release: another feature is planned for this branch before
chore(release): 1.20.0.Test plan
uv run ruff check .anduv run ruff format --check .-m "not slow"): 3504 passed; the only failures are the four case-delete tests that need Qdrant, absent on the dev machinetests/test_demo_detector_coverage_clickhouse.pygreen: all three detectors find their signal in both frames, and the gate offers all threeverifyskill recipe): plan verdicts, findings in both frames, cache hit, 422 on bad zone / bad bucket / one-field correlation, persisted-run snapshots carryingpartition_field,min_transitions,timezone,bucket_minutes,rule_confidence,min_support🤖 Generated with Claude Code