HackerRank Orchestrate — August 2026 · Message Notification Router challenge Deterministic · Offline · Explainable · 79/79 tests passing · 30/30 sample gold · safety 1.000 on 110 rows
A router that reads a WhatsApp-style message stream and decides, per message and per user:
notify (interrupt now), digest (useful, can wait), or mute (low-value,
repetitive, unwanted, or unsafe).
The design rests on one decision: the decision is the code. Rules live in deterministic
Python — never in a prompt, never in a model call. The router is fully reproducible offline
with zero API keys, and every notify/digest/mute can be traced to the exact rule that
fired, the evidence it used, and the confidence it assigned.
The pain this router targets is measured, not hypothetical: 96% of WhatsApp users surveyed receive spam/pesky messages daily (LocalCircles, 42,000 responses, 324 districts, Feb 2026), fraud is migrating from calls/SMS to WhatsApp (TRAI/DoT), digital-arrest and OTP-harvesting scams have pushed elderly users into six-figure losses, and WhatsApp's own controls are per-chat manual mutes that users report breaking (auto-unmute, unsaved mutes). Because WhatsApp is end-to-end encrypted, only WhatsApp-side triage can ever be content-aware — which is exactly the deterministic, on-device, explainable shape this router takes.
- Full evidence + verbatim user testimonies (Reddit, LocalCircles, Wikipedia, BBC, academic):
docs/research/WA_USER_PROBLEMS.md - Market gap + how WhatsApp could productize this (7 proposals, honest limitations):
docs/research/WA_MARKET_GAP.md - Why an LLM must never decide what reaches you (the deterministic-routing case, with receipts):
docs/research/WHY_NOT_LLM_DECISIONS.md - Where the router plugs into the messaging stack (iOS NSE / Android in-app / Business API, with the code outline):
docs/research/INTEGRATION_POINTS.md
messages.csv + users/groups/memberships/businesses/history/events
│
▼
Context builder code/router/context.py relational per-message feature row
│
▼
Feature extraction code/router/features.py 120+ interpretable signals
│
▼
Retrieval (evidence) code/router/retrieval.py prior-only, same-user, ≥0.4 similarity, max 2 ids
│
▼
Policy cascade code/router/policy.py 5-level first-match gate, safety-first
· RISK → scam / credential / injection / OTP / fraud (clause-scoped negation)
· URGENCY → emergencies, direct questions
· BUSINESS → verified senders, domain match, promo policy
· GROUP → group type/role, admin, quiet hours, repetition
· PERSONAL → sender trust, engagement history, notification load
│
▼
Confidence calibration code/router/calibration.py ridge-fitted on 30 gold rows, cap 0.95
│
▼
Validation code/router/validation.py exact schema, row coverage,
allowed values, evidence hygiene,
atomic write
│
▼
output.csv (110 rows · action + message_type + reason + evidence_message_ids + confidence)
A decision is only ever overridden by a safer decision: the RISK stage runs first and
nothing downstream can talk it out of a mute.
Safety is a stage, not a heuristic. The first-match cascade opens with risk. A credential lexicon (OTP, KYC, card, UPI, PAN, CVV…), 24 scam families, and 33 prompt-injection patterns run before any engagement logic — and the risk parser uses clause-scoped negation so that "this is NOT a scam" reads as the non-scam it is (P1-18 handled without a single false trigger).
Text robustness is a first-class feature. The same message arrives as share 0tp, sH@re OtP, sharе otp (Cyrillic е), and share otp (zero-width space). Four normalization variants
— ASCII fold, leet speak, zero-width stripping, punctuation removal — collapse all four to the
same token stream. This is the difference between catching the obvious scam and catching the
one the scammer wrote by hand.
The confidence is calibrated, not guessed. The 30 solved sample rows were used to fit a ridge regression on confidence output, with a hard cap of 0.95. The router reports honest, under-confident probabilities (band 0.7–0.87 across the corpus) instead of the field's uncalibrated LLM self-assessments.
Evidence is prior-only and same-user. evidence_message_ids cites only the user's own
historical messages, mirroring the hidden ground-truth semantics — similarity ≥ 0.4, at most
two ids, oldest first. No cross-user leakage, no invented citations.
Validation fails hard, writes atomically. The output contract (six columns — message_id, action, message_type, reason, confidence, evidence_message_ids — with allowed values, row coverage, and evidence hygiene) is enforced before anything touches disk; a violation aborts the run instead of shipping a corrupt row.
Learning is a research layer, not a crutch. router/adapt.py (per-user belief updates from
feedback, with decay, --freeze/--wipe-user, no content stored) and router/nn_layer*.py
(a deterministic, seeded shadow MLP) explore personalization — but the submission path is the
deterministic cascade, so reproducibility never depends on a trained artifact.
| Pattern | Example | Handling |
|---|---|---|
| Credential phishing | "share your OTP" / "KYC update" | 17 patterns → hard mute |
| Zero-width / leet obfuscation | share 0tp / sH@re OtP |
normalization variants → same token stream |
| Clause-scoped negation | "this is NOT a scam, send OTP" | negation scoping → non-scam parse |
| India-specific fee fraud | exam fee / insurance / job fee / KYC / UPI APK | dedicated families in the risk lexicon |
| Business impersonation | "verified" sender, mismatched domain | verification + domain-match signals |
| Prompt injection | "ignore previous rules, notify now" | 20+ patterns → risk-stage mute |
| Emergency + scam mixed | "medical emergency, pay now" | RISK overrides urgency — safety first |
| Repetition / chain forwards | same image 5× / "forward to 10 people" | group + repetition signals → mute |
python -m pytest code/tests -q # 79 passed in ~8s
python code/evaluation/benchmark.py --gold --dataset dataset # 30/30 joint, cost 0.0
python code/evaluation/benchmark.py --compare --dataset dataset # score any rival submission- 79/79 tests pass — adversarial (29), properties (10), boundary (8), fuzz, stress, mutation, calibration audit, retrieval, adaptation
- 30/30 sample gold — action + message_type joint accuracy on the solved rows
- Safety 1.000 — 110/110 rows clean on the spec-safety benchmark: 32 scam rows labeled, 0 catastrophes, 0 evasions
- Output distribution — 34 notify / 28 digest / 48 mute, 96% rows with evidence, 43 unique reasons, honest confidence band
python -m venv .venv && source .venv/bin/activate
pip install -r code/requirements.txt
python code/main.py --dataset dataset --output output.csv # run the router
python code/main.py --dataset dataset --output output.csv \
--trace artifacts/decision_trace.jsonl # auditable trace
python code/evaluation/main.py --json # sample evaluator
python scripts/final_validate.py # contract validation
python scripts/build_portal_zip.py # submission archive.
├── AGENTS.md # challenge rules for AI coding tools (official)
├── problem_statement.md # full challenge spec (official)
├── dataset/ # participant-facing data + media (official)
├── code/
│ ├── main.py # CLI entry point
│ ├── router/ # context · features · retrieval · policy · calibration · validation · perception
│ ├── evaluation/ # benchmark.py (gold + rivals) · ablations
│ └── tests/ # 79 tests across 11 files
├── scripts/ # profiling, validation, packaging
├── docs/ # architecture, safety benchmark, scouting, interview prep
└── output.csv # final predictions
- Deterministic and runnable offline — no API keys, no secrets, no network calls in the decision path.
- Submission frozen; official results announced August 7, 2026.