When the authorization service disappears, does your application deny the request — or quietly grant it?
A local red/blue lab for one vulnerability class: the security control that fails open. It ships the exploit, the root cause, the fix, the detection rule, and the regression test that keeps the fix honest.
Quick start · The finding · The fix · Detection · Tests · Docs
CONTROL ONLINE CONTROL OFFLINE
(policy service times out)
GET /api/admin/secrets GET /api/admin/secrets
X-User: alice X-User: alice
│ │
▼ ▼
policy: DENY policy: ……… ?
│ │
▼ ▼
403 Forbidden 200 OK
BLACKOUT-DEMO-SECRET-001
Not one byte of the request changed. No payload, no injection, no encoding trick. The only thing that changed is that the service holding the authorization decision stopped answering — and the application decided that “I could not find out whether this is allowed” was close enough to “allowed.”
That is the entire vulnerability, and it is one except block long:
try:
decision = await self.client.authorize(...)
except PolicyDependencyError as exc:
# Availability fallback: the policy service being down should not
# take the whole product down with it.
log.warning("policy service unavailable; allowing request")
return ALLOW # ← the findingNothing about that code is exotic. It has no backdoor, no missing check, no inverted comparison. It was written to close out a previous incident — “the application must not hard-depend on the policy service” — and it did exactly that, while quietly making authorization optional.
Three frames from the console. The request is identical in all three — only the dependency and the implementation change.
1 · Control online. The policy service answers. alice is not an admin, so she is
refused. This is the baseline; without it the next frame proves nothing.
2 · The control disappears. The timeout breaker is tripped and the same button is
pressed again. The authorize wire opens with an arc gap, the console drains from amber to
red, and privileged data comes back to a user the policy explicitly denies.
3 · The fix, under the same outage. Nothing is repaired — the breaker is still
tripped. Only the implementation changed. UNKNOWN no longer routes to the happy path,
so nothing is granted and the response says exactly why.
And all three in the trace log, folded from the security event stream — one user, one endpoint, three answers:
The console is a substation panel for a single circuit — identity, application, decision point — wired to one power state. While the decision point answers, current runs along the authorize wire and the room is lit amber. When it stops answering, the room browns out.
One deliberate inversion:
200is the red one. In this lab a refusal is the control working, so403reads mint and503reads amber. The colour follows the security meaning, not the HTTP class.
The whole console — full-page captures and mobile
| 01 · Control online | Healthy breaker, 403, no alerts, fail-open grants 0. |
| 02 · Blackout | Breaker tripped, same request, red 200, BLACKOUT-001 raised. |
| 03 · Fail closed | Nothing repaired — only the mode lever moved. Amber 503 and no new alert. |
| 04 · Mobile | The single-line diagram rotates vertical and keeps the arc gap. |
Every capture, and how to regenerate it: evidence/SCREENSHOTS.md.
No installation required: open docs/console-preview.html
in any browser. It is a self-contained copy of the real console — same markup, same
stylesheet, same script — with the gateway calls stubbed by a small simulator, so the
breakers, the mode lever and the request buttons all work offline.
Most application security exercises need the attacker to send something: a quote, a
../, a script tag, a serialised object. This one does not.
| It has no signature | There is nothing in the request to block. A WAF, an input filter and a schema validator all pass it, because it is a valid request. |
| It arrives with an incident | The bypass window opens exactly when everyone is staring at availability dashboards and paging each other. Nobody is reading access logs. |
| It is written by good intentions | The code that causes it is usually the remediation for a previous outage. That is what makes it survive code review. |
| The bar is “slow”, not “down” | An attacker does not need to kill the policy service. Adding POLICY_TIMEOUT seconds of latency is enough — and latency is far easier to cause than an outage. |
Prerequisites: Python 3.11+ and nothing else. No database, no broker, no cloud account, no internet access at runtime.
git clone https://github.com/het-P301204/BlackOut.git
cd BlackOut
make install
make demomake demo is the whole story in 19 steps and about 57 seconds, no interaction: it
starts four services, proves the vulnerability, prints the correlated telemetry, runs the
detector, applies the fix while the dependency is still broken, proves the fix,
restores the lab, and finishes by running the security regression tests. Every number it
prints comes from a real response or a real log line.
make run # four services + console on http://127.0.0.1:9000
make scenarios # every failure mode × both implementations, with assertions
make test # 230 tests
make help # everything elseWindows, or any shell without make
Every make target has an identical PowerShell target:
./make.ps1 install
./make.ps1 demo
./make.ps1 help # lists them allOr run the commands directly — each target is a one-liner:
py -m venv .venv
.venv/Scripts/python -m pip install -r requirements.txt -r requirements-dev.txt
.venv/Scripts/python scripts/demo.pyOn Linux/macOS use python3 -m venv .venv and .venv/bin/python.
Docker
make docker-up # console: http://127.0.0.1:9000
make docker-logs
make docker-downFour services, one image, health checks on all of them, non-root user. Only ports
9000, 9001, 9003, 9004 are published and every one is bound to 127.0.0.1; the
policy data plane is reachable only from the API over the compose network.
Docker is not installed on the machine this lab was developed on, so the compose file is structurally validated but has not been run. See limitations.
Drive it by hand with curl
# 1. the control works
curl -i -H "X-User: alice" http://127.0.0.1:9001/api/admin/secrets # 403
# 2. break the dependency
curl -X POST http://127.0.0.1:9003/control/failure-mode \
-H 'Content-Type: application/json' -d '{"mode":"timeout"}'
# 3. exactly the same request
curl -i -H "X-User: alice" http://127.0.0.1:9001/api/admin/secrets # 200
# 4. ask the detector what it thinks
curl -s http://127.0.0.1:9004/alerts | python -m json.tool
# 5. switch to the fixed implementation, leave the dependency broken
curl -X POST http://127.0.0.1:9001/control/mode \
-H 'Content-Type: application/json' -d '{"mode":"secure"}'
# 6. same request again
curl -i -H "X-User: alice" http://127.0.0.1:9001/api/admin/secrets # 503BLACKOUT is built as one full pass around a security engineering loop, and every stage ships as something you can run rather than something you can read.
| Stage | What it is here | Where |
|---|---|---|
| Break | Six real dependency failure modes on a fault-injection control plane | policy_service/ · exploit.md |
| Prove | Reproducible evidence packages, one per case, machine-readable | evidence/ |
| Understand | The two-state decision model, argued rather than asserted | root-cause.md |
| Fix | Four states, targeted handlers, honest status codes | api/app/secure/ · remediation.md |
| Detect | BLACKOUT-001, with seven false-positive cases and a demonstrated blind spot |
detector/ · detection.md |
| Test | 230 tests, including the invariant across every failure mode | tests/ |
| Document | Threat model, architecture, limitations, and an AppSec-style report | docs/ · report |
flowchart LR
C["CLIENT<br/>X-User: alice"] -->|GET /api/admin/secrets| A
subgraph API["BLACKOUT API :9001"]
A["route handler"] --> V["vulnerable<br/>authorizer"]
A --> S["secure<br/>authorizer"]
end
V -->|POST /v1/authorize| P
S -->|POST /v1/authorize| P
subgraph POL["POLICY SERVICE"]
P["data plane :9002<br/>ALLOW / DENY"]
F["control plane :9003<br/>fault injection"]
end
API -->|"JSON events<br/>(one trace_id)"| L[("security-events.jsonl")]
P --> L
L --> D["DETECTOR :9004<br/>BLACKOUT-001"]
D --> DASH["CONSOLE :9000"]
API --> DASH
F --> DASH
| Component | Port | Responsibility |
|---|---|---|
| blackout-api | 9001 | The application under test. Holds both the vulnerable and the secure authorization implementation behind one interface. |
| policy-service | 9002 data · 9003 control | External decision point. Returns ALLOW/DENY, and can be broken in six ways on demand. |
| detector | 9004 | Tails the security event log, correlates by trace ID, raises BLACKOUT-001. |
| console | 9000 | Operator UI and single-origin gateway. The only port you actually need. |
Two design notes that carry weight:
- The fault injector lives on its own listener.
connection_refusedgenuinely closes the authorize socket, so a lab you can no longer control after breaking it would be useless. Control plane and data plane are separate servers. - Both implementations share one policy client. They receive identical inputs — the
same validated decision on success, the same exception on failure. That removes every
explanation for the two different outcomes except the
exceptblock itself.
Full detail, including trace propagation and the telemetry schema: docs/architecture.md.
Every one is real. Nothing here returns a canned error string, because the whole finding is about what the caller's exception handling does — and you cannot exercise exception handling without producing exceptions.
curl -X POST http://127.0.0.1:9003/control/failure-mode \
-H 'Content-Type: application/json' -d '{"mode":"timeout"}'| Mode | What actually happens | Vulnerable | Secure |
|---|---|---|---|
healthy |
Decisions are returned normally | 403 / 200 | 403 / 200 |
timeout |
The handler really sleeps past the caller's budget | 200 | 503 |
connection_refused |
The listening socket is really closed | 200 | 503 |
malformed |
HTTP 200 whose body contains no decision | 200 | 503 |
http_500 |
The decision point returns a real 500 | 200 | 503 |
disabled |
Provider is reachable and configured, but not enabled | 200 | 503 |
Five different faults — transport, protocol, schema, state — and one identical outcome in the vulnerable implementation: privileged data returned to a user the policy explicitly denies.
$ make scenarios
[2/17] TIMEOUT
vulnerable alice /api/admin/secrets
-> HTTP 200 privileged data returned
authorization_state=UNKNOWN trace=8ab6e2873306
BLACKOUT-001 raised (timeout)
PASS fail-open: no decision, access granted
scenarios 18
passed 18
failed 0
alerts 5 BLACKOUT-001
The implementation modelled authorization as a boolean, so a dependency failure had
nowhere to go. Under availability pressure it was routed to ALLOW.
A remote authorization decision has four outcomes, and the last two are not opinions about the subject at all — they are statements about our own ignorance:
ALLOW the policy engine authorized this subject → grant
DENY the policy engine refused this subject → refuse
UNKNOWN no decision was obtained → refuse
ERROR our own decision path failed while finding out → refuse
The invariant lives in exactly one place, so there is one line to audit:
class AuthorizationState(StrEnum):
ALLOW = "ALLOW"
DENY = "DENY"
UNKNOWN = "UNKNOWN"
ERROR = "ERROR"
@property
def grants_access(self) -> bool:
return self is AuthorizationState.ALLOW…and each dependency failure is handled by name, at the security decision boundary:
except PolicyProviderDisabled as exc: # reachable, but not enabled
return self._deny_on_failure(...)
except PolicyInvalidResponse as exc: # answered, but not with a decision
return self._deny_on_failure(...)
except PolicyTimeout as exc: # took the question, never answered
return self._deny_on_failure(...)
except PolicyUnreachable as exc: # nothing accepted the connection
return self._deny_on_failure(...)
except PolicyUpstreamError as exc: # answered with an error status
return self._deny_on_failure(...)
except PolicyDependencyError as exc: # a failure type added later — still denies
return self._deny_on_failure(...)All six deny — the security answer is identical — but each emits a distinct event, so “our policy engine is throwing 500s” stays distinguishable from “someone disabled the provider” at 3am. The base-class clause is the safety net: a failure type added next quarter denies by default instead of depending on someone remembering this file.
Why 503 and not 403 when no decision was obtained
A 403 asserts something untrue — that authorization was evaluated and the subject was
refused. That lie is not academic:
- it tells the caller “you will never be allowed to do this”, when they may well be;
- it makes
Retry-Aftermeaningless, so well-behaved clients cannot back off and recover; - it hides a control-plane outage inside the denial rate, which is the single most useful signal for telling “your users are being denied” apart from “your authorization system is broken”;
- it under-reports the incident. A fail-closed system genuinely is unavailable during a
decision-point outage, and saying so is how that cost gets budgeted and engineered away
instead of hidden by an
exceptblock.
The availability trade-off, honestly
Security Availability during dependency failure
fail-open lower higher
fail-closed higher lower
Failing closed means returning 503 to everyone, including entitled users, for the
duration of a policy outage. That is a real availability incident, and pretending
otherwise is how teams get talked back into fail-open. It is still the right trade: the
availability incident is visible, bounded and recoverable; the bypass is invisible,
unbounded, and irreversible once the data has left.
Buy the availability back with mechanisms whose security properties can be reviewed one at a time — dependency redundancy, decision caching with bounded staleness, circuit breakers, an SLO with a named owner, a runbook that says “503 during a policy outage is by design” — and, if genuinely required, break-glass access that is explicitly enabled, separately authenticated, audited, time limited, tightly scoped and loud.
A fail-open is a break-glass mechanism with every safeguard removed and access granted to everybody instead of one accountable person.
Full treatment: docs/remediation.md.
CRITICAL — An authorization dependency failure and a successful privileged response inside the same request trace.
IF an authorization-failure event exists
(dependency_error | invalid_response | provider_disabled | internal_error)
AND a privileged_response event exists
AND both carry the same trace_id
AND that response is 2xx
AND its route is a privileged route
THEN raise BLACKOUT-001 (CRITICAL)
What it deliberately does not alert on:
- a dependency failure alone — a timeout followed by a fail-closed
503is a correctly handled outage, and paging on it trains people to ignore the alert; - a privileged success alone — that is Tuesday;
- a fail-open on a non-privileged route —
/api/profilefails open too, and it crosses no authorization boundary.
The conjunction is an event a correct implementation cannot produce, so there is no threshold to tune and no baseline to learn: one occurrence is a finding. The trace ID is what makes it provable rather than circumstantial.
Every field in that card is evidence, not decoration: the trace ties the two events
together, failure_type says which fault produced it, and authorization: UNKNOWN next
to response: 200 is the contradiction itself.
The detector is a pure function of the event log, so any captured evidence can be re-judged with nothing running:
python -m detector --once --events evidence/fail-open-timeout/events.jsonlThe blind spot — demonstrated, not hand-waved
If a fallback path emits no telemetry, there is nothing to correlate and the rule cannot fire. The detector does not infer a fail-open from the absence of a decision event, because inferring from absence would turn every dropped log line into a critical alert.
Set VULN_EMIT_FALLBACK_TELEMETRY=false and the attacker still gets the data while the
detector stays silent — it is the last scenario in make scenarios:
[18/18] TIMEOUT + SILENT FALLBACK (vulnerable, no telemetry)
-> HTTP 200 privileged data returned
detector silent
PASS detector cannot correlate an event nobody emitted
The conclusion is a design requirement, not a caveat:
Security fallback paths must emit distinguishable telemetry. Observability is part of the control, not reporting about it.
make test # 230 tests
make unit-test # 114 no services required
make integration-test # 47 real policy service, real injected faults
make security-test # 33 the invariant
make detector-test # 36 BLACKOUT-001 true and false positivesThe one that matters most:
# tests/security/test_authorization_failure_never_grants_access.py
@pytest.mark.parametrize("failure_mode", ["timeout", "connection_refused",
"malformed", "http_500", "disabled"])
@pytest.mark.parametrize("subject", ["alice", "bob", "admin"])
def test_authorization_failure_never_grants_access(...):
"""No dependency failure grants privileged access — to anyone."""
response = client.get("/api/admin/secrets", headers={"X-User": subject})
assert response.status_code == 503
assert response.headers["X-Authorization-State"] == "UNKNOWN"
assert "BLACKOUT-DEMO-SECRET-001" not in response.textadmin is in that list on purpose. During a policy outage the application does not
know that admin is an admin. A fix that special-cases known privileged users is not a
fix; it is the same bug with a smaller blast radius.
Two more that keep the lab honest:
test_healthy_dependency_still_enforces_the_policy— proves the fix is not simply “deny everything”, which would pass every other security test.test_the_exploit_is_real_in_the_vulnerable_implementation— if someone “tidies up” the vulnerable authorizer, the demonstration silently becomes a no-op and every other test still passes. This one fails instead.
make demo writes a reproducible evidence package. Each directory holds
evidence.json (machine-readable), evidence.txt (what you paste into a report) and
events.jsonl (the raw correlated events).
evidence/
├── healthy-deny/ control online, alice denied 403
├── healthy-allow/ control online, admin allowed 200
├── fail-open-timeout/ control offline, alice granted 200 ← the finding
├── fail-open-refused/ socket closed, alice granted 200
├── fail-open-malformed/ no decision in body, alice granted 200
├── fail-open-500/ decision point 500, alice granted 200
├── disabled-provider/ provider not enabled, alice granted 200
├── secure-timeout/ same outage, fixed code 503
├── normal-traffic/ control restored 403
└── detector-alert/ BLACKOUT-001 as raised
TRACE: 8ab6e287330649c1a2caa41a6bf6dfab
REQUEST: GET /api/admin/secrets (X-User: alice)
POLICY: TIMEOUT
AUTHORIZATION: UNKNOWN
RESPONSE: 200 OK
ALERT: BLACKOUT-001 (CRITICAL)
Each row is argued in docs/root-cause.md, because a mapping without an argument is decoration.
| Standard | Why it applies |
|---|---|
| OWASP Top 10:2025 A10 — Mishandling of Exceptional Conditions | The finding exists only on the exceptional path. Normal operation is correct; the defect is entirely in what happens when the expected path cannot complete. |
| CWE-636 — Not Failing Securely (“Failing Open”) | The most precise mapping. Two behaviours are available on failure and the system reverts to the less secure one. |
| CWE-703 — Improper Check or Handling of Exceptional Conditions | The parent weakness. The exception is checked — it is caught — and then handled incorrectly. |
| CWE-754 — Improper Check for Unusual or Exceptional Conditions | Describes malformed and disabled exactly: a 200 arrives and the code never checks for the unusual condition that it contains no usable decision. |
| CWE-755 — Improper Handling of Exceptional Conditions | Focused on the handler: it continues the request instead of terminating it. |
Deliberately not claimed: CWE-285 / CWE-862 (Improper / Missing Authorization). The check is present, correct, bound to the subject, and enforced on every request where it can run. Filing this under “missing authorization” would describe a different bug and hide what is interesting about this one.
No CVE is reproduced or claimed. Real-world parallels — such as OCSP soft-fail, a documented browser design decision made for exactly these availability reasons — are discussed as conceptual parallels only.
| It is | It is not |
|---|---|
|
|
Known limits, stated plainly:
- Authentication is a single
X-Userheader. It is a stub, and labelled as one. POST /control/modeswitches the authorization implementation at runtime. In a real system that would be a critical finding by itself; it exists so the demo can change the code path during an outage, and it is gated byALLOW_RUNTIME_MODE_SWITCH=false.- The policy engine is a dictionary lookup. The lab is about what happens when the rules cannot be consulted, not about the rules.
- Docker Compose and the Makefile were validated but not executed on the development
machine (neither
dockernormakeis installed there); the PowerShell shim and every underlying Python command were.
The complete list, including the detection blind spot and one environment-specific networking quirk: docs/limitations.md.
api/app/ the application under test
├── policy/ models.py (4-state lattice) · errors.py (taxonomy) · client.py
├── vulnerable/ authorizer.py ← the finding, one except block
├── secure/ authorizer.py ← the fix
└── routes/ admin · health · control · meta
policy_service/ decision point + six real failure modes, two listeners
detector/ rules.py (BLACKOUT-001) · tail.py · service.py · CLI
dashboard/ single-origin gateway + the console (no build step)
common/ JSONL event emitter · counters · trace IDs
scripts/ demo.py · run_scenarios.py · serve.py · lab.py · build_preview.py
tests/ unit · integration · security · detector
docs/ architecture · threat model · exploit · root cause ·
remediation · detection · limitations · console-preview.html
report/ BLACKOUT-report.md
evidence/ generated by `make demo`, plus committed screenshots
| Document | Contents |
|---|---|
| architecture.md | Components, ports, request flow, trace propagation, telemetry schema, why the control plane is separate |
| threat-model.md | Assets, actor, preconditions, trust boundaries, what is in and out of scope |
| exploit.md | The exploit step by step, with captured evidence |
| root-cause.md | The two-state decision model, and the standards mapping argued rather than asserted |
| remediation.md | The fix, status codes, targeted vs global exception handling, availability trade-off, break-glass |
| detection.md | BLACKOUT-001, event schema, tuning, false-positive analysis, the blind spot |
| limitations.md | What this lab does not prove |
| BLACKOUT-report.md | The finding written up as a professional AppSec report |
| SECURITY.md | Why this repo is intentionally vulnerable, and how to report a defect in the lab itself |
- OWASP Top 10:2025 — A10 Mishandling of Exceptional Conditions
- MITRE CWE-636, CWE-703, CWE-754, CWE-755
- RFC 9110 §15.5.4 (403 Forbidden) and §15.6.4 (503 Service Unavailable) — why “no decision” is not “forbidden”
- OWASP Application Security Verification Standard — access control verification requirements
A security control that disappears during an exceptional condition is still a security vulnerability.
The safest answer to “I don’t know whether this request is authorized” is never “yes.”
UNKNOWN ≠ ALLOW · ERROR ≠ ALLOW
MIT licensed · synthetic data only · no real secret exists anywhere in this repository





