Skip to content
het-P301204Public

About

Security control failure & detection lab: an authorization fail-open, its exploit, root cause, fix, detection rule and regression test. Local red/blue lab, synthetic data only.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

BLACKOUT

Security Control Failure & Detection Lab

When the authorization service disappears, does your application deny the request — or quietly grant it?

A local red/blue lab for one vulnerability class: the security control that fails open. It ships the exploit, the root cause, the fix, the detection rule, and the regression test that keeps the fix honest.

CI tests python OWASP CWE license

Quick start · The finding · The fix · Detection · Tests · Docs


The same request, twice

   CONTROL ONLINE                             CONTROL OFFLINE
                                              (policy service times out)

   GET /api/admin/secrets                     GET /api/admin/secrets
   X-User: alice                              X-User: alice
            │                                          │
            ▼                                          ▼
      policy: DENY                              policy: ……… ?
            │                                          │
            ▼                                          ▼
       403 Forbidden                              200 OK
                                              BLACKOUT-DEMO-SECRET-001

Not one byte of the request changed. No payload, no injection, no encoding trick. The only thing that changed is that the service holding the authorization decision stopped answering — and the application decided that “I could not find out whether this is allowed” was close enough to “allowed.”

That is the entire vulnerability, and it is one except block long:

try:
    decision = await self.client.authorize(...)
except PolicyDependencyError as exc:
    # Availability fallback: the policy service being down should not
    # take the whole product down with it.
    log.warning("policy service unavailable; allowing request")
    return ALLOW          # ← the finding

Nothing about that code is exotic. It has no backdoor, no missing check, no inverted comparison. It was written to close out a previous incident — “the application must not hard-depend on the policy service” — and it did exactly that, while quietly making authorization optional.


See it happen

Three frames from the console. The request is identical in all three — only the dependency and the implementation change.


1 · Control online. The policy service answers. alice is not an admin, so she is refused. This is the baseline; without it the next frame proves nothing.

Control online: healthy authorize wire, and a 403 refused by policy


2 · The control disappears. The timeout breaker is tripped and the same button is pressed again. The authorize wire opens with an arc gap, the console drains from amber to red, and privileged data comes back to a user the policy explicitly denies.

Blackout: the authorize wire is broken and a 200 returns the admin secrets to alice


3 · The fix, under the same outage. Nothing is repaired — the breaker is still tripped. Only the implementation changed. UNKNOWN no longer routes to the happy path, so nothing is granted and the response says exactly why.

Failing closed: the same broken wire, and a 503 with no decision obtained


And all three in the trace log, folded from the security event stream — one user, one endpoint, three answers:

Trace log showing 403 healthy, 200 fail-open, and 503 fail-closed for the same request


The console is a substation panel for a single circuit — identity, application, decision point — wired to one power state. While the decision point answers, current runs along the authorize wire and the room is lit amber. When it stops answering, the room browns out.

One deliberate inversion: 200 is the red one. In this lab a refusal is the control working, so 403 reads mint and 503 reads amber. The colour follows the security meaning, not the HTTP class.

The whole console — full-page captures and mobile

The full console during a blackout

01 · Control online Healthy breaker, 403, no alerts, fail-open grants 0.
02 · Blackout Breaker tripped, same request, red 200, BLACKOUT-001 raised.
03 · Fail closed Nothing repaired — only the mode lever moved. Amber 503 and no new alert.
04 · Mobile The single-line diagram rotates vertical and keeps the arc gap.

Every capture, and how to regenerate it: evidence/SCREENSHOTS.md.

No installation required: open docs/console-preview.html in any browser. It is a self-contained copy of the real console — same markup, same stylesheet, same script — with the gateway calls stubbed by a small simulator, so the breakers, the mode lever and the request buttons all work offline.


Why this class deserves a lab

Most application security exercises need the attacker to send something: a quote, a ../, a script tag, a serialised object. This one does not.

It has no signature There is nothing in the request to block. A WAF, an input filter and a schema validator all pass it, because it is a valid request.
It arrives with an incident The bypass window opens exactly when everyone is staring at availability dashboards and paging each other. Nobody is reading access logs.
It is written by good intentions The code that causes it is usually the remediation for a previous outage. That is what makes it survive code review.
The bar is “slow”, not “down” An attacker does not need to kill the policy service. Adding POLICY_TIMEOUT seconds of latency is enough — and latency is far easier to cause than an outage.

Quick start

Prerequisites: Python 3.11+ and nothing else. No database, no broker, no cloud account, no internet access at runtime.

git clone https://github.com/het-P301204/BlackOut.git
cd BlackOut
make install
make demo

make demo is the whole story in 19 steps and about 57 seconds, no interaction: it starts four services, proves the vulnerability, prints the correlated telemetry, runs the detector, applies the fix while the dependency is still broken, proves the fix, restores the lab, and finishes by running the security regression tests. Every number it prints comes from a real response or a real log line.

make run          # four services + console on http://127.0.0.1:9000
make scenarios    # every failure mode × both implementations, with assertions
make test         # 230 tests
make help         # everything else
Windows, or any shell without make

Every make target has an identical PowerShell target:

./make.ps1 install
./make.ps1 demo
./make.ps1 help      # lists them all

Or run the commands directly — each target is a one-liner:

py -m venv .venv
.venv/Scripts/python -m pip install -r requirements.txt -r requirements-dev.txt
.venv/Scripts/python scripts/demo.py

On Linux/macOS use python3 -m venv .venv and .venv/bin/python.

Docker
make docker-up      # console: http://127.0.0.1:9000
make docker-logs
make docker-down

Four services, one image, health checks on all of them, non-root user. Only ports 9000, 9001, 9003, 9004 are published and every one is bound to 127.0.0.1; the policy data plane is reachable only from the API over the compose network.

Docker is not installed on the machine this lab was developed on, so the compose file is structurally validated but has not been run. See limitations.

Drive it by hand with curl
# 1. the control works
curl -i -H "X-User: alice" http://127.0.0.1:9001/api/admin/secrets     # 403

# 2. break the dependency
curl -X POST http://127.0.0.1:9003/control/failure-mode \
     -H 'Content-Type: application/json' -d '{"mode":"timeout"}'

# 3. exactly the same request
curl -i -H "X-User: alice" http://127.0.0.1:9001/api/admin/secrets     # 200

# 4. ask the detector what it thinks
curl -s http://127.0.0.1:9004/alerts | python -m json.tool

# 5. switch to the fixed implementation, leave the dependency broken
curl -X POST http://127.0.0.1:9001/control/mode \
     -H 'Content-Type: application/json' -d '{"mode":"secure"}'

# 6. same request again
curl -i -H "X-User: alice" http://127.0.0.1:9001/api/admin/secrets     # 503

The complete lifecycle

BLACKOUT is built as one full pass around a security engineering loop, and every stage ships as something you can run rather than something you can read.

Stage What it is here Where
Break Six real dependency failure modes on a fault-injection control plane policy_service/ · exploit.md
Prove Reproducible evidence packages, one per case, machine-readable evidence/
Understand The two-state decision model, argued rather than asserted root-cause.md
Fix Four states, targeted handlers, honest status codes api/app/secure/ · remediation.md
Detect BLACKOUT-001, with seven false-positive cases and a demonstrated blind spot detector/ · detection.md
Test 230 tests, including the invariant across every failure mode tests/
Document Threat model, architecture, limitations, and an AppSec-style report docs/ · report

Architecture

flowchart LR
    C["CLIENT<br/>X-User: alice"] -->|GET /api/admin/secrets| A

    subgraph API["BLACKOUT API :9001"]
        A["route handler"] --> V["vulnerable<br/>authorizer"]
        A --> S["secure<br/>authorizer"]
    end

    V -->|POST /v1/authorize| P
    S -->|POST /v1/authorize| P

    subgraph POL["POLICY SERVICE"]
        P["data plane :9002<br/>ALLOW / DENY"]
        F["control plane :9003<br/>fault injection"]
    end

    API -->|"JSON events<br/>(one trace_id)"| L[("security-events.jsonl")]
    P --> L
    L --> D["DETECTOR :9004<br/>BLACKOUT-001"]
    D --> DASH["CONSOLE :9000"]
    API --> DASH
    F --> DASH
Loading
Component Port Responsibility
blackout-api 9001 The application under test. Holds both the vulnerable and the secure authorization implementation behind one interface.
policy-service 9002 data · 9003 control External decision point. Returns ALLOW/DENY, and can be broken in six ways on demand.
detector 9004 Tails the security event log, correlates by trace ID, raises BLACKOUT-001.
console 9000 Operator UI and single-origin gateway. The only port you actually need.

Two design notes that carry weight:

  • The fault injector lives on its own listener. connection_refused genuinely closes the authorize socket, so a lab you can no longer control after breaking it would be useless. Control plane and data plane are separate servers.
  • Both implementations share one policy client. They receive identical inputs — the same validated decision on success, the same exception on failure. That removes every explanation for the two different outcomes except the except block itself.

Full detail, including trace propagation and the telemetry schema: docs/architecture.md.


The six failure modes

Every one is real. Nothing here returns a canned error string, because the whole finding is about what the caller's exception handling does — and you cannot exercise exception handling without producing exceptions.

curl -X POST http://127.0.0.1:9003/control/failure-mode \
     -H 'Content-Type: application/json' -d '{"mode":"timeout"}'
Mode What actually happens Vulnerable Secure
healthy Decisions are returned normally 403 / 200 403 / 200
timeout The handler really sleeps past the caller's budget 200 503
connection_refused The listening socket is really closed 200 503
malformed HTTP 200 whose body contains no decision 200 503
http_500 The decision point returns a real 500 200 503
disabled Provider is reachable and configured, but not enabled 200 503

Five different faults — transport, protocol, schema, state — and one identical outcome in the vulnerable implementation: privileged data returned to a user the policy explicitly denies.

$ make scenarios

[2/17] TIMEOUT
      vulnerable alice /api/admin/secrets
      -> HTTP 200   privileged data returned
      authorization_state=UNKNOWN  trace=8ab6e2873306
      BLACKOUT-001 raised  (timeout)
      PASS   fail-open: no decision, access granted

  scenarios   18
  passed      18
  failed      0
  alerts      5 BLACKOUT-001

The fix

The implementation modelled authorization as a boolean, so a dependency failure had nowhere to go. Under availability pressure it was routed to ALLOW.

A remote authorization decision has four outcomes, and the last two are not opinions about the subject at all — they are statements about our own ignorance:

ALLOW     the policy engine authorized this subject        →  grant
DENY      the policy engine refused this subject           →  refuse
UNKNOWN   no decision was obtained                         →  refuse
ERROR     our own decision path failed while finding out   →  refuse

The invariant lives in exactly one place, so there is one line to audit:

class AuthorizationState(StrEnum):
    ALLOW = "ALLOW"
    DENY = "DENY"
    UNKNOWN = "UNKNOWN"
    ERROR = "ERROR"

    @property
    def grants_access(self) -> bool:
        return self is AuthorizationState.ALLOW

…and each dependency failure is handled by name, at the security decision boundary:

except PolicyProviderDisabled as exc:   # reachable, but not enabled
    return self._deny_on_failure(...)
except PolicyInvalidResponse as exc:    # answered, but not with a decision
    return self._deny_on_failure(...)
except PolicyTimeout as exc:            # took the question, never answered
    return self._deny_on_failure(...)
except PolicyUnreachable as exc:        # nothing accepted the connection
    return self._deny_on_failure(...)
except PolicyUpstreamError as exc:      # answered with an error status
    return self._deny_on_failure(...)
except PolicyDependencyError as exc:    # a failure type added later — still denies
    return self._deny_on_failure(...)

All six deny — the security answer is identical — but each emits a distinct event, so “our policy engine is throwing 500s” stays distinguishable from “someone disabled the provider” at 3am. The base-class clause is the safety net: a failure type added next quarter denies by default instead of depending on someone remembering this file.

Why 503 and not 403 when no decision was obtained

A 403 asserts something untrue — that authorization was evaluated and the subject was refused. That lie is not academic:

  • it tells the caller “you will never be allowed to do this”, when they may well be;
  • it makes Retry-After meaningless, so well-behaved clients cannot back off and recover;
  • it hides a control-plane outage inside the denial rate, which is the single most useful signal for telling “your users are being denied” apart from “your authorization system is broken”;
  • it under-reports the incident. A fail-closed system genuinely is unavailable during a decision-point outage, and saying so is how that cost gets budgeted and engineered away instead of hidden by an except block.
The availability trade-off, honestly
                 Security        Availability during dependency failure
fail-open        lower           higher
fail-closed      higher          lower

Failing closed means returning 503 to everyone, including entitled users, for the duration of a policy outage. That is a real availability incident, and pretending otherwise is how teams get talked back into fail-open. It is still the right trade: the availability incident is visible, bounded and recoverable; the bypass is invisible, unbounded, and irreversible once the data has left.

Buy the availability back with mechanisms whose security properties can be reviewed one at a time — dependency redundancy, decision caching with bounded staleness, circuit breakers, an SLO with a named owner, a runbook that says “503 during a policy outage is by design” — and, if genuinely required, break-glass access that is explicitly enabled, separately authenticated, audited, time limited, tightly scoped and loud.

A fail-open is a break-glass mechanism with every safeguard removed and access granted to everybody instead of one accountable person.

Full treatment: docs/remediation.md.


Detection — BLACKOUT-001

CRITICAL — An authorization dependency failure and a successful privileged response inside the same request trace.

IF    an authorization-failure event exists
        (dependency_error | invalid_response | provider_disabled | internal_error)
AND   a privileged_response event exists
AND   both carry the same trace_id
AND   that response is 2xx
AND   its route is a privileged route
THEN  raise BLACKOUT-001 (CRITICAL)

What it deliberately does not alert on:

  • a dependency failure alone — a timeout followed by a fail-closed 503 is a correctly handled outage, and paging on it trains people to ignore the alert;
  • a privileged success alone — that is Tuesday;
  • a fail-open on a non-privileged route — /api/profile fails open too, and it crosses no authorization boundary.

The conjunction is an event a correct implementation cannot produce, so there is no threshold to tune and no baseline to learn: one occurrence is a finding. The trace ID is what makes it provable rather than circumstantial.

BLACKOUT-001 critical alert with trace, user, endpoint, failure type and response

Every field in that card is evidence, not decoration: the trace ties the two events together, failure_type says which fault produced it, and authorization: UNKNOWN next to response: 200 is the contradiction itself.

The detector is a pure function of the event log, so any captured evidence can be re-judged with nothing running:

python -m detector --once --events evidence/fail-open-timeout/events.jsonl
The blind spot — demonstrated, not hand-waved

If a fallback path emits no telemetry, there is nothing to correlate and the rule cannot fire. The detector does not infer a fail-open from the absence of a decision event, because inferring from absence would turn every dropped log line into a critical alert.

Set VULN_EMIT_FALLBACK_TELEMETRY=false and the attacker still gets the data while the detector stays silent — it is the last scenario in make scenarios:

[18/18] TIMEOUT + SILENT FALLBACK (vulnerable, no telemetry)
      -> HTTP 200   privileged data returned
      detector silent
      PASS   detector cannot correlate an event nobody emitted

The conclusion is a design requirement, not a caveat:

Security fallback paths must emit distinguishable telemetry. Observability is part of the control, not reporting about it.


Tests

make test              # 230 tests
make unit-test         # 114  no services required
make integration-test  #  47  real policy service, real injected faults
make security-test     #  33  the invariant
make detector-test     #  36  BLACKOUT-001 true and false positives

The one that matters most:

# tests/security/test_authorization_failure_never_grants_access.py

@pytest.mark.parametrize("failure_mode", ["timeout", "connection_refused",
                                          "malformed", "http_500", "disabled"])
@pytest.mark.parametrize("subject", ["alice", "bob", "admin"])
def test_authorization_failure_never_grants_access(...):
    """No dependency failure grants privileged access — to anyone."""
    response = client.get("/api/admin/secrets", headers={"X-User": subject})
    assert response.status_code == 503
    assert response.headers["X-Authorization-State"] == "UNKNOWN"
    assert "BLACKOUT-DEMO-SECRET-001" not in response.text

admin is in that list on purpose. During a policy outage the application does not know that admin is an admin. A fix that special-cases known privileged users is not a fix; it is the same bug with a smaller blast radius.

Two more that keep the lab honest:

  • test_healthy_dependency_still_enforces_the_policy — proves the fix is not simply “deny everything”, which would pass every other security test.
  • test_the_exploit_is_real_in_the_vulnerable_implementation — if someone “tidies up” the vulnerable authorizer, the demonstration silently becomes a no-op and every other test still passes. This one fails instead.

Evidence

make demo writes a reproducible evidence package. Each directory holds evidence.json (machine-readable), evidence.txt (what you paste into a report) and events.jsonl (the raw correlated events).

evidence/
├── healthy-deny/          control online, alice denied            403
├── healthy-allow/         control online, admin allowed           200
├── fail-open-timeout/     control offline, alice granted          200   ← the finding
├── fail-open-refused/     socket closed, alice granted            200
├── fail-open-malformed/   no decision in body, alice granted      200
├── fail-open-500/         decision point 500, alice granted       200
├── disabled-provider/     provider not enabled, alice granted     200
├── secure-timeout/        same outage, fixed code                 503
├── normal-traffic/        control restored                        403
└── detector-alert/        BLACKOUT-001 as raised
TRACE:          8ab6e287330649c1a2caa41a6bf6dfab
REQUEST:        GET /api/admin/secrets   (X-User: alice)
POLICY:         TIMEOUT
AUTHORIZATION:  UNKNOWN
RESPONSE:       200 OK
ALERT:          BLACKOUT-001 (CRITICAL)

Standards mapping

Each row is argued in docs/root-cause.md, because a mapping without an argument is decoration.

Standard Why it applies
OWASP Top 10:2025 A10 — Mishandling of Exceptional Conditions The finding exists only on the exceptional path. Normal operation is correct; the defect is entirely in what happens when the expected path cannot complete.
CWE-636 — Not Failing Securely (“Failing Open”) The most precise mapping. Two behaviours are available on failure and the system reverts to the less secure one.
CWE-703 — Improper Check or Handling of Exceptional Conditions The parent weakness. The exception is checked — it is caught — and then handled incorrectly.
CWE-754 — Improper Check for Unusual or Exceptional Conditions Describes malformed and disabled exactly: a 200 arrives and the code never checks for the unusual condition that it contains no usable decision.
CWE-755 — Improper Handling of Exceptional Conditions Focused on the handler: it continues the request instead of terminating it.

Deliberately not claimed: CWE-285 / CWE-862 (Improper / Missing Authorization). The check is present, correct, bound to the subject, and enforced on every request where it can run. Filing this under “missing authorization” would describe a different bug and hide what is interesting about this one.

No CVE is reproduced or claimed. Real-world parallels — such as OCSP soft-fail, a documented browser design decision made for exactly these availability reasons — are discussed as conceptual parallels only.


What this is, and what it is not

It isIt is not
  • A local lab you can run in 60 seconds
  • One vulnerability class, done end to end
  • Real faults: real timeouts, a really closed socket
  • Honest about every simplification
  • Not a scanner or a product
  • Not a reproduction of any CVE
  • Not safe to expose to a network
  • Not an unauthenticated bypass — it is privilege escalation for an authenticated user

Known limits, stated plainly:

  • Authentication is a single X-User header. It is a stub, and labelled as one.
  • POST /control/mode switches the authorization implementation at runtime. In a real system that would be a critical finding by itself; it exists so the demo can change the code path during an outage, and it is gated by ALLOW_RUNTIME_MODE_SWITCH=false.
  • The policy engine is a dictionary lookup. The lab is about what happens when the rules cannot be consulted, not about the rules.
  • Docker Compose and the Makefile were validated but not executed on the development machine (neither docker nor make is installed there); the PowerShell shim and every underlying Python command were.

The complete list, including the detection blind spot and one environment-specific networking quirk: docs/limitations.md.


Repository map

api/app/                the application under test
├── policy/             models.py (4-state lattice) · errors.py (taxonomy) · client.py
├── vulnerable/         authorizer.py    ← the finding, one except block
├── secure/             authorizer.py    ← the fix
└── routes/             admin · health · control · meta
policy_service/         decision point + six real failure modes, two listeners
detector/               rules.py (BLACKOUT-001) · tail.py · service.py · CLI
dashboard/              single-origin gateway + the console (no build step)
common/                 JSONL event emitter · counters · trace IDs
scripts/                demo.py · run_scenarios.py · serve.py · lab.py · build_preview.py
tests/                  unit · integration · security · detector
docs/                   architecture · threat model · exploit · root cause ·
                        remediation · detection · limitations · console-preview.html
report/                 BLACKOUT-report.md
evidence/               generated by `make demo`, plus committed screenshots

Documentation

Document Contents
architecture.md Components, ports, request flow, trace propagation, telemetry schema, why the control plane is separate
threat-model.md Assets, actor, preconditions, trust boundaries, what is in and out of scope
exploit.md The exploit step by step, with captured evidence
root-cause.md The two-state decision model, and the standards mapping argued rather than asserted
remediation.md The fix, status codes, targeted vs global exception handling, availability trade-off, break-glass
detection.md BLACKOUT-001, event schema, tuning, false-positive analysis, the blind spot
limitations.md What this lab does not prove
BLACKOUT-report.md The finding written up as a professional AppSec report
SECURITY.md Why this repo is intentionally vulnerable, and how to report a defect in the lab itself

References

  • OWASP Top 10:2025 — A10 Mishandling of Exceptional Conditions
  • MITRE CWE-636, CWE-703, CWE-754, CWE-755
  • RFC 9110 §15.5.4 (403 Forbidden) and §15.6.4 (503 Service Unavailable) — why “no decision” is not “forbidden”
  • OWASP Application Security Verification Standard — access control verification requirements

A security control that disappears during an exceptional condition is still a security vulnerability.

The safest answer to “I don’t know whether this request is authorized” is never “yes.”


UNKNOWN ≠ ALLOW  ·  ERROR ≠ ALLOW


MIT licensed · synthetic data only · no real secret exists anywhere in this repository

About

Security control failure & detection lab: an authorization fail-open, its exploit, root cause, fix, detection rule and regression test. Local red/blue lab, synthetic data only.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages