bulwark

A local-first LLM guardrails toolkit. Wrap any LLM call in a layered defense, then measure how well each guardrail works against a labeled dataset.

No API key required for the core: every built-in validator is rule- or schema-based, so it runs offline, in CI, and at low latency. The optional LLM-as-judge layer is additive, not a dependency.

from bulwark import Guard, InjectionValidator, PIIValidator

guard = Guard([InjectionValidator(), PIIValidator()])
result = guard.check("Ignore previous instructions. Email me at jane@corp.com.")

result.blocked      # True  -> prompt injection detected
result.text         # "Ignore previous instructions. Email me at [REDACTED_EMAIL]."
result.findings     # structured, explainable reasons for every action

Validating structured output against a real schema works the same way:

from bulwark import Guard, JSONSchemaValidator

schema = {
    "type": "object",
    "required": ["name", "age"],
    "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
}
guard = Guard([JSONSchemaValidator(schema=schema)])

guard.check('{"name": "Ada", "age": 36}').passed    # True
guard.check('{"name": "Ada"}').findings[0].message
# "output does not match schema: 'age' is a required property"

Why

Deploying an LLM is easy; containing one is the hard part that shows up in production. bulwark is a small, transparent baseline for the guardrails teams reach for first:

Validator	Guards	Default action
PIIValidator	emails, phones, SSNs, credit cards (Luhn-checked), IPs, IBANs (mod-97 checked), passports	redact
InjectionValidator	prompt-injection / jailbreak attempts	block
SecretsValidator	leaked API keys & tokens (AWS, OpenAI, GitHub, Slack, Google, PEM keys)	block
JSONSchemaValidator	malformed / off-schema structured output	block
ToxicityValidator	abusive language (threats, insults, profanity) via a curated lexicon	block
LengthValidator	output outside a configured character or word count	block
TopicValidator	denylisted topics, or (allow mode) anything outside an allowlist	block

Guardrails are only as good as their measured error rate, so bulwark ships an eval harness rather than asking you to trust the rules. See the full API in docs/validators.md, one runnable snippet per validator.

Install

pip install -e ".[dev]"      # from source

Evaluate

Every detector is scored against a labeled dataset with precision / recall / F1 and a false-positive rate, the number that actually matters in production. See docs/evaluation.md for what each of those numbers means and why some evals score lower than others on purpose.

python -m evals.eval_injection
# n=20  TP=9 FP=1 TN=9 FN=1
# precision=0.900  recall=0.900  f1=0.900  false-positive-rate=0.100

python -m evals.eval_pii
# n=28  TP=15 FP=1 TN=10 FN=2
# precision=0.938  recall=0.882  f1=0.909  false-positive-rate=0.091

python -m evals.eval_combined
# end-to-end: precision=1.000 recall=1.000 f1=1.000
# plus a per-validator catch-rate breakdown

Those numbers are the honest current baseline, not a hand-picked best case, eval_pii.py's dataset includes real failure modes (an intl phone format the North-American-only regex misses, a bare 9-digit ID that collides with the passport pattern's shape). Improving them without inflating false positives is tracked in ROADMAP.md. eval_combined.py's job is different: it checks the validators compose correctly as a pipeline, not re-measuring what the per-validator evals already cover.

Design

                    ┌─────────────────────────────────┐
   text  ─────────▶ │              Guard               │ ─────────▶ GuardResult
                    │  ┌───────────┐  ┌───────────┐    │            .text (redacted)
                    │  │Validator 1│  │Validator 2│ …  │            .findings
                    │  └─────┬─────┘  └─────┬─────┘    │            .passed / .blocked
                    │        └──────┬───────┘          │            .by_validator(name)
                    │       apply redactions            │
                    │     (one place, right-to-left)    │
                    └─────────────────────────────────┘

• Validators are pure and declarative. They never mutate input; a finding describes what to do (ALLOW / REDACT / BLOCK) and, for redactions, the span and replacement. The Guard applies transforms in one place, so overlapping spans and ordering are handled correctly once.
• Composable. Guard([...]) runs validators as a pipeline; guard.check returns a single GuardResult you can branch on (result.passed, result.redacted, result.by_validator("pii")).
• Explainable. Every action carries the pattern/reason that produced it, so false positives are debuggable, not mysterious.

Writing a custom validator

A validator is a class with a name and a check(text) -> list[Finding] method, that's the entire contract, and the built-ins follow it too:

from bulwark import Action, Finding, Guard, Severity, Validator

class BannedPhraseValidator(Validator):
    name = "banned_phrase"

    def __init__(self, phrases: list[str]) -> None:
        self.phrases = phrases

    def check(self, text: str) -> list[Finding]:
        return [
            Finding(validator=self.name, action=Action.BLOCK,
                    message=f"banned phrase: {phrase!r}", severity=Severity.HIGH)
            for phrase in self.phrases if phrase in text
        ]

result = Guard([BannedPhraseValidator(["guaranteed returns"])]).check(
    "This fund offers guaranteed returns."
)
result.blocked   # True

See examples/custom_validator.py for a fuller, runnable version with redaction and composition with a built-in validator.

Threat model & scope

bulwark is a rule-based, local-first baseline. What that means in practice:

• What it catches: the common, pattern-shaped cases, well-formed emails and phone numbers, known secret-key formats, the obvious prompt-injection phrasings, structurally invalid JSON. Every claim is backed by a measured eval, not a vibe, see the numbers above.
• What it does not catch: novel or heavily obfuscated prompt-injection phrasing the pattern library hasn't seen, PII in formats outside what's implemented (the phone pattern is North-American only, by design, not an oversight), secrets in formats not in SecretsValidator's pattern list, or anything that requires actual semantic understanding rather than pattern matching. The ToxicityValidator is an explicit baseline, not a state-of-the-art classifier, see its own docstring.
• Known ambiguities, documented not hidden: the passport pattern matches any 9-digit string, which means it can't distinguish a real passport number from an unrelated 9-digit ID by shape alone. That tradeoff is in the code comment, the eval dataset, and here.
• Not a replacement for a real content-moderation or DLP system in anything handling genuinely sensitive data at scale. It's a fast, offline, explainable first layer, use it as one.

Testing

pytest -q        # 206 tests, runs in <0.1s, no network

Contributing

See CONTRIBUTING.md for the project's ground rules (zero core dependencies, the validator contract, how to add a validator with a real eval). CHANGELOG.md tracks what shipped when.

Roadmap

See ROADMAP.md. Near-term: growing the injection dataset with genuinely hard adversarial cases, and an LLMJudgeValidator for checks that need real semantic understanding.

License

MIT