Add E2E story coverage for the fail2ban action family

[vladimirrott]

Half of this landed in #401 on 2026-09-15. Recounted at b2c823e6 on 2026-09-16.
The four stories exist. What the issue still asks for is the live Ubuntu pass
behind them, and no cassette carries a fail2ban exchange:

$ ls tests/e2e/stories/story-13[4-7].sh
tests/e2e/stories/story-134.sh
tests/e2e/stories/story-135.sh
tests/e2e/stories/story-136.sh
tests/e2e/stories/story-137.sh
$ for f in tests/e2e/cassettes/*.json; do printf '%-45s fail2ban_hits=%s\n' "$(basename $f)" "$(grep -oic fail2ban "$f")"; done
ubuntu-jammy-gpt-oss-120b.json                fail2ban_hits=0
ubuntu-noble-gpt-oss-120b.json                fail2ban_hits=0
ubuntu-resolute-gpt-oss-120b.json             fail2ban_hits=0

The paragraph below is how the tree looked when I filed this, kept so the
thread reads in order. Read the Scope section for what is left.

As filed, 2026-08-17. The fail2ban action family has no end-to-end story. ConfigureFail2banJail, Fail2banBanIp, Fail2banUnbanIp and Fail2banStatus are all absent from tests/e2e/stories/, and grep -li fail2ban tests/e2e/stories/ returns nothing.

The construction path is well covered: crates/sysknife-daemon/src/actions/fail2ban.rs carries 20 unit tests, including the IpAddr::from_str validation that rejects a malformed address before the ActionSpec is built. What is missing is the other half, proof that the planner selects these actions from a natural-language intent on a real Ubuntu host.

Why it is worth doing

fail2ban is a security action family, so the planner picking the wrong action here has a sharper cost than a mislabelled disk query. A story like "ban 203.0.113.7 in the sshd jail" exercises the whole chain: intent to Fail2banBanIp with the right jail and IP params, the daemon's validation, and the approval gate. That is exactly what the E2E suite exists to catch and what unit tests cannot.

Scope

Done in #401. Four stories under tests/e2e/stories/, one per action, each
tagged for the family and the risk level:

$ for f in 134 135 136 137; do sed -n '2p' tests/e2e/stories/story-$f.sh; done
# Story 134 (ubuntu, read-only): Active fail2ban jail status
# Story 135 (ubuntu, medium-risk): Unban a documented IP from sshd
# Story 136 (ubuntu, high-risk): Ban a documented IP in sshd
# Story 137 (ubuntu, high-risk): Enable the sshd fail2ban jail

Left. Record and replay the four on Ubuntu the way the rest of the suite
runs: UBUNTU_RELEASE=noble ./tests/e2e/ubuntu-vm.sh for the live pass, then
commit the cassette so the suite still runs offline. #401 says plainly that its
environment had no QEMU and no VM harness, which is why this half is still here.
Until a cassette carries these four, the stories assert against a planner nobody
has watched answer them.

Difficulty

medium. The harness and the pattern exist; the work is writing the intents and recording a clean pass.

Getting started

CONTRIBUTING.md has the build and test commands, and the E2E section covers the VM harness. No CLA and no copyright waiver. The project is MIT.

[0xZKc0de]

Hi @vladimirrott !
I'd like to work on this issue. As suggested in the description, I will start with a small PR for Fail2banStatus first to test the harness.

[vladimirrott]

Thanks for picking this up, and welcome. I have marked the issue as claimed so nobody starts the same work beside you. (GitHub will not let me put you in the assignee field until you are a repo collaborator, which is a separate conversation if you want it.)

Starting with Fail2banStatus on its own is exactly the right instinct. It is the only Low-risk action in the family, so it needs no approval receipt, which means you get to exercise the whole harness without the approval dance in the way. One small story that records and replays cleanly teaches you more about this suite than reading it would.

A few things I should have put in the issue and did not. None of them are hard, but two of them are quiet rather than loud, so I would rather you hear them from me than find them at 1am.

Your story number is 134. The files run 1 to 133 with no gaps.

The header is what assigns your story to a distro family, and getting it wrong fails silently. The shape is:

# Story 134 (ubuntu, read-only): <name>

run-stories.sh and the claim guard in scripts/check_evidence_claims.py both read the family out of those parentheses, and a story counts as ubuntu only when the word ubuntu is in there. Anything else lands in the atomic family. So a missing tag will not error, it will quietly file your Ubuntu story under Fedora and show up later as a family size that looks wrong for no visible reason. story-101.sh is the closest read-only example to copy.

You can run only your own story. run-stories.sh takes story numbers as arguments, so there is no need to sit through the other 79.

Your own provider key is fine here. The runner uses whichever of OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, GROQ_API_KEY, DEEPSEEK_API_KEY, MISTRAL_API_KEY or XAI_API_KEY it finds, so you do not need anything from me. (#217 is the issue that does need credentials, because it records a full 54-story suite.)

Read the rate-limit count in the summary, not only the passes. Two limiters can stop a story: ours, which SYSKNIFE_MAX_RPM raises, and the provider's, which is usually tokens per minute and cannot be raised from here. That is why the recording delay defaults to 4 seconds. It matters because a throttled run can look healthy: stories 91 to 93 accept an empty plan by design, so rate limiting makes them pass while proving nothing. Recording three releases at once once produced a very believable, entirely wrong 48/50 this way.

You should not need to touch docs/distro-support.md. The 79/79 figures there quote the committed cassette runs, and the guard accepts a recorded run's total as well as a current family size, so adding a story leaves them valid. Deciding when to re-record the suite is on me.

If the first one goes well and you want more, the rest of the family is already mapped out for you, and the risk levels tell you which story tag each one needs:

action	risk	story tag
Fail2banStatus	Low	read-only
Fail2banUnbanIp	Medium	medium-risk
Fail2banBanIp	High	high-risk
ConfigureFail2banJail	High	high-risk

Those three would close the issue completely, and they are more interesting than the first because they run through the approval gate rather than around it.

Beyond fail2ban, #216 is the one I would point you at next. It asks for the Observer-level actions to be exposed as direct MCP tools, and the code is mechanical once you have settled the part that needs settling: what counts as read-only, given that AptUpdate is classified Low and still runs sudo apt-get update. That is a judgement call about a security boundary rather than a wiring job, and I would want to talk it through with whoever takes it.

Ask here whenever the harness argues with you, including if it argues for a silly reason. TDD is the rhythm in this repo, so in your case the story is the test.

[vladimirrott]

Checking in, with nothing attached to it. You claimed this on 17 August and I have not seen a branch, which is fine; the issue is not going anywhere.

Two things in case either is what stalled it.

Your story number is still 134. The files still run 1 to 133 with no gaps, so nothing has moved under you since I wrote that. If you started and hit something that did not match the issue text, tell me and I will fix the text rather than leave you working around it.

You need nothing from me to run it. run-stories.sh uses whichever provider key you already have in your environment, and it takes story numbers as arguments, so you never sit through the other 79.

If you would rather hand it back, say so and I will drop the claimed label with no hard feelings. It is a good first issue and someone else may want it. If you are still on it, it stays yours and I will stop asking.

[vladimirrott]

Releasing the claim on this one, with no hard feelings and nothing owed.

You claimed #219 on 17 August and I checked in on 1 September. Life happens, and an unanswered check-in is a perfectly normal answer. I am removing the claimed label so the issue is visible to whoever comes looking next, rather than sitting reserved.

If you do want it back, say so here and it is yours again. The fail2ban story family has not moved and nobody else has started it.

[vladimirrott]

@bferanmi806-sketch reserved for you as a follow-up to your recent work; the why is on your PR thread. No pressure and no deadline. Reply here and it is formally yours.

[bferanmi806-sketch]

I’d like to take this back, yeah. I’ll start with the fail2ban story family from the current tree and work through the harness properly rather than picking up from any old assumptions.

[vladimirrott]

@bferanmi806-sketch #401 covers all four actions, and one thing in it was not asked for: each story requires exactly one matching plan step instead of accepting an unrelated alternative. A story that passes because the model said something proves nothing, and yours cannot pass that way.

I have edited the body. It opened on a premise your PR made false, and somebody arriving cold would have read the first line and started work you had already finished:

$ grep -li fail2ban tests/e2e/stories/ -r
tests/e2e/stories/story-136.sh
tests/e2e/stories/story-135.sh
tests/e2e/stories/story-137.sh
tests/e2e/stories/story-134.sh

The one blocking item for closing this is the half #401 says its environment could not reach. No cassette carries a fail2ban exchange:

$ for f in tests/e2e/cassettes/*.json; do printf '%-45s fail2ban_hits=%s\n' "$(basename $f)" "$(grep -oic fail2ban "$f")"; done
ubuntu-jammy-gpt-oss-120b.json                fail2ban_hits=0
ubuntu-noble-gpt-oss-120b.json                fail2ban_hits=0
ubuntu-resolute-gpt-oss-120b.json             fail2ban_hits=0

So the four stories have never been replayed against a recorded planner answer. That needs a box with QEMU on it rather than more code, and re-recording the suite is mine, as I said on this thread on 17 August. The issue stays yours; tell me if you would rather I took the recording off your hands now, and I will.

Optional, and only if you want it: #233 tracks the same gap for the rest of the catalogue, and your four stories moved its numbers. I have posted the recount there.