Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,10 @@ table below.

| Area | Why it matters | Difficulty |
|---|---|---|
| **Ubuntu LTS support** | All three LTS releases are validated against the full story suite on a live VM, each with a committed replay twin that reproduces it: 22.04, 24.04 and 26.04 all at 79/79. `ubuntu-vm.sh` accepts `UBUNTU_RELEASE=jammy\|noble\|resolute`. Remaining: story coverage for the cross-family actions, and five Debian-only ones that still have none: the four fail2ban actions and `GrubSetKargs`. | medium |
| **Ubuntu LTS support** | The current suite is 83 Ubuntu stories. Committed live-VM evidence covers 79 of those Ubuntu stories on each LTS release, with a committed replay twin that reproduces each run: 22.04, 24.04 and 26.04 all at 79/79. The four additional stories are not yet included in a committed live-VM run. `ubuntu-vm.sh` accepts `UBUNTU_RELEASE=jammy\|noble\|resolute`. Remaining: story coverage for the cross-family actions, and one Debian-only action still has no story: `GrubSetKargs`. | medium |
| **Distro detection coverage** | Robust `/etc/os-release` parsing for every release we claim to support. Pure-function tests against real fixture files, no integration mocks. The existing fixtures at the bottom of `crates/sysknife-core/src/distro.rs` show the shape. | easy |
| **Action catalogue gaps** | Add a typed action (for example `EnableFirewallZone`). Small and isolated, and every PR carries the policy entry, the risk level and the tests. | easy |
| **E2E story coverage** | Real prompts, real LLM, real daemon. The suite is 133 stories: 54 atomic + 79 Ubuntu. What is left is the cross-family middle: of the action names available on both families, 59 are still untouched by any story, plus 10 Fedora-only and 5 Ubuntu-only ones. See #233 for the clustered map. | medium |
| **E2E story coverage** | Real prompts, real LLM, real daemon. The suite is 137 stories: 54 atomic + 83 Ubuntu. What is left is the cross-family middle: of the action names available on both families, 59 are still untouched by any story, plus 10 Fedora-only and 1 Ubuntu-only ones. See #233 for the clustered map. | medium |
| **Fedora Atomic validation** | The action families exist and `DistroId::is_supported()` returns true for Atomic 41 and up. Nobody has run `tests/e2e/atomic-vm.sh` against a current release. Needs Fedora Atomic hardware or a VM host. | tedious |
| **Demo recording on real hardware** | Replace the bundled demo GIF with a 30-second recording on real Ubuntu 26.04 with rollback visible. | easy |

Expand Down
11 changes: 7 additions & 4 deletions docs/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,13 @@ SysKnife, not an afterthought.

> **ℹ️ Distro support**
>
> All three Ubuntu LTS releases have a committed live-VM run of the 79-story
> Ubuntu suite, in `tests/evidence/story-runs/`: 22.04, 24.04 and 26.04 all at
> 79/79. Each run has a replay twin that reproduces it, serving every call with
> zero misses. Five Debian-only actions still have no story.
> The current suite is 83 Ubuntu stories. All three Ubuntu LTS releases have a
> committed live-VM run covering 79 of those Ubuntu stories, in
> `tests/evidence/story-runs/`:
> 22.04, 24.04 and 26.04 all at 79/79. The four additional stories are not yet
> included in a committed live-VM run. Each run has a replay twin that
> reproduces it, serving every call with zero misses.
> One Debian-only action still has no story.
> Fedora Atomic is supported by the rpm-ostree action family, but a current
> Silverblue 44 VM run is a release gate. Plain Fedora remains experimental
> until the `dnf` action family ships.
Expand Down
2 changes: 1 addition & 1 deletion scripts/check_evidence_claims.py
Original file line number Diff line number Diff line change
Expand Up @@ -433,7 +433,7 @@ def _claim_count(value: str) -> int:
DEBIAN_GAP_PROSE = (
r"\b(?P<count>\d+|zero|one|two|three|four|five|six|seven|eight|"
r"nine|ten)\s+Debian-only\s+(?:actions?|ones?)\s+"
r"(?:that\s+)?still\s+have\s+no(?:ne|\s+story)\b"
r"(?:that\s+)?still\s+(?:have|has)\s+no(?:ne|\s+story)\b"
)
DEBIAN_EVERY_CLAIM = re.compile(
r"\bEvery\s+Debian-only\s+action\s+(?:now\s+)?has\s+(?:one|a\s+story)\b",
Expand Down
19 changes: 19 additions & 0 deletions tests/e2e/stories/story-134.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/usr/bin/env bash
# Story 134 (ubuntu, read-only): Active fail2ban jail status
# Intent: "show me which fail2ban jails are active"
# Distro: ubuntu
set -euo pipefail
INTENT="show me which fail2ban jails are active"
echo "=== Story 134 (ubuntu): Fail2banStatus ==="
PLAN=$(sysknife --dry-run --json "$INTENT" 2>/tmp/sysknife-story-134-stderr.log)
echo "$PLAN" | jq .

STEP_COUNT=$(echo "$PLAN" | jq '.plan.steps | length')
if [[ "$STEP_COUNT" != "1" ]]; then echo "FAIL: expected 1 step, got $STEP_COUNT"; exit 1; fi
STEP=$(echo "$PLAN" | jq '.plan.steps[0] | select(.action == "Fail2banStatus")')
if [[ -z "$STEP" || "$STEP" == "null" ]]; then echo "FAIL: expected Fail2banStatus"; exit 1; fi
RISK=$(echo "$STEP" | jq -r '.risk')
if [[ "$RISK" != "low" ]]; then echo "FAIL: expected risk low, got $RISK"; exit 1; fi
PARAMS=$(echo "$STEP" | jq -c '.params')
if [[ "$PARAMS" != "{}" ]]; then echo "FAIL: expected no jail parameter, got $PARAMS"; exit 1; fi
echo "PASS: Story 134"
21 changes: 21 additions & 0 deletions tests/e2e/stories/story-135.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# Story 135 (ubuntu, medium-risk): Unban a documented IP from sshd
# Intent: "unban 203.0.113.7 from the sshd fail2ban jail"
# Distro: ubuntu
set -euo pipefail
INTENT="unban 203.0.113.7 from the sshd fail2ban jail"
echo "=== Story 135 (ubuntu): Fail2banUnbanIp ==="
PLAN=$(sysknife --dry-run --json "$INTENT" 2>/tmp/sysknife-story-135-stderr.log)
echo "$PLAN" | jq .

STEP_COUNT=$(echo "$PLAN" | jq '.plan.steps | length')
if [[ "$STEP_COUNT" != "1" ]]; then echo "FAIL: expected 1 step, got $STEP_COUNT"; exit 1; fi
STEP=$(echo "$PLAN" | jq '.plan.steps[0] | select(.action == "Fail2banUnbanIp")')
if [[ -z "$STEP" || "$STEP" == "null" ]]; then echo "FAIL: expected Fail2banUnbanIp"; exit 1; fi
RISK=$(echo "$STEP" | jq -r '.risk')
if [[ "$RISK" != "medium" ]]; then echo "FAIL: expected risk medium, got $RISK"; exit 1; fi
JAIL=$(echo "$STEP" | jq -r '.params.jail // ""')
if [[ "$JAIL" != "sshd" ]]; then echo "FAIL: expected jail=sshd, got $JAIL"; exit 1; fi
IP=$(echo "$STEP" | jq -r '.params.ip // ""')
if [[ "$IP" != "203.0.113.7" ]]; then echo "FAIL: expected ip=203.0.113.7, got $IP"; exit 1; fi
echo "PASS: Story 135"
21 changes: 21 additions & 0 deletions tests/e2e/stories/story-136.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# Story 136 (ubuntu, high-risk): Ban a documented IP in sshd
# Intent: "ban 203.0.113.7 in the sshd fail2ban jail"
# Distro: ubuntu
set -euo pipefail
INTENT="ban 203.0.113.7 in the sshd fail2ban jail"
echo "=== Story 136 (ubuntu): Fail2banBanIp ==="
PLAN=$(sysknife --dry-run --json "$INTENT" 2>/tmp/sysknife-story-136-stderr.log)
echo "$PLAN" | jq .

STEP_COUNT=$(echo "$PLAN" | jq '.plan.steps | length')
if [[ "$STEP_COUNT" != "1" ]]; then echo "FAIL: expected 1 step, got $STEP_COUNT"; exit 1; fi
STEP=$(echo "$PLAN" | jq '.plan.steps[0] | select(.action == "Fail2banBanIp")')
if [[ -z "$STEP" || "$STEP" == "null" ]]; then echo "FAIL: expected Fail2banBanIp"; exit 1; fi
RISK=$(echo "$STEP" | jq -r '.risk')
if [[ "$RISK" != "high" ]]; then echo "FAIL: expected risk high, got $RISK"; exit 1; fi
JAIL=$(echo "$STEP" | jq -r '.params.jail // ""')
if [[ "$JAIL" != "sshd" ]]; then echo "FAIL: expected jail=sshd, got $JAIL"; exit 1; fi
IP=$(echo "$STEP" | jq -r '.params.ip // ""')
if [[ "$IP" != "203.0.113.7" ]]; then echo "FAIL: expected ip=203.0.113.7, got $IP"; exit 1; fi
echo "PASS: Story 136"
22 changes: 22 additions & 0 deletions tests/e2e/stories/story-137.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env bash
# Story 137 (ubuntu, high-risk): Enable the sshd fail2ban jail
# Intent: "enable the sshd jail in fail2ban"
# Distro: ubuntu
set -euo pipefail
INTENT="enable the sshd jail in fail2ban"
echo "=== Story 137 (ubuntu): ConfigureFail2banJail ==="
PLAN=$(sysknife --dry-run --json "$INTENT" 2>/tmp/sysknife-story-137-stderr.log)
echo "$PLAN" | jq .

STEP_COUNT=$(echo "$PLAN" | jq '.plan.steps | length')
if [[ "$STEP_COUNT" != "1" ]]; then echo "FAIL: expected 1 step, got $STEP_COUNT"; exit 1; fi
STEP=$(echo "$PLAN" | jq '.plan.steps[0] | select(.action == "ConfigureFail2banJail")')
if [[ -z "$STEP" || "$STEP" == "null" ]]; then echo "FAIL: expected ConfigureFail2banJail"; exit 1; fi
RISK=$(echo "$STEP" | jq -r '.risk')
if [[ "$RISK" != "high" ]]; then echo "FAIL: expected risk high, got $RISK"; exit 1; fi
NAME=$(echo "$STEP" | jq -r '.params.name // ""')
if [[ "$NAME" != "sshd" ]]; then echo "FAIL: expected name=sshd, got $NAME"; exit 1; fi
if ! echo "$STEP" | jq -e '.params.enabled == true' >/dev/null; then
echo "FAIL: expected enabled=true, got $(echo "$STEP" | jq -c '.params.enabled // null')"; exit 1
fi
echo "PASS: Story 137"
2 changes: 1 addition & 1 deletion tests/release/public-claims.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -398,7 +398,7 @@ if [[ -z "${intro_published:-}" || -z "${intro_new:-}" ]]; then
printf 'FAIL: introduction-count mutation produced no values\n' >&2
exit 1
fi
if ! grep -Eq "$intro_new Debian-only actions still have no story" \
if ! grep -Eq "$intro_new Debian-only (actions still have|action still has) no story" \
"$fixture/docs/introduction.md"; then
printf 'FAIL: introduction-count mutation did not apply\n' >&2
exit 1
Expand Down
Loading