Story coverage map: 64 Ubuntu and cross-family actions have no story, clustered into takeable pieces

[vladimirrott]

Recounted 2026-09-21 at ba1a7b8f. Every figure below still holds. One
sentence in the caveat section said all three named actions sit in the
uncovered set; GrubSetKargs left it when story 138 landed, which the cluster
list already said. Corrected so the two halves agree.

Recounted at 30b9e315 on 2026-09-17. #447 landed story 138 and closed the
GRUB cluster, so every figure below moved again: uncovered 62 to 61, off-Ubuntu
96 to 95, 83 ubuntu-tagged files to 84, next free id 138 to 139. The numbers in
the first version of this issue (164 / 64 / 98, 189 rows, 79 ubuntu-tagged
files, next id 134) and the 2026-09-16 recount at b2c823e6 (62 / 96 / 83 /
138) are both superseded by the ones now in the text.

61 of the 166 Ubuntu-and-cross-family actions are named by no story at all, and 95 of them are named by no story that runs on Ubuntu. This issue is the map, so the work can be taken one cluster at a time.

Derivation

python3 - <<'PY'
import re, glob
from collections import Counter
rows = re.findall(r'^\| `([A-Za-z0-9_]+)` \|.*?\| (All|Ubuntu|Fedora) \|',
                  open('docs/action-reference.md').read(), re.M)
U = {n for n, d in rows if d in ('All', 'Ubuntu')}
named = set()
for f in glob.glob('tests/e2e/stories/*.sh'):
    named |= set(re.findall(r'"([A-Za-z0-9_]+)"', open(f, errors='replace').read()))
ub = [f for f in glob.glob('tests/e2e/stories/*.sh')
      if re.search(r'^# Story \d+ \([^)]*ubuntu[^)]*\)', open(f, errors='replace').read(), re.M)]
named_ub = set()
for f in ub:
    named_ub |= set(re.findall(r'"([A-Za-z0-9_]+)"', open(f, errors='replace').read()))
print('|U| =', len(U), 'uncovered =', len(U - named), 'not on ubuntu =', len(U - named_ub))
PY
|U| = 166 uncovered = 61 not on ubuntu = 95

Catalogue split: 191 tabled rows, All 99, Ubuntu 67, Fedora 25. 84 story files carry an ubuntu header tag.

One honest caveat, and one I withdraw.

I wrote that the derivation over-counts, because it treats a quoted literal
anywhere in a story as coverage, and named ConfigureWifi, GrubSetKargs and
SetLocale as actions carried only by "must not confuse with" comments. That is
wrong and was wrong when I filed it. All three appear unquoted in prose comments,
so the regex never saw them. ConfigureWifi and SetLocale are still in the
uncovered set where they belong; GrubSetKargs left it when story 138 landed in
#447, and the cluster list below records that. No action in the catalogue has its only quoted occurrence on
a comment line, so the figure is not inflated by that mechanism at all.

The caveat that does hold: 34 Ubuntu-and-All actions are covered only by stories 1 to 54, the atomic family, which no recorded Ubuntu run includes, so AptListUpgradable, a Debian-only action, has its sole coverage in a Fedora-family story.

Clusters, each one a reasonable PR

Ubuntu-family exclusive, highest priority:

• fail2ban (4): stories 134-137 landed in test(e2e): cover fail2ban action family #401 on 2026-09-15. Add E2E story coverage for the fail2ban action family #219 stays open for the live recording behind them.
• GRUB kargs (1): GrubSetKargs landed as story 138 in test(e2e): cover GrubSetKargs with story 138 #447 on 2026-09-17. No Ubuntu-family-exclusive action is left without a story.

Cross-family (All):

• sysctl and auditd (5): GetSysctl, SetSysctl, GetAuditRules, AddAuditRule, RemoveAuditRule
• PAM policy and lockout (6): GetPasswordAging, SetPasswordAging, SetPasswordPolicy, SetAccountLockout, LockUserAccount, UnlockUserAccount
• Scoped sudoers.d (3): GetSudoGrants, GrantSudoAccess, RevokeSudoAccess
• Groups (2): CreateGroup, DeleteGroup
• Mounts and swap (5): GetMounts, AddMount, RemoveMount, AddSwap, RemoveSwap
• LVM (4): GetLvmReport, CreateLogicalVolume, ExtendLogicalVolume, CreateLvSnapshot
• Log management (5): GetLogrotateStatus, ConfigureLogRotation, RemoveLogRotation, ConfigureRemoteSyslog, RemoveRemoteSyslog
• Journald and diagnostics (3): GetJournalLog, VacuumJournal, CollectDiagnostics
• Container lifecycle (4): CreateContainer, StartContainer, StopContainer, RemoveContainer
• Flatpak remotes and info (4): ListFlatpakRemotes, AddFlatpakRemote, RemoveFlatpakRemote, GetFlatpakAppInfo
• Toolbox (2): ListToolboxes, RemoveToolbox
• certbot and ACME (3): GetCertificates, ObtainCertificate, RenewCertificates
• DNS, wifi, sockets (4): ResolvectlStatus, ResolvectlSetDns, ConfigureWifi, GetListeningPorts
• systemd extras (4): UnmaskService, CreateScheduledJob, GetServiceResourceLimits, SetServiceResourceLimits
• Time and locale (3): GetDateTime, SetNtp, SetLocale
• Singletons (2): SetSshdOption, SignalProcess
• Firewall and nftables observation (2): GetFirewallBackendState, GetNftablesRuleset, added by feat(network): report nftables and firewall backend observations #413 after this map was drawn

How to take one

Pick a cluster, write one story per action, and record on Ubuntu the way the current 84 run:

UBUNTU_RELEASE=noble ./tests/e2e/ubuntu-vm.sh

Then commit the cassette and its replay twin so the suite still runs offline. run-stories.sh accepts story numbers, so you can record only yours. Any of the seven *_API_KEY variables the runner recognises works; no credentials are needed from the maintainer.

The header decides the family and a wrong one fails silently, so use # Story <id> (ubuntu, <read-only|medium-risk|high-risk>): <name> and check that bash tests/e2e/run-stories.sh --metadata puts your story in the ubuntu family. Next free id is 139.

Start with a read-only cluster if this is your first story. GetMounts, GetSysctl and GetListeningPorts need no approval receipt and exercise the whole chain.

Worth fixing alongside

Nothing compares the story set against the action catalogue, which is why this drifted unnoticed. The derivation above is small enough to become a gate, and it would keep the number in #229 honest at the same time.

Difficulty

medium per cluster. The harness and the pattern exist; the work is writing intents and recording a clean pass.

Getting started

CONTRIBUTING.md has the build and test commands, and its E2E section covers the VM harness. No CLA and no copyright waiver. The project is MIT.

[vladimirrott]

@bferanmi806-sketch reserved for you as a follow-up to your recent work; the why is on your PR thread. No pressure and no deadline. Reply here and it is formally yours.

[bferanmi806-sketch]

I’m in for this one too. I’ll treat it as the coverage map rather than trying to tackle everything at once, and work through it cluster by cluster. #219 should be a good first piece of it.

[vladimirrott]

@bferanmi806-sketch your four stories moved this map, so I recounted it and edited the body rather than leave you to trip over the drift.

I ran the derivation exactly as the body has it, with three extra print lines on the end:

$ python3 - <<'PY'
... the body's derivation, unchanged, then:
print('tabled rows =', len(rows), Counter(d for _,d in rows))
print('ubuntu-tagged story files =', len(ub))
print('uncovered set =', sorted(U - named))
PY
|U| = 166 uncovered = 62 not on ubuntu = 96
tabled rows = 191 Counter({'All': 99, 'Ubuntu': 67, 'Fedora': 25})
ubuntu-tagged story files = 83

62 = 64 − 4 + 2. #401 covered the four fail2ban actions, and #413 added GetFirewallBackendState and GetNftablesRuleset, both of which arrived with no story. The catalogue grew by those same two rows, 189 → 191. Next free story id is 138.

GrubSetKargs is now the only Ubuntu-family-exclusive action with no story, which is what #401 claimed and what I get:

$ python3 - <<'PY'
import re, glob
rows = re.findall(r'^\| `([A-Za-z0-9_]+)` \|.*?\| (All|Ubuntu|Fedora) \|', open('docs/action-reference.md').read(), re.M)
named=set()
for f in glob.glob('tests/e2e/stories/*.sh'):
    named |= set(re.findall(r'"([A-Za-z0-9_]+)"', open(f, errors='replace').read()))
ubuntu_only = {n for n,d in rows if d=='Ubuntu'}
print('Ubuntu-only uncovered:', sorted(ubuntu_only - named))
print('new-since-#233 All actions:', [n for n,d in rows if n in ('GetFirewallBackendState','GetNftablesRuleset')])
PY
Ubuntu-only uncovered: ['GrubSetKargs']
new-since-#233 All actions: ['GetNftablesRuleset', 'GetFirewallBackendState']

I have withdrawn one of the two caveats, because I had it wrong. I wrote that the derivation over-counts, and named ConfigureWifi, GrubSetKargs and SetLocale as actions carried only by "must not confuse with" comments. Those three appear in prose without quotes, so the regex never matched them, on the day I filed this or now:

$ for n in ConfigureWifi GrubSetKargs SetLocale; do printf '%-16s quoted-hits=%s\n' "$n" "$(grep -rh "\"$n\"" tests/e2e/stories/ | wc -l)"; done
ConfigureWifi    quoted-hits=0
GrubSetKargs     quoted-hits=0
SetLocale        quoted-hits=0

All three sit in the uncovered set, where they belong. I then checked the mechanism instead of the three examples: no action in the catalogue has its only quoted occurrence on a comment line, so 62 is not inflated that way. The other caveat holds unchanged. 34 of these actions are covered only by the 54 atomic-family stories, and AptListUpgradable is still a Debian-only action whose sole story is a Fedora one.

Nothing here changes what the issue asks for, and no deadline moved.

[vladimirrott]

Recounted at ba1a7b8f, running the body's own derivation unchanged:

$ python3 - <<'PY'
import re, glob
from collections import Counter
rows = re.findall(r'^\| `([A-Za-z0-9_]+)` \|.*?\| (All|Ubuntu|Fedora) \|',
                  open('docs/action-reference.md').read(), re.M)
U = {n for n, d in rows if d in ('All', 'Ubuntu')}
named = set()
for f in glob.glob('tests/e2e/stories/*.sh'):
    named |= set(re.findall(r'"([A-Za-z0-9_]+)"', open(f, errors='replace').read()))
ub = [f for f in glob.glob('tests/e2e/stories/*.sh')
      if re.search(r'^# Story \d+ \([^)]*ubuntu[^)]*\)', open(f, errors='replace').read(), re.M)]
named_ub = set()
for f in ub:
    named_ub |= set(re.findall(r'"([A-Za-z0-9_]+)"', open(f, errors='replace').read()))
print('|U| =', len(U), 'uncovered =', len(U - named), 'not on ubuntu =', len(U - named_ub))
print('tabled rows =', len(rows), Counter(d for _,d in rows))
print('ubuntu-tagged story files =', len(ub))
print('GrubSetKargs in uncovered set?', 'GrubSetKargs' in (U - named))
PY
|U| = 166 uncovered = 61 not on ubuntu = 95
tabled rows = 191 Counter({'All': 99, 'Ubuntu': 67, 'Fedora': 25})
ubuntu-tagged story files = 84
GrubSetKargs in uncovered set? False

Every published figure matches: 166, 61, 95, 191 rows split 99/67/25, and 84 ubuntu-tagged story files.

One sentence contradicted the rest of the issue. The caveat section said ConfigureWifi, GrubSetKargs and SetLocale "all three sit in the uncovered set where they belong", while the cluster list four lines below correctly struck the GRUB cluster after story 138 landed:

$ for n in ConfigureWifi GrubSetKargs SetLocale; do printf '%-16s quoted-hits=%s\n' "$n" "$(grep -rh "\"$n\"" tests/e2e/stories/ | wc -l)"; done
ConfigureWifi    quoted-hits=0
GrubSetKargs     quoted-hits=1
SetLocale        quoted-hits=0
$ sed -n '2p' tests/e2e/stories/story-138.sh
# Story 138 (ubuntu, high-risk): Append quiet and remove splash in GRUB

Fixed so both halves agree. ConfigureWifi and SetLocale are still uncovered.

@bferanmi806-sketch story 138 is the one that moved this, so the correction is a consequence of your own merge rather than anything you need to do.

[bferanmi806-sketch]

Hi Vladimir! I've opened #506 with stories 139 and 140 for the firewall observation cluster.

The static checks and CI are passing. Live recording is still blocked by the QEMU environment, so I've left the cassettes untouched.

Happy to make any adjustments if needed. Thanks!