Build out volamos's own CI-safe test corpus (own tools/compare_amitools_suite.py's real coverage, MIT/Apache-licensed)

[sidick]

Motivation

tools/compare_amitools_suite.py (PR #50) has been the single highest-value bug-finding tool this project has: across a handful of sessions it found and led to fixing real volamos bugs in dos.library (#45, #46, #47, #55), exec.library/formatting (#48), and the math libraries (#51, #53) -- more real bugs than fixtures/'s own hand-authored corpus (tools/compare_vamos.py, ~10 fixtures) has found in its entire history. It also correctly attributed several other differences to vamos's own divergences (#49, #52, #54) rather than volamos bugs.

But it's local-only, permanently -- amitools is GPLv2, volamos is MIT/Apache-2.0, so test/bin/test/src/test/data can never be vendored or run in CI (see that script's own module doc). Its bug-finding power comes from covering real library surface fixtures/ doesn't touch at all: AnchorPath/MatchFirst/MatchNext, ReadArgs/FindArg, Seek, RawDoFmt's less-common format specifiers, and all four math libraries (mathffp/mathtrans/mathieeedoubbas/mathieeedoubtrans). None of that is exercised by anything that runs in CI today.

Proposal

Grow fixtures/'s own corpus (hand-authored .s/vasm sources + gen_*.py, exactly the existing pattern in fixtures/README.md) to cover the same kinds of calls amitools' corpus exercises -- not by copying amitools' own test source (that would just re-import the GPL problem), but by writing new, project-owned equivalents from the NDK autodocs directly, the same way the existing fixtures were built. Candidates, roughly in the order amitools' corpus proved valuable this session:

• AnchorPath/MatchFirst/MatchNext/MatchEnd -- a wildcard scan, a non-wildcard direct match, and at least one APF_DODIR recursive descent (this is exactly the kind of test that would have caught MatchFirst/MatchNext's ap_Buf strips the device prefix entirely instead of only deduplicating a doubled one #46 and surfaced MatchFirst/MatchNext: APF_DirChanged (bit 6) set by volamos but never by vamos -- needs real-hardware arbitration #58's still-open root-report-ordering question in CI, rather than needing an ad hoc Copperline session to find it).
• ReadArgs/FreeArgs/FindArg -- template parsing, /M, NAME=ABBREV aliasing.
• Seek/Read/Write round-trip including an out-of-range Seek() (the exact Seek() doesn't reject a target position beyond end-of-file #47 regression).
• RawDoFmt's %b/%x paths (the exact RawDoFmt's %b (BSTR) format doesn't convert the BPTR to a real address #45/Investigate: RawDoFmt/Printf %x hex digit case (lowercase in volamos, uppercase in vamos) -- needs real-hardware check #48 regressions).
• mathffp/mathtrans/mathieeedoubbas/mathieeedoubtrans -- at minimum the FFP encoding round-trip and the argument-order quirks (SPSub/SPDiv/SPPow) that mathffp.library/mathtrans.library (single-precision FFP): results diverge extensively from vamos -- needs dedicated investigation #53 covered.

Each new fixture should go through tools/compare_vamos.py (the existing CI-safe two-way harness) from day one, so growing this corpus directly grows CI coverage -- no separate promotion step needed, unlike compare_amitools_suite.py's GPL-blocked one.

Multi-version and multi-implementation verification (own harness, not GPL-blocked)

Because this corpus is volamos's own (not GPL-encumbered), it can be run against more than just vamos:

• Multiple real Kickstart versions via Copperline -- 1.3, 2.0, 3.0, 3.1, and 3.2 -- not just the single 3.1 (40.72) ROM this session's investigations used. Use Copperline's [lide] device (a lide.device-compatible Zorro II IDE board) for this, not [ide]: [ide] needs a Gayle or A4000 IDE port, so it only works on the A600/A1200/A3000/A4000 models this session used, but [lide] "works on any machine model", explicitly including Kickstart 1.3, since it's a Zorro II expansion board rather than chipset-integrated IDE -- some of the target Kickstart versions predate Gayle IDE entirely. Same directory-mount-to-in-memory-FFS/OFS trick applies ([lide] drive0 = { path = "...", name = "...", bootpri = N, filesystem = "ffs"|"ofs" } -- a genuine in-memory volume with a real Rigid Disk Block, discarded on exit; see the MatchFirst/MatchNext's ap_Buf strips the device prefix entirely instead of only deduplicating a doubled one #46/MatchFirst/MatchNext: APF_DirChanged (bit 6) set by volamos but never by vamos -- needs real-hardware arbitration #58 fix commits for the [ide] version of this recipe), so the same fixture corpus can run against every Kickstart version through one consistent storage config. This would catch real version-skew bugs (a call behaving differently across OS versions, or a fixture that accidentally only passes on the one version it was written against) that a single-ROM check can't. Only Kickstart 3.1 (40.72) is confirmed available locally right now (~/src/amibake/assets/hyperion); 1.3/2.0/3.0/3.2 need sourcing before this dimension can run end-to-end -- not a blocker for starting the corpus itself (target 3.1 first, matching volamos's own stated Phase 1 compatibility target), but worth tracking as a follow-up once fixtures exist to run against them.
• AROS as a further, zero-setup oracle -- Copperline bundles its own AROS Kickstart replacement by default (no ROM file needed, works out of the box), and as a free/open reimplementation it sidesteps any Kickstart-ROM redistribution question entirely, unlike the real Hyperion-copyrighted ROMs (which stay local-only, never bundled with volamos, same posture as this project's existing amitools/AmiBake assets).

Non-goals

• Not a replacement for compare_amitools_suite.py -- keep running that locally too; it's still valuable for cross-checking against a second, independently-authored corpus and for the specific compiler-variety angle (gcc/sc/vc/agcc) fixtures/ doesn't have.
• Not attempting to match every one of amitools' ~34 test programs one-for-one -- prioritize by what's found real bugs already (see the ordering above) over exhaustive parity.

Suggested first step

Start with the AnchorPath fixture -- it's the single area that's found the most real bugs (#14, #46, #58) and is completely unrepresented in fixtures/ today.

🤖 Filed by Claude Code at the user's request

https://claude.ai/code/session_0171vaHw8HbkiLzUph9bAA2p

[sidick]

Good addition from the user: since this would be volamos's own corpus (not GPL-encumbered), it can also be run against multiple real Kickstart versions via Copperline -- 1.3, 2.0, 3.0, 3.1, and 3.2 -- not just against vamos. This session's own [ide]-directory-mount technique (a genuine in-memory FFS/OFS volume synthesized from a host directory, real RDB, discarded on exit -- see the #46/#58 fix commits for the recipe) generalizes directly to any of these ROMs, and would catch real version-skew bugs (a call that behaves differently across OS versions, or a fixture that accidentally only works on the exact version it was written against) that a single-ROM check can't.

Also: AROS is worth including as a further, zero-setup oracle -- Copperline bundles its own AROS Kickstart replacement by default (no ROM file needed at all, works out of the box), and as a free/open reimplementation it sidesteps any Kickstart-ROM redistribution question entirely, unlike the real Hyperion-copyrighted ROMs (which stay local-only, never bundled with volamos, same posture as this project's existing amitools/AmiBake assets).

Practical note: only Kickstart 3.1 (40.72) is confirmed available locally right now (~/src/amibake/assets/hyperion); 1.3/2.0/3.0/3.2 would need sourcing before this part of the harness can run end-to-end. Not a blocker for starting the corpus itself (which should target 3.1 first, matching volamos's own stated Phase 1 compatibility target), but worth tracking as a follow-up dimension once the fixture corpus exists to run against them.

[sidick]

Correction/refinement on the multi-Kickstart-version approach: use Copperline's `[lide]` device (a `lide.device`-compatible Zorro II IDE board), not `[ide]`, for the OS-version comparison. `[ide]` needs a Gayle or A4000 IDE port -- so it only works on the A600/A1200/A3000/A4000 models this session used -- but `[lide]` "works on any machine model", explicitly including Kickstart 1.3, since it's a Zorro II expansion board rather than chipset-integrated IDE. Same directory-mount-to-in-memory-FFS/OFS trick applies (`drive0 = { path = "...", name = "...", bootpri = N, filesystem = "ffs"|"ofs" }`), so testing the SAME fixture corpus across 1.3/2.0/3.0/3.1/3.2 -- some of which predate Gayle IDE entirely -- should standardize on `[lide]` rather than needing a different storage config per Kickstart version.

[codewiz]

I'm happy to relicense all of my amitools contributions as MIT / Apache 2.0 / 0-BSD or whatever you prefer :-)

[sidick]

Planning out the actual list of candidate fixtures for this corpus, using amitools' own test/src/*.c programs (34 total, 16 already covered by the local, GPL-blocked tools/compare_amitools_suite.py) as examples of the type of coverage each should aim for -- not as source to copy.

Also worth flagging: @codewiz's offer above to relicense their amitools contributions is a potentially bigger unlock than writing everything from scratch. Caveat: amitools has multiple contributors, so one person's offer may not cover the whole test/src tree -- worth confirming exact scope (which files, whose copyright) before leaning on it. Either way, the plan below stands as the coverage target regardless of whether fixtures end up hand-written from the NDK autodocs or adapted from a properly-relicensed source.

Tier 1 -- proven bug-finders, write these first (already named above):

• dos_match -> AnchorPath/MatchFirst/MatchNext/MatchEnd (wildcard scan, non-wildcard direct match, one APF_DODIR recursive descent) -- found AnchorPath's ap_Buf includes the device-name prefix; real dos.library's is volume-relative #14, MatchFirst/MatchNext's ap_Buf strips the device prefix entirely instead of only deduplicating a doubled one #46, MatchFirst/MatchNext: APF_DirChanged (bit 6) set by volamos but never by vamos -- needs real-hardware arbitration #58; the suggested first step above
• dos_readargs + dos_readargs2 + dos_findarg -> ReadArgs/FreeArgs/FindArg (template parsing, /M, NAME=ABBREV aliasing)
• dos_seek -> Seek/Read/Write round-trip including an out-of-range Seek() (the exact Seek() doesn't reject a target position beyond end-of-file #47 regression shape)
• exec_rawdofmt -> RawDoFmt's %b/%x/%lx paths (the exact RawDoFmt's %b (BSTR) format doesn't convert the BPTR to a real address #45/Investigate: RawDoFmt/Printf %x hex digit case (lowercase in volamos, uppercase in vamos) -- needs real-hardware check #48 regression shape)
• math_double/math_fast/math_single (+ _trans) -> FFP/IEEE encoding round-trip and the SPSub/SPDiv/SPPow/IEEE*Pow argument-order quirk family (mathffp.library/mathtrans.library (single-precision FFP): results diverge extensively from vamos -- needs dedicated investigation #53, and Implement mathieeesingbas.library/mathieeesingtrans.library #62's IEEESPPow/IEEEDPPow finding)

Tier 2 -- real, currently-untested surface, moderate effort:

• exec_semaphore -> SignalSemaphore API ordering/nesting -- implemented but with no dedicated end-to-end fixture today
• locks -> Lock/UnLock/DupLock (SameLock semantics) -- currently only unit-tested, not exercised via a real compiled binary
• dos_loadseg -> LoadSeg/UnLoadSeg round-trip
• dos_runcmd -> RunCommand -- deliberately touches the still-open real-overlay-semantics gap (issue RunCommand should overlay the calling process, not spawn a nested Runtime #8); a fixture here documents the gap in CI rather than closing it, which is valuable on its own
• proc_args -> CLI argument passing into a launched guest program (partial overlap with fixtures/echoargs -- worth diffing against before duplicating)
• dos_system/dos_program -> System()/Execute() nested guest programs

Tier 3 -- deliberately-unimplemented gap or low value, track but don't prioritize:

• exec_makelib/exec_makefuncs/exec_libfuncs/exec_initres/exec_initstruct -> MakeLibrary/SetFunction -- on the Changelog's own "What's not done yet" list; a fixture here is a regression-tracker for a known non-goal, not a bug-finder
• lib_testnix/lib_testsc -> needs a companion shared-library build step per compiler; matches this issue's own explicitly-deferred "phase 2" scope
• test_hello/test_raise/test_devtimer/test_execpy/util_muldiv -> trivial, or already well-covered by existing hand-written fixtures/unit tests; low marginal value

🤖 Posted by Claude Code at the user's request

https://claude.ai/code/session_0171vaHw8HbkiLzUph9bAA2p

[sidick]

The multi-Kickstart-version dimension discussed above now exists: tools/compare_kickstart_versions.py runs fixtures/exectest/echoargs against volamos plus every real Kickstart ROM found in assets/*.rom, via Copperline's copperhf.device (an emulator-only virtual hardfile controller, no ROM/board-fitment quirks -- see the script's own module doc for why this was picked over [lide]/[ide]; needs Copperline 0.20+, copperhf.device wasn't stable before that).

Currently covers 4 of the 5 target versions end-to-end (2.0/3.0/3.1/3.2 all agree on exectest); Kickstart 1.3 hits a real boot-time "Software error -- task held" system requester via copperhf -- Copperline's own docs mark its 1.3 copperhf autoboot tests #[ignore]d with "a skipped test is not evidence that its configuration works," so this looks like a Copperline-side gap rather than a volamos issue. It's excluded from the required-agreement set for now but still surfaced in the output, not silently dropped.

Also already found a real, currently-unexplained divergence this way: fixtures/echoargs prints its arguments correctly under volamos but prints nothing at all under every real Kickstart ROM this harness reaches -- filed as #63.

Local-only (needs real Kickstart ROM images under assets/, gitignored, same posture as compare_three_way.py), same as the other Copperline-based harnesses in this repo.

🤖 Posted by Claude Code at the user's request

https://claude.ai/code/session_0171vaHw8HbkiLzUph9bAA2p

[sidick]

The multi-Kickstart-version dimension of this proposal is done: tools/compare_kickstart_versions.py runs cleanly across every real Kickstart ROM this harness reaches (2.0/3.0/3.1/3.2, via copperhf.device), and along the way it already earned its keep -- it found and led to fixing two real bugs (a real-hardware-unsafe A0 assumption in fixtures/echoargs.s/libcall.s, and a genuine Runtime::new command-line-buffer bug missing AmigaOS's trailing-space convention), plus correctly identified a genuine Kickstart 3.1-vs-3.2 behavioral difference as a real OS-version divergence rather than a volamos bug. Full writeup in #63.

Remaining from this issue's original proposal, not yet started: growing the actual fixture corpus itself (the tiered AnchorPath/ReadArgs/Seek/RawDoFmt/math-library list from the earlier planning comment above) beyond the two fixtures (exectest, echoargs) this harness currently exercises, Kickstart 1.3 support (blocked on what looks like a Copperline-side copperhf gap, not a volamos issue), and the AROS oracle dimension. Leaving this issue open to track those.

🤖 Posted by Claude Code at the user's request

https://claude.ai/code/session_0171vaHw8HbkiLzUph9bAA2p

[sidick]

The AROS oracle dimension is also done now: tools/compare_kickstart_versions.py includes an aros entry that needs no ROM file at all -- Copperline boots its own bundled AROS replacement whenever its config omits rom = ... entirely, so this is genuinely zero-setup, and (being APL 1.1, free/open) has no redistribution question the way the real Hyperion ROMs do.

Confirmed it agrees with volamos and every real Kickstart 2.0/3.0/3.1 ROM on both corpus entries, including the trailing-space command-line convention from #63.

Still remaining from this issue's original proposal: growing the actual fixture corpus beyond exectest/echoargs, and Kickstart 1.3 support (still blocked on what looks like a Copperline-side copperhf gap).

🤖 Posted by Claude Code at the user's request

https://claude.ai/code/session_0171vaHw8HbkiLzUph9bAA2p