Support HUNK_OVERLAY executables (LoadSeg overlay-continuation)

[sidick]

Summary

loader.rs currently rejects HUNK_OVERLAY (0x3F5) binaries outright (see module docs, line ~32-33). This blocks real linker-overlay executables such as AExplorer (Aminet, vendored at ~/src/amiga_explorer_api/vendor/amiga-explorer-adf/AExplorer), which uses the standard SAS/C SLink-style hierarchical overlay manager.

Unlike issue #8 (RunCommand overlay-the-caller semantics), this does not need reentrant guest-code invocation. The overlay manager runs entirely as ordinary guest code inside the program's own process; volamos only needs to service its library calls (LoadSeg, Seek, OpenLibrary, FreeMem, Alert) correctly and extend LoadSeg's NULL-name overlay-continuation contract.

Ground truth (verified via disassembly, not guessed)

Disassembled AExplorer's 76-longword root hunk 0 (named "Overlay") with capstone and cross-checked byte-for-byte against the RKRM AmigaDOS Manual's "Overlays" chapter (1.4.1-1.4.9), which includes a full reference implementation. AExplorer's manager matches the documented SLink hierarchical manager almost exactly:

struct OverlayHeader (first hunk, first longwords), confirmed at these offsets:

• oh_Jump (0x00) — branch around the struct, not interpreted by the loader
• oh_Magic (0x04) = 0x0000ABCD — loader does NOT fill this; it's part of the hunk data itself. Checked by UnLoadSeg() to recognize overlay root segments.
• oh_FileHandle (0x08) — loader-filled: BPTR to the file handle the root node was loaded from. Must stay open for the program's lifetime.
• oh_OVTab (0x0c) — loader-filled: pointer to the HUNK_OVERLAY payload (starting at the tree-depth element, i.e. after the l length longword).
• oh_Segments (0x10) — loader-filled: BPTR to an array of segment BPTRs, one entry per segment counting from 0 (hunk 0 = the manager itself). Array size = t_size from the root HUNK_HEADER.
• oh_GV (0x14) — loader-filled: BPTR to dos.library's public Global Vector (BCPL only; unused by C/asm managers, can be left 0/unpopulated for now).
• Followed by an optional length-prefixed identifier BSTR (AExplorer: dc.b 7,"Overlay") — not interpreted by the loader.

HUNK_OVERLAY (0x3F5) payload — table Table 1.4.3.0.1 in the RKRM skill's binary-file-structure.md:

• longword l: table is l+1 longwords total (off-by-one convention, confirmed against AExplorer's raw bytes)
• longword od: tree depth including root
• od-1 zero longwords: the "currently loaded ordinate per level" array (oh_OVTab points here at runtime, i.e. at the second element, skipping od itself per the RKRM text)
• then a SymTab[] array, 7 longwords per entry: ot_FilePosition(0), ot_Reserved[2](4,8), ot_Level(0xc), ot_Ordinate(0x10), ot_FirstSegment(0x14), ot_SymbolSegment(0x18), ot_SymbolOffset(0x1c) — all 7 offsets independently confirmed against AExplorer's manager code, which reads every one of them at exactly these offsets.

HUNK_BREAK (0x3F6): terminates a non-root overlay node (no payload). Not required after the root node — HUNK_OVERLAY already signals that stop.

Overlay-node continuation load — LoadSeg(name, table, fh) with name=NULL as the overlay-continuation trigger, confirmed via AExplorer's real call:

• D1 = 0 (NULL name — this is the discriminator vs. a normal named load)
• D2 = oh_Segments (BPTR to the segment table)
• D3 = oh_FileHandle (BPTR to the already-open file handle)
• Caller must Seek(fh, ot_FilePosition, OFFSET_BEGINNING) immediately before this call — the manager does so itself (confirmed: Seek LVO -66 with d3=-1).
• Loads hunks starting at that file position as a normal HUNK_HEADER...HUNK_END/HUNK_BREAK node, but numbers/patches its segments into the shared global segment table (oh_Segments) starting at the node's declared first-segment index, and any HUNK_RELOC32 in the node can reference already-resident root/ancestor hunks by their global segment number.
• Returns the new node's first segment BPTR in D0 (0 on failure), links it into the segment list, same as regular LoadSeg.

LVOs used by the manager (all independently confirmed against exec/dos LVO tables via the disassembly): OpenLibrary (-552), Seek (-66), LoadSeg (-150), FreeMem (-210), Alert (-108).

Implementation sketch

1. loader.rs: on encountering HUNK_OVERLAY, stop the normal hunk-loading loop (root hunks 0..N become the initial seglist), read the l/od/ordinate-array/SymTab[] payload, and retain the open file handle rather than closing it after load (today's loader presumably closes the file once LoadSeg finishes — needs to keep it open when a HUNK_OVERLAY was seen).
2. Header patching at LoadSeg time: after loading the root node's hunk 0, locate the 0x0000ABCD magic and write the four loader-filled fields (oh_FileHandle, oh_OVTab, oh_Segments, oh_GV) at their fixed offsets (0x08/0x0c/0x10/0x14 relative to the magic... actually relative to hunk start, magic is at +4) into guest memory.
3. Global segment table: allocate/populate oh_Segments as a guest-memory array of segment BPTRs sized from the root HUNK_HEADER's t_size, filled with hunks 0..N as they're loaded (root) and appended to for each overlay node loaded later.
4. dosseg.rs (LoadSeg handler): detect the D1==0 (NULL name) overlay-continuation form; when present, use D2/D3 instead of resolving a filename, seek the given handle to the caller-specified position (already done by caller before the call per the confirmed contract — no extra seek needed loader-side), and load one node's hunks, numbering them starting at the segment-table slot the node's own HUNK_HEADER implies, with HUNK_RELOC32 resolving against the shared/global hunk address table instead of a fresh per-load one.
5. UnLoadSeg: recognize the 0xABCD magic in the first segment to know it must also Close() oh_FileHandle and release the segment table (per RKRM 1.4.8/1.4.9) in addition to walking the segment chain as normal.

Test binary

AExplorer at ~/src/amiga_explorer_api/vendor/amiga-explorer-adf/AExplorer — root node is hunks 0-2 (manager/code/data), one HUNK_OVERLAY node (a good minimal first test case), uses the exact manager described above. Good candidate for the three-way Copperline harness once implemented.

Why this doesn't need issue #8

Issue #8 is about RunCommand needing to reentrantly invoke guest code from inside a library-call handler. This overlay manager is itself guest code that makes ordinary library calls (LoadSeg/Seek/OpenLibrary/FreeMem/Alert) volamos already handles or can extend without any reentrant Runtime::run invocation — the manager's jmp (a1) tail-call into the resolved symbol happens entirely within the guest CPU's own control flow, not something volamos's handlers need to orchestrate.