From 0ccab536c8ff9cb7f7d97547a7fd674b146d2cf5 Mon Sep 17 00:00:00 2001 From: "John William Creighton (s243a)" Date: Sat, 12 Sep 2026 20:42:21 -0600 Subject: [PATCH 01/10] feat(pkg_resolver): symbol-level ABI-compatibility store lane (abi/) Extend resolution from coarse version constraints to fine-grained ABI compatibility at the SYMBOL level: for a binary, compute the min AND max (newest) compatible library version via versioned-symbol-set containment -- the [min,max] range ldd won't give you. A driver above the frozen spec; resolver.pl / resolver_store.pl untouched. Model: a library version PROVIDES exported versioned symbols {sym@ver} + soname; a binary REQUIRES referenced versioned symbols + NEEDED sonames; compatible = soname match AND requires subset provides. Each symbol is stored once as a validity interval [intro, inf) within a soname -- the same interval store as snapshot membership, one level down. - ingest_symbols.mjs -- 3-tier ingest -> P/2 interval JSONL: (1) Packages `Depends` = the coarse min, free from the repo index (noted); (2) `.symbols` control-member -> symprov intervals [soname|sym, intro#inf] (curated `minimum-version` = intro; zero binary download; the common Debian/Ubuntu path); `symbols-dir` batches a whole dpkg info dir; (3) readelf fallback -- `elf`/`requires` for any ELF anywhere. - abi_resolve.pl -- new module (reuses resolver:version_lt/2): abi_min (the verneed floor), provides_at (intro =< V interval lookup), abi_compatible (name-intro containment, the max-side predicate), newest_abi_compatible, abi_range. + abi_cli.pl driver. - test_abi.pl + run_abi_verify.sh -- real-data harness. - README.md + SYMBOL_ABI_HOWTO.md -- model, the 3 ingest tiers, the min/max epistemics (symbol-absence = hard veto, presence = defeasible maybe), Debian/Ubuntu/any-ELF coverage, package-manager vs coding-agent consumers. Verified on this machine (Ubuntu 22.04), 8/8 checks: libc6 .symbols -> 4827 symprov rows (3006 for libc.so.6); /bin/ls -> 112 symreq, NEEDED [libselinux.so.1, libc.so.6]; abi_min(/bin/ls, libc.so.6) = 2.34; abi_range = range(2.34, 2.35); simulating getenv removed at 2.30 caps the max at 2.29, so min 2.34 > max 2.29 -> no_candidate. Batch across all 562 .symbols = 256,225 interval rows / 2,231 sonames -- the genuine memory-pressure dataset. Note: readelf-derived intro vs curated .symbols agree on 91.7% of shared libc symbols; every divergence is the glibc 2.34 pthread/rt-into-libc merge (curated .symbols dates those to 2.34, when they entered libc.so.6's stable ABI; raw ELF keeps the historical libpthread tag) -- the curated metadata is the more ABI-faithful source, which is why the .symbols tier is preferred. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01RoXjhStCqoig6944pVNBGe --- examples/pkg_resolver/abi/.gitignore | 4 + examples/pkg_resolver/abi/README.md | 138 +++++++++ examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md | 134 +++++++++ examples/pkg_resolver/abi/abi_cli.pl | 49 ++++ examples/pkg_resolver/abi/abi_resolve.pl | 241 ++++++++++++++++ examples/pkg_resolver/abi/ingest_symbols.mjs | 269 ++++++++++++++++++ examples/pkg_resolver/abi/run_abi_verify.sh | 63 ++++ examples/pkg_resolver/abi/test_abi.pl | 83 ++++++ 8 files changed, 981 insertions(+) create mode 100644 examples/pkg_resolver/abi/.gitignore create mode 100644 examples/pkg_resolver/abi/README.md create mode 100644 examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md create mode 100644 examples/pkg_resolver/abi/abi_cli.pl create mode 100644 examples/pkg_resolver/abi/abi_resolve.pl create mode 100644 examples/pkg_resolver/abi/ingest_symbols.mjs create mode 100755 examples/pkg_resolver/abi/run_abi_verify.sh create mode 100644 examples/pkg_resolver/abi/test_abi.pl diff --git a/examples/pkg_resolver/abi/.gitignore b/examples/pkg_resolver/abi/.gitignore new file mode 100644 index 000000000..3d1a51dc3 --- /dev/null +++ b/examples/pkg_resolver/abi/.gitignore @@ -0,0 +1,4 @@ +# Generated symbol-ABI stores and verification artifacts (rebuild via +# run_abi_verify.sh). Never commit machine-specific ingested data. +.out/ +*.jsonl diff --git a/examples/pkg_resolver/abi/README.md b/examples/pkg_resolver/abi/README.md new file mode 100644 index 000000000..57db8ad07 --- /dev/null +++ b/examples/pkg_resolver/abi/README.md @@ -0,0 +1,138 @@ + + + +# Symbol-level ABI-compatibility lane (`pkg_resolver/abi`) + +This lane promotes the coarse package resolver's `provides` (a package provides +a virtual name / satisfies `libfoo (>= 2.0)`) to **fine-grained ABI +compatibility at the *symbol* level**. It computes, for a binary, the **MIN** +and **MAX (newest)** compatible library version via symbol-set containment — +the real `[min, max]` range that `ldd` will not tell you. + +It reuses the SAME interval-store idea one level down: instead of a package +having a version tenure, each exported **symbol** has a validity interval +`[intro, inf)` *within a soname*. A library version `V` provides a symbol iff +`intro(sym) =< V` (until a soname bump / removal). The frozen `resolver.pl` / +`resolver_store.pl` are **not** edited — version comparison is delegated to +`resolver:version_lt/2`. + +## The model (one generalization of `provides`) + +- A **library version PROVIDES** a set of exported versioned symbols plus a + `soname` (its ABI generation, e.g. `libc.so.6`). +- A **binary REQUIRES** a set of referenced versioned symbols plus the sonames + it links (`DT_NEEDED`). +- `compatible(Bin, LibVer)` ⟺ `soname matches` AND `requires ⊆ provides`. + +## Files + +| File | Role | +|------|------| +| `ingest_symbols.mjs` | Ingest symbol metadata → P/2 JSONL interval store (3 tiers). | +| `abi_resolve.pl` | Prolog resolver: intervals, containment, min/max. New module; reuses `resolver:version_lt/2`. | +| `abi_cli.pl` | Callable driver: `min` / `newest` / `range` / `compat`. | +| `test_abi.pl` | Real-data assertions (the PoC's numbers). | +| `run_abi_verify.sh` | End-to-end: build the store from this machine's data + verify. | + +Generated stores land in `./.out` (gitignored) — rebuild with +`./run_abi_verify.sh`. + +## The three ingest tiers (cheapest first) + +The full per-symbol interval data is *curated metadata, not the binary* — so the +common path needs **zero package download**. + +1. **`Packages` `Depends:` — coarse min for free.** Debian/Ubuntu's package + index already carries `libc6 (>= 2.34)`, computed *from the symbols* by + `dpkg-shlibdeps` at build time. The coarse floor needs no download at all. + (Consumed by the coarse resolver; this lane refines it.) +2. **`.symbols` control member — curated intervals (`ingest_symbols.mjs + symbols-file`).** Library packages ship a `.symbols` file whose lines are + literally `symbol@Base minimum-version` — *exactly our `[sym, intro]` + interval store, precomputed by the maintainer*. It lives in the package's + small `control.tar.*` member (a few KB), not the multi-MB data archive. On an + installed system it is already on disk under `/var/lib/dpkg/info/*.symbols`. +3. **`readelf` fallback — any ELF (`ingest_symbols.mjs elf` / `requires`).** If + no `.symbols` is published, `readelf -W --dyn-syms` gives a `.so`'s exported + versioned symbols (`intro` = the symbol's own version tag), and a binary's + `readelf -V` verneed table attributes each *required* symbol to its soname. + Applies to **any ELF**, so the lane works for Debian, Ubuntu, and arbitrary + binaries with no repo metadata at all. + +Applies to **Debian AND Ubuntu** unchanged (same `dpkg` / `.symbols` format), +plus the `readelf` path for any ELF on any distro. + +### Store shape (P/2 JSONL, `[key, value]`, `load_p2_jsonl`-compatible) + +``` +symprov.jsonl : ["|", "#inf"] # interval; to=inf within soname +symreq.jsonl : ["|", "#"] # verneed floor, per soname +needed.jsonl : ["", ""] # DT_NEEDED +``` + +## The resolver (`abi_resolve.pl`) + +- `provides_at(SoName, Sym, V)` — true iff `intro(SoName|Sym) =< V` (interval + lookup; `to=inf` within a soname). +- `abi_min(Binary, SoName, Min)` — the **verneed floor**: the highest required + version among the binary's required symbols of that soname. Hard, derived, + exact. +- `abi_compatible(Binary, SoName, V[, Drop])` — containment: every required + symbol of that soname is provided at `V`. `Drop = drop(Sym, At)` simulates a + soname-era ABI break (symbol removed at `At`) for testing the max. +- `newest_abi_compatible(Binary, SoName, Candidates[, Drop], Result)` — newest + `V` in `Candidates` (descending) with `abi_compatible` → `compatible(V)`, else + `no_candidate`. `Candidates` = `soname_candidates/2` (the distinct intro + versions in the store — the soname's version axis). +- `abi_range(Binary, SoName, Candidates[, Drop], Result)` — combines the derived + MIN with the newest-compatible MAX → `range(Min, Max)`, or `no_candidate(...)` + when `Min > Max` (a removed symbol squeezed the range empty). + +Note the division of labor (matching the PoC): `abi_compatible` is *name-intro +containment* and drives the **MAX** search; the **MIN** is the separate verneed +floor from `abi_min`. So `abi_compatible(2.33)` can be true by name even though +the binary references the `GLIBC_2.34` node — `abi_range` applies the floor and +reports `2.34` as the effective lower bound. + +## Min vs max — the epistemics + +- **min** = the verneed floor (highest required symver). **Hard, derived, + exact.** +- **max** = a **defeasible default** = "the soname generation" (no upper bound + for backward-compatible libs), refined *downward* only by hard evidence: a + needed symbol removed, or a soname bump. Symbol analysis can only ever NARROW + the default. +- **Symbol ABSENCE / soname mismatch = a hard veto** (proof of "no") — it can + override an optimistic or wrong declared dependency. +- **Symbol PRESENCE = a defeasible "might work"** — necessary but not sufficient + (semantics unverified); it never outranks a declared/tested dep, it only + widens the candidate set with a low-confidence maybe. + +Confidence order: `tested > declared repo dep > ELF-hard-veto / ELF-maybe > +soname default`. See `SYMBOL_ABI_HOWTO.md` for the full design and worked +examples. + +## Verify on this machine + +``` +./run_abi_verify.sh # builds the store from /var/lib/dpkg + /bin/ls, runs all checks +``` + +Real-data results (Ubuntu 22.04, libc6, `/bin/ls`): + +- `libc6` `.symbols` → **4827** symprov interval rows across 20 sonames + (**3006** for `libc.so.6`); `/bin/ls` → **112** symreq rows + `NEEDED = + [libselinux.so.1, libc.so.6]`. +- Tier-2 cross-check: `readelf(libc.so.6)` vs curated `.symbols` agree on the + intro axis for **2241 / 2443 (91.7%)** shared symbols. Every disagreement is + the **glibc 2.34 pthread/rt-into-libc merge** — `.symbols` conservatively + dates those symbols to 2.34 (when they entered `libc.so.6`'s stable ABI) while + the raw ELF keeps the historical libpthread version node. +- `abi_min(/bin/ls, libc.so.6)` = **2.34** (set by + `__libc_start_main@GLIBC_2.34`). +- `abi_compatible(/bin/ls, libc.so.6, 2.35)` = **TRUE** (0 missing); + `newest_abi_compatible` = `compatible(2.35)`; `abi_range` = **range(2.34, + 2.35)**. +- Simulated removal of `getenv` at 2.30 (a symbol `/bin/ls` needs): the newest + version still exporting `getenv` caps at **2.29**; combined with min **2.34** + → **min > max squeeze** → **`no_candidate`**. diff --git a/examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md b/examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md new file mode 100644 index 000000000..ea03da480 --- /dev/null +++ b/examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md @@ -0,0 +1,134 @@ + + +# Symbol-level ABI resolution — how it works, with examples + +This extends the package resolver from coarse version constraints +(`libfoo (>= 2.0)`) to **fine-grained ABI compatibility** at the *symbol* +level, so we can compute the real *[min, max]* compatible library-version range +for a binary — the thing `ldd` won't tell you. + +## The model (one generalization of "provides") + +- A **library version PROVIDES** a set of exported versioned symbols + `{sym@ver}` plus a `soname` (its ABI generation, e.g. `libc.so.6`). +- A **binary/package REQUIRES** a set of referenced versioned symbols `{sym@ver}` + plus the `sonames` it links (`NEEDED`). +- **Compatible(bin, libver)** ⟺ `soname matches` AND `requires ⊆ provides`. + +Everything below is just this predicate plus how the versions are stored. + +## The tools (PoC) + +- `elf_symbols.mjs extract|rows|compat ` — read provides/requires/soname + from any ELF via `readelf`; `compat` checks `requires ⊆ provides`. +- `sym_abi.mjs intervals|newest ` — treat each symbol's version tag as + the version it was *introduced* in, giving each symbol a validity interval + `[intro, inf)` within the soname; `newest` computes the min/max compatible + version. + +## Worked examples (real output, this machine) + +### 1. A library's provides, a binary's requires +``` +$ elf_symbols.mjs extract libc.so.6 -> n_provides: 2970, soname: libc.so.6 +$ elf_symbols.mjs extract /bin/ls -> n_requires: 112, needed: [libselinux.so.1, libc.so.6], + min_versions: { GLIBC: 2.34, LIBSELINUX: 1.0 } +``` +The **minimum** version is *derived*, not declared: it's the highest version +tag among the required symbols (`/bin/ls` references a `@GLIBC_2.34` symbol, so +it needs glibc ≥ 2.34). `ldd` never shows this; `readelf` does. + +### 2. Compatibility = set containment +``` +$ elf_symbols.mjs compat /bin/ls libc.so.6 libselinux.so.1 + binary_requires: 112, libs_provide: 3207, compatible: true, missing_count: 0 +``` +All 112 needs are covered → compatible. If any were missing → `missing > 0` → +a **hard incompatibility** (proof of "no"). + +### 3. Symbols as validity intervals (the real ABI-growth curve) +``` +$ sym_abi.mjs intervals libc.so.6 + 2443 symbols across 35 versions [2.2.5 .. 2.35] + introduced per version: ... 2.33: 12 2.34: 212 2.35: 4 + row example: ["pthread_setname_np","2.12","inf"] +``` +Each symbol is stored ONCE as `[sym, intro, inf]` — a validity interval within +the soname. (glibc 2.34 adding 212 symbols is the real pthread/rt-into-libc +merge.) "Does version V provide sym?" = `intro(sym) ≤ V`. This is the same +interval store we built for snapshot membership, one level down — and it's the +memory-pressure dataset (thousands of symbols × versions × snapshots). + +### 4. Newest-compatible, no ABI break +``` +$ sym_abi.mjs newest /bin/ls libc.so.6 + min_compatible_version: 2.34 + newest_compatible_version: 2.35 (max unbounded within soname libc.so.6) +``` +Newest version whose provides still cover ls's needs. With a backward-compatible +library there's no upper bound inside the soname — the default max is "the +soname generation." + +### 5. Where a real MAX comes from — a symbol removal +``` +$ sym_abi.mjs newest /bin/ls libc.so.6 --drop=getenv@2.30 # simulate ABI break + min = 2.34, binary needs getenv: true, effective MAX = 2.29 + => min (2.34) > max (2.29) => NO compatible version (correct no_candidate) +``` +If a newer version *removes* a symbol the binary needs (or bumps the soname), +that caps the max. Here ls needs both 2.34-era symbols *and* `getenv`, so +removing `getenv` at 2.30 makes the range empty — the resolver correctly returns +no candidate. + +## Min vs max — the epistemics (defeasible defaults + hard vetoes) + +- **min** = the verneed floor (highest required symver). Hard, derived, exact. +- **max** = a *defeasible default* = "the soname generation" (no upper bound for + backward-compatible libs), *refined downward* only by hard evidence: a needed + symbol removed, or a soname bump. Symbol analysis can only ever NARROW the + default. +- **Symbol ABSENCE / soname mismatch = a hard veto** (proof of "no") — this can + override an optimistic or wrong declared dependency. +- **Symbol PRESENCE = a defeasible "might work"** — necessary but not sufficient + (semantics unverified). It never outranks a declared/tested dependency; it only + widens the candidate set with a low-confidence maybe. + +Confidence order: `tested > declared repo dep > ELF-hard-veto / ELF-maybe > +soname default`. + +## "Is the symbol info in the repo, so we don't download the whole package?" + +Largely yes — in three tiers, cheapest first: + +1. **The min-version dependency is already in the repo index.** Debian's + `Packages` file (fetched by `apt update`) carries each package's `Depends:`, + e.g. `libc6 (>= 2.34)` — and that bound was computed *from the symbols* by + `dpkg-shlibdeps` at build time. So the coarse min needs **zero** package + download. +2. **The full per-symbol interval data is curated metadata, not the binary.** + Debian library packages ship a `.symbols` file whose lines are literally + `symbol@Base minimum-version` — i.e. *exactly our `[sym, intro]` interval + store, precomputed by the maintainer*. It lives in the package's small + **control member** (`control.tar.*`), so you fetch that member (a few KB), + not the multi-MB data. (A mirror could also publish a symbols index directly.) +3. **Only if neither is published** do you fetch the binary (or HTTP + range-fetch just its `.dynsym`/`.gnu.version_*` sections) and run `readelf` — + which is what this PoC does — then cache the result. + +Either way it's a **one-time ingest** per `(pkg,ver)`, deduped across snapshots +by the interval encoding — never a per-resolution download. And on a real +machine the "locked set" libraries are already installed, so their provides are +free; only the *candidates* you're weighing need any fetch, lazily, newest-first. + +## How it lands in the store / resolver (no new engine) + +- Ingest → `symprov(soname|sym -> intro#inf)` and `symreq(bin|sym -> ver)` rows, + interval-encoded and deduped exactly like the package pool + membership. +- `Compatible` is the existing `provides ⊇ requires` containment; the + already-built `newest_compatible_snap` generalizes from coarse version + constraints to symbol-set containment — "newest ABI-compatible library version + across snapshots" becomes a real query on real data. +- Confidence tag per edge (`declared | elf_hard | elf_maybe | soname_default | + tested`) so a package manager prunes on hard + prefers declared, and a coding + agent can turn an `elf_maybe` into `tested` by actually building — the Level-3 + oracle a package manager lacks. diff --git a/examples/pkg_resolver/abi/abi_cli.pl b/examples/pkg_resolver/abi/abi_cli.pl new file mode 100644 index 000000000..4822d818d --- /dev/null +++ b/examples/pkg_resolver/abi/abi_cli.pl @@ -0,0 +1,49 @@ +:- encoding(utf8). +% SPDX-License-Identifier: MIT OR Apache-2.0 +% Copyright (c) 2026 John William Creighton (@s243a) +% +% abi_cli.pl -- a small callable driver over the ABI resolver. +% +% swipl -q -g main -t halt examples/pkg_resolver/abi/abi_cli.pl -- \ +% +% +% Commands: +% min derived verneed floor +% newest [DropSym DropAt] newest ABI-compatible version +% range [DropSym DropAt] [min, max] or no_candidate +% compat containment check at a version + +:- use_module(abi_resolve). + +main :- + current_prolog_flag(argv, Argv), + ( Argv = [Dir, Cmd | Args] -> true ; usage, halt(2) ), + load_abi_store(Dir), + run(Cmd, Args). + +usage :- + format(user_error, + "usage: abi_cli.pl -- min|newest|range|compat ~n", []). + +drop_of([], none). +drop_of([Sym, At], drop(SymA, At)) :- atom_string(SymA, Sym). + +run(min, [Bin, So]) :- !, + ( abi_min(Bin, So, Min) -> format("min ~w~n", [Min]) ; format("min none~n", []) ). +run(newest, [Bin, So | DropArgs]) :- !, + soname_candidates(So, Cands), + drop_of(DropArgs, Drop), + newest_abi_compatible(Bin, So, Cands, Drop, R), + format("newest ~w~n", [R]). +run(range, [Bin, So | DropArgs]) :- !, + soname_candidates(So, Cands), + drop_of(DropArgs, Drop), + abi_range(Bin, So, Cands, Drop, R), + format("range ~w~n", [R]). +run(compat, [Bin, So, V]) :- !, + ( abi_compatible(Bin, So, V) + -> format("compatible ~w yes~n", [V]) + ; missing_syms(Bin, So, V, none, M), length(M, N), + format("compatible ~w no (missing ~w)~n", [V, N]) + ). +run(_, _) :- usage, halt(2). diff --git a/examples/pkg_resolver/abi/abi_resolve.pl b/examples/pkg_resolver/abi/abi_resolve.pl new file mode 100644 index 000000000..37cbb58c0 --- /dev/null +++ b/examples/pkg_resolver/abi/abi_resolve.pl @@ -0,0 +1,241 @@ +:- encoding(utf8). +% SPDX-License-Identifier: MIT OR Apache-2.0 +% Copyright (c) 2026 John William Creighton (@s243a) +% +% abi_resolve.pl -- symbol-level ABI-compatibility resolver, the fine-grained +% generalization of the package resolver's coarse `provides`. It reuses the +% SAME interval-store idea one level down: instead of a package having a +% version tenure, each exported SYMBOL has a validity interval [intro, inf) +% WITHIN a soname (a library version V provides sym iff intro(sym) =< V, until +% a soname bump / removal). Given a binary's referenced versioned symbols +% (verneed floor) and a library's exported-symbol intervals, it computes the +% MIN and MAX (newest) compatible library version via symbol-set containment. +% +% Frozen resolver.pl / resolver_store.pl are NOT edited; version comparison is +% delegated to resolver:version_lt/2. +% +% Store rows (P/2 JSONL, load_p2_jsonl shape), produced by ingest_symbols.mjs: +% symprov(Key, Val) Key = 'SoName|Sym' Val = 'Intro#inf' (interval) +% symreq(Key, Val) Key = 'Binary|Sym' Val = 'SoName#Ver' (verneed) +% needed(Binary, SoName) (DT_NEEDED) + +:- module(abi_resolve, [ + load_abi_store/1, + abi_store_clear/0, + provides_at/3, + symprov_intro/3, + req_sym/4, + abi_min/3, + abi_compatible/3, + abi_compatible/4, + missing_syms/5, + soname_candidates/2, + newest_providing/5, + newest_abi_compatible/4, + newest_abi_compatible/5, + abi_range/4, + abi_range/5 +]). + +:- use_module('../resolver', [version_lt/2]). +:- use_module(library(http/json)). + +:- dynamic symprov/2. +:- dynamic symreq/2. +:- dynamic needed/2. + +% --------------------------------------------------------------------------- +% Store loading (mirrors resolver_store:load_p2_jsonl / load_pairs) +% --------------------------------------------------------------------------- + +abi_store_clear :- + retractall(symprov(_, _)), + retractall(symreq(_, _)), + retractall(needed(_, _)). + +load_abi_store(Dir) :- + abi_store_clear, + load_pairs(Dir, 'symprov.jsonl', symprov), + load_pairs(Dir, 'symreq.jsonl', symreq), + load_pairs(Dir, 'needed.jsonl', needed). + +load_pairs(Dir, File, Pred) :- + atomic_list_concat([Dir, '/', File], Path), + ( exists_file(Path) + -> setup_call_cleanup(open(Path, read, S), + load_pair_lines(S, Pred), + close(S)) + ; true + ). + +load_pair_lines(S, Pred) :- + read_line_to_string(S, Line), + ( Line == end_of_file + -> true + ; ( Line == "" + -> true + ; atom_string(Atom, Line), + atom_json_term(Atom, [K, V], [value_string_as(atom)]), + Fact =.. [Pred, K, V], + assertz(Fact) + ), + load_pair_lines(S, Pred) + ). + +% --------------------------------------------------------------------------- +% Version comparison (delegated to frozen resolver:version_lt/2) +% --------------------------------------------------------------------------- +% Dotted numeric versions ("2.34", "2.2.5") -> v(A,B,C), padded with 0 so the +% 2- and 3-component glibc tags compare correctly. + +ver_term(V, v(A, B, C)) :- + ( atom(V) -> atom_string(V, S) ; V = S ), + split_string(S, ".", "", Parts0), + exclude(==(""), Parts0, Parts), + nums_pad(Parts, A, B, C). + +nums_pad(Parts, A, B, C) :- + ( nth0(0, Parts, P0) -> to_num(P0, A) ; A = 0 ), + ( nth0(1, Parts, P1) -> to_num(P1, B) ; B = 0 ), + ( nth0(2, Parts, P2) -> to_num(P2, C) ; C = 0 ). + +to_num(S, N) :- ( number(S) -> N = S ; number_string(N0, S) -> N = N0 ; N = 0 ). + +ver_lt(A, B) :- ver_term(A, TA), ver_term(B, TB), resolver:version_lt(TA, TB). +ver_le(A, B) :- \+ ver_lt(B, A). + +% max version of a non-empty list (highest wins). +max_ver([V|Vs], Max) :- foldl(max_ver_1, Vs, V, Max). +max_ver_1(V, Acc, Out) :- ( ver_lt(Acc, V) -> Out = V ; Out = Acc ). + +% --------------------------------------------------------------------------- +% Store accessors +% --------------------------------------------------------------------------- + +% intro(SoName|Sym) -- the symbol's introduced version (interval floor). +symprov_intro(SoName, Sym, Intro) :- + symprov(Key, Val), + split_key(Key, SoName, Sym), + split_hash(Val, Intro, _To). + +% req_sym(Binary, SoName, Sym, Ver) -- a versioned symbol the binary needs. +req_sym(Binary, SoName, Sym, Ver) :- + symreq(Key, Val), + split_key(Key, Binary, Sym), + split_hash(Val, SoName, Ver). + +split_key(Key, A, B) :- + ( atom(Key) -> atom_string(Key, S) ; S = Key ), + sub_string(S, Before, _, After, "|"), + !, + sub_string(S, 0, Before, _, AS), + sub_string(S, _, After, 0, BS), + atom_string(A, AS), + atom_string(B, BS). + +split_hash(Val, A, B) :- + ( atom(Val) -> atom_string(Val, S) ; S = Val ), + split_string(S, "#", "", [AS, BS | _]), + atom_string(A, AS), + atom_string(B, BS). + +% --------------------------------------------------------------------------- +% Core ABI predicates +% --------------------------------------------------------------------------- + +% provides_at(SoName, Sym, V): library version V exports Sym, i.e. it was +% introduced at or before V (to = inf within a soname). +provides_at(SoName, Sym, V) :- + symprov_intro(SoName, Sym, Intro), + ver_le(Intro, V). + +% provides_at with a simulated removal: drop(DropSym, At) removes DropSym for +% every V >= At (a soname-era ABI break, for testing the max). +provides_at_drop(SoName, Sym, V, none) :- !, + provides_at(SoName, Sym, V). +provides_at_drop(_SoName, Sym, V, drop(Sym, At)) :- + ver_le(At, V), !, + fail. +provides_at_drop(SoName, Sym, V, _Drop) :- + provides_at(SoName, Sym, V). + +% abi_min(Binary, SoName, Min): the verneed floor -- the highest required +% version among the binary's required symbols of that soname. Hard/derived. +abi_min(Binary, SoName, Min) :- + findall(Ver, req_sym(Binary, SoName, _Sym, Ver), Vers), + Vers \== [], + max_ver(Vers, Min). + +% missing_syms(Binary, SoName, V, Drop, Missing): required symbols of SoName +% NOT provided at library version V (containment failures = hard vetoes). +missing_syms(Binary, SoName, V, Drop, Missing) :- + findall(Sym, + ( req_sym(Binary, SoName, Sym, _), + \+ provides_at_drop(SoName, Sym, V, Drop) + ), + Missing0), + sort(Missing0, Missing). + +% abi_compatible(Binary, SoName, V): every required symbol of SoName is +% provided at library version V (set containment). +abi_compatible(Binary, SoName, V) :- + abi_compatible(Binary, SoName, V, none). + +abi_compatible(Binary, SoName, V, Drop) :- + missing_syms(Binary, SoName, V, Drop, []). + +% soname_candidates(SoName, Descending): the soname's version axis = the +% distinct intro versions in the store, highest first. +soname_candidates(SoName, Desc) :- + findall(Intro, symprov_intro(SoName, _Sym, Intro), Intros0), + sort(Intros0, Uniq), + predsort(cmp_ver_desc, Uniq, Desc). + +cmp_ver_desc(Order, A, B) :- + ( ver_lt(A, B) -> Order = (>) + ; ver_lt(B, A) -> Order = (<) + ; Order = (=) + ). + +% newest_providing(SoName, Sym, Candidates, Drop, Result): the newest V in +% Candidates (descending) that still exports Sym under Drop -> compatible(V); +% else no_candidate. With Drop = drop(Sym, At) this is the "effective max cap" +% a symbol removal imposes (the newest version just below the removal). +newest_providing(_SoName, _Sym, [], _Drop, no_candidate) :- !. +newest_providing(SoName, Sym, [V|Vs], Drop, Result) :- + ( provides_at_drop(SoName, Sym, V, Drop) + -> Result = compatible(V) + ; newest_providing(SoName, Sym, Vs, Drop, Result) + ). + +% newest_abi_compatible(Binary, SoName, Candidates, Result): +% newest V in Candidates (descending) with abi_compatible -> compatible(V); +% else no_candidate. +newest_abi_compatible(Binary, SoName, Candidates, Result) :- + newest_abi_compatible(Binary, SoName, Candidates, none, Result). + +newest_abi_compatible(_Binary, _SoName, [], _Drop, no_candidate) :- !. +newest_abi_compatible(Binary, SoName, [V|Vs], Drop, Result) :- + ( abi_compatible(Binary, SoName, V, Drop) + -> Result = compatible(V) + ; newest_abi_compatible(Binary, SoName, Vs, Drop, Result) + ). + +% abi_range(Binary, SoName, Candidates, Result): combine the derived MIN +% (verneed floor) with the newest compatible MAX. If min > max (a removed +% symbol squeezed the range) -> no_candidate. +abi_range(Binary, SoName, Candidates, Result) :- + abi_range(Binary, SoName, Candidates, none, Result). + +abi_range(Binary, SoName, Candidates, Drop, Result) :- + ( abi_min(Binary, SoName, Min) -> true ; Min = none ), + newest_abi_compatible(Binary, SoName, Candidates, Drop, Newest), + ( Newest = compatible(Max) + -> ( Min == none + -> Result = range(none, Max) + ; ver_le(Min, Max) + -> Result = range(Min, Max) + ; Result = no_candidate(min_gt_max(Min, Max)) + ) + ; Result = no_candidate(no_compatible_version) + ). diff --git a/examples/pkg_resolver/abi/ingest_symbols.mjs b/examples/pkg_resolver/abi/ingest_symbols.mjs new file mode 100644 index 000000000..a431b8c85 --- /dev/null +++ b/examples/pkg_resolver/abi/ingest_symbols.mjs @@ -0,0 +1,269 @@ +#!/usr/bin/env node +// SPDX-License-Identifier: MIT OR Apache-2.0 +// Copyright (c) 2026 John William Creighton (@s243a) +// +// ingest_symbols.mjs -- ingest symbol-level ABI metadata into the interval +// store as P/2 JSONL rows, mirroring the pkg_resolver store shape one level +// down (symbol tenures instead of package tenures). Three ingest tiers: +// +// symbols-file parse a Debian/Ubuntu `.symbols` control member -> +// symprov interval rows. ZERO binary download; the +// maintainer already computed the [sym, intro] table. +// elf readelf fallback for a `.so` with no `.symbols`: +// exported versioned symbols -> symprov rows +// (intro = the symbol's own version tag). +// requires readelf: the binary's referenced versioned symbols +// -> symreq rows (attributed to their soname via the +// verneed/.gnu.version_r table) + NEEDED sonames. +// symbols-dir batch: ingest every *.symbols under . +// +// Store rows are P/2 pairs `[key, value]` (the D43 indexer / load_p2_jsonl +// shape). Intervals are `intro#inf` within a soname (to="inf"). +// +// symprov.jsonl : ["|", "#inf"] +// symreq.jsonl : ["|", "#"] +// needed.jsonl : ["", ""] +// +// Usage: +// node ingest_symbols.mjs symbols-file [--out DIR] [--stdout] +// node ingest_symbols.mjs elf [--out DIR] [--stdout] +// node ingest_symbols.mjs requires [--out DIR] [--stdout] +// node ingest_symbols.mjs symbols-dir [--out DIR] + +import { execFileSync } from "node:child_process"; +import { readFileSync, appendFileSync, writeFileSync, mkdirSync, readdirSync } from "node:fs"; +import { dirname, join } from "node:path"; + +function readelf(args, file) { + try { + return execFileSync("readelf", [...args, file], { encoding: "utf8", maxBuffer: 1 << 26 }); + } catch { + return ""; + } +} + +function pair(k, v) { + return JSON.stringify([k, v]); +} + +// A versioned symbol's numeric intro version, extracted from a version tag such +// as GLIBC_2.34 / LIBSELINUX_1.0 -> "2.34" / "1.0". Non-numeric namespaces +// (e.g. GLIBC_PRIVATE, @Base) return null so they never set a bound. +function verNum(ver) { + const m = String(ver).match(/_([0-9][0-9.]*)$/); + return m ? m[1] : null; +} + +// --------------------------------------------------------------------------- +// Tier 1: parse a Debian/Ubuntu `.symbols` control member. +// --------------------------------------------------------------------------- +// Format (per soname block): +// #MINVER# <- header (no leading space) +// | libc6 (>> 2.35), libc6 (<< 2.36) <- alt-dep template (skip) +// * Build-Depends-Package: libc6-dev <- meta field (skip) +// @ [] <- symbol (leading space) +// A symbol line may carry leading `(tag=value|...)` selectors which we strip. +// The `minimum-version` field IS the symbol's introduced version (our intro). +function parseSymbolsFile(path) { + const text = readFileSync(path, "utf8"); + let soname = null; + const rows = []; // {soname, sym, intro} + const sonames = new Set(); + let symCount = 0, skipped = 0; + for (const raw of text.split("\n")) { + if (!raw) continue; + // Header / meta lines are NOT indented. + if (!/^[ \t]/.test(raw)) { + const c = raw[0]; + if (c === "|" || c === "*" || c === "#") continue; // alt-dep / meta / comment + soname = raw.trim().split(/\s+/)[0]; + if (soname) sonames.add(soname); + continue; + } + if (!soname) continue; + let line = raw.trim(); + if (!line || line[0] === "|" || line[0] === "*" || line[0] === "#") continue; + // Strip a leading (tag=value|...) selector group, e.g. "(optional)sym@Base". + if (line[0] === "(") { + const close = line.indexOf(")"); + if (close >= 0) line = line.slice(close + 1).trimStart(); + } + // "@ [id]" -- symbol names have no '@'. + const at = line.indexOf("@"); + if (at < 0) { skipped++; continue; } + const sym = line.slice(0, at); + const rest = line.slice(at + 1).split(/\s+/); + const version = rest[0]; // the symver tag, e.g. GLIBC_2.34 or Base + const minver = rest[1]; // the introduced-version field + if (!sym || minver === undefined) { skipped++; continue; } + // intro = the curated minimum-version field. "0" (private) stays 0.0.0. + const intro = /^[0-9]/.test(minver) ? minver : (verNum(version) || "0"); + rows.push({ soname, sym, intro }); + symCount++; + } + return { rows, sonames: [...sonames], symCount, skipped }; +} + +// --------------------------------------------------------------------------- +// readelf helpers (Tiers 2 & 3). +// --------------------------------------------------------------------------- +// Defined versioned syms = provides; UND versioned syms = requires. +function dynsyms(file) { + const out = readelf(["-W", "--dyn-syms"], file); + const defined = [], undef = []; + for (const raw of out.split("\n")) { + const t = raw.trim().split(/\s+/); + if (t.length < 8 || !/^\d+:$/.test(t[0])) continue; + const ndx = t[6], name = t[7]; + if (!name.includes("@")) continue; + const [sym, ver] = name.replace("@@", "@").split("@"); + if (!sym || !ver) continue; + (ndx === "UND" ? undef : defined).push([sym, ver]); + } + return { defined, undef }; +} + +function dynamic(file) { + const out = readelf(["-d"], file); + let soname = null; + const needed = []; + for (const raw of out.split("\n")) { + let m; + if ((m = raw.match(/\(SONAME\)\s+Library soname: \[([^\]]+)\]/))) soname = m[1]; + else if ((m = raw.match(/\(NEEDED\)\s+Shared library: \[([^\]]+)\]/))) needed.push(m[1]); + } + return { soname, needed }; +} + +// Map each version tag (GLIBC_2.34) to the soname (File) that provides it, +// from the `.gnu.version_r` (verneed) table. +function verneedMap(file) { + const out = readelf(["-V"], file); + const map = new Map(); // versionName -> soname + let inNeeds = false, curFile = null; + for (const raw of out.split("\n")) { + if (/Version needs section/.test(raw)) { inNeeds = true; continue; } + if (inNeeds && /Version (definition|symbols) section/.test(raw)) inNeeds = false; + if (!inNeeds) continue; + let m; + if ((m = raw.match(/File:\s+(\S+)/))) curFile = m[1]; + if ((m = raw.match(/Name:\s+(\S+)/)) && curFile) map.set(m[1], curFile); + } + return map; +} + +// --------------------------------------------------------------------------- +// Output sink. +// --------------------------------------------------------------------------- +function makeSink(outDir, toStdout) { + const buffers = { symprov: [], symreq: [], needed: [] }; + return { + add(store, k, v) { buffers[store].push(pair(k, v)); }, + flush(append) { + if (outDir) mkdirSync(outDir, { recursive: true }); + for (const store of Object.keys(buffers)) { + const lines = buffers[store]; + if (!lines.length) continue; + if (outDir) { + const f = join(outDir, store + ".jsonl"); + const body = lines.join("\n") + "\n"; + if (append) appendFileSync(f, body); + else writeFileSync(f, body); + } + if (toStdout) for (const l of lines) process.stdout.write(l + "\n"); + } + }, + }; +} + +// --------------------------------------------------------------------------- +// Commands. +// --------------------------------------------------------------------------- +function parseArgs(rest) { + const opts = { out: null, stdout: false, append: false, positional: [] }; + for (let i = 0; i < rest.length; i++) { + const a = rest[i]; + if (a === "--out") opts.out = rest[++i]; + else if (a === "--stdout") opts.stdout = true; + else if (a === "--append") opts.append = true; + else opts.positional.push(a); + } + return opts; +} + +const [cmd, ...rest] = process.argv.slice(2); +const opts = parseArgs(rest); +// symbols-dir manages its own append semantics; other modes default to +// overwrite unless --append. If neither --out nor --stdout, echo to stdout. +if (!opts.out && !opts.stdout) opts.stdout = true; + +if (cmd === "symbols-file") { + const path = opts.positional[0]; + const { rows, sonames, symCount, skipped } = parseSymbolsFile(path); + const sink = makeSink(opts.out, opts.stdout); + for (const { soname, sym, intro } of rows) sink.add("symprov", soname + "|" + sym, intro + "#inf"); + sink.flush(opts.append); + process.stderr.write( + `symbols-file ${path}: symprov=${symCount} sonames=${sonames.length} [${sonames.slice(0, 6).join(", ")}${sonames.length > 6 ? ", ..." : ""}] skipped=${skipped}\n` + ); +} else if (cmd === "elf") { + const path = opts.positional[0]; + const { defined } = dynsyms(path); + const { soname } = dynamic(path); + const so = soname || path; + // intro(sym) = earliest version tag seen on the exported symbol. + const intro = new Map(); + for (const [sym, ver] of defined) { + const n = verNum(ver); + if (!n) continue; + if (!intro.has(sym) || cmpVer(n, intro.get(sym)) < 0) intro.set(sym, n); + } + const sink = makeSink(opts.out, opts.stdout); + for (const [sym, v] of intro) sink.add("symprov", so + "|" + sym, v + "#inf"); + sink.flush(opts.append); + process.stderr.write(`elf ${path}: symprov=${intro.size} soname=${so}\n`); +} else if (cmd === "requires") { + const path = opts.positional[0]; + const { undef } = dynsyms(path); + const { needed } = dynamic(path); + const vmap = verneedMap(path); + const sink = makeSink(opts.out, opts.stdout); + let n = 0; + for (const [sym, ver] of undef) { + const num = verNum(ver); + if (!num) continue; // ignore GLIBC_PRIVATE etc. + const so = vmap.get(ver) || "?"; // soname from verneed + sink.add("symreq", path + "|" + sym, so + "#" + num); + n++; + } + for (const so of needed) sink.add("needed", path, so); + sink.flush(opts.append); + process.stderr.write(`requires ${path}: symreq=${n} needed=[${needed.join(", ")}]\n`); +} else if (cmd === "symbols-dir") { + const dir = opts.positional[0]; + const files = readdirSync(dir).filter((f) => f.endsWith(".symbols")).map((f) => join(dir, f)); + if (opts.out) { mkdirSync(opts.out, { recursive: true }); writeFileSync(join(opts.out, "symprov.jsonl"), ""); } + let total = 0, nfiles = 0; + for (const f of files) { + let parsed; + try { parsed = parseSymbolsFile(f); } catch { continue; } + const sink = makeSink(opts.out, false); + for (const { soname, sym, intro } of parsed.rows) sink.add("symprov", soname + "|" + sym, intro + "#inf"); + sink.flush(true); // append across all files + total += parsed.symCount; + nfiles++; + } + process.stderr.write(`symbols-dir ${dir}: files=${nfiles} symprov=${total}\n`); +} else { + process.stderr.write("usage: ingest_symbols.mjs symbols-file|elf|requires|symbols-dir [--out DIR] [--stdout] [--append]\n"); + process.exit(2); +} + +function cmpVer(a, b) { + const pa = a.split("."), pb = b.split("."); + for (let i = 0; i < Math.max(pa.length, pb.length); i++) { + const x = +(pa[i] || 0), y = +(pb[i] || 0); + if (x !== y) return x - y; + } + return 0; +} diff --git a/examples/pkg_resolver/abi/run_abi_verify.sh b/examples/pkg_resolver/abi/run_abi_verify.sh new file mode 100755 index 000000000..46a420dfc --- /dev/null +++ b/examples/pkg_resolver/abi/run_abi_verify.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MIT OR Apache-2.0 +# Copyright (c) 2026 John William Creighton (@s243a) +# +# run_abi_verify.sh -- build the symbol-level ABI store from THIS machine's +# real Ubuntu/Debian data and verify it against the PoC's numbers. +# +# Tiers exercised: +# 1. .symbols control member -> symprov intervals (zero binary download) +# 2. readelf on libc.so.6 -> symprov (fallback), cross-checked vs tier 1 +# 3. readelf on /bin/ls -> symreq (verneed floor) + NEEDED sonames +# +# Everything lands in ./.out (gitignored). Requires: node, readelf, swipl. +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +OUT="$HERE/.out" +STORE="$OUT/store" +BINARY="${ABI_BINARY:-/bin/ls}" +SONAME="${ABI_SONAME:-libc.so.6}" +LIBSO="${ABI_LIBSO:-/lib/x86_64-linux-gnu/libc.so.6}" + +# Locate libc6's .symbols control member (arch-qualified on multiarch). +SYMFILE="" +for c in /var/lib/dpkg/info/libc6:amd64.symbols /var/lib/dpkg/info/libc6.symbols; do + [ -f "$c" ] && SYMFILE="$c" && break +done +[ -z "$SYMFILE" ] && SYMFILE="$(ls /var/lib/dpkg/info/libc6*.symbols 2>/dev/null | grep -v i386 | head -1)" +[ -z "$SYMFILE" ] && { echo "no libc6 .symbols file found" >&2; exit 1; } + +rm -rf "$OUT" +mkdir -p "$STORE" "$OUT/cmp" + +echo "== Tier 1: ingest $SYMFILE (.symbols control member) -> symprov ==" +node "$HERE/ingest_symbols.mjs" symbols-file "$SYMFILE" --out "$STORE" + +echo +echo "== Tier 3: ingest $BINARY requires -> symreq + NEEDED ==" +node "$HERE/ingest_symbols.mjs" requires "$BINARY" --out "$STORE" --append + +echo +echo "== Tier 2 cross-check: readelf($LIBSO) vs .symbols intros ==" +node "$HERE/ingest_symbols.mjs" symbols-file "$SYMFILE" --out "$OUT/cmp/sym" >/dev/null 2>&1 +node "$HERE/ingest_symbols.mjs" elf "$LIBSO" --out "$OUT/cmp/elf" >/dev/null 2>&1 +node -e ' +const fs=require("fs"); +const load=(f,so)=>{const m=new Map();for(const l of fs.readFileSync(f,"utf8").split("\n")){if(!l)continue;const [k,v]=JSON.parse(l);if(!k.startsWith(so+"|"))continue;m.set(k.slice(so.length+1),v.split("#")[0]);}return m;}; +const so=process.argv[3]; +const s=load(process.argv[1]+"/symprov.jsonl",so), e=load(process.argv[2]+"/symprov.jsonl",so); +let shared=0,agree=0; const dis=[]; +for(const [sym,iv] of e){ if(s.has(sym)){shared++; if(s.get(sym)===iv)agree++; else if(dis.length<5)dis.push(sym+" (.symbols="+s.get(sym)+" elf="+iv+")");}} +const pct=100*agree/shared; +console.log(" .symbols libc.so.6 syms:",s.size,"| readelf syms:",e.size,"| shared:",shared); +console.log(" intro agreement:",agree+"/"+shared,"("+pct.toFixed(1)+"%)"); +console.log(" sample disagreements (glibc 2.34 pthread/rt merge):",JSON.stringify(dis)); +if(pct<85){console.error(" FAIL: intro agreement below 85%");process.exit(1);} +if(s.get("getenv")!==e.get("getenv")){console.error(" FAIL: getenv intro mismatch");process.exit(1);} +console.log(" PASS: curated .symbols and readelf agree on the intro axis"); +' "$OUT/cmp/sym" "$OUT/cmp/elf" "$SONAME" + +echo +echo "== Prolog resolver verification ==" +swipl -q -g run -t halt "$HERE/test_abi.pl" -- "$STORE" diff --git a/examples/pkg_resolver/abi/test_abi.pl b/examples/pkg_resolver/abi/test_abi.pl new file mode 100644 index 000000000..aeb9ddab0 --- /dev/null +++ b/examples/pkg_resolver/abi/test_abi.pl @@ -0,0 +1,83 @@ +:- encoding(utf8). +% SPDX-License-Identifier: MIT OR Apache-2.0 +% Copyright (c) 2026 John William Creighton (@s243a) +% +% test_abi.pl -- verify the symbol-level ABI lane on THIS machine's real data. +% The store must already be built (see run_abi_verify.sh) into a directory +% passed as the first program argument (default: ./.out/store). +% +% swipl -q -g run -t halt examples/pkg_resolver/abi/test_abi.pl -- +% +% Asserts (the PoC's numbers, libc6 / /bin/ls on Ubuntu 22.04): +% abi_min(/bin/ls, libc.so.6) == 2.34 +% abi_compatible(/bin/ls, libc.so.6, 2.35) is TRUE (0 missing) +% newest_abi_compatible(no drop) == compatible(2.35), range(2.34,2.35) +% simulated removal of getenv at 2.30 -> cap 2.29, min 2.34 > max 2.29 +% -> no_candidate (the min>max squeeze) + +:- use_module(abi_resolve). + +:- dynamic pass_count/1, fail_count/1. +pass_count(0). +fail_count(0). + +check(Name, Goal) :- + ( catch(Goal, E, (print_message(error, E), fail)) + -> format(" PASS ~w~n", [Name]), + retract(pass_count(P)), P1 is P + 1, assertz(pass_count(P1)) + ; format(" FAIL ~w~n", [Name]), + retract(fail_count(F)), F1 is F + 1, assertz(fail_count(F1)) + ). + +store_dir(Dir) :- + ( current_prolog_flag(argv, [D | _]), D \== [] -> Dir = D ; Dir = './.out/store' ). + +run :- + store_dir(Dir), + format("~n== ABI lane verification (store: ~w) ==~n", [Dir]), + load_abi_store(Dir), + aggregate_all(count, abi_resolve:symprov(_,_), NP), + aggregate_all(count, abi_resolve:symreq(_,_), NR), + aggregate_all(count, ( abi_resolve:symprov(K,_), atom_concat('libc.so.6|', _, K) ), NLibc), + format(" store: symprov=~w (libc.so.6=~w) symreq=~w~n~n", [NP, NLibc, NR]), + + Bin = '/bin/ls', So = 'libc.so.6', + soname_candidates(So, Cands), + length(Cands, NC), + format(" libc.so.6 version axis: ~w distinct versions~n", [NC]), + + check('abi_min(/bin/ls, libc.so.6) == 2.34', + ( abi_min(Bin, So, Min), report(' abi_min', Min), Min == '2.34' )), + + check('abi_compatible(/bin/ls, libc.so.6, 2.35) is TRUE', + abi_compatible(Bin, So, '2.35')), + + check('0 missing symbols at 2.35', + ( missing_syms(Bin, So, '2.35', none, M), length(M, LM), + report(' missing@2.35', LM), LM =:= 0 )), + + check('newest_abi_compatible (no drop) == compatible(2.35)', + ( newest_abi_compatible(Bin, So, Cands, R), + report(' newest', R), R == compatible('2.35') )), + + check('abi_range (no drop) == range(2.34, 2.35)', + ( abi_range(Bin, So, Cands, RR), + report(' range', RR), RR == range('2.34', '2.35') )), + + % Simulated ABI break: remove getenv at 2.30 (a symbol /bin/ls needs). + check('/bin/ls needs getenv', + req_sym(Bin, So, getenv, _)), + + check('removal cap: newest version still exporting getenv == 2.29', + ( newest_providing(So, getenv, Cands, drop(getenv, '2.30'), Cap), + report(' getenv cap', Cap), Cap == compatible('2.29') )), + + check('simulated removal -> no_candidate (min 2.34 > max 2.29 squeeze)', + ( abi_range(Bin, So, Cands, drop(getenv, '2.30'), RD), + report(' range(drop)', RD), RD = no_candidate(_) )), + + pass_count(P), fail_count(F), + format("~n== ~w passed, ~w failed ==~n", [P, F]), + ( F =:= 0 -> true ; halt(1) ). + +report(Label, Val) :- format("~w = ~w~n", [Label, Val]). From 3dac22083efe7f1bfd0390ad3a2250175428fd5b Mon Sep 17 00:00:00 2001 From: "John William Creighton (s243a)" Date: Sat, 12 Sep 2026 23:28:59 -0600 Subject: [PATCH 02/10] Redesign symbol-level ABI lane: exact sym@node identity, separate axes, explicit evidence Addresses Astra's REQUEST-CHANGES review of #4262. Direction unchanged (three evidence tiers; a driver above the frozen resolver); model + ingest rebuilt. 1. Identity is the exact (soname, symbol, version-node) triple, stored and matched by string equality; no collapse to bare names, no numeric intro. 2. Every obligation is preserved (non-numeric nodes, unversioned and weak references); NEEDED is checked; evidence completeness is explicit (evidence.jsonl) and a missing/unreadable ELF is a loud exit-3 failure that records INCOMPLETE evidence instead of an empty success. Verdicts are tri-state: compatible(exact|extrapolated) / incompatible / unknown. 3. Requirements are attributed via the ELF version INDEX (.gnu.version -> .gnu.version_r), so two libraries sharing a node name (COMMON_1) no longer collide. 4. The Debian package-version axis reuses debian/deb_parse + the frozen resolver:version_lt/2 on deb/3 (epoch, tilde, revision); ELF nodes are an unordered label axis. `.symbols` minimums are documented as curated lower bounds, not introduction dates (below_floor is reported distinctly). 5. Ranges are computed over the actual release axis (apt-cache madison + dpkg); [min,max] ends are compatible by construction (range(1,5) bug gone). 6. Cross-check compares exact sym@node sets: 3006/3006 (100%); corrected per-name earliest-row figure 2478/2478 (100%). The false "glibc 2.34 merge divergence" explanation is removed from the docs. 7. `.symbols` templates: (optional)/(arch=..) processed (arch needs --arch); (symver)/(regex)/(c++)/quoted patterns/#include rejected loudly. Verification (Ubuntu 22.04.5, libc6 2.35-0ubuntu3.15, /bin/ls): computed floors 2.34 / 3.1~ equal coreutils' declared Pre-Depends; gcc-built fixtures for foo@LIB_1-vs-foo@LIB_2 (loader confirms "undefined symbol: foo, version LIB_1"), COMMON_1 cross-attribution, PUBLIC + unversioned obligations, and a missing ELF. test_abi.pl: 63 passed, 0 failed. REVIEW_NOTES.md maps each review point to code + proving fixture. Frozen resolver files untouched. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01RoXjhStCqoig6944pVNBGe --- examples/pkg_resolver/abi/.gitignore | 3 +- examples/pkg_resolver/abi/README.md | 233 ++++--- examples/pkg_resolver/abi/REVIEW_NOTES.md | 161 +++++ examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md | 267 ++++---- examples/pkg_resolver/abi/abi_cli.pl | 55 +- examples/pkg_resolver/abi/abi_resolve.pl | 544 ++++++++++------ examples/pkg_resolver/abi/crosscheck.mjs | 73 +++ examples/pkg_resolver/abi/fixtures/alpha.c | 2 + examples/pkg_resolver/abi/fixtures/beta.c | 2 + examples/pkg_resolver/abi/fixtures/common.map | 1 + examples/pkg_resolver/abi/fixtures/foo.c | 7 + examples/pkg_resolver/abi/fixtures/foo_v1.map | 1 + examples/pkg_resolver/abi/fixtures/foo_v2.map | 2 + examples/pkg_resolver/abi/fixtures/plain.c | 3 + examples/pkg_resolver/abi/fixtures/pub.c | 2 + examples/pkg_resolver/abi/fixtures/pub.map | 1 + .../pkg_resolver/abi/fixtures/simple.symbols | 11 + .../abi/fixtures/tmpl_arch.symbols | 6 + .../abi/fixtures/tmpl_cxx.symbols | 3 + .../abi/fixtures/tmpl_symver.symbols | 3 + .../pkg_resolver/abi/fixtures/usecommon.c | 7 + examples/pkg_resolver/abi/fixtures/usefoo.c | 4 + examples/pkg_resolver/abi/fixtures/usepub.c | 6 + examples/pkg_resolver/abi/ingest_symbols.mjs | 581 ++++++++++++------ examples/pkg_resolver/abi/run_abi_verify.sh | 157 +++-- examples/pkg_resolver/abi/test_abi.pl | 411 +++++++++++-- 26 files changed, 1854 insertions(+), 692 deletions(-) create mode 100644 examples/pkg_resolver/abi/REVIEW_NOTES.md create mode 100644 examples/pkg_resolver/abi/crosscheck.mjs create mode 100644 examples/pkg_resolver/abi/fixtures/alpha.c create mode 100644 examples/pkg_resolver/abi/fixtures/beta.c create mode 100644 examples/pkg_resolver/abi/fixtures/common.map create mode 100644 examples/pkg_resolver/abi/fixtures/foo.c create mode 100644 examples/pkg_resolver/abi/fixtures/foo_v1.map create mode 100644 examples/pkg_resolver/abi/fixtures/foo_v2.map create mode 100644 examples/pkg_resolver/abi/fixtures/plain.c create mode 100644 examples/pkg_resolver/abi/fixtures/pub.c create mode 100644 examples/pkg_resolver/abi/fixtures/pub.map create mode 100644 examples/pkg_resolver/abi/fixtures/simple.symbols create mode 100644 examples/pkg_resolver/abi/fixtures/tmpl_arch.symbols create mode 100644 examples/pkg_resolver/abi/fixtures/tmpl_cxx.symbols create mode 100644 examples/pkg_resolver/abi/fixtures/tmpl_symver.symbols create mode 100644 examples/pkg_resolver/abi/fixtures/usecommon.c create mode 100644 examples/pkg_resolver/abi/fixtures/usefoo.c create mode 100644 examples/pkg_resolver/abi/fixtures/usepub.c diff --git a/examples/pkg_resolver/abi/.gitignore b/examples/pkg_resolver/abi/.gitignore index 3d1a51dc3..65b2c4210 100644 --- a/examples/pkg_resolver/abi/.gitignore +++ b/examples/pkg_resolver/abi/.gitignore @@ -1,4 +1,3 @@ -# Generated symbol-ABI stores and verification artifacts (rebuild via +# Generated stores, built fixtures and verification artifacts (rebuild via # run_abi_verify.sh). Never commit machine-specific ingested data. .out/ -*.jsonl diff --git a/examples/pkg_resolver/abi/README.md b/examples/pkg_resolver/abi/README.md index 57db8ad07..8164358aa 100644 --- a/examples/pkg_resolver/abi/README.md +++ b/examples/pkg_resolver/abi/README.md @@ -3,136 +3,133 @@ # Symbol-level ABI-compatibility lane (`pkg_resolver/abi`) -This lane promotes the coarse package resolver's `provides` (a package provides -a virtual name / satisfies `libfoo (>= 2.0)`) to **fine-grained ABI -compatibility at the *symbol* level**. It computes, for a binary, the **MIN** -and **MAX (newest)** compatible library version via symbol-set containment — -the real `[min, max]` range that `ldd` will not tell you. - -It reuses the SAME interval-store idea one level down: instead of a package -having a version tenure, each exported **symbol** has a validity interval -`[intro, inf)` *within a soname*. A library version `V` provides a symbol iff -`intro(sym) =< V` (until a soname bump / removal). The frozen `resolver.pl` / -`resolver_store.pl` are **not** edited — version comparison is delegated to -`resolver:version_lt/2`. - -## The model (one generalization of `provides`) - -- A **library version PROVIDES** a set of exported versioned symbols plus a - `soname` (its ABI generation, e.g. `libc.so.6`). -- A **binary REQUIRES** a set of referenced versioned symbols plus the sonames - it links (`DT_NEEDED`). -- `compatible(Bin, LibVer)` ⟺ `soname matches` AND `requires ⊆ provides`. +This lane refines the coarse package resolver's `provides` (`libc6 (>= 2.34)`) +to **ABI compatibility at the exact versioned-symbol level**: for a binary and +a shared library it answers *compatible / incompatible / unknown* per candidate +release, and computes the `[min, max]` compatible release range that `ldd` +cannot tell you. It is a driver **above** the frozen resolver: `resolver.pl` +and `resolver_store.pl` are not edited; the Debian-version comparison is +delegated to `resolver:version_lt/2` on `deb/3` terms from +`debian/deb_parse.pl`. + +The lane was redesigned after the PR #4262 review; `REVIEW_NOTES.md` maps each +review point to the code and the fixture that proves it. + +## The model + +Identity is the exact **(soname, symbol, version-node)** triple. Two axes are +kept strictly apart: + +| Axis | Values | Ordering | Used for | +|------|--------|----------|----------| +| ELF version node | `GLIBC_2.34`, `LIBSELINUX_1.0`, `COMMON_1`, `PUBLIC`, `Base` | **none** — opaque labels matched by string equality | *Does release R export `Sym@Node`?* A requirement `foo@LIB_1` is satisfied only by a provider row `foo@LIB_1` on the same soname. `foo@LIB_2` is not a match (the loader agrees: `undefined symbol: foo, version LIB_1`). `Base` is dpkg's spelling of "unversioned". | +| Debian package version | `2.34`, `3.1~`, `1:2.3-1`, `2.35-0ubuntu3.15` | `resolver:version_lt/2` over `deb(Epoch, Upstream, Revision)` (epoch, `~`, revision, any number of components) | the `.symbols` minimum-version field (a curated **lower bound**, not an introduction date) and the **release candidates**. | + +Evidence is explicit and tri-state: + +- `prov_evidence(So, symbols|elf, R0, complete)` — the soname's export set is + completely known at evidence release `R0`. +- `req_evidence(Bin, readelf, complete | missing_file | readelf_failed | inconsistent, Detail)`. +- Provider bounds: `since(Min)` from `.symbols` (exported at every release + `>= Min`, exact up to `R0`, *extrapolated* beyond it); `at(R0)` from readelf + (exact at `R0`, extrapolated beyond, **unknown** before). + +Verdict for `(Bin, So, Rel)`: + +``` +compatible(exact | extrapolated) presence — defeasible "structurally possible" +incompatible([missing(Sym@Node) | below_floor(Sym@Node, Min) | soname_mismatch(...)]) + HARD veto — only reachable with complete, attributed evidence +unknown([no_requires_evidence | requires_evidence(Status,_) | no_provider_evidence(So) + | unknown(Sym@Node, evidence_release(R0)) | ...]) +not_needed(So) +``` + +Documented defeasible assumption: within a soname, exports do not disappear +(removing one is an ABI break that requires a soname bump), so presence at +`R0` extrapolates to later releases and absence from a complete export set is +a veto for the soname. `drop(Sym, Node, At)` models a violation of that +assumption to exercise the upper bound. ## Files | File | Role | |------|------| -| `ingest_symbols.mjs` | Ingest symbol metadata → P/2 JSONL interval store (3 tiers). | -| `abi_resolve.pl` | Prolog resolver: intervals, containment, min/max. New module; reuses `resolver:version_lt/2`. | -| `abi_cli.pl` | Callable driver: `min` / `newest` / `range` / `compat`. | -| `test_abi.pl` | Real-data assertions (the PoC's numbers). | -| `run_abi_verify.sh` | End-to-end: build the store from this machine's data + verify. | - -Generated stores land in `./.out` (gitignored) — rebuild with -`./run_abi_verify.sh`. - -## The three ingest tiers (cheapest first) - -The full per-symbol interval data is *curated metadata, not the binary* — so the -common path needs **zero package download**. - -1. **`Packages` `Depends:` — coarse min for free.** Debian/Ubuntu's package - index already carries `libc6 (>= 2.34)`, computed *from the symbols* by - `dpkg-shlibdeps` at build time. The coarse floor needs no download at all. - (Consumed by the coarse resolver; this lane refines it.) -2. **`.symbols` control member — curated intervals (`ingest_symbols.mjs - symbols-file`).** Library packages ship a `.symbols` file whose lines are - literally `symbol@Base minimum-version` — *exactly our `[sym, intro]` - interval store, precomputed by the maintainer*. It lives in the package's - small `control.tar.*` member (a few KB), not the multi-MB data archive. On an - installed system it is already on disk under `/var/lib/dpkg/info/*.symbols`. -3. **`readelf` fallback — any ELF (`ingest_symbols.mjs elf` / `requires`).** If - no `.symbols` is published, `readelf -W --dyn-syms` gives a `.so`'s exported - versioned symbols (`intro` = the symbol's own version tag), and a binary's - `readelf -V` verneed table attributes each *required* symbol to its soname. - Applies to **any ELF**, so the lane works for Debian, Ubuntu, and arbitrary - binaries with no repo metadata at all. - -Applies to **Debian AND Ubuntu** unchanged (same `dpkg` / `.symbols` format), -plus the `readelf` path for any ELF on any distro. - -### Store shape (P/2 JSONL, `[key, value]`, `load_p2_jsonl`-compatible) +| `ingest_symbols.mjs` | Ingest: `.symbols` (since bounds), `readelf` provides (`at` bounds), `readelf` requires (attributed via the ELF version **index**), NEEDED, evidence, release axis. Loud failures (exit 3), never an empty success. | +| `abi_resolve.pl` | Resolver: store loading, the two axes, per-requirement status, verdicts, floor, range over the real release axis. | +| `abi_cli.pl` | Driver: `verdict` / `status` / `floor` / `axis` / `range`. | +| `crosscheck.mjs` | readelf-vs-`.symbols` cross-check on exact identity (+ the corrected legacy per-name figure). | +| `test_abi.pl` | Assertions: real data (A), version axes (B), model fixtures (C), gcc-built ELF fixtures + `.symbols` template fixtures (D). | +| `run_abi_verify.sh` | End-to-end: build the real store, cross-check, build/ingest fixtures, run the tests. | +| `fixtures/` | C sources + version scripts for the ELF fixtures; `.symbols` fixtures (simple cases, template rejects, arch selectors). | + +Generated stores land in `./.out` (gitignored). + +## Store shape (P/2 JSONL, `[key, value]`) ``` -symprov.jsonl : ["|", "#inf"] # interval; to=inf within soname -symreq.jsonl : ["|", "#"] # verneed floor, per soname -needed.jsonl : ["", ""] # DT_NEEDED +symprov.jsonl ["|@", ["since", ""]] # .symbols tier + ["|@", ["at", ""]] # readelf tier +symreq.jsonl ["|@", ["", "GLOBAL"|"WEAK"]] + ["|", ["", "GLOBAL"|"WEAK"]] # unversioned reference +needed.jsonl ["", ""] +evidence.jsonl ["provides|", ["symbols"|"elf", "", "complete", ""]] + ["requires|", ["readelf", "complete"|"missing_file"|"readelf_failed"|"inconsistent", ""]] +releases.jsonl ["", ""] # the candidate axis (actual releases) ``` -## The resolver (`abi_resolve.pl`) - -- `provides_at(SoName, Sym, V)` — true iff `intro(SoName|Sym) =< V` (interval - lookup; `to=inf` within a soname). -- `abi_min(Binary, SoName, Min)` — the **verneed floor**: the highest required - version among the binary's required symbols of that soname. Hard, derived, - exact. -- `abi_compatible(Binary, SoName, V[, Drop])` — containment: every required - symbol of that soname is provided at `V`. `Drop = drop(Sym, At)` simulates a - soname-era ABI break (symbol removed at `At`) for testing the max. -- `newest_abi_compatible(Binary, SoName, Candidates[, Drop], Result)` — newest - `V` in `Candidates` (descending) with `abi_compatible` → `compatible(V)`, else - `no_candidate`. `Candidates` = `soname_candidates/2` (the distinct intro - versions in the store — the soname's version axis). -- `abi_range(Binary, SoName, Candidates[, Drop], Result)` — combines the derived - MIN with the newest-compatible MAX → `range(Min, Max)`, or `no_candidate(...)` - when `Min > Max` (a removed symbol squeezed the range empty). - -Note the division of labor (matching the PoC): `abi_compatible` is *name-intro -containment* and drives the **MAX** search; the **MIN** is the separate verneed -floor from `abi_min`. So `abi_compatible(2.33)` can be true by name even though -the binary references the `GLIBC_2.34` node — `abi_range` applies the floor and -reports `2.34` as the effective lower bound. - -## Min vs max — the epistemics - -- **min** = the verneed floor (highest required symver). **Hard, derived, - exact.** -- **max** = a **defeasible default** = "the soname generation" (no upper bound - for backward-compatible libs), refined *downward* only by hard evidence: a - needed symbol removed, or a soname bump. Symbol analysis can only ever NARROW - the default. -- **Symbol ABSENCE / soname mismatch = a hard veto** (proof of "no") — it can - override an optimistic or wrong declared dependency. -- **Symbol PRESENCE = a defeasible "might work"** — necessary but not sufficient - (semantics unverified); it never outranks a declared/tested dep, it only - widens the candidate set with a low-confidence maybe. - -Confidence order: `tested > declared repo dep > ELF-hard-veto / ELF-maybe > -soname default`. See `SYMBOL_ABI_HOWTO.md` for the full design and worked -examples. +## The three provider tiers (cheapest first) + +1. **`Packages` `Depends:`** — the coarse floor (`libc6 (>= 2.34)`), consumed + by the coarse resolver. This lane's `abi_floor/3` recomputes it from the + symbols and, on this machine, matches coreutils' declared + `libc6 (>= 2.34), libselinux1 (>= 3.1~)` exactly. +2. **`.symbols` control member** (`ingest_symbols.mjs symbols-file`) — every + exported `sym@node` with a curated minimum package version. Already on disk + under `/var/lib/dpkg/info/*.symbols`; a few KB per package, no binary + download. Binary-form files are fully supported; source-template constructs + are processed where their semantics are known (`(optional)`, `(arch=...)` + with `--arch`) and **rejected loudly** otherwise (`(symver)`, `(regex)`, + quoted C++ patterns, `#include`). +3. **`readelf`** (`ingest_symbols.mjs elf` / `requires`) — any ELF. Provides + are exact for the file's own release; requires are joined to their soname + through `.gnu.version` -> `.gnu.version_r` by index, so two libraries using + the same node name never collide. ## Verify on this machine ``` -./run_abi_verify.sh # builds the store from /var/lib/dpkg + /bin/ls, runs all checks +./run_abi_verify.sh ``` -Real-data results (Ubuntu 22.04, libc6, `/bin/ls`): - -- `libc6` `.symbols` → **4827** symprov interval rows across 20 sonames - (**3006** for `libc.so.6`); `/bin/ls` → **112** symreq rows + `NEEDED = - [libselinux.so.1, libc.so.6]`. -- Tier-2 cross-check: `readelf(libc.so.6)` vs curated `.symbols` agree on the - intro axis for **2241 / 2443 (91.7%)** shared symbols. Every disagreement is - the **glibc 2.34 pthread/rt-into-libc merge** — `.symbols` conservatively - dates those symbols to 2.34 (when they entered `libc.so.6`'s stable ABI) while - the raw ELF keeps the historical libpthread version node. -- `abi_min(/bin/ls, libc.so.6)` = **2.34** (set by - `__libc_start_main@GLIBC_2.34`). -- `abi_compatible(/bin/ls, libc.so.6, 2.35)` = **TRUE** (0 missing); - `newest_abi_compatible` = `compatible(2.35)`; `abi_range` = **range(2.34, - 2.35)**. -- Simulated removal of `getenv` at 2.30 (a symbol `/bin/ls` needs): the newest - version still exporting `getenv` caps at **2.29**; combined with min **2.34** - → **min > max squeeze** → **`no_candidate`**. +Real-data results (Ubuntu 22.04.5, libc6 2.35-0ubuntu3.15, libselinux1 3.3-1build2, `/bin/ls`): + +- Ingest: libc6 `.symbols` -> **4827** `symprov` rows over 20 sonames + (**3006** for `libc.so.6`); libselinux1 -> 238; `/bin/ls` -> **112 versioned + + 3 unversioned (weak)** requirements, `NEEDED = [libselinux.so.1, libc.so.6]`. +- Cross-check `readelf(libc.so.6)` vs `.symbols`: **3006/3006 exact + `sym@node` identities agree (100%)**. The corrected legacy per-name + comparison (earliest row on *both* sides) is **2478/2478 (100%)**; the + earlier 91.7% was an aggregation bug (last curated row vs earliest ELF row), + not a glibc 2.34 merge effect. +- `floor /bin/ls libc.so.6` = **2.34**; `floor /bin/ls libselinux.so.1` = + **3.1~** — equal to coreutils' declared Pre-Depends. +- Release axis (from `apt-cache madison` + dpkg): `[2.35-0ubuntu3, + 2.35-0ubuntu3.15]`; `range /bin/ls libc.so.6` = **[2.35-0ubuntu3, + 2.35-0ubuntu3.15]**, both ends `compatible(exact)`. +- Hypothetical older release `2.31-0ubuntu9.9` -> + `incompatible([below_floor(__libc_start_main@GLIBC_2.34, 2.34), below_floor(lstat@GLIBC_2.33, 2.33)])`; + an axis extended below the floor yields min `2.34-0ubuntu3`, never the + lowest release. +- Hypothetical removal of `getenv@GLIBC_2.2.5` at `2.35-0ubuntu3.15` caps the + range at `[2.35-0ubuntu3, 2.35-0ubuntu3]`; removing + `__libc_start_main@GLIBC_2.34` at the oldest release -> `no_candidate`. +- Offering `libc.so.7` -> `incompatible([soname_mismatch(...)])`; a binary with + no requirement evidence -> `unknown([no_requires_evidence(...)])`. +- ELF fixtures (gcc): `foo@LIB_1` required vs `foo@LIB_2` exported under the + same soname -> `incompatible([missing(foo@LIB_1)])`, and the loader agrees + (`undefined symbol: foo, version LIB_1`); `COMMON_1` in two libraries + attributed correctly; `pub_fn@PUBLIC` + unversioned `plain_fn` preserved; + missing ELF -> `unknown([requires_evidence(missing_file, _)])`. + +`test_abi.pl`: 63 checks, all passing. diff --git a/examples/pkg_resolver/abi/REVIEW_NOTES.md b/examples/pkg_resolver/abi/REVIEW_NOTES.md new file mode 100644 index 000000000..bd1beb9a9 --- /dev/null +++ b/examples/pkg_resolver/abi/REVIEW_NOTES.md @@ -0,0 +1,161 @@ + + + +# Review notes — redesign of the symbol-level ABI lane (PR #4262 follow-up) + +Astra's REQUEST-CHANGES review found the first implementation too lossy: it +collapsed `sym@node` to bare names with a numeric "intro", dropped non-numeric +and unversioned obligations, attributed requirements by version-*name*, mixed +the Debian package-version axis with the ELF node axis, computed an +unsatisfiable lower bound, mis-aggregated the cross-check (91.7%), and +silently mis-ingested `.symbols` templates. The model and ingest were +redesigned; the direction (three evidence tiers, a driver above the frozen +resolver) is unchanged. Frozen `resolver.pl` / `resolver_store.pl` / +`debian/` are untouched (`git diff origin/main -- examples/pkg_resolver/{resolver.pl,resolver_store.pl,debian}` is empty). + +Run `./run_abi_verify.sh` (needs node, readelf, swipl, gcc). Expected tail: +`== 63 passed, 0 failed, 0 skipped ==`. Check names in `test_abi.pl` carry the +review point they prove, e.g. `(#3)`. + +## Sol's checklist + +### (a) Exact version-node identity end-to-end — Astra #1 + +- **Ingest** (`ingest_symbols.mjs`): `.symbols` rows are split at the last + `@` into `(sym, node)` and stored as `"|@"` with the minimum + version *verbatim* (`["since", ""]`). readelf provides carry the + node from `.gnu.version_d` by index (`["at", ""]`). There is no + `verNum()` anymore; nothing is parsed out of a node name. +- **Store** (`abi_resolve.pl`): `symprov(So, Sym, Node, Bound)`, + `symreq(Bin, Sym, Node, So, Bind)`. Loading rejects an empty node. +- **Match** (`versioned_status/7`): `symprov(So, Sym, Node, _)` — same + soname, same symbol, same node, by unification. No fallback to `symprov(So, + Sym, _, _)` for versioned requirements. +- **Fixtures**: C1/C1b (in-memory: `foo@LIB_1` vs `foo@LIB_2` -> + `incompatible([missing(foo@LIB_1)])`, then exact row -> `compatible(exact)`); + **D1** (gcc-built `libfoo.so.1` v1/v2, real ingest; the loader is run as + ground truth and must print `undefined symbol: foo, version LIB_1`); + A17/A18 on real libc (`getenv@GLIBC_2.99` -> missing; + `pthread_setname_np@GLIBC_2.12` (real non-default node) -> provided, + `@GLIBC_2.13` -> missing); A7 (no node-less provider rows); B8 (node + labels are never ordered numerically). + +### (b) verneed attribution via version INDEX — Astra #3 + +- `elfTables()` parses `readelf -W -V`: the `.gnu.version` array (per-dynsym + index, hidden bit `h`), `.gnu.version_d` (index -> name) and + `.gnu.version_r` (index -> `{file, name, weak}`). `elfRequires()` joins each + UND symbol by its **index** to `(file, node)`; the `name@VER` text is only + used as a consistency assertion (mismatch -> `inconsistent` evidence, exit 3). +- Also enforced: the verneed file must be in DT_NEEDED; a symbol whose index + has no verneed entry is an ingest error, not a `?` soname. +- **Fixture D3**: `libalpha.so.1` and `libbeta.so.1` both define `COMMON_1`; + `usecommon` needs one symbol from each. Asserted: `alpha_fn@COMMON_1 -> + libalpha.so.1`, `beta_fn@COMMON_1 -> libbeta.so.1`, and the negatives. D3b: + both verdicts `compatible(exact)`. A4/A5 on `/bin/ls`: `__libc_start_main@GLIBC_2.34 + -> libc.so.6`, `freecon@LIBSELINUX_1.0 -> libselinux.so.1`. + +### (c) Two axes separate; deb parsing reused — Astra #4 + +- Package versions are parsed once at load by `debian/deb_parse:parse_deb_version/2` + (`rel_term/2`, `assert_symprov/2`) and ordered by the frozen + `resolver:version_lt/2` on `deb/3` (`rel_lt/2`, `rel_le/2`). No local + version arithmetic exists in the lane. `since(Deb, Atom)` keeps the original + atom so `3.1~` is reported as `3.1~`, not re-formatted. +- ELF nodes never enter `rel_term/2` as versions: a node string that is not a + Debian version becomes `label(Atom)` and only matches itself (B7/B8). +- **Fixtures**: B1–B6 (`3.1~ < 3.1`, `3.1~ < 3.1~rc1`, `1:2.3-1 > 3.1`, + `2.35 < 2.35-0ubuntu3 < 2.35-0ubuntu3.15`, `2.2.5 < 2.2.5.1 < 2.2.6`, `0` + lowest); D7 (`.symbols` fixture with `1:0.9-2` and `1.2~rc1` minimums loaded + as `deb(1, ...)` / preserved atoms); **A8–A10** on real data: the computed + floors `2.34` and `3.1~` equal coreutils' declared + `libc6 (>= 2.34), libselinux1 (>= 3.1~)` read from dpkg. +- Documented (`README.md`, `SYMBOL_ABI_HOWTO.md`, header comment of + `abi_resolve.pl`): `.symbols` minimum versions are curated lower bounds, + raisable by policy, not introduction dates; a release below one is reported + as `below_floor`, distinct from `missing`. + +### (d) Incomplete evidence never becomes a false veto or false compat — Astra #2 + +- `evidence.jsonl` rows are emitted by every successful ingest; a missing or + unreadable ELF makes `requires` exit 3 **and** record + `["requires|", ["readelf", "missing_file", ...]]` (no empty success); + `elf` on a missing file exits 3 with nothing written. +- `abi_verdict/5` short-circuits to `unknown(...)` when: no `req_evidence`, + `req_evidence` not `complete`, or no complete `prov_evidence` for the + soname; `at(R0)` evidence yields `unknown(Sym@Node, evidence_release(R0))` + for older releases; an unversioned obligation is `unknown` while any NEEDED + object lacks evidence. `needed/2` is checked (`not_needed`, and + `soname_mismatch` is a hard veto when the offered soname differs from the + NEEDED one with the same stem). +- Non-numeric nodes and unversioned references are kept (`pub_fn@PUBLIC`, + `plain_fn`), weak references are flagged `WEAK` and classified + `weak_unresolved` (never a veto). +- **Fixtures**: C3/C3b (at-evidence: older -> unknown), C4–C4e (no provider + evidence -> unknown; `missing_file` -> unknown; unversioned with one lib + unevidenced -> unknown, with all evidenced and nobody exporting -> + `missing`, exported by another NEEDED lib -> compatible), C5/C5b, C6; + **D4–D6** through the real pipeline (`usepub` / `libpub` / `libplain`; + `store_pub_nolibpub`, `store_pub_noplain`, `store_missing`); A1–A3, A6, + A19–A22 on real data. + +### (e) `[min, max]` satisfiable at both ends — Astra #5 + +- The candidate axis is `releases.jsonl` (actual releases; on this machine + from `apt-cache madison libc6` + dpkg), never symbol-intro points. + `abi_range/5` evaluates **every** release and takes min/max from the + releases whose verdict is `compatible(_)` (`range_min_max/3`), so both ends + are compatible by construction; releases that add no symbols are still + evaluated (C2d). +- **Fixtures**: **C2** — releases 1..5, `a since 1`, `b since 5` -> + `range(5.0, 5.0)`, verdict at 1.0 `incompatible([below_floor(b@L, 5.0)])` + (the old code returned `range(1,5)`); C2c (6.0 -> `compatible(extrapolated)`, + `range(5.0, 6.0)`); C3b (unknown releases are reported, not counted); A13 + (real axis, both ends compatible); **A15** (axis extended below the floor: + min is `2.34-0ubuntu3`, not `2.31-0ubuntu9.9`); A23/A24 (hypothetical + removal caps the max / squeezes to `no_candidate`); D2 (`range(1.0-1, 1.0-1)`). + +### (f) Cross-check aggregation fixed; false 2.34-merge claim removed — Astra #6 + +- `crosscheck.mjs` compares the two tiers on **exact `sym@node` sets**: + `.symbols=3006 readelf=3006 shared=3006`, 100%. It also reproduces the + legacy per-name figure with earliest-row aggregation on *both* sides: + 2478/2478 (100%) (the review's 2443/2443 used the old script's symbol + filter; either way it is 100%). Both must be 100% or the script fails. +- `README.md` and `SYMBOL_ABI_HOWTO.md` no longer claim a glibc 2.34 + pthread/rt-merge divergence; they state the 91.7% was the last-row-vs- + earliest-row bug. + +### (g) Frozen files untouched + +`git diff origin/main -- examples/pkg_resolver/resolver.pl +examples/pkg_resolver/resolver_store.pl examples/pkg_resolver/debian/` is +empty. The lane only *imports* `resolver:version_lt/2` and +`deb_parse:parse_deb_version/2`. + +### Astra #7 — `.symbols` templates handled or rejected loudly + +- `parseSymbolsFile()` processes `(optional)` (kept), `(ignore-blacklist)` + (kept), `(arch=…)`, `(arch-bits=…)`, `(arch-endian=…)` (row kept iff it + selects `--arch`; **rejected** without `--arch`), and rejects `(symver)`, + `(regex)`, `(c++…)`, quoted pattern rows, `#include`, malformed rows and + non-Debian minimum versions — the whole file is refused (exit 3) with line + numbers; nothing partial is written. `#PACKAGE#` headers are accepted but + then `--release` is mandatory (the evidence release cannot be looked up). +- The simple cases keep working: multiple sonames, `|` alt-dep lines, `*` + meta fields, `#` comments, `#MINVER#`, private minimum `0`, dep-id column. +- **Fixtures**: D7 (`fixtures/simple.symbols`), D8 + (`tmpl_arch.symbols` with `--arch amd64`: `!amd64` row dropped, `(optional)` + kept), D9 x3 (`tmpl_symver`, `tmpl_cxx`, `tmpl_arch` without `--arch` -> + no store written; `run_abi_verify.sh` also asserts the non-zero exit). + +## Things worth a second look + +- `unversioned_status/7` treats an unversioned reference as satisfied by *any* + exported node of the symbol in *any* NEEDED object (the loader's default- + version rule is not modelled). Documented in the HOWTO "Limits". +- `bound_holds(since(Min,_), R0, Rel, extrapolated)` for `Rel > R0` and + `at(R0)` for `Rel > R0` both rely on the in-soname monotone-export + assumption; the `Basis` value makes that visible to callers. +- `abi_floor/3` fails (rather than guessing) when a requirement has no + `since()` row — readelf-only provider evidence has no package floor. diff --git a/examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md b/examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md index ea03da480..452510c08 100644 --- a/examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md +++ b/examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md @@ -1,134 +1,181 @@ - + + -# Symbol-level ABI resolution — how it works, with examples +# Symbol-level ABI resolution — how it works, with real examples This extends the package resolver from coarse version constraints -(`libfoo (>= 2.0)`) to **fine-grained ABI compatibility** at the *symbol* -level, so we can compute the real *[min, max]* compatible library-version range -for a binary — the thing `ldd` won't tell you. +(`libfoo (>= 2.0)`) to ABI compatibility at the *exact versioned symbol* level, +so we can compute the real `[min, max]` compatible release range for a binary. + +## The model + +- A **library release PROVIDES** a set of exported `sym@node` identities under + a `soname` (its ABI generation, e.g. `libc.so.6`). +- A **binary REQUIRES** a set of `sym@node` identities, each attributed to the + soname its ELF verneed table names, plus the sonames it links (`DT_NEEDED`), + plus any unversioned references. +- **Compatible(bin, release)** ⟺ every NEEDED soname is offered under its exact + name AND every required `sym@node` is exported by that soname at that + release (unversioned references: exported under any node by any NEEDED + object). + +Two axes, never mixed: + +1. **ELF version nodes** are labels. `GLIBC_2.34` is not the number 2.34 — + it is the name of a node in `.gnu.version_d`. A requirement `foo@LIB_1` is + satisfied by `foo@LIB_1` and by nothing else; in particular not by + `foo@LIB_2` on the same soname (the loader fails with + `undefined symbol: foo, version LIB_1`). Non-numeric nodes (`PUBLIC`, + `GLIBC_PRIVATE`) and unversioned symbols (`Base`) are first-class identities. +2. **Debian package versions** order the release candidates and the `.symbols` + minimum-version field, through the frozen resolver's `deb/3` comparison + (`debian/deb_parse.pl` + `resolver:version_lt/2`): epochs (`1:2.3-1`), + tildes (`3.1~ < 3.1`), revisions (`2.35 < 2.35-0ubuntu3`), any number of + components. + +## Where the evidence comes from (three tiers, cheapest first) + +1. **`Packages` `Depends:`** — `libc6 (>= 2.34)` was computed by + `dpkg-shlibdeps` from the symbols at build time. Zero download; the coarse + resolver consumes it. `abi_floor/3` recomputes it from tier 2 and, on this + machine, reproduces coreutils' `libc6 (>= 2.34), libselinux1 (>= 3.1~)`. +2. **`.symbols` control member** — one row per `sym@node` with a curated + minimum package version: + ``` + libc.so.6 libc6 #MINVER# + | libc6 (>> 2.35), libc6 (<< 2.36) + getenv@GLIBC_2.2.5 2.2.5 + pthread_setname_np@GLIBC_2.12 2.12 + pthread_setname_np@GLIBC_2.34 2.34 + __libc_enable_secure@GLIBC_PRIVATE 0 1 + ``` + The minimum version is a **lower bound** a dependent must declare, *not* a + ground-truth introduction date: Debian policy lets a maintainer raise it + after a compatible behaviour change. The lane therefore stores it as + `since(Min)` and reports a release below it as `below_floor` (the same + conservative floor `dpkg-shlibdeps` emits), never as proof the symbol was + absent. +3. **`readelf`** on the ELF itself — exact for that file's release + (`at(R0)`), unknown for older releases, extrapolated for newer ones. + +## Worked examples (real output, Ubuntu 22.04.5, after `./run_abi_verify.sh`) + +### 1. Requirements attributed through the version index -## The model (one generalization of "provides") - -- A **library version PROVIDES** a set of exported versioned symbols - `{sym@ver}` plus a `soname` (its ABI generation, e.g. `libc.so.6`). -- A **binary/package REQUIRES** a set of referenced versioned symbols `{sym@ver}` - plus the `sonames` it links (`NEEDED`). -- **Compatible(bin, libver)** ⟺ `soname matches` AND `requires ⊆ provides`. - -Everything below is just this predicate plus how the versions are stored. +``` +$ node ingest_symbols.mjs requires /bin/ls --stdout | grep -E 'start_main|freecon|gmon' +["/bin/ls|__libc_start_main@GLIBC_2.34",["libc.so.6","GLOBAL"]] +["/bin/ls|freecon@LIBSELINUX_1.0",["libselinux.so.1","GLOBAL"]] +["/bin/ls|__gmon_start__",["","WEAK"]] +``` +Each undefined symbol's `.gnu.version` index is looked up in `.gnu.version_r`, +giving *(file, node)*. Two libraries that both define a node called `COMMON_1` +therefore cannot be confused (fixture `usecommon`: `alpha_fn@COMMON_1 -> +libalpha.so.1`, `beta_fn@COMMON_1 -> libbeta.so.1`). Unversioned and weak +references are kept, flagged, and never silently dropped. -## The tools (PoC) +### 2. The curated floor equals the declared dependency -- `elf_symbols.mjs extract|rows|compat ` — read provides/requires/soname - from any ELF via `readelf`; `compat` checks `requires ⊆ provides`. -- `sym_abi.mjs intervals|newest ` — treat each symbol's version tag as - the version it was *introduced* in, giving each symbol a validity interval - `[intro, inf)` within the soname; `newest` computes the min/max compatible - version. +``` +$ swipl -q -g main -t halt abi_cli.pl -- .out/store floor /bin/ls libc.so.6 +floor /bin/ls libc.so.6: 2.34 +$ swipl -q -g main -t halt abi_cli.pl -- .out/store floor /bin/ls libselinux.so.1 +floor /bin/ls libselinux.so.1: 3.1~ +$ dpkg-query -W -f '${Pre-Depends}\n' coreutils +libacl1 (>= 2.2.23), libattr1 (>= 1:2.4.44), libc6 (>= 2.34), libgmp10 (>= 2:6.2.1+dfsg), libselinux1 (>= 3.1~) +``` -## Worked examples (real output, this machine) +### 3. Verdicts on the actual release axis -### 1. A library's provides, a binary's requires ``` -$ elf_symbols.mjs extract libc.so.6 -> n_provides: 2970, soname: libc.so.6 -$ elf_symbols.mjs extract /bin/ls -> n_requires: 112, needed: [libselinux.so.1, libc.so.6], - min_versions: { GLIBC: 2.34, LIBSELINUX: 1.0 } +$ ... axis libc.so.6 +axis libc.so.6: [2.35-0ubuntu3,2.35-0ubuntu3.15] # apt-cache madison + dpkg +$ ... range /bin/ls libc.so.6 +range /bin/ls libc.so.6: [2.35-0ubuntu3, 2.35-0ubuntu3.15] + 2.35-0ubuntu3: compatible(exact) + 2.35-0ubuntu3.15: compatible(exact) +$ ... verdict /bin/ls libc.so.6 2.31-0ubuntu9.9 +verdict /bin/ls libc.so.6 2.31-0ubuntu9.9: incompatible([below_floor('__libc_start_main'@'GLIBC_2.34','2.34'),below_floor(lstat@'GLIBC_2.33','2.33')]) ``` -The **minimum** version is *derived*, not declared: it's the highest version -tag among the required symbols (`/bin/ls` references a `@GLIBC_2.34` symbol, so -it needs glibc ≥ 2.34). `ldd` never shows this; `readelf` does. +The candidates are releases, not symbol-introduction points; `range/3` reports +a `[min, max]` whose ends both carry `compatible` verdicts. With an axis +extended below the floor (`2.31-0ubuntu9.9, 2.34-0ubuntu3, ...`) the min is +`2.34-0ubuntu3` — the first release that actually satisfies the requirements. -### 2. Compatibility = set containment +### 4. Exact node identity (the case the old model got wrong) + +Fixture: `libfoo.so.1` v1 exports `foo@LIB_1`; v2 keeps a `LIB_1` node but +exports `foo` only as `foo@LIB_2`. `usefoo` is linked against v1. ``` -$ elf_symbols.mjs compat /bin/ls libc.so.6 libselinux.so.1 - binary_requires: 112, libs_provide: 3207, compatible: true, missing_count: 0 +$ LD_LIBRARY_PATH=.out/fx/v2 .out/fx/usefoo +usefoo: symbol lookup error: usefoo: undefined symbol: foo, version LIB_1 +$ ... .out/fx/store_foo_v2 verdict .out/fx/usefoo libfoo.so.1 2.0-1 +verdict ... libfoo.so.1 2.0-1: incompatible([missing(foo@'LIB_1')]) ``` -All 112 needs are covered → compatible. If any were missing → `missing > 0` → -a **hard incompatibility** (proof of "no"). +A name-level model with a numeric intro would have said "compatible" here. + +### 5. Where a real MAX comes from — a removal (hypothetical) -### 3. Symbols as validity intervals (the real ABI-growth curve) ``` -$ sym_abi.mjs intervals libc.so.6 - 2443 symbols across 35 versions [2.2.5 .. 2.35] - introduced per version: ... 2.33: 12 2.34: 212 2.35: 4 - row example: ["pthread_setname_np","2.12","inf"] +$ ... range /bin/ls libc.so.6 getenv GLIBC_2.2.5 2.35-0ubuntu3.15 +range /bin/ls libc.so.6: [2.35-0ubuntu3, 2.35-0ubuntu3] + 2.35-0ubuntu3: compatible(exact) + 2.35-0ubuntu3.15: incompatible([missing(getenv@'GLIBC_2.2.5',hypothetical_drop)]) +$ ... range /bin/ls libc.so.6 __libc_start_main GLIBC_2.34 2.35-0ubuntu3 +range /bin/ls libc.so.6: no_candidate ``` -Each symbol is stored ONCE as `[sym, intro, inf]` — a validity interval within -the soname. (glibc 2.34 adding 212 symbols is the real pthread/rt-into-libc -merge.) "Does version V provide sym?" = `intro(sym) ≤ V`. This is the same -interval store we built for snapshot membership, one level down — and it's the -memory-pressure dataset (thousands of symbols × versions × snapshots). -### 4. Newest-compatible, no ABI break +### 6. Incomplete evidence is "unknown", not "compatible" + ``` -$ sym_abi.mjs newest /bin/ls libc.so.6 - min_compatible_version: 2.34 - newest_compatible_version: 2.35 (max unbounded within soname libc.so.6) +$ node ingest_symbols.mjs requires /nonexistent/bin --out .out/x ; echo exit=$? +ingest_symbols: requires /nonexistent/bin: missing_file; recorded INCOMPLETE evidence, no requirement rows +exit=3 +$ ... verdict /nonexistent/bin libc.so.6 2.35-0ubuntu3 +verdict /nonexistent/bin libc.so.6 2.35-0ubuntu3: unknown([no_requires_evidence('/nonexistent/bin')]) ``` -Newest version whose provides still cover ls's needs. With a backward-compatible -library there's no upper bound inside the soname — the default max is "the -soname generation." +Likewise a NEEDED soname with no provider evidence yields +`unknown([no_provider_evidence(So)])`, and readelf-only evidence yields +`unknown` for releases older than the file's own. + +### 7. Cross-check: the two provider tiers agree exactly -### 5. Where a real MAX comes from — a symbol removal ``` -$ sym_abi.mjs newest /bin/ls libc.so.6 --drop=getenv@2.30 # simulate ABI break - min = 2.34, binary needs getenv: true, effective MAX = 2.29 - => min (2.34) > max (2.29) => NO compatible version (correct no_candidate) + exact sym@node identity: .symbols=3006 readelf=3006 shared=3006 only-readelf=0 only-.symbols=0 + identity agreement: 3006/3006 (100.0%) + legacy per-name earliest-row comparison (corrected aggregation): 2478/2478 (100.0%) ``` -If a newer version *removes* a symbol the binary needs (or bumps the soname), -that caps the max. Here ls needs both 2.34-era symbols *and* `getenv`, so -removing `getenv` at 2.30 makes the range empty — the resolver correctly returns -no candidate. - -## Min vs max — the epistemics (defeasible defaults + hard vetoes) - -- **min** = the verneed floor (highest required symver). Hard, derived, exact. -- **max** = a *defeasible default* = "the soname generation" (no upper bound for - backward-compatible libs), *refined downward* only by hard evidence: a needed - symbol removed, or a soname bump. Symbol analysis can only ever NARROW the - default. -- **Symbol ABSENCE / soname mismatch = a hard veto** (proof of "no") — this can - override an optimistic or wrong declared dependency. -- **Symbol PRESENCE = a defeasible "might work"** — necessary but not sufficient - (semantics unverified). It never outranks a declared/tested dependency; it only - widens the candidate set with a low-confidence maybe. - -Confidence order: `tested > declared repo dep > ELF-hard-veto / ELF-maybe > +The earlier "91.7% agreement, explained by the glibc 2.34 pthread/rt merge" +was an aggregation bug: the last curated row of a symbol was compared with the +earliest ELF node of that symbol, so any symbol with two nodes +(`pthread_setname_np@GLIBC_2.12` + `@GLIBC_2.34`) "disagreed". Compared +consistently, everything agrees; the merge explanation was false. + +## Epistemics + +- **Absence / soname mismatch = hard veto** (`incompatible`) — valid only + when requirement evidence is complete and provider evidence for that soname + is complete and attributed by index. Otherwise the answer is `unknown`. +- **Presence = defeasible "structurally possible"** (`compatible(exact | + extrapolated)`) — necessary, not sufficient; semantics are unverified. It + never outranks a declared or tested dependency; it widens the candidate set + with a low-confidence maybe. +- **`below_floor`** is a curated floor, not proof of absence, and is reported + distinctly so a caller can choose to treat it as declared-dependency + strength rather than ELF-hard strength. +- **`extrapolated`** relies on the in-soname monotone-export assumption and is + reported distinctly from `exact`. + +Confidence order: `tested > declared repo dep > ELF hard veto / ELF maybe > soname default`. -## "Is the symbol info in the repo, so we don't download the whole package?" - -Largely yes — in three tiers, cheapest first: - -1. **The min-version dependency is already in the repo index.** Debian's - `Packages` file (fetched by `apt update`) carries each package's `Depends:`, - e.g. `libc6 (>= 2.34)` — and that bound was computed *from the symbols* by - `dpkg-shlibdeps` at build time. So the coarse min needs **zero** package - download. -2. **The full per-symbol interval data is curated metadata, not the binary.** - Debian library packages ship a `.symbols` file whose lines are literally - `symbol@Base minimum-version` — i.e. *exactly our `[sym, intro]` interval - store, precomputed by the maintainer*. It lives in the package's small - **control member** (`control.tar.*`), so you fetch that member (a few KB), - not the multi-MB data. (A mirror could also publish a symbols index directly.) -3. **Only if neither is published** do you fetch the binary (or HTTP - range-fetch just its `.dynsym`/`.gnu.version_*` sections) and run `readelf` — - which is what this PoC does — then cache the result. - -Either way it's a **one-time ingest** per `(pkg,ver)`, deduped across snapshots -by the interval encoding — never a per-resolution download. And on a real -machine the "locked set" libraries are already installed, so their provides are -free; only the *candidates* you're weighing need any fetch, lazily, newest-first. - -## How it lands in the store / resolver (no new engine) - -- Ingest → `symprov(soname|sym -> intro#inf)` and `symreq(bin|sym -> ver)` rows, - interval-encoded and deduped exactly like the package pool + membership. -- `Compatible` is the existing `provides ⊇ requires` containment; the - already-built `newest_compatible_snap` generalizes from coarse version - constraints to symbol-set containment — "newest ABI-compatible library version - across snapshots" becomes a real query on real data. -- Confidence tag per edge (`declared | elf_hard | elf_maybe | soname_default | - tested`) so a package manager prunes on hard + prefers declared, and a coding - agent can turn an `elf_maybe` into `tested` by actually building — the Level-3 - oracle a package manager lacks. +## Limits + +- Hidden (`@`, non-default) versions are matched like default (`@@`) ones, + which is what the loader does for an exact versioned reference; the "which + node does an *unversioned* reference bind to" rule is simplified to "any + exported node". +- `.symbols` source-template constructs that need the binary to expand + (`(symver)`, `(regex)`, quoted C++ patterns) are rejected, not interpreted. +- The release axis is an input (from `apt-cache madison`, dpkg, a snapshot + store); the lane never invents candidates from symbol data. diff --git a/examples/pkg_resolver/abi/abi_cli.pl b/examples/pkg_resolver/abi/abi_cli.pl index 4822d818d..5b7b672b0 100644 --- a/examples/pkg_resolver/abi/abi_cli.pl +++ b/examples/pkg_resolver/abi/abi_cli.pl @@ -8,10 +8,14 @@ % % % Commands: -% min derived verneed floor -% newest [DropSym DropAt] newest ABI-compatible version -% range [DropSym DropAt] [min, max] or no_candidate -% compat containment check at a version +% verdict [DropSym DropNode DropAt] +% compatible(exact|extrapolated) | incompatible([...]) | unknown([...]) | not_needed(_) +% status one line per requirement (provided / missing / ...) +% floor curated lower bound implied by `.symbols` (= dpkg-shlibdeps' dep) +% axis the ingested release candidates, ascending +% range [DropSym DropNode DropAt] +% range(Min, Max, ...) over the release axis (both ends compatible), or +% no_candidate / unknown / no_releases :- use_module(abi_resolve). @@ -19,31 +23,40 @@ current_prolog_flag(argv, Argv), ( Argv = [Dir, Cmd | Args] -> true ; usage, halt(2) ), load_abi_store(Dir), - run(Cmd, Args). + atom_string(CmdA, Cmd), + run(CmdA, Args). usage :- format(user_error, - "usage: abi_cli.pl -- min|newest|range|compat ~n", []). + "usage: abi_cli.pl -- verdict|status|floor|axis|range ~n", []). drop_of([], none). -drop_of([Sym, At], drop(SymA, At)) :- atom_string(SymA, Sym). +drop_of([Sym, Node, At], drop(Sym, Node, At)). -run(min, [Bin, So]) :- !, - ( abi_min(Bin, So, Min) -> format("min ~w~n", [Min]) ; format("min none~n", []) ). -run(newest, [Bin, So | DropArgs]) :- !, - soname_candidates(So, Cands), +run(verdict, [Bin, So, Rel | DropArgs]) :- !, drop_of(DropArgs, Drop), - newest_abi_compatible(Bin, So, Cands, Drop, R), - format("newest ~w~n", [R]). + abi_verdict(Bin, So, Rel, Drop, V), + format("verdict ~w ~w ~w: ~q~n", [Bin, So, Rel, V]). +run(status, [Bin, So, Rel]) :- !, + rel_term(Rel, R), + forall(req_status(Bin, So, R, none, S), format(" ~q~n", [S])). +run(floor, [Bin, So]) :- !, + ( abi_floor(Bin, So, F) + -> format("floor ~w ~w: ~w~n", [Bin, So, F]) + ; format("floor ~w ~w: none (a requirement has no since() provider row)~n", [Bin, So]) + ). +run(axis, [So]) :- !, + release_axis(So, Rels), + format("axis ~w: ~w~n", [So, Rels]). run(range, [Bin, So | DropArgs]) :- !, - soname_candidates(So, Cands), drop_of(DropArgs, Drop), - abi_range(Bin, So, Cands, Drop, R), - format("range ~w~n", [R]). -run(compat, [Bin, So, V]) :- !, - ( abi_compatible(Bin, So, V) - -> format("compatible ~w yes~n", [V]) - ; missing_syms(Bin, So, V, none, M), length(M, N), - format("compatible ~w no (missing ~w)~n", [V, N]) + abi_range(Bin, So, Drop, R), + ( R = range(Min, Max, Pairs) + -> format("range ~w ~w: [~w, ~w]~n", [Bin, So, Min, Max]), + forall(member(A-V, Pairs), format(" ~w: ~q~n", [A, V])) + ; R =.. [Kind, Pairs], is_list(Pairs) + -> format("range ~w ~w: ~w~n", [Bin, So, Kind]), + forall(member(A-V, Pairs), format(" ~w: ~q~n", [A, V])) + ; format("range ~w ~w: ~q~n", [Bin, So, R]) ). run(_, _) :- usage, halt(2). diff --git a/examples/pkg_resolver/abi/abi_resolve.pl b/examples/pkg_resolver/abi/abi_resolve.pl index 37cbb58c0..d9c38abe2 100644 --- a/examples/pkg_resolver/abi/abi_resolve.pl +++ b/examples/pkg_resolver/abi/abi_resolve.pl @@ -2,240 +2,428 @@ % SPDX-License-Identifier: MIT OR Apache-2.0 % Copyright (c) 2026 John William Creighton (@s243a) % -% abi_resolve.pl -- symbol-level ABI-compatibility resolver, the fine-grained -% generalization of the package resolver's coarse `provides`. It reuses the -% SAME interval-store idea one level down: instead of a package having a -% version tenure, each exported SYMBOL has a validity interval [intro, inf) -% WITHIN a soname (a library version V provides sym iff intro(sym) =< V, until -% a soname bump / removal). Given a binary's referenced versioned symbols -% (verneed floor) and a library's exported-symbol intervals, it computes the -% MIN and MAX (newest) compatible library version via symbol-set containment. +% abi_resolve.pl -- symbol-level ABI-compatibility resolver (redesigned after +% the PR #4262 review; see REVIEW_NOTES.md for the point-by-point map). % -% Frozen resolver.pl / resolver_store.pl are NOT edited; version comparison is -% delegated to resolver:version_lt/2. +% MODEL +% Two independent axes: +% * ELF version-node axis: a node (GLIBC_2.34, LIBSELINUX_1.0, COMMON_1, +% PUBLIC, Base) is an opaque label. A requirement `Sym@Node` on soname So +% is satisfied only by a provider row with the SAME (So, Sym, Node) -- +% string equality, never numeric ordering, never a bare-name fallback. +% `Base` = dpkg's spelling of "unversioned"; an unversioned REQUIREMENT +% (no node) is satisfied by any exported node of that symbol. +% * Debian package-version axis: the `.symbols` minimum-version and the +% release candidates are deb/3 terms produced by debian/deb_parse.pl and +% ordered by the frozen resolver:version_lt/2 (epoch, ~, revision all +% handled there). A non-deb release id is kept as label(Atom) and only +% ever matches itself. +% Evidence is explicit: +% prov_evidence(So, Src, R0, complete) -- So's export set is known +% completely at evidence release R0 (Src = symbols | elf). +% req_evidence(Bin, Src, Status, Detail) -- Bin's requirement set is +% complete, or why not (missing_file / readelf_failed / inconsistent). +% Provider bounds: +% since(Min, MinAtom) -- from `.symbols`: exported at R0 and, by the +% curated lower bound, at every release >= Min. Min is NOT a ground-truth +% introduction date (Debian policy lets it be raised); R < Min is treated +% as `below_floor` (the same conservative floor dpkg-shlibdeps emits). +% at(R0) -- from readelf: exported at exactly R0. +% Extrapolation (defeasible, documented): within a soname, exports do not +% disappear (removing one is an ABI break that requires a soname bump), so +% presence at R0 is extrapolated to R > R0 with basis `extrapolated`, and +% absence from a complete export set is a veto for every release of the +% soname. Presence never becomes a guarantee: compatible(_) is defeasible. +% For at(R0) evidence, R < R0 is UNKNOWN (readelf says nothing about older +% releases). A hypothetical drop(Sym, Node, At) models an in-soname removal +% (violating the assumption) to exercise the upper bound. % -% Store rows (P/2 JSONL, load_p2_jsonl shape), produced by ingest_symbols.mjs: -% symprov(Key, Val) Key = 'SoName|Sym' Val = 'Intro#inf' (interval) -% symreq(Key, Val) Key = 'Binary|Sym' Val = 'SoName#Ver' (verneed) -% needed(Binary, SoName) (DT_NEEDED) +% VERDICTS abi_verdict(Bin, So, Rel, Verdict): +% compatible(exact | extrapolated) +% incompatible([missing(Sym@Node) | below_floor(Sym@Node, MinAtom) +% | soname_mismatch(offered(So), needed(N)) ...]) -- HARD veto, +% only reachable when requirement AND provider evidence are complete +% unknown([no_requires_evidence(Bin) | requires_evidence(Status, Detail) +% | no_provider_evidence(So) | unknown(Sym@Node, evidence_release(R0)) +% | unknown(Sym, no_provider_evidence(S)) ...]) +% not_needed(So) -- Bin has no DT_NEEDED entry for So (and no stem clash) +% +% RANGE abi_range(Bin, So, Releases, Result): every release in the ACTUAL +% candidate axis is evaluated; range(Min, Max, Pairs) has compatible verdicts +% at BOTH ends by construction (Min/Max are drawn from the compatible set). +% +% Frozen resolver.pl / resolver_store.pl are NOT edited. :- module(abi_resolve, [ + op(200, xfx, @), load_abi_store/1, abi_store_clear/0, - provides_at/3, - symprov_intro/3, - req_sym/4, - abi_min/3, - abi_compatible/3, - abi_compatible/4, - missing_syms/5, - soname_candidates/2, - newest_providing/5, - newest_abi_compatible/4, - newest_abi_compatible/5, + symprov/4, + symreq/5, + needed/2, + prov_evidence/4, + req_evidence/4, + release/3, + rel_term/2, + rel_le/2, + rel_lt/2, + provides_at/5, + req_status/5, + abi_verdict/4, + abi_verdict/5, + abi_floor/3, + soname_offer/3, + release_axis/2, + abi_range/3, abi_range/4, - abi_range/5 + abi_range/5, + range_min_max/3 ]). +:- op(200, xfx, @). % Sym@Node terms in statuses/verdicts + :- use_module('../resolver', [version_lt/2]). +:- use_module('../debian/deb_parse', [parse_deb_version/2]). :- use_module(library(http/json)). +:- use_module(library(lists)). +:- use_module(library(apply)). -:- dynamic symprov/2. -:- dynamic symreq/2. -:- dynamic needed/2. +:- dynamic symprov/4. % symprov(SoName, Sym, Node, Bound) Bound = since(Deb, Atom) | at(Rel) +:- dynamic symreq/5. % symreq(Binary, Sym, Node, SoName, Bind) Node/SoName = none if unversioned +:- dynamic needed/2. % needed(Binary, SoName) +:- dynamic prov_evidence/4. % prov_evidence(SoName, Src, Rel, Status) +:- dynamic req_evidence/4. % req_evidence(Binary, Src, Status, Detail) +:- dynamic release/3. % release(SoName, Rel, Atom) % --------------------------------------------------------------------------- -% Store loading (mirrors resolver_store:load_p2_jsonl / load_pairs) +% Store loading (P/2 JSONL: [Key, Value] per line; values may be JSON arrays) % --------------------------------------------------------------------------- abi_store_clear :- - retractall(symprov(_, _)), - retractall(symreq(_, _)), - retractall(needed(_, _)). + retractall(symprov(_, _, _, _)), + retractall(symreq(_, _, _, _, _)), + retractall(needed(_, _)), + retractall(prov_evidence(_, _, _, _)), + retractall(req_evidence(_, _, _, _)), + retractall(release(_, _, _)). load_abi_store(Dir) :- abi_store_clear, - load_pairs(Dir, 'symprov.jsonl', symprov), - load_pairs(Dir, 'symreq.jsonl', symreq), - load_pairs(Dir, 'needed.jsonl', needed). + load_rows(Dir, 'symprov.jsonl', assert_symprov), + load_rows(Dir, 'symreq.jsonl', assert_symreq), + load_rows(Dir, 'needed.jsonl', assert_needed), + load_rows(Dir, 'evidence.jsonl', assert_evidence), + load_rows(Dir, 'releases.jsonl', assert_release). -load_pairs(Dir, File, Pred) :- +load_rows(Dir, File, Handler) :- atomic_list_concat([Dir, '/', File], Path), ( exists_file(Path) -> setup_call_cleanup(open(Path, read, S), - load_pair_lines(S, Pred), + load_row_lines(S, Path, 1, Handler), close(S)) ; true ). -load_pair_lines(S, Pred) :- +load_row_lines(S, Path, N, Handler) :- read_line_to_string(S, Line), ( Line == end_of_file -> true ; ( Line == "" -> true ; atom_string(Atom, Line), - atom_json_term(Atom, [K, V], [value_string_as(atom)]), - Fact =.. [Pred, K, V], - assertz(Fact) + ( catch(atom_json_term(Atom, [K, V], [value_string_as(atom)]), _, fail), + catch(call(Handler, K, V), _, fail) + -> true + ; throw(error(abi_store_row(Path, N, Line), load_abi_store/1)) + ) ), - load_pair_lines(S, Pred) + N1 is N + 1, + load_row_lines(S, Path, N1, Handler) + ). + +% "|@" -> since(Deb, Atom) | at(Rel) +assert_symprov(K, [Kind, V]) :- + split_first(K, '|', So, Ident), + split_last(Ident, '@', Sym, Node), + Node \== '', + ( Kind == since + -> parse_deb_version(V, Deb), Bound = since(Deb, V) + ; Kind == at + -> rel_term(V, Rel), Bound = at(Rel) + ), + assertz(symprov(So, Sym, Node, Bound)). + +% "|[@]" -> symreq(Bin, Sym, Node, SoName, Bind) +assert_symreq(K, [So0, Bind]) :- + split_first(K, '|', Bin, Ident), + ( split_last(Ident, '@', Sym, Node) + -> Node \== '', So0 \== '', So = So0 + ; Sym = Ident, Node = none, So = none + ), + memberchk(Bind, ['GLOBAL', 'WEAK']), + assertz(symreq(Bin, Sym, Node, So, Bind)). + +assert_needed(Bin, So) :- + atom(So), + assertz(needed(Bin, So)). + +assert_evidence(K, V) :- + split_first(K, '|', Kind, Subject), + ( Kind == provides + -> V = [Src, RelAtom, Status, _Source], + rel_term(RelAtom, Rel), + assertz(prov_evidence(Subject, Src, Rel, Status)) + ; Kind == requires + -> V = [Src, Status, Detail], + assertz(req_evidence(Subject, Src, Status, Detail)) ). +assert_release(So, V) :- + rel_term(V, Rel), + ( release(So, Rel, _) -> true ; assertz(release(So, Rel, V)) ). + +split_first(Atom, Sep, Before, After) :- + sub_atom(Atom, B, _, A, Sep), !, + sub_atom(Atom, 0, B, _, Before), + sub_atom(Atom, _, A, 0, After). + +split_last(Atom, Sep, Before, After) :- + sub_atom(Atom, B, _, A, Sep), + \+ ( sub_atom(Atom, B2, _, _, Sep), B2 > B ), !, + sub_atom(Atom, 0, B, _, Before), + sub_atom(Atom, _, A, 0, After). + % --------------------------------------------------------------------------- -% Version comparison (delegated to frozen resolver:version_lt/2) +% Release axis (Debian package versions; labels only match themselves) % --------------------------------------------------------------------------- -% Dotted numeric versions ("2.34", "2.2.5") -> v(A,B,C), padded with 0 so the -% 2- and 3-component glibc tags compare correctly. -ver_term(V, v(A, B, C)) :- - ( atom(V) -> atom_string(V, S) ; V = S ), - split_string(S, ".", "", Parts0), - exclude(==(""), Parts0, Parts), - nums_pad(Parts, A, B, C). +% rel_term(+Atom, -Rel): deb/3 via the frozen parser, else label(Atom). +rel_term(Atom, Rel) :- + ( catch(parse_deb_version(Atom, Deb), _, fail), + Deb = deb(_, [s([], _)|_], _) % upstream starts with a digit (Policy 5.6.12) + -> Rel = Deb + ; Rel = label(Atom) + ). + +rel_lt(deb(E1, U1, R1), deb(E2, U2, R2)) :- + version_lt(deb(E1, U1, R1), deb(E2, U2, R2)). -nums_pad(Parts, A, B, C) :- - ( nth0(0, Parts, P0) -> to_num(P0, A) ; A = 0 ), - ( nth0(1, Parts, P1) -> to_num(P1, B) ; B = 0 ), - ( nth0(2, Parts, P2) -> to_num(P2, C) ; C = 0 ). +rel_le(A, B) :- + ( A = deb(_, _, _), B = deb(_, _, _) + -> \+ version_lt(B, A) + ; A == B + ). -to_num(S, N) :- ( number(S) -> N = S ; number_string(N0, S) -> N = N0 ; N = 0 ). +rel_cmp(Order, A-_, B-_) :- + ( rel_lt(A, B) -> Order = (<) + ; rel_lt(B, A) -> Order = (>) + ; A = deb(_, _, _), B = label(_) -> Order = (<) + ; A = label(_), B = deb(_, _, _) -> Order = (>) + ; A == B -> Order = (=) + ; compare(Order, A, B) + ). -ver_lt(A, B) :- ver_term(A, TA), ver_term(B, TB), resolver:version_lt(TA, TB). -ver_le(A, B) :- \+ ver_lt(B, A). +% release_axis(SoName, AscendingAtoms): the actual release candidates known for +% the soname (ingested `releases` rows), ascending, deduplicated. +release_axis(So, Atoms) :- + findall(R-A, release(So, R, A), Pairs0), + predsort(rel_cmp, Pairs0, Pairs), + pairs_values(Pairs, Atoms). -% max version of a non-empty list (highest wins). -max_ver([V|Vs], Max) :- foldl(max_ver_1, Vs, V, Max). -max_ver_1(V, Acc, Out) :- ( ver_lt(Acc, V) -> Out = V ; Out = Acc ). +% --------------------------------------------------------------------------- +% Provision at a release +% --------------------------------------------------------------------------- + +% provides_at(So, Sym, Node, Rel, Basis): So exports exactly Sym@Node at Rel; +% Basis = exact (inside the evidence) | extrapolated (beyond the evidence +% release, under the in-soname monotone-export assumption). +provides_at(So, Sym, Node, Rel, Basis) :- + symprov(So, Sym, Node, Bound), + prov_evidence(So, _, R0, complete), + bound_holds(Bound, R0, Rel, Basis). + +bound_holds(since(Min, _), R0, Rel, Basis) :- + rel_le(Min, Rel), + ( rel_le(Rel, R0) -> Basis = exact ; Basis = extrapolated ). +bound_holds(at(R0), _, Rel, Basis) :- + ( Rel == R0 -> Basis = exact + ; rel_lt(R0, Rel) -> Basis = extrapolated + ). + +hyp_dropped(drop(Sym, Node, At), Sym, Node, Rel) :- + rel_term(At, AtRel), + rel_le(AtRel, Rel). % --------------------------------------------------------------------------- -% Store accessors +% Per-requirement status % --------------------------------------------------------------------------- -% intro(SoName|Sym) -- the symbol's introduced version (interval floor). -symprov_intro(SoName, Sym, Intro) :- - symprov(Key, Val), - split_key(Key, SoName, Sym), - split_hash(Val, Intro, _To). - -% req_sym(Binary, SoName, Sym, Ver) -- a versioned symbol the binary needs. -req_sym(Binary, SoName, Sym, Ver) :- - symreq(Key, Val), - split_key(Key, Binary, Sym), - split_hash(Val, SoName, Ver). - -split_key(Key, A, B) :- - ( atom(Key) -> atom_string(Key, S) ; S = Key ), - sub_string(S, Before, _, After, "|"), - !, - sub_string(S, 0, Before, _, AS), - sub_string(S, _, After, 0, BS), - atom_string(A, AS), - atom_string(B, BS). - -split_hash(Val, A, B) :- - ( atom(Val) -> atom_string(Val, S) ; S = Val ), - split_string(S, "#", "", [AS, BS | _]), - atom_string(A, AS), - atom_string(B, BS). +% req_status(Bin, So, Rel, Hyp, Status) enumerates one Status per requirement +% of Bin that concerns So (versioned requirements attributed to So via the +% version index, plus Bin's unversioned requirements, which the loader +% resolves against any NEEDED object). +req_status(Bin, So, Rel, Hyp, Status) :- + symreq(Bin, Sym, Node, So, Bind), + Node \== none, + versioned_status(So, Sym, Node, Bind, Rel, Hyp, Status). +req_status(Bin, So, Rel, Hyp, Status) :- + symreq(Bin, Sym, none, none, Bind), + unversioned_status(Bin, So, Sym, Bind, Rel, Hyp, Status). + +versioned_status(So, Sym, Node, Bind, Rel, Hyp, Status) :- + ( hyp_dropped(Hyp, Sym, Node, Rel) + -> Status = missing(Sym@Node, hypothetical_drop) + ; symprov(So, Sym, Node, Bound) + -> prov_evidence(So, _, R0, complete), + ( bound_holds(Bound, R0, Rel, Basis) + -> Status = provided(Sym@Node, Basis) + ; Bound = since(_, MinAtom) + -> Status = below_floor(Sym@Node, MinAtom) + ; Status = unknown(Sym@Node, evidence_release(R0)) + ) + ; Bind == 'WEAK' + -> Status = weak_unresolved(Sym@Node) + ; Status = missing(Sym@Node) + ). + +% An unversioned reference binds to any exported node of Sym in any NEEDED +% object. Against the queried So we evaluate at Rel; against the other NEEDED +% objects at their own evidence release. Complete absence is a veto only when +% every NEEDED object has complete provider evidence. +unversioned_status(Bin, So, Sym, Bind, Rel, Hyp, Status) :- + ( \+ hyp_dropped_any(Hyp, Sym), + ( provides_at(So, Sym, _, Rel, Basis) + -> Status = provided(Sym, any_node(So, Basis)) + ; needed(Bin, S), S \== So, prov_evidence(S, _, R0, complete), + provides_at(S, Sym, _, R0, _) + -> Status = provided(Sym, any_node(S, exact)) + ; fail + ) + -> true + ; Bind == 'WEAK' + -> Status = weak_unresolved(Sym) % a weak ref never vetoes, so missing evidence is moot + ; needed(Bin, S), \+ prov_evidence(S, _, _, complete) + -> Status = unknown(Sym, no_provider_evidence(S)) + ; Status = missing(Sym) + ). + +hyp_dropped_any(drop(Sym, _, _), Sym). % --------------------------------------------------------------------------- -% Core ABI predicates +% Verdict % --------------------------------------------------------------------------- -% provides_at(SoName, Sym, V): library version V exports Sym, i.e. it was -% introduced at or before V (to = inf within a soname). -provides_at(SoName, Sym, V) :- - symprov_intro(SoName, Sym, Intro), - ver_le(Intro, V). - -% provides_at with a simulated removal: drop(DropSym, At) removes DropSym for -% every V >= At (a soname-era ABI break, for testing the max). -provides_at_drop(SoName, Sym, V, none) :- !, - provides_at(SoName, Sym, V). -provides_at_drop(_SoName, Sym, V, drop(Sym, At)) :- - ver_le(At, V), !, - fail. -provides_at_drop(SoName, Sym, V, _Drop) :- - provides_at(SoName, Sym, V). - -% abi_min(Binary, SoName, Min): the verneed floor -- the highest required -% version among the binary's required symbols of that soname. Hard/derived. -abi_min(Binary, SoName, Min) :- - findall(Ver, req_sym(Binary, SoName, _Sym, Ver), Vers), - Vers \== [], - max_ver(Vers, Min). - -% missing_syms(Binary, SoName, V, Drop, Missing): required symbols of SoName -% NOT provided at library version V (containment failures = hard vetoes). -missing_syms(Binary, SoName, V, Drop, Missing) :- - findall(Sym, - ( req_sym(Binary, SoName, Sym, _), - \+ provides_at_drop(SoName, Sym, V, Drop) - ), - Missing0), - sort(Missing0, Missing). - -% abi_compatible(Binary, SoName, V): every required symbol of SoName is -% provided at library version V (set containment). -abi_compatible(Binary, SoName, V) :- - abi_compatible(Binary, SoName, V, none). - -abi_compatible(Binary, SoName, V, Drop) :- - missing_syms(Binary, SoName, V, Drop, []). - -% soname_candidates(SoName, Descending): the soname's version axis = the -% distinct intro versions in the store, highest first. -soname_candidates(SoName, Desc) :- - findall(Intro, symprov_intro(SoName, _Sym, Intro), Intros0), - sort(Intros0, Uniq), - predsort(cmp_ver_desc, Uniq, Desc). - -cmp_ver_desc(Order, A, B) :- - ( ver_lt(A, B) -> Order = (>) - ; ver_lt(B, A) -> Order = (<) - ; Order = (=) +abi_verdict(Bin, So, RelAtom, Verdict) :- + abi_verdict(Bin, So, RelAtom, none, Verdict). + +abi_verdict(Bin, So, RelAtom, Hyp, Verdict) :- + rel_term(RelAtom, Rel), + ( \+ req_evidence(Bin, _, _, _) + -> Verdict = unknown([no_requires_evidence(Bin)]) + ; req_evidence(Bin, _, Status, Detail), Status \== complete + -> Verdict = unknown([requires_evidence(Status, Detail)]) + ; soname_offer(Bin, So, mismatch(N)) + -> Verdict = incompatible([soname_mismatch(offered(So), needed(N))]) + ; \+ needed(Bin, So) + -> Verdict = not_needed(So) + ; \+ prov_evidence(So, _, _, complete) + -> Verdict = unknown([no_provider_evidence(So)]) + ; findall(S, req_status(Bin, So, Rel, Hyp, S), Ss), + aggregate_statuses(Ss, Verdict) ). -% newest_providing(SoName, Sym, Candidates, Drop, Result): the newest V in -% Candidates (descending) that still exports Sym under Drop -> compatible(V); -% else no_candidate. With Drop = drop(Sym, At) this is the "effective max cap" -% a symbol removal imposes (the newest version just below the removal). -newest_providing(_SoName, _Sym, [], _Drop, no_candidate) :- !. -newest_providing(SoName, Sym, [V|Vs], Drop, Result) :- - ( provides_at_drop(SoName, Sym, V, Drop) - -> Result = compatible(V) - ; newest_providing(SoName, Sym, Vs, Drop, Result) +aggregate_statuses(Ss, Verdict) :- + include(hard_veto, Ss, Hard), + include(is_unknown, Ss, Unk), + ( Hard \== [] + -> Verdict = incompatible(Hard) + ; Unk \== [] + -> Verdict = unknown(Unk) + ; memberchk(provided(_, extrapolated), Ss) + -> Verdict = compatible(extrapolated) + ; memberchk(provided(_, any_node(_, extrapolated)), Ss) + -> Verdict = compatible(extrapolated) + ; Verdict = compatible(exact) ). -% newest_abi_compatible(Binary, SoName, Candidates, Result): -% newest V in Candidates (descending) with abi_compatible -> compatible(V); -% else no_candidate. -newest_abi_compatible(Binary, SoName, Candidates, Result) :- - newest_abi_compatible(Binary, SoName, Candidates, none, Result). - -newest_abi_compatible(_Binary, _SoName, [], _Drop, no_candidate) :- !. -newest_abi_compatible(Binary, SoName, [V|Vs], Drop, Result) :- - ( abi_compatible(Binary, SoName, V, Drop) - -> Result = compatible(V) - ; newest_abi_compatible(Binary, SoName, Vs, Drop, Result) +hard_veto(missing(_)). +hard_veto(missing(_, _)). +hard_veto(below_floor(_, _)). +is_unknown(unknown(_, _)). + +% soname_offer(Bin, So, Offer): needed | mismatch(NeededSoName) | not_needed. +% Offering libfoo.so.2 to a binary whose DT_NEEDED says libfoo.so.1 is a hard +% veto: the loader matches DT_NEEDED by exact soname string. +soname_offer(Bin, So, Offer) :- + ( needed(Bin, So) + -> Offer = needed + ; so_stem(So, Stem), + needed(Bin, N), so_stem(N, Stem) + -> Offer = mismatch(N) + ; Offer = not_needed ). -% abi_range(Binary, SoName, Candidates, Result): combine the derived MIN -% (verneed floor) with the newest compatible MAX. If min > max (a removed -% symbol squeezed the range) -> no_candidate. -abi_range(Binary, SoName, Candidates, Result) :- - abi_range(Binary, SoName, Candidates, none, Result). - -abi_range(Binary, SoName, Candidates, Drop, Result) :- - ( abi_min(Binary, SoName, Min) -> true ; Min = none ), - newest_abi_compatible(Binary, SoName, Candidates, Drop, Newest), - ( Newest = compatible(Max) - -> ( Min == none - -> Result = range(none, Max) - ; ver_le(Min, Max) - -> Result = range(Min, Max) - ; Result = no_candidate(min_gt_max(Min, Max)) - ) - ; Result = no_candidate(no_compatible_version) +so_stem(So, Stem) :- + ( sub_atom(So, B, _, _, '.so') + -> sub_atom(So, 0, B, _, Stem) + ; Stem = So + ). + +% --------------------------------------------------------------------------- +% Floor: the curated lower bound implied by `.symbols` (= dpkg-shlibdeps' dep) +% --------------------------------------------------------------------------- + +% abi_floor(Bin, So, FloorAtom): the highest `.symbols` minimum-version among +% the provider rows matched (exactly, by node) by Bin's requirements on So. +% Fails if any versioned requirement on So has no since() provider row +% (missing symbol, or readelf-only evidence). +abi_floor(Bin, So, Floor) :- + findall(Sym-Node, ( symreq(Bin, Sym, Node, So, _), Node \== none ), Reqs), + Reqs \== [], + maplist(req_floor(So), Reqs, Mins), + max_deb(Mins, _-Floor). + +req_floor(So, Sym-Node, Deb-Atom) :- + symprov(So, Sym, Node, since(Deb, Atom)). + +max_deb([M|Ms], Max) :- foldl(max_deb_1, Ms, M, Max). +max_deb_1(D-A, D0-A0, Out) :- ( rel_lt(D0, D) -> Out = D-A ; Out = D0-A0 ). + +% --------------------------------------------------------------------------- +% Range over the actual release axis +% --------------------------------------------------------------------------- + +abi_range(Bin, So, Result) :- + release_axis(So, Rels), + abi_range(Bin, So, Rels, none, Result). + +% abi_range(Bin, So, Hyp, Result): store axis, hypothetical drop(Sym, Node, At). +abi_range(Bin, So, Hyp, Result) :- + release_axis(So, Rels), + abi_range(Bin, So, Rels, Hyp, Result). + +% abi_range(Bin, So, RelAtoms, Hyp, Result): +% range(Min, Max, Pairs) -- Min/Max are releases with compatible(_) verdicts +% no_candidate(Pairs) -- every release incompatible / not needed +% unknown(Pairs) -- no compatible release, some unknown +% no_releases -- empty axis +% Pairs = [RelAtom-Verdict ...] ascending. +abi_range(_Bin, _So, [], _Hyp, no_releases) :- !. +abi_range(Bin, So, RelAtoms, Hyp, Result) :- + maplist(rel_pair, RelAtoms, P0), + predsort(rel_cmp, P0, Sorted), + pairs_values(Sorted, Asc), + findall(A-V, ( member(A, Asc), abi_verdict(Bin, So, A, Hyp, V) ), Pairs), + range_min_max(Pairs, Pairs, Result). + +rel_pair(A, R-A) :- rel_term(A, R). + +range_min_max(Pairs, Detail, Result) :- + findall(A, member(A-compatible(_), Pairs), Compat), + ( Compat = [Min|_] + -> last(Compat, Max), + Result = range(Min, Max, Detail) + ; memberchk(_-unknown(_), Pairs) + -> Result = unknown(Detail) + ; Result = no_candidate(Detail) ). diff --git a/examples/pkg_resolver/abi/crosscheck.mjs b/examples/pkg_resolver/abi/crosscheck.mjs new file mode 100644 index 000000000..2307bbd04 --- /dev/null +++ b/examples/pkg_resolver/abi/crosscheck.mjs @@ -0,0 +1,73 @@ +#!/usr/bin/env node +// SPDX-License-Identifier: MIT OR Apache-2.0 +// Copyright (c) 2026 John William Creighton (@s243a) +// +// crosscheck.mjs -- compare the two provider tiers for one soname: +// readelf(lib.so) -> symprov rows ["so|sym@node", ["at", rel]] +// .symbols file -> symprov rows ["so|sym@node", ["since", minver]] +// +// (1) EXACT IDENTITY: the sets of sym@node must be equal. This is the check +// the model actually relies on (a requirement matches a provider by the +// exact triple), and on Ubuntu 22.04 libc6 it is 3006/3006. +// (2) LEGACY PER-NAME COMPARISON, CORRECTED: the original script compared the +// LAST curated row of a symbol against the EARLIEST ELF node of that +// symbol, so every symbol with two nodes (e.g. pthread_setname_np@GLIBC_2.12 +// + @GLIBC_2.34 after the 2.34 libpthread merge) "disagreed" (91.7%). With +// the same earliest-row aggregation on both sides the figure is 100%; the +// "glibc 2.34 merge divergence" explanation was an artifact. Note this +// comparison mixes axes (node label number vs package version) and is kept +// only as the corrected regression figure. +// +// usage: crosscheck.mjs + +import { readFileSync } from "node:fs"; + +const [symFile, elfFile, so] = process.argv.slice(2); + +function loadRows(file, tag) { + const rows = []; // {sym, node, v} + for (const l of readFileSync(file, "utf8").split("\n")) { + if (!l) continue; + const [k, v] = JSON.parse(l); + if (!k.startsWith(so + "|") || v[0] !== tag) continue; + const ident = k.slice(so.length + 1); + const at = ident.lastIndexOf("@"); + rows.push({ sym: ident.slice(0, at), node: ident.slice(at + 1), v: v[1] }); + } + return rows; +} + +const S = loadRows(symFile, "since"), E = loadRows(elfFile, "at"); +const sKeys = new Set(S.map((r) => `${r.sym}@${r.node}`)); +const eKeys = new Set(E.map((r) => `${r.sym}@${r.node}`)); +let both = 0; const onlyE = [], onlyS = []; +for (const k of eKeys) (sKeys.has(k) ? both++ : onlyE.push(k)); +for (const k of sKeys) if (!eKeys.has(k)) onlyS.push(k); +console.log(` exact sym@node identity: .symbols=${sKeys.size} readelf=${eKeys.size} shared=${both} only-readelf=${onlyE.length} only-.symbols=${onlyS.length}`); +if (onlyE.length) console.log(` only-readelf sample: ${onlyE.slice(0, 5).join(", ")}`); +if (onlyS.length) console.log(` only-.symbols sample: ${onlyS.slice(0, 5).join(", ")}`); +const identityPct = (100 * both) / Math.max(sKeys.size, eKeys.size); +console.log(` identity agreement: ${both}/${Math.max(sKeys.size, eKeys.size)} (${identityPct.toFixed(1)}%)`); + +// Legacy per-name figure, corrected: EARLIEST row on BOTH sides. +function nodeNum(node) { const m = node.match(/_([0-9][0-9.]*)$/); return m ? m[1] : null; } +function cmpDotted(a, b) { + const pa = a.split("."), pb = b.split("."); + for (let i = 0; i < Math.max(pa.length, pb.length); i++) { const x = +(pa[i] || 0), y = +(pb[i] || 0); if (x !== y) return x - y; } + return 0; +} +function earliest(rows, pick) { + const m = new Map(); + for (const r of rows) { const n = pick(r); if (n === null) continue; if (!m.has(r.sym) || cmpDotted(n, m.get(r.sym)) < 0) m.set(r.sym, n); } + return m; +} +const sE = earliest(S, (r) => nodeNum(r.node)); // earliest numeric node per name (curated rows) +const eE = earliest(E, (r) => nodeNum(r.node)); // earliest numeric node per name (readelf rows) +let shared = 0, agree = 0; const dis = []; +for (const [sym, n] of eE) { if (!sE.has(sym)) continue; shared++; if (sE.get(sym) === n) agree++; else if (dis.length < 5) dis.push(`${sym} (.symbols=${sE.get(sym)} readelf=${n})`); } +const pct = (100 * agree) / shared; +console.log(` legacy per-name earliest-row comparison (corrected aggregation): ${agree}/${shared} (${pct.toFixed(1)}%)`); +if (dis.length) console.log(` disagreements: ${dis.join(", ")}`); +if (identityPct < 100) { console.error(" FAIL: exact identity sets differ"); process.exit(1); } +if (pct < 100) { console.error(" FAIL: corrected per-name comparison below 100%"); process.exit(1); } +console.log(" PASS: readelf and .symbols agree on every exact sym@node identity"); diff --git a/examples/pkg_resolver/abi/fixtures/alpha.c b/examples/pkg_resolver/abi/fixtures/alpha.c new file mode 100644 index 000000000..1da6d417f --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/alpha.c @@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: MIT OR Apache-2.0 */ +int alpha_fn(void) { return 1; } diff --git a/examples/pkg_resolver/abi/fixtures/beta.c b/examples/pkg_resolver/abi/fixtures/beta.c new file mode 100644 index 000000000..cfc77ab98 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/beta.c @@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: MIT OR Apache-2.0 */ +int beta_fn(void) { return 2; } diff --git a/examples/pkg_resolver/abi/fixtures/common.map b/examples/pkg_resolver/abi/fixtures/common.map new file mode 100644 index 000000000..5a5d2859b --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/common.map @@ -0,0 +1 @@ +COMMON_1 { global: alpha_fn; beta_fn; local: *; }; diff --git a/examples/pkg_resolver/abi/fixtures/foo.c b/examples/pkg_resolver/abi/fixtures/foo.c new file mode 100644 index 000000000..ce374cf29 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/foo.c @@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: MIT OR Apache-2.0 */ +/* libfoo fixture: built twice under the SAME soname (libfoo.so.1) with two + * version scripts. v1 exports foo@LIB_1; v2 keeps the LIB_1 node (foo_legacy) + * but moves foo to LIB_2 only -- so a binary linked against v1 needs foo@LIB_1 + * and the loader rejects v2: "undefined symbol: foo, version LIB_1". */ +int foo(void) { return 1; } +int foo_legacy(void) { return 0; } diff --git a/examples/pkg_resolver/abi/fixtures/foo_v1.map b/examples/pkg_resolver/abi/fixtures/foo_v1.map new file mode 100644 index 000000000..4c55d00f6 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/foo_v1.map @@ -0,0 +1 @@ +LIB_1 { global: foo; foo_legacy; local: *; }; diff --git a/examples/pkg_resolver/abi/fixtures/foo_v2.map b/examples/pkg_resolver/abi/fixtures/foo_v2.map new file mode 100644 index 000000000..0c4f24e0d --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/foo_v2.map @@ -0,0 +1,2 @@ +LIB_1 { global: foo_legacy; local: *; }; +LIB_2 { global: foo; } LIB_1; diff --git a/examples/pkg_resolver/abi/fixtures/plain.c b/examples/pkg_resolver/abi/fixtures/plain.c new file mode 100644 index 000000000..4beb0795d --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/plain.c @@ -0,0 +1,3 @@ +/* SPDX-License-Identifier: MIT OR Apache-2.0 */ +/* Built WITHOUT a version script: an unversioned export (dpkg spells it @Base). */ +int plain_fn(void) { return 4; } diff --git a/examples/pkg_resolver/abi/fixtures/pub.c b/examples/pkg_resolver/abi/fixtures/pub.c new file mode 100644 index 000000000..19b7f83a5 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/pub.c @@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: MIT OR Apache-2.0 */ +int pub_fn(void) { return 3; } diff --git a/examples/pkg_resolver/abi/fixtures/pub.map b/examples/pkg_resolver/abi/fixtures/pub.map new file mode 100644 index 000000000..e53c6468a --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/pub.map @@ -0,0 +1 @@ +PUBLIC { global: pub_fn; local: *; }; diff --git a/examples/pkg_resolver/abi/fixtures/simple.symbols b/examples/pkg_resolver/abi/fixtures/simple.symbols new file mode 100644 index 000000000..234e9f6cf --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/simple.symbols @@ -0,0 +1,11 @@ +# A binary-form .symbols with the simple cases: two sonames, an alt-dep line, +# a meta field, comments, a private (minimum 0) row, tilde and epoch minimums. +libsimple.so.1 libsimple1 #MINVER# +| libsimple1 (>> 1.2), libsimple1 (<< 1.3) +* Build-Depends-Package: libsimple-dev + SIMPLE_1.0@SIMPLE_1.0 1.0 + simple_new@SIMPLE_1.2 1.2~rc1 + simple_old@SIMPLE_1.0 1:0.9-2 + _private_thing@SIMPLE_PRIVATE 0 1 +libsimple-extra.so.0 libsimple1 #MINVER# + extra_fn@Base 1.1 diff --git a/examples/pkg_resolver/abi/fixtures/tmpl_arch.symbols b/examples/pkg_resolver/abi/fixtures/tmpl_arch.symbols new file mode 100644 index 000000000..d91aa4e2b --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/tmpl_arch.symbols @@ -0,0 +1,6 @@ +libtmpl.so.1 #PACKAGE# #MINVER# + (arch=amd64)only_amd64@Base 1.0 + (arch=!amd64)not_amd64@Base 1.0 + (arch-bits=64)bits64@Base 1.0 + (optional)maybe_fn@Base 1.0 + plain_fn@Base 1.0 diff --git a/examples/pkg_resolver/abi/fixtures/tmpl_cxx.symbols b/examples/pkg_resolver/abi/fixtures/tmpl_cxx.symbols new file mode 100644 index 000000000..62efeb085 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/tmpl_cxx.symbols @@ -0,0 +1,3 @@ +libtmpl.so.1 #PACKAGE# #MINVER# + (c++)"Foo::bar(int)@Base" 1.0 + plain_fn@Base 1.0 diff --git a/examples/pkg_resolver/abi/fixtures/tmpl_symver.symbols b/examples/pkg_resolver/abi/fixtures/tmpl_symver.symbols new file mode 100644 index 000000000..773b054d7 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/tmpl_symver.symbols @@ -0,0 +1,3 @@ +libtmpl.so.1 #PACKAGE# #MINVER# + (symver)LIB_1 1.0 + plain_fn@LIB_1 1.0 diff --git a/examples/pkg_resolver/abi/fixtures/usecommon.c b/examples/pkg_resolver/abi/fixtures/usecommon.c new file mode 100644 index 000000000..df82a9ae4 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/usecommon.c @@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: MIT OR Apache-2.0 */ +/* Needs alpha_fn@COMMON_1 from libalpha.so.1 AND beta_fn@COMMON_1 from + * libbeta.so.1: the same version-node NAME in two verneed files. Attribution + * must go through the per-symbol version INDEX, not the node name. */ +int alpha_fn(void); +int beta_fn(void); +int main(void) { return alpha_fn() + beta_fn() == 3 ? 0 : 1; } diff --git a/examples/pkg_resolver/abi/fixtures/usefoo.c b/examples/pkg_resolver/abi/fixtures/usefoo.c new file mode 100644 index 000000000..9a47301c6 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/usefoo.c @@ -0,0 +1,4 @@ +/* SPDX-License-Identifier: MIT OR Apache-2.0 */ +/* Linked against libfoo v1: requires foo@LIB_1 from libfoo.so.1. */ +int foo(void); +int main(void) { return foo() == 1 ? 0 : 1; } diff --git a/examples/pkg_resolver/abi/fixtures/usepub.c b/examples/pkg_resolver/abi/fixtures/usepub.c new file mode 100644 index 000000000..10a6f5e79 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/usepub.c @@ -0,0 +1,6 @@ +/* SPDX-License-Identifier: MIT OR Apache-2.0 */ +/* Needs pub_fn@PUBLIC (a non-numeric node) and plain_fn (unversioned). Both + * are real loader obligations and must survive ingestion. */ +int pub_fn(void); +int plain_fn(void); +int main(void) { return pub_fn() + plain_fn() == 7 ? 0 : 1; } diff --git a/examples/pkg_resolver/abi/ingest_symbols.mjs b/examples/pkg_resolver/abi/ingest_symbols.mjs index a431b8c85..e4454632f 100644 --- a/examples/pkg_resolver/abi/ingest_symbols.mjs +++ b/examples/pkg_resolver/abi/ingest_symbols.mjs @@ -2,173 +2,366 @@ // SPDX-License-Identifier: MIT OR Apache-2.0 // Copyright (c) 2026 John William Creighton (@s243a) // -// ingest_symbols.mjs -- ingest symbol-level ABI metadata into the interval -// store as P/2 JSONL rows, mirroring the pkg_resolver store shape one level -// down (symbol tenures instead of package tenures). Three ingest tiers: +// ingest_symbols.mjs -- ingest symbol-level ABI evidence into the store as P/2 +// JSONL rows (`[key, value]`, the load_p2_jsonl shape). // -// symbols-file parse a Debian/Ubuntu `.symbols` control member -> -// symprov interval rows. ZERO binary download; the -// maintainer already computed the [sym, intro] table. -// elf readelf fallback for a `.so` with no `.symbols`: -// exported versioned symbols -> symprov rows -// (intro = the symbol's own version tag). -// requires readelf: the binary's referenced versioned symbols -// -> symreq rows (attributed to their soname via the -// verneed/.gnu.version_r table) + NEEDED sonames. -// symbols-dir batch: ingest every *.symbols under . +// DESIGN (post-review redesign; see REVIEW_NOTES.md): // -// Store rows are P/2 pairs `[key, value]` (the D43 indexer / load_p2_jsonl -// shape). Intervals are `intro#inf` within a soname (to="inf"). +// * Identity is the exact (soname, symbol, version-node) triple. A version +// node (`GLIBC_2.34`, `LIBSELINUX_1.0`, `COMMON_1`, `PUBLIC`, `Base`) is an +// opaque ELF label matched by string equality -- never parsed into a number, +// never collapsed to the bare symbol name. `Base` is dpkg's spelling of "no +// version" (an unversioned export); ELF unversioned exports are recorded +// under that same spelling so the two provider tiers agree. +// * The Debian PACKAGE-version axis (the `.symbols` minimum-version field, +// the release candidates) is a separate axis, carried verbatim as the deb +// version string and parsed/ordered on the Prolog side by the frozen +// resolver's deb/3 machinery (debian/deb_parse.pl + resolver:version_lt/2). +// * Every obligation is preserved: versioned requirements with non-numeric +// nodes, unversioned requirements, weak references (flagged, not dropped). +// * Requirements are attributed to their soname through the per-symbol +// version INDEX (.gnu.version -> .gnu.version_r), not through the version +// NAME, so two libraries sharing a node name (COMMON_1) never collide. +// * Evidence completeness is explicit: every successful ingest emits an +// `evidence` row; a missing / unreadable ELF is a loud failure (exit 3) +// that -- when --out is given -- records a failure evidence row instead +// of an empty "success". // -// symprov.jsonl : ["|", "#inf"] -// symreq.jsonl : ["|", "#"] -// needed.jsonl : ["", ""] +// Store rows (all JSON arrays `[key, value]`): +// symprov.jsonl ["|@", ["since", ""]] (.symbols) +// ["|@", ["at", ""]] (readelf) +// symreq.jsonl ["|@", ["", "GLOBAL"|"WEAK"]] +// ["|", ["", "GLOBAL"|"WEAK"]] (unversioned) +// needed.jsonl ["", ""] +// evidence.jsonl ["provides|", ["symbols"|"elf", "", "complete", ""]] +// ["requires|", ["readelf", "complete"|"missing_file"|"readelf_failed"|"inconsistent", ""]] +// releases.jsonl ["", ""] (candidate axis) // // Usage: -// node ingest_symbols.mjs symbols-file [--out DIR] [--stdout] -// node ingest_symbols.mjs elf [--out DIR] [--stdout] -// node ingest_symbols.mjs requires [--out DIR] [--stdout] -// node ingest_symbols.mjs symbols-dir [--out DIR] +// node ingest_symbols.mjs symbols-file [--release V] [--arch A] [--out DIR] [--append] [--stdout] +// node ingest_symbols.mjs symbols-dir [--arch A] --out DIR +// node ingest_symbols.mjs elf [--release V] [--out DIR] [--append] [--stdout] +// node ingest_symbols.mjs requires [--out DIR] [--append] [--stdout] +// node ingest_symbols.mjs releases ... [--out DIR] [--append] [--stdout] +// +// Exit codes: 0 ok; 2 usage; 3 evidence failure (missing file, readelf failure, +// unsupported .symbols template, unknown evidence release). import { execFileSync } from "node:child_process"; -import { readFileSync, appendFileSync, writeFileSync, mkdirSync, readdirSync } from "node:fs"; -import { dirname, join } from "node:path"; +import { readFileSync, existsSync, appendFileSync, writeFileSync, mkdirSync, readdirSync } from "node:fs"; +import { join } from "node:path"; -function readelf(args, file) { - try { - return execFileSync("readelf", [...args, file], { encoding: "utf8", maxBuffer: 1 << 26 }); - } catch { - return ""; - } +const EXIT_EVIDENCE = 3; + +function die(msg, code = EXIT_EVIDENCE) { + process.stderr.write(`ingest_symbols: ${msg}\n`); + process.exit(code); } function pair(k, v) { return JSON.stringify([k, v]); } -// A versioned symbol's numeric intro version, extracted from a version tag such -// as GLIBC_2.34 / LIBSELINUX_1.0 -> "2.34" / "1.0". Non-numeric namespaces -// (e.g. GLIBC_PRIVATE, @Base) return null so they never set a bound. -function verNum(ver) { - const m = String(ver).match(/_([0-9][0-9.]*)$/); - return m ? m[1] : null; +function run(cmd, args) { + return execFileSync(cmd, args, { encoding: "utf8", maxBuffer: 1 << 26, stdio: ["ignore", "pipe", "ignore"] }); +} + +// readelf: returns null (never "") on failure so callers cannot mistake a +// failed read for an empty section. +function readelf(args, file) { + if (!existsSync(file)) return null; + try { return run("readelf", [...args, file]); } catch { return null; } +} + +// --------------------------------------------------------------------------- +// Debian package-version helpers (ingestion edge only; ordering is Prolog's). +// --------------------------------------------------------------------------- +// The evidence release of a provider = the package version the evidence was +// taken from. For an installed .symbols file that is the installed package +// version (dpkg-query); for an ELF file it is the owning package's version +// (dpkg -S). Callers may override with --release. +function dpkgVersion(pkg) { + try { return run("dpkg-query", ["-W", "-f", "${Version}", pkg]).trim() || null; } catch { return null; } +} + +function dpkgOwner(path) { + try { + const out = run("dpkg", ["-S", path]).trim(); + const m = out.match(/^([^:\s]+(?::[^:\s]+)?):\s/); + return m ? m[1] : null; + } catch { return null; } } +// A syntactically valid Debian version: [epoch:]upstream[-revision], upstream +// starts with a digit (Policy 5.6.12). We only validate; parsing is Prolog's. +const DEB_VERSION_RE = /^(?:\d+:)?\d[A-Za-z0-9.+~:-]*$/; + // --------------------------------------------------------------------------- -// Tier 1: parse a Debian/Ubuntu `.symbols` control member. +// Tier: parse a Debian/Ubuntu `.symbols` file (binary control member form). // --------------------------------------------------------------------------- -// Format (per soname block): -// #MINVER# <- header (no leading space) -// | libc6 (>> 2.35), libc6 (<< 2.36) <- alt-dep template (skip) -// * Build-Depends-Package: libc6-dev <- meta field (skip) -// @ [] <- symbol (leading space) -// A symbol line may carry leading `(tag=value|...)` selectors which we strip. -// The `minimum-version` field IS the symbol's introduced version (our intro). -function parseSymbolsFile(path) { +// Binary form (dpkg-gensymbols output, /var/lib/dpkg/info/*.symbols): +// #MINVER# header (no leading whitespace) +// | libc6 (>> 2.35), libc6 (<< 2.36) alternative dependency template +// * Build-Depends-Package: libc6-dev meta field +// @ [] symbol row (indented) +// +// SOURCE-TEMPLATE syntax (debian/*.symbols in source packages) differs and is +// only partially supportable without the binary at hand. We process the +// semantics we can and REJECT the rest loudly (never silently mis-ingest): +// (optional) processed: the row is kept (the tag only relaxes +// dpkg-gensymbols' diff, it does not change the ABI fact) +// (arch=..)/(arch-bits=..)/(arch-endian=..) +// processed when --arch is given (row kept iff it +// selects the arch; arch-bits/endian derived from it); +// rejected otherwise +// (symver) rejected: `(symver)NODE minver` expands to "every +// symbol under NODE", which needs the binary to expand +// (regex) rejected: pattern rows need the binary to expand +// (c++) / (c++11) ... rejected: demangled C++ patterns (quoted) are not +// ELF symbol identities +// (ignore-blacklist) processed (ignored; does not affect identity) +// #include "file" rejected (template include) +// #PACKAGE# accepted in the header; the package name is then +// unknown so --release becomes mandatory +// Minimum-version semantics: a curated LOWER BOUND on the package version a +// dependent needs (Debian policy lets maintainers raise it after a compatible +// behaviour change), NOT a ground-truth introduction date. Stored verbatim. + +const KNOWN_ARCHES_64 = new Set(["amd64", "arm64", "ppc64el", "s390x", "riscv64", "ia64", "mips64el", "sparc64", "ppc64", "alpha", "loong64"]); +const BIG_ENDIAN = new Set(["s390x", "ppc64", "sparc64", "hppa", "m68k", "mips", "powerpc"]); + +function archSelects(spec, arch) { + // spec: "amd64 !i386 any-arm linux-any" -- dpkg-architecture style; we + // support exact names, `!` negation, and the `any`/`linux-any` wildcards. + const terms = spec.trim().split(/\s+/); + let selected = null; + for (let t of terms) { + let neg = false; + if (t.startsWith("!")) { neg = true; t = t.slice(1); } + const hit = t === "any" || t === "linux-any" || t === arch || t === `linux-${arch}` || + (t.startsWith("any-") && arch.endsWith(t.slice(4))); + if (neg) { if (hit) return false; if (selected === null) selected = true; } + else if (hit) selected = true; + else if (selected === null) selected = false; + } + return selected === null ? true : selected; +} + +function parseTags(line) { + // Leading `(tag|tag=value|...)` group(s). Returns {tags: Map, rest}. + const tags = new Map(); + let rest = line; + while (rest[0] === "(") { + const close = rest.indexOf(")"); + if (close < 0) break; + for (const t of rest.slice(1, close).split("|")) { + const eq = t.indexOf("="); + if (eq < 0) tags.set(t.trim(), true); + else tags.set(t.slice(0, eq).trim(), t.slice(eq + 1).trim()); + } + rest = rest.slice(close + 1); + } + return { tags, rest }; +} + +const UNSUPPORTED_TAGS = ["symver", "regex", "c++", "c++11", "c++14", "c++17", "c++20"]; + +function parseSymbolsFile(path, { arch = null } = {}) { + if (!existsSync(path)) die(`symbols file not found: ${path}`); const text = readFileSync(path, "utf8"); - let soname = null; - const rows = []; // {soname, sym, intro} - const sonames = new Set(); - let symCount = 0, skipped = 0; + const blocks = []; // {soname, package, rows: [{sym, node, minver}]} + const errors = []; // unsupported template constructs (line numbers) + let cur = null; + let lineNo = 0; for (const raw of text.split("\n")) { - if (!raw) continue; - // Header / meta lines are NOT indented. + lineNo++; + if (!raw.trim()) continue; if (!/^[ \t]/.test(raw)) { const c = raw[0]; - if (c === "|" || c === "*" || c === "#") continue; // alt-dep / meta / comment - soname = raw.trim().split(/\s+/)[0]; - if (soname) sonames.add(soname); + if (c === "|" || c === "*") continue; // alt-dep template / meta field + if (c === "#") { + if (/^#include\b/.test(raw)) errors.push(`${lineNo}: template #include`); + continue; // comment + } + const [soname, pkg] = raw.trim().split(/\s+/); + if (!soname) { errors.push(`${lineNo}: malformed header`); continue; } + cur = { soname, package: pkg && pkg !== "#PACKAGE#" ? pkg : null, rows: [] }; + blocks.push(cur); continue; } - if (!soname) continue; + if (!cur) { errors.push(`${lineNo}: symbol row before any soname header`); continue; } let line = raw.trim(); - if (!line || line[0] === "|" || line[0] === "*" || line[0] === "#") continue; - // Strip a leading (tag=value|...) selector group, e.g. "(optional)sym@Base". - if (line[0] === "(") { - const close = line.indexOf(")"); - if (close >= 0) line = line.slice(close + 1).trimStart(); + if (line[0] === "|" || line[0] === "*" || line[0] === "#") continue; + const { tags, rest } = parseTags(line); + line = rest.trimStart(); + // Reject template-only semantics loudly. + for (const u of UNSUPPORTED_TAGS) { + if (tags.has(u)) { errors.push(`${lineNo}: unsupported template tag (${u}): ${raw.trim()}`); tags.clear(); line = null; break; } + } + if (line === null) continue; + if (line[0] === '"') { errors.push(`${lineNo}: quoted (pattern) symbol needs the binary to expand: ${raw.trim()}`); continue; } + // Architecture selectors. + let archOk = true; + for (const key of ["arch", "arch-bits", "arch-endian"]) { + if (!tags.has(key)) continue; + if (!arch) { errors.push(`${lineNo}: (${key}=...) selector but no --arch given: ${raw.trim()}`); archOk = null; break; } + const v = String(tags.get(key)); + if (key === "arch") archOk = archOk && archSelects(v, arch); + else if (key === "arch-bits") archOk = archOk && (v === (KNOWN_ARCHES_64.has(arch) ? "64" : "32")); + else archOk = archOk && (v === (BIG_ENDIAN.has(arch) ? "big" : "little")); } - // "@ [id]" -- symbol names have no '@'. - const at = line.indexOf("@"); - if (at < 0) { skipped++; continue; } - const sym = line.slice(0, at); - const rest = line.slice(at + 1).split(/\s+/); - const version = rest[0]; // the symver tag, e.g. GLIBC_2.34 or Base - const minver = rest[1]; // the introduced-version field - if (!sym || minver === undefined) { skipped++; continue; } - // intro = the curated minimum-version field. "0" (private) stays 0.0.0. - const intro = /^[0-9]/.test(minver) ? minver : (verNum(version) || "0"); - rows.push({ soname, sym, intro }); - symCount++; + if (archOk === null) continue; + if (!archOk) continue; // row does not apply to this arch + // "@ []" -- split at the LAST '@' (symbol + // names never contain '@'; node names never do either). + const parts = line.split(/\s+/); + const ident = parts[0], minver = parts[1]; + const at = ident.lastIndexOf("@"); + if (at <= 0 || minver === undefined) { errors.push(`${lineNo}: malformed symbol row: ${raw.trim()}`); continue; } + const sym = ident.slice(0, at), node = ident.slice(at + 1); + if (!node) { errors.push(`${lineNo}: empty version node: ${raw.trim()}`); continue; } + if (!DEB_VERSION_RE.test(minver)) { errors.push(`${lineNo}: minimum-version is not a Debian version: ${raw.trim()}`); continue; } + cur.rows.push({ sym, node, minver }); } - return { rows, sonames: [...sonames], symCount, skipped }; + return { blocks, errors }; } // --------------------------------------------------------------------------- -// readelf helpers (Tiers 2 & 3). +// readelf: version-index-aware symbol tables (Tiers 2 & 3). // --------------------------------------------------------------------------- -// Defined versioned syms = provides; UND versioned syms = requires. -function dynsyms(file) { - const out = readelf(["-W", "--dyn-syms"], file); - const defined = [], undef = []; - for (const raw of out.split("\n")) { - const t = raw.trim().split(/\s+/); - if (t.length < 8 || !/^\d+:$/.test(t[0])) continue; - const ndx = t[6], name = t[7]; - if (!name.includes("@")) continue; - const [sym, ver] = name.replace("@@", "@").split("@"); - if (!sym || !ver) continue; - (ndx === "UND" ? undef : defined).push([sym, ver]); +// readelf -W --dyn-syms rows: " 7: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __libc_start_main@GLIBC_2.34 (5)" +// readelf -W -V: +// Version symbols section '.gnu.version' ... +// 000: 0 (*local*) 2 (GLIBC_2.3) 3 (GLIBC_2.2.5) 5h (GLIBC_2.34) <- versym per .dynsym index; 'h' = hidden +// Version definition section '.gnu.version_d' ... +// 0x0000: Rev: 1 Flags: base Index: 1 Cnt: 1 Name: libc.so.6 +// 0x001c: Rev: 1 Flags: none Index: 2 Cnt: 1 Name: GLIBC_2.2.5 +// Version needs section '.gnu.version_r' ... +// 0x0000: Version: 1 File: libc.so.6 Cnt: 9 +// 0x0010: Name: GLIBC_2.28 Flags: none Version: 11 +function elfTables(file) { + const symOut = readelf(["-W", "--dyn-syms"], file); + const verOut = readelf(["-W", "-V"], file); + const dynOut = readelf(["-W", "-d"], file); + if (symOut === null || verOut === null || dynOut === null) return null; + + const syms = []; // {idx, bind, ndx, name, verName (from name@VER), hiddenFromName} + for (const raw of symOut.split("\n")) { + const m = raw.match(/^\s*(\d+):\s+\S+\s+\S+\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)?(?:\s+\((\d+)\))?\s*$/); + if (!m) continue; + const [, idx, , bind, , ndx, name0] = m; + if (!name0) continue; + let name = name0, verName = null; + const dd = name0.indexOf("@@"), d = name0.indexOf("@"); + if (dd >= 0) { name = name0.slice(0, dd); verName = name0.slice(dd + 2); } + else if (d > 0) { name = name0.slice(0, d); verName = name0.slice(d + 1); } + syms.push({ idx: +idx, bind, ndx, name, verName }); + } + + // .gnu.version: index -> {ver, hidden} + const versym = new Map(); + const verdef = new Map(); // verIdx -> name (definitions; Index 1 = base/unversioned) + const verneed = new Map(); // verIdx -> {file, name, weak} + let section = null, curFile = null; + for (const raw of verOut.split("\n")) { + if (/^Version symbols section/.test(raw)) { section = "sym"; continue; } + if (/^Version definition section/.test(raw)) { section = "def"; continue; } + if (/^Version needs section/.test(raw)) { section = "need"; continue; } + if (section === "sym") { + const m = raw.match(/^\s*([0-9a-f]+):\s+(.*)$/); + if (!m) continue; + const base = parseInt(m[1], 16); + const re = /([0-9a-f]+)(h?)\s*\(([^)]*)\)/g; // "23 (GLIBC_2.34)" or hidden "10h(GLIBC_2.12)" + let e, i = 0; + while ((e = re.exec(m[2]))) { versym.set(base + i, { ver: parseInt(e[1], 16), hidden: e[2] === "h" }); i++; } + } else if (section === "def") { + const m = raw.match(/Index:\s+(\d+)\s+Cnt:\s+\d+\s+Name:\s+(\S+)/); + if (m) verdef.set(+m[1], m[2]); + } else if (section === "need") { + let m; + if ((m = raw.match(/Version:\s+\d+\s+File:\s+(\S+)\s+Cnt:/))) { curFile = m[1]; continue; } + if ((m = raw.match(/Name:\s+(\S+)\s+Flags:\s+([^\s]+(?:\s+[^\s]+)*?)\s+Version:\s+(\d+)/)) && curFile) { + verneed.set(+m[3], { file: curFile, name: m[1], weak: /WEAK/.test(m[2]) }); + } + } } - return { defined, undef }; -} -function dynamic(file) { - const out = readelf(["-d"], file); let soname = null; const needed = []; - for (const raw of out.split("\n")) { + for (const raw of dynOut.split("\n")) { let m; if ((m = raw.match(/\(SONAME\)\s+Library soname: \[([^\]]+)\]/))) soname = m[1]; else if ((m = raw.match(/\(NEEDED\)\s+Shared library: \[([^\]]+)\]/))) needed.push(m[1]); } - return { soname, needed }; + return { syms, versym, verdef, verneed, soname, needed, hasVersioning: versym.size > 0 }; } -// Map each version tag (GLIBC_2.34) to the soname (File) that provides it, -// from the `.gnu.version_r` (verneed) table. -function verneedMap(file) { - const out = readelf(["-V"], file); - const map = new Map(); // versionName -> soname - let inNeeds = false, curFile = null; - for (const raw of out.split("\n")) { - if (/Version needs section/.test(raw)) { inNeeds = true; continue; } - if (inNeeds && /Version (definition|symbols) section/.test(raw)) inNeeds = false; - if (!inNeeds) continue; - let m; - if ((m = raw.match(/File:\s+(\S+)/))) curFile = m[1]; - if ((m = raw.match(/Name:\s+(\S+)/)) && curFile) map.set(m[1], curFile); +// Provides: defined dynamic symbols with their exact version node. +function elfProvides(t) { + const rows = []; // {sym, node} + const problems = []; + for (const s of t.syms) { + if (s.ndx === "UND" || s.ndx === "Ndx") continue; + if (s.bind !== "GLOBAL" && s.bind !== "WEAK") continue; // LOCAL never exported + let node; + if (!t.hasVersioning) node = "Base"; + else { + const v = t.versym.get(s.idx); + if (!v) { problems.push(`no versym entry for dynsym ${s.idx} (${s.name})`); continue; } + if (v.ver === 0) continue; // *local* + if (v.ver === 1) node = "Base"; // *global* = unversioned + else { + node = t.verdef.get(v.ver); + if (!node) { problems.push(`dynsym ${s.idx} (${s.name}) versym ${v.ver} has no verdef entry`); continue; } + if (s.verName && s.verName !== node) problems.push(`dynsym ${s.idx}: name says @${s.verName} but verdef index says ${node}`); + } + } + rows.push({ sym: s.name, node }); } - return map; + return { rows, problems }; +} + +// Requires: undefined dynamic symbols, each attributed to (file, node) via its +// version INDEX. Never keyed by node name. +function elfRequires(t) { + const rows = []; // {sym, node|null, soname|"", bind} + const problems = []; + for (const s of t.syms) { + if (s.ndx !== "UND" || !s.name) continue; + if (s.bind !== "GLOBAL" && s.bind !== "WEAK") continue; + let bind = s.bind; + if (!t.hasVersioning) { rows.push({ sym: s.name, node: null, soname: "", bind }); continue; } + const v = t.versym.get(s.idx); + if (!v) { problems.push(`no versym entry for dynsym ${s.idx} (${s.name})`); continue; } + if (v.ver <= 1) { rows.push({ sym: s.name, node: null, soname: "", bind }); continue; } // unversioned reference + const need = t.verneed.get(v.ver); + if (!need) { problems.push(`dynsym ${s.idx} (${s.name}) versym ${v.ver} has no verneed entry`); continue; } + if (s.verName && s.verName !== need.name) problems.push(`dynsym ${s.idx}: name says @${s.verName} but verneed index ${v.ver} says ${need.name}`); + if (!t.needed.includes(need.file)) problems.push(`verneed file ${need.file} (for ${s.name}@${need.name}) is not in DT_NEEDED`); + if (need.weak) bind = "WEAK"; + rows.push({ sym: s.name, node: need.name, soname: need.file, bind }); + } + return { rows, problems }; } // --------------------------------------------------------------------------- // Output sink. // --------------------------------------------------------------------------- +const STORES = ["symprov", "symreq", "needed", "evidence", "releases"]; + function makeSink(outDir, toStdout) { - const buffers = { symprov: [], symreq: [], needed: [] }; + const buffers = Object.fromEntries(STORES.map((s) => [s, []])); return { add(store, k, v) { buffers[store].push(pair(k, v)); }, flush(append) { if (outDir) mkdirSync(outDir, { recursive: true }); - for (const store of Object.keys(buffers)) { + for (const store of STORES) { const lines = buffers[store]; if (!lines.length) continue; if (outDir) { const f = join(outDir, store + ".jsonl"); const body = lines.join("\n") + "\n"; - if (append) appendFileSync(f, body); - else writeFileSync(f, body); + if (append) appendFileSync(f, body); else writeFileSync(f, body); } if (toStdout) for (const l of lines) process.stdout.write(l + "\n"); } @@ -176,94 +369,140 @@ function makeSink(outDir, toStdout) { }; } -// --------------------------------------------------------------------------- -// Commands. -// --------------------------------------------------------------------------- function parseArgs(rest) { - const opts = { out: null, stdout: false, append: false, positional: [] }; + const opts = { out: null, stdout: false, append: false, release: null, arch: null, positional: [] }; for (let i = 0; i < rest.length; i++) { const a = rest[i]; if (a === "--out") opts.out = rest[++i]; else if (a === "--stdout") opts.stdout = true; else if (a === "--append") opts.append = true; + else if (a === "--release") opts.release = rest[++i]; + else if (a === "--arch") opts.arch = rest[++i]; else opts.positional.push(a); } + if (!opts.out && !opts.stdout) opts.stdout = true; return opts; } +function requireRelease(opts, guess, what) { + const rel = opts.release || guess; + if (!rel) die(`${what}: evidence release unknown (not owned by an installed package); pass --release `); + if (!DEB_VERSION_RE.test(rel)) die(`${what}: --release '${rel}' is not a Debian version`); + return rel; +} + +// --------------------------------------------------------------------------- +// Commands. +// --------------------------------------------------------------------------- +function cmdSymbolsFile(opts, path, sink, { batch = false } = {}) { + const { blocks, errors } = parseSymbolsFile(path, { arch: opts.arch }); + if (errors.length) { + process.stderr.write(`ingest_symbols: ${path}: ${errors.length} unsupported/malformed row(s) -- rejecting the file:\n`); + for (const e of errors.slice(0, 20)) process.stderr.write(` ${e}\n`); + if (errors.length > 20) process.stderr.write(` ... ${errors.length - 20} more\n`); + return null; + } + if (!blocks.length) { process.stderr.write(`ingest_symbols: ${path}: no soname blocks\n`); return null; } + let n = 0; + const sonames = []; + for (const b of blocks) { + const guess = b.package ? dpkgVersion(b.package) : null; + const rel = opts.release || guess; + if (!rel) { + process.stderr.write(`ingest_symbols: ${path}: ${b.soname}: evidence release unknown (package ${b.package || "#PACKAGE#"} not installed); pass --release\n`); + if (!batch) return null; else continue; + } + for (const { sym, node, minver } of b.rows) sink.add("symprov", `${b.soname}|${sym}@${node}`, ["since", minver]); + sink.add("evidence", `provides|${b.soname}`, ["symbols", rel, "complete", path]); + n += b.rows.length; + sonames.push(b.soname); + } + return { n, sonames }; +} + const [cmd, ...rest] = process.argv.slice(2); const opts = parseArgs(rest); -// symbols-dir manages its own append semantics; other modes default to -// overwrite unless --append. If neither --out nor --stdout, echo to stdout. -if (!opts.out && !opts.stdout) opts.stdout = true; if (cmd === "symbols-file") { const path = opts.positional[0]; - const { rows, sonames, symCount, skipped } = parseSymbolsFile(path); + if (!path) die("symbols-file: missing path", 2); const sink = makeSink(opts.out, opts.stdout); - for (const { soname, sym, intro } of rows) sink.add("symprov", soname + "|" + sym, intro + "#inf"); + const r = cmdSymbolsFile(opts, path, sink); + if (!r) process.exit(EXIT_EVIDENCE); sink.flush(opts.append); - process.stderr.write( - `symbols-file ${path}: symprov=${symCount} sonames=${sonames.length} [${sonames.slice(0, 6).join(", ")}${sonames.length > 6 ? ", ..." : ""}] skipped=${skipped}\n` - ); + process.stderr.write(`symbols-file ${path}: symprov=${r.n} sonames=${r.sonames.length} [${r.sonames.slice(0, 6).join(", ")}${r.sonames.length > 6 ? ", ..." : ""}]\n`); +} else if (cmd === "symbols-dir") { + const dir = opts.positional[0]; + if (!dir || !opts.out) die("symbols-dir: needs and --out DIR", 2); + const files = readdirSync(dir).filter((f) => f.endsWith(".symbols")).map((f) => join(dir, f)); + mkdirSync(opts.out, { recursive: true }); + for (const s of ["symprov", "evidence"]) writeFileSync(join(opts.out, s + ".jsonl"), ""); + let total = 0, ok = 0, rejected = 0; + for (const f of files) { + const sink = makeSink(opts.out, false); + const r = cmdSymbolsFile(opts, f, sink, { batch: true }); + if (!r) { rejected++; continue; } + sink.flush(true); + total += r.n; ok++; + } + process.stderr.write(`symbols-dir ${dir}: files=${ok} rejected=${rejected} symprov=${total}\n`); + if (rejected) process.exit(EXIT_EVIDENCE); } else if (cmd === "elf") { const path = opts.positional[0]; - const { defined } = dynsyms(path); - const { soname } = dynamic(path); - const so = soname || path; - // intro(sym) = earliest version tag seen on the exported symbol. - const intro = new Map(); - for (const [sym, ver] of defined) { - const n = verNum(ver); - if (!n) continue; - if (!intro.has(sym) || cmpVer(n, intro.get(sym)) < 0) intro.set(sym, n); - } + if (!path) die("elf: missing path", 2); + const t = elfTables(path); + if (!t) die(`elf: cannot read ${path} (missing file or readelf failure); no evidence emitted`); + const so = t.soname || path; + const owner = dpkgOwner(path); + const rel = requireRelease(opts, owner ? dpkgVersion(owner) : null, `elf ${path}`); + const { rows, problems } = elfProvides(t); + if (problems.length) die(`elf ${path}: inconsistent version tables:\n ${problems.slice(0, 10).join("\n ")}`); const sink = makeSink(opts.out, opts.stdout); - for (const [sym, v] of intro) sink.add("symprov", so + "|" + sym, v + "#inf"); + const seen = new Set(); + for (const { sym, node } of rows) { + const k = `${so}|${sym}@${node}`; + if (seen.has(k)) continue; // @ and @@ of the same node are one identity + seen.add(k); + sink.add("symprov", k, ["at", rel]); + } + sink.add("evidence", `provides|${so}`, ["elf", rel, "complete", path]); sink.flush(opts.append); - process.stderr.write(`elf ${path}: symprov=${intro.size} soname=${so}\n`); + process.stderr.write(`elf ${path}: soname=${so} release=${rel} symprov=${seen.size}\n`); } else if (cmd === "requires") { const path = opts.positional[0]; - const { undef } = dynsyms(path); - const { needed } = dynamic(path); - const vmap = verneedMap(path); + if (!path) die("requires: missing path", 2); + const t = elfTables(path); const sink = makeSink(opts.out, opts.stdout); - let n = 0; - for (const [sym, ver] of undef) { - const num = verNum(ver); - if (!num) continue; // ignore GLIBC_PRIVATE etc. - const so = vmap.get(ver) || "?"; // soname from verneed - sink.add("symreq", path + "|" + sym, so + "#" + num); - n++; + if (!t) { + const status = existsSync(path) ? "readelf_failed" : "missing_file"; + sink.add("evidence", `requires|${path}`, ["readelf", status, path]); + sink.flush(opts.append); + die(`requires ${path}: ${status}; recorded INCOMPLETE evidence, no requirement rows`); } - for (const so of needed) sink.add("needed", path, so); - sink.flush(opts.append); - process.stderr.write(`requires ${path}: symreq=${n} needed=[${needed.join(", ")}]\n`); -} else if (cmd === "symbols-dir") { - const dir = opts.positional[0]; - const files = readdirSync(dir).filter((f) => f.endsWith(".symbols")).map((f) => join(dir, f)); - if (opts.out) { mkdirSync(opts.out, { recursive: true }); writeFileSync(join(opts.out, "symprov.jsonl"), ""); } - let total = 0, nfiles = 0; - for (const f of files) { - let parsed; - try { parsed = parseSymbolsFile(f); } catch { continue; } - const sink = makeSink(opts.out, false); - for (const { soname, sym, intro } of parsed.rows) sink.add("symprov", soname + "|" + sym, intro + "#inf"); - sink.flush(true); // append across all files - total += parsed.symCount; - nfiles++; + const { rows, problems } = elfRequires(t); + if (problems.length) { + sink.add("evidence", `requires|${path}`, ["readelf", "inconsistent", problems[0]]); + sink.flush(opts.append); + die(`requires ${path}: inconsistent version tables:\n ${problems.slice(0, 10).join("\n ")}`); + } + let nv = 0, nu = 0; + for (const { sym, node, soname, bind } of rows) { + if (node === null) { sink.add("symreq", `${path}|${sym}`, ["", bind]); nu++; } + else { sink.add("symreq", `${path}|${sym}@${node}`, [soname, bind]); nv++; } } - process.stderr.write(`symbols-dir ${dir}: files=${nfiles} symprov=${total}\n`); + for (const so of t.needed) sink.add("needed", path, so); + sink.add("evidence", `requires|${path}`, ["readelf", "complete", path]); + sink.flush(opts.append); + process.stderr.write(`requires ${path}: symreq=${nv} versioned + ${nu} unversioned; needed=[${t.needed.join(", ")}]\n`); +} else if (cmd === "releases") { + const [so, ...vers] = opts.positional; + if (!so || !vers.length) die("releases: needs ...", 2); + for (const v of vers) if (!DEB_VERSION_RE.test(v)) die(`releases: '${v}' is not a Debian version`); + const sink = makeSink(opts.out, opts.stdout); + for (const v of vers) sink.add("releases", so, v); + sink.flush(opts.append); + process.stderr.write(`releases ${so}: ${vers.length} candidate(s)\n`); } else { - process.stderr.write("usage: ingest_symbols.mjs symbols-file|elf|requires|symbols-dir [--out DIR] [--stdout] [--append]\n"); + process.stderr.write("usage: ingest_symbols.mjs symbols-file|symbols-dir|elf|requires|releases [--release V] [--arch A] [--out DIR] [--append] [--stdout]\n"); process.exit(2); } - -function cmpVer(a, b) { - const pa = a.split("."), pb = b.split("."); - for (let i = 0; i < Math.max(pa.length, pb.length); i++) { - const x = +(pa[i] || 0), y = +(pb[i] || 0); - if (x !== y) return x - y; - } - return 0; -} diff --git a/examples/pkg_resolver/abi/run_abi_verify.sh b/examples/pkg_resolver/abi/run_abi_verify.sh index 46a420dfc..cf87b05f0 100755 --- a/examples/pkg_resolver/abi/run_abi_verify.sh +++ b/examples/pkg_resolver/abi/run_abi_verify.sh @@ -3,61 +3,140 @@ # Copyright (c) 2026 John William Creighton (@s243a) # # run_abi_verify.sh -- build the symbol-level ABI store from THIS machine's -# real Ubuntu/Debian data and verify it against the PoC's numbers. +# real Ubuntu/Debian data plus the review fixtures, then verify. # -# Tiers exercised: -# 1. .symbols control member -> symprov intervals (zero binary download) -# 2. readelf on libc.so.6 -> symprov (fallback), cross-checked vs tier 1 -# 3. readelf on /bin/ls -> symreq (verneed floor) + NEEDED sonames +# 1. Real data: libc6 + libselinux1 `.symbols` -> symprov (since bounds); +# /bin/ls -> symreq (attributed via the ELF version index) + NEEDED; +# the release axis from `apt-cache madison` + dpkg (real candidates). +# 2. Cross-check: readelf(libc.so.6) vs `.symbols` on EXACT sym@node +# identity (must be 100%), plus the corrected per-name earliest-row +# comparison the old script got wrong (must also be 100%). +# 3. ELF fixtures (gcc): foo@LIB_1 vs foo@LIB_2 under one soname (with the +# loader as ground truth), COMMON_1 cross-attribution, PUBLIC + +# unversioned obligations, a missing ELF, template `.symbols` rejects. +# 4. test_abi.pl: the Prolog assertions over all of the above. # -# Everything lands in ./.out (gitignored). Requires: node, readelf, swipl. +# Everything lands in ./.out (gitignored). Requires: node, readelf, swipl; +# gcc for the ELF fixtures (ABI_ALLOW_SKIP=1 tolerates a missing gcc). set -euo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" OUT="$HERE/.out" STORE="$OUT/store" +FX="$OUT/fx" +SRC="$HERE/fixtures" +INGEST="$HERE/ingest_symbols.mjs" BINARY="${ABI_BINARY:-/bin/ls}" -SONAME="${ABI_SONAME:-libc.so.6}" LIBSO="${ABI_LIBSO:-/lib/x86_64-linux-gnu/libc.so.6}" +ARCH="$(dpkg --print-architecture 2>/dev/null || echo amd64)" -# Locate libc6's .symbols control member (arch-qualified on multiarch). -SYMFILE="" -for c in /var/lib/dpkg/info/libc6:amd64.symbols /var/lib/dpkg/info/libc6.symbols; do - [ -f "$c" ] && SYMFILE="$c" && break -done -[ -z "$SYMFILE" ] && SYMFILE="$(ls /var/lib/dpkg/info/libc6*.symbols 2>/dev/null | grep -v i386 | head -1)" -[ -z "$SYMFILE" ] && { echo "no libc6 .symbols file found" >&2; exit 1; } +find_symbols() { # -> path of its installed .symbols control member + local p + for p in "/var/lib/dpkg/info/$1:$ARCH.symbols" "/var/lib/dpkg/info/$1.symbols"; do + [ -f "$p" ] && { echo "$p"; return 0; } + done + return 1 +} +LIBC_SYM="$(find_symbols libc6)" || { echo "no libc6 .symbols file found" >&2; exit 1; } +SELINUX_SYM="$(find_symbols libselinux1 || true)" rm -rf "$OUT" -mkdir -p "$STORE" "$OUT/cmp" +mkdir -p "$STORE" "$OUT/cmp" "$FX" + +echo "== 1. Real data: .symbols (since bounds) + $BINARY requires + release axis ==" +node "$INGEST" symbols-file "$LIBC_SYM" --out "$STORE" +if [ -n "$SELINUX_SYM" ]; then + node "$INGEST" symbols-file "$SELINUX_SYM" --out "$STORE" --append +fi +node "$INGEST" requires "$BINARY" --out "$STORE" --append -echo "== Tier 1: ingest $SYMFILE (.symbols control member) -> symprov ==" -node "$HERE/ingest_symbols.mjs" symbols-file "$SYMFILE" --out "$STORE" +# Release axis: every libc6 candidate the package index knows (madison) plus +# the installed version. These are actual releases, not symbol-intro points. +mapfile -t LIBC_RELS < <( { apt-cache madison libc6 2>/dev/null | awk -F'|' '{gsub(/ /,"",$2); print $2}'; dpkg-query -W -f '${Version}\n' libc6; } | sort -u ) +node "$INGEST" releases libc.so.6 "${LIBC_RELS[@]}" --out "$STORE" --append +if [ -n "$SELINUX_SYM" ]; then + mapfile -t SEL_RELS < <( { apt-cache madison libselinux1 2>/dev/null | awk -F'|' '{gsub(/ /,"",$2); print $2}'; dpkg-query -W -f '${Version}\n' libselinux1; } | sort -u ) + node "$INGEST" releases libselinux.so.1 "${SEL_RELS[@]}" --out "$STORE" --append +fi +# The declared dependency dpkg-shlibdeps generated for the binary's package: +# ground truth for abi_floor (test_abi.pl compares against this file). +OWNER="$(dpkg -S "$BINARY" 2>/dev/null | head -1 | cut -d: -f1 || true)" +if [ -n "$OWNER" ]; then + dpkg-query -W -f '${Pre-Depends}, ${Depends}\n' "$OWNER" > "$OUT/declared_depends.txt" + echo "declared deps of $OWNER: $(cat "$OUT/declared_depends.txt")" +fi echo -echo "== Tier 3: ingest $BINARY requires -> symreq + NEEDED ==" -node "$HERE/ingest_symbols.mjs" requires "$BINARY" --out "$STORE" --append +echo "== 2. Cross-check: readelf($LIBSO) vs $LIBC_SYM ==" +node "$INGEST" elf "$LIBSO" --out "$OUT/cmp/elf" 2>/dev/null +node "$INGEST" symbols-file "$LIBC_SYM" --out "$OUT/cmp/sym" 2>/dev/null +node "$HERE/crosscheck.mjs" "$OUT/cmp/sym/symprov.jsonl" "$OUT/cmp/elf/symprov.jsonl" libc.so.6 | tee "$OUT/crosscheck.txt" echo -echo "== Tier 2 cross-check: readelf($LIBSO) vs .symbols intros ==" -node "$HERE/ingest_symbols.mjs" symbols-file "$SYMFILE" --out "$OUT/cmp/sym" >/dev/null 2>&1 -node "$HERE/ingest_symbols.mjs" elf "$LIBSO" --out "$OUT/cmp/elf" >/dev/null 2>&1 -node -e ' -const fs=require("fs"); -const load=(f,so)=>{const m=new Map();for(const l of fs.readFileSync(f,"utf8").split("\n")){if(!l)continue;const [k,v]=JSON.parse(l);if(!k.startsWith(so+"|"))continue;m.set(k.slice(so.length+1),v.split("#")[0]);}return m;}; -const so=process.argv[3]; -const s=load(process.argv[1]+"/symprov.jsonl",so), e=load(process.argv[2]+"/symprov.jsonl",so); -let shared=0,agree=0; const dis=[]; -for(const [sym,iv] of e){ if(s.has(sym)){shared++; if(s.get(sym)===iv)agree++; else if(dis.length<5)dis.push(sym+" (.symbols="+s.get(sym)+" elf="+iv+")");}} -const pct=100*agree/shared; -console.log(" .symbols libc.so.6 syms:",s.size,"| readelf syms:",e.size,"| shared:",shared); -console.log(" intro agreement:",agree+"/"+shared,"("+pct.toFixed(1)+"%)"); -console.log(" sample disagreements (glibc 2.34 pthread/rt merge):",JSON.stringify(dis)); -if(pct<85){console.error(" FAIL: intro agreement below 85%");process.exit(1);} -if(s.get("getenv")!==e.get("getenv")){console.error(" FAIL: getenv intro mismatch");process.exit(1);} -console.log(" PASS: curated .symbols and readelf agree on the intro axis"); -' "$OUT/cmp/sym" "$OUT/cmp/elf" "$SONAME" +echo "== 3. Fixtures ==" +echo "-- .symbols templates: simple cases accepted; unsupported template rows rejected loudly" +node "$INGEST" symbols-file "$SRC/simple.symbols" --release 1.2-1 --out "$FX/simple" +for t in tmpl_symver tmpl_cxx tmpl_arch; do + if node "$INGEST" symbols-file "$SRC/$t.symbols" --release 1.0 --out "$FX/$t" 2>"$FX/$t.err"; then + echo "FAIL: $t.symbols was accepted without --arch / with template rows" >&2; exit 1 + else + echo " rejected $t.symbols (exit $?): $(head -2 "$FX/$t.err" | tail -1)" + fi +done +node "$INGEST" symbols-file "$SRC/tmpl_arch.symbols" --release 1.0 --arch amd64 --out "$FX/tmpl_arch_amd64" +echo " tmpl_arch with --arch amd64 -> $(tr -d '\n' < "$FX/tmpl_arch_amd64/symprov.jsonl" | sed 's/\]\[/] [/g')" + +echo "-- missing ELF: loud failure + INCOMPLETE evidence row (never an empty success)" +if node "$INGEST" requires "$FX/does-not-exist" --out "$FX/store_missing"; then + echo "FAIL: missing ELF ingested successfully" >&2; exit 1 +fi +echo " evidence: $(cat "$FX/store_missing/evidence.jsonl")" + +if command -v gcc >/dev/null 2>&1; then + echo "-- building ELF fixtures with gcc" + mkdir -p "$FX/v1" "$FX/v2" "$FX/lib" + gcc -shared -fPIC -Wl,--version-script="$SRC/foo_v1.map" -Wl,-soname,libfoo.so.1 -o "$FX/v1/libfoo.so.1" "$SRC/foo.c" + gcc -shared -fPIC -Wl,--version-script="$SRC/foo_v2.map" -Wl,-soname,libfoo.so.1 -o "$FX/v2/libfoo.so.1" "$SRC/foo.c" + gcc -Wl,-z,now -o "$FX/usefoo" "$SRC/usefoo.c" -L"$FX/v1" -l:libfoo.so.1 + gcc -shared -fPIC -Wl,--version-script="$SRC/common.map" -Wl,-soname,libalpha.so.1 -o "$FX/lib/libalpha.so.1" "$SRC/alpha.c" + gcc -shared -fPIC -Wl,--version-script="$SRC/common.map" -Wl,-soname,libbeta.so.1 -o "$FX/lib/libbeta.so.1" "$SRC/beta.c" + gcc -Wl,-z,now -o "$FX/usecommon" "$SRC/usecommon.c" -L"$FX/lib" -l:libalpha.so.1 -l:libbeta.so.1 + gcc -shared -fPIC -Wl,--version-script="$SRC/pub.map" -Wl,-soname,libpub.so.1 -o "$FX/lib/libpub.so.1" "$SRC/pub.c" + gcc -shared -fPIC -Wl,-soname,libplain.so.1 -o "$FX/lib/libplain.so.1" "$SRC/plain.c" + gcc -Wl,-z,now -o "$FX/usepub" "$SRC/usepub.c" -L"$FX/lib" -l:libpub.so.1 -l:libplain.so.1 + + echo "-- loader ground truth for foo@LIB_1 vs foo@LIB_2 (same soname libfoo.so.1)" + LD_LIBRARY_PATH="$FX/v1" "$FX/usefoo" && echo " v1: runs (foo@LIB_1 present)" + if LD_LIBRARY_PATH="$FX/v2" "$FX/usefoo" 2>"$FX/usefoo_v2.err"; then + echo "FAIL: loader accepted libfoo v2 (expected: undefined symbol foo, version LIB_1)" >&2; exit 1 + fi + echo " v2: loader rejects -> $(sed 's/^[^:]*: //' "$FX/usefoo_v2.err")" + grep -q "undefined symbol: foo, version LIB_1" "$FX/usefoo_v2.err" || { echo "FAIL: unexpected loader error" >&2; exit 1; } + + echo "-- ingesting fixture stores" + node "$INGEST" requires "$FX/usefoo" --out "$FX/store_foo_v1" + node "$INGEST" elf "$FX/v1/libfoo.so.1" --release 1.0-1 --out "$FX/store_foo_v1" --append + node "$INGEST" releases libfoo.so.1 1.0-1 --out "$FX/store_foo_v1" --append + node "$INGEST" requires "$FX/usefoo" --out "$FX/store_foo_v2" + node "$INGEST" elf "$FX/v2/libfoo.so.1" --release 2.0-1 --out "$FX/store_foo_v2" --append + node "$INGEST" releases libfoo.so.1 2.0-1 --out "$FX/store_foo_v2" --append + node "$INGEST" requires "$FX/usecommon" --out "$FX/store_common" + node "$INGEST" elf "$FX/lib/libalpha.so.1" --release 1.0-1 --out "$FX/store_common" --append + node "$INGEST" elf "$FX/lib/libbeta.so.1" --release 1.0-1 --out "$FX/store_common" --append + echo " usecommon symreq: $(cat "$FX/store_common/symreq.jsonl" | tr '\n' ' ')" + node "$INGEST" requires "$FX/usepub" --out "$FX/store_pub" + node "$INGEST" elf "$FX/lib/libpub.so.1" --release 1.0-1 --out "$FX/store_pub" --append + node "$INGEST" elf "$FX/lib/libplain.so.1" --release 1.0-1 --out "$FX/store_pub" --append + node "$INGEST" requires "$FX/usepub" --out "$FX/store_pub_nolibpub" + node "$INGEST" elf "$FX/lib/libplain.so.1" --release 1.0-1 --out "$FX/store_pub_nolibpub" --append + node "$INGEST" requires "$FX/usepub" --out "$FX/store_pub_noplain" + node "$INGEST" elf "$FX/lib/libpub.so.1" --release 1.0-1 --out "$FX/store_pub_noplain" --append + echo " usepub symreq: $(grep -E 'pub_fn|plain_fn' "$FX/store_pub/symreq.jsonl" | tr '\n' ' ')" + touch "$FX/BUILT" +else + echo "WARNING: gcc not found; ELF fixtures not built (test_abi.pl will SKIP them only with ABI_ALLOW_SKIP=1)" >&2 +fi echo -echo "== Prolog resolver verification ==" -swipl -q -g run -t halt "$HERE/test_abi.pl" -- "$STORE" +echo "== 4. Prolog resolver verification ==" +swipl -q -g run -t halt "$HERE/test_abi.pl" -- "$OUT" diff --git a/examples/pkg_resolver/abi/test_abi.pl b/examples/pkg_resolver/abi/test_abi.pl index aeb9ddab0..b253b2c13 100644 --- a/examples/pkg_resolver/abi/test_abi.pl +++ b/examples/pkg_resolver/abi/test_abi.pl @@ -2,82 +2,387 @@ % SPDX-License-Identifier: MIT OR Apache-2.0 % Copyright (c) 2026 John William Creighton (@s243a) % -% test_abi.pl -- verify the symbol-level ABI lane on THIS machine's real data. -% The store must already be built (see run_abi_verify.sh) into a directory -% passed as the first program argument (default: ./.out/store). +% test_abi.pl -- verify the redesigned symbol-level ABI lane. % -% swipl -q -g run -t halt examples/pkg_resolver/abi/test_abi.pl -- +% swipl -q -g run -t halt examples/pkg_resolver/abi/test_abi.pl -- % -% Asserts (the PoC's numbers, libc6 / /bin/ls on Ubuntu 22.04): -% abi_min(/bin/ls, libc.so.6) == 2.34 -% abi_compatible(/bin/ls, libc.so.6, 2.35) is TRUE (0 missing) -% newest_abi_compatible(no drop) == compatible(2.35), range(2.34,2.35) -% simulated removal of getenv at 2.30 -> cap 2.29, min 2.34 > max 2.29 -% -> no_candidate (the min>max squeeze) +% is run_abi_verify.sh's ./.out: it holds the real-data store +% (/store), the fixture stores (/fx/store_*) and the declared +% dependency ground truth (/declared_depends.txt). +% +% Sections (each check names the review point it proves; see REVIEW_NOTES.md): +% A. real data -- libc6 / libselinux1 .symbols + /bin/ls on this machine +% B. version axes -- Debian package axis via the frozen deb/3 machinery +% C. model -- in-memory store fixtures for the epistemic contract +% D. ELF fixtures -- gcc-built libraries ingested through the real pipeline :- use_module(abi_resolve). +:- use_module('../debian/deb_parse', [parse_deb_version/2]). -:- dynamic pass_count/1, fail_count/1. +:- dynamic pass_count/1, fail_count/1, skip_count/1. pass_count(0). fail_count(0). +skip_count(0). check(Name, Goal) :- ( catch(Goal, E, (print_message(error, E), fail)) - -> format(" PASS ~w~n", [Name]), - retract(pass_count(P)), P1 is P + 1, assertz(pass_count(P1)) - ; format(" FAIL ~w~n", [Name]), - retract(fail_count(F)), F1 is F + 1, assertz(fail_count(F1)) + -> format(" PASS ~w~n", [Name]), bump(pass_count) + ; format(" FAIL ~w~n", [Name]), bump(fail_count) ). -store_dir(Dir) :- - ( current_prolog_flag(argv, [D | _]), D \== [] -> Dir = D ; Dir = './.out/store' ). +skip(Name) :- + format(" SKIP ~w~n", [Name]), bump(skip_count). + +bump(C) :- G0 =.. [C, N], retract(G0), N1 is N + 1, G1 =.. [C, N1], assertz(G1). + +report(Label, Val) :- format(" ~w = ~q~n", [Label, Val]). + +out_dir(Dir) :- + ( current_prolog_flag(argv, [D | _]), D \== [] -> Dir = D ; Dir = './.out' ). run :- - store_dir(Dir), - format("~n== ABI lane verification (store: ~w) ==~n", [Dir]), - load_abi_store(Dir), - aggregate_all(count, abi_resolve:symprov(_,_), NP), - aggregate_all(count, abi_resolve:symreq(_,_), NR), - aggregate_all(count, ( abi_resolve:symprov(K,_), atom_concat('libc.so.6|', _, K) ), NLibc), - format(" store: symprov=~w (libc.so.6=~w) symreq=~w~n~n", [NP, NLibc, NR]), + out_dir(Out), + format("~n== ABI lane verification (out: ~w) ==~n", [Out]), + section_real(Out), + section_axes, + section_model, + section_elf_fixtures(Out), + pass_count(P), fail_count(F), skip_count(S), + format("~n== ~w passed, ~w failed, ~w skipped ==~n", [P, F, S]), + ( F =:= 0 -> true ; halt(1) ). + +% =========================================================================== +% A. Real data +% =========================================================================== +section_real(Out) :- + atom_concat(Out, '/store', Store), + format("~n-- A. real data (~w) --~n", [Store]), + load_abi_store(Store), + aggregate_all(count, symprov(_, _, _, _), NP), + aggregate_all(count, symprov('libc.so.6', _, _, _), NLibc), + aggregate_all(count, symreq(_, _, _, _, _), NR), + aggregate_all(count, symreq(_, _, none, none, _), NU), + format(" store: symprov=~w (libc.so.6=~w) symreq=~w (~w unversioned)~n", [NP, NLibc, NR, NU]), Bin = '/bin/ls', So = 'libc.so.6', - soname_candidates(So, Cands), - length(Cands, NC), - format(" libc.so.6 version axis: ~w distinct versions~n", [NC]), - check('abi_min(/bin/ls, libc.so.6) == 2.34', - ( abi_min(Bin, So, Min), report(' abi_min', Min), Min == '2.34' )), + check('A1 requirement evidence for /bin/ls is complete (#2)', + req_evidence(Bin, readelf, complete, _)), + check('A2 provider evidence for libc.so.6 is complete, from .symbols (#2)', + prov_evidence(So, symbols, _, complete)), + check('A3 NEEDED sonames ingested and checked (#2)', + ( needed(Bin, So), needed(Bin, 'libselinux.so.1') )), + check('A4 exact identity: __libc_start_main@GLIBC_2.34 attributed to libc.so.6 via version index (#1,#3)', + symreq(Bin, '__libc_start_main', 'GLIBC_2.34', So, 'GLOBAL')), + check('A5 libselinux requirement attributed to libselinux.so.1, not libc (#3)', + symreq(Bin, freecon, 'LIBSELINUX_1.0', 'libselinux.so.1', _)), + check('A6 unversioned (weak) requirements preserved, not dropped (#2)', + symreq(Bin, '__gmon_start__', none, none, 'WEAK')), + check('A7 no bare-name provider rows: every symprov row carries a node (#1)', + \+ ( symprov(_, _, N, _), ( N == '' ; N == none ) )), - check('abi_compatible(/bin/ls, libc.so.6, 2.35) is TRUE', - abi_compatible(Bin, So, '2.35')), + % Curated floors on the Debian package axis == dpkg-shlibdeps' declared deps. + check('A8 abi_floor(/bin/ls, libc.so.6) == 2.34 (#4)', + ( abi_floor(Bin, So, F1), report(floor_libc, F1), F1 == '2.34' )), + check('A9 abi_floor(/bin/ls, libselinux.so.1) == 3.1~ (tilde preserved) (#4)', + ( abi_floor(Bin, 'libselinux.so.1', F2), report(floor_selinux, F2), F2 == '3.1~' )), + atom_concat(Out, '/declared_depends.txt', DepFile), + ( exists_file(DepFile) + -> read_file_to_string(DepFile, DepS, []), + check('A10 floors equal the package''s declared Pre-Depends/Depends (ground truth) (#4)', + ( abi_floor(Bin, So, FL), abi_floor(Bin, 'libselinux.so.1', FS), + format(atom(E1), "libc6 (>= ~w)", [FL]), + format(atom(E2), "libselinux1 (>= ~w)", [FS]), + sub_string(DepS, _, _, _, E1), sub_string(DepS, _, _, _, E2) )) + ; skip('A10 declared_depends.txt not present') + ), - check('0 missing symbols at 2.35', - ( missing_syms(Bin, So, '2.35', none, M), length(M, LM), - report(' missing@2.35', LM), LM =:= 0 )), + % Verdicts over the ACTUAL release axis. + release_axis(So, Axis), + report(libc_release_axis, Axis), + check('A11 release axis is non-empty and made of real package versions (#5)', + ( Axis = [_|_], forall(member(A, Axis), (rel_term(A, deb(_, _, _)))) )), + check('A12 verdict at the .symbols evidence release is compatible(exact)', + ( prov_evidence(So, symbols, R0, complete), release(So, R0, R0A), + abi_verdict(Bin, So, R0A, V), report(verdict_at_evidence_release, R0A-V), + V == compatible(exact) )), + check('A13 abi_range over the real axis: both ends carry compatible verdicts (#5)', + ( abi_range(Bin, So, RR), RR = range(Min, Max, Pairs), + report(range, Min-Max), + memberchk(Min-compatible(_), Pairs), memberchk(Max-compatible(_), Pairs) )), + check('A14 older hypothetical release 2.31-0ubuntu9.9 -> incompatible(below_floor __libc_start_main@GLIBC_2.34) (#4,#5)', + ( abi_verdict(Bin, So, '2.31-0ubuntu9.9', V31), report(verdict_2_31, V31), + V31 = incompatible(L31), + memberchk(below_floor('__libc_start_main'@'GLIBC_2.34', '2.34'), L31) )), + check('A15 valid lower bound: axis extended below the floor -> min is the first COMPATIBLE release, not the lowest (#5)', + ( abi_range(Bin, So, ['2.31-0ubuntu9.9', '2.34-0ubuntu3' | Axis], none, RX), + RX = range(MinX, MaxX, PX), report(extended_range, MinX-MaxX), + MinX == '2.34-0ubuntu3', last(Axis, MaxX), + memberchk('2.31-0ubuntu9.9'-incompatible(_), PX) )), + check('A16 beyond the evidence release -> compatible(extrapolated), never exact', + ( abi_verdict(Bin, So, '99:1.0', VX), VX == compatible(extrapolated) )), - check('newest_abi_compatible (no drop) == compatible(2.35)', - ( newest_abi_compatible(Bin, So, Cands, R), - report(' newest', R), R == compatible('2.35') )), + % Exact node identity on real data: same symbol, other node -> hard veto. + check('A17 getenv@GLIBC_2.99 (symbol present, node absent) -> incompatible(missing) (#1)', + with_extra_req(Bin, getenv, 'GLIBC_2.99', So, + ( abi_verdict(Bin, So, '2.35-0ubuntu3', VG), report(verdict_getenv_2_99, VG), + VG = incompatible(LG), memberchk(missing(getenv@'GLIBC_2.99'), LG) ))), + check('A18 pthread_setname_np@GLIBC_2.12 (non-default node, real) -> provided; @GLIBC_2.13 -> missing (#1)', + ( with_extra_req(Bin, pthread_setname_np, 'GLIBC_2.12', So, + abi_verdict(Bin, So, '2.35-0ubuntu3', compatible(_))), + with_extra_req(Bin, pthread_setname_np, 'GLIBC_2.13', So, + abi_verdict(Bin, So, '2.35-0ubuntu3', incompatible(_))) )), - check('abi_range (no drop) == range(2.34, 2.35)', - ( abi_range(Bin, So, Cands, RR), - report(' range', RR), RR == range('2.34', '2.35') )), + check('A19 soname mismatch is a hard veto: offering libc.so.7 for a libc.so.6 NEEDED (#2)', + ( abi_verdict(Bin, 'libc.so.7', '2.35-0ubuntu3', VS), report(verdict_libc7, VS), + VS == incompatible([soname_mismatch(offered('libc.so.7'), needed('libc.so.6'))]) )), + check('A20 a soname the binary does not need -> not_needed', + abi_verdict(Bin, 'libz.so.1', '1.0', not_needed('libz.so.1'))), + check('A21 binary with no requirement evidence -> unknown, not compatible (#2)', + abi_verdict('/nonexistent/bin', So, '2.35-0ubuntu3', unknown([no_requires_evidence(_)]))), + check('A22 weak unversioned refs never veto (statuses are weak_unresolved)', + ( prov_evidence(So, symbols, R0b, complete), + findall(S, ( req_status(Bin, So, R0b, none, S), S \= provided(_, _) ), Odd), + report(non_provided_statuses, Odd), + forall(member(O, Odd), O = weak_unresolved(_)) )), - % Simulated ABI break: remove getenv at 2.30 (a symbol /bin/ls needs). - check('/bin/ls needs getenv', - req_sym(Bin, So, getenv, _)), + % Hypothetical in-soname removal: the max is refined downward, min stays valid. + last(Axis, Top), Axis = [Bottom|_], + check('A23 drop getenv@GLIBC_2.2.5 at the newest release -> max refined below it (or no_candidate on a 1-release axis)', + ( abi_range(Bin, So, drop(getenv, 'GLIBC_2.2.5', Top), RD), report(range_drop_getenv, RD), + ( RD = range(_, MaxD, _) -> rel_term(MaxD, MD), rel_term(Top, TD), rel_lt(MD, TD) + ; RD = no_candidate(_), Axis = [_] ) )), + check('A24 drop __libc_start_main@GLIBC_2.34 at the oldest release -> no_candidate (squeeze)', + ( abi_range(Bin, So, drop('__libc_start_main', 'GLIBC_2.34', Bottom), RD2), + RD2 = no_candidate(_) )). - check('removal cap: newest version still exporting getenv == 2.29', - ( newest_providing(So, getenv, Cands, drop(getenv, '2.30'), Cap), - report(' getenv cap', Cap), Cap == compatible('2.29') )), +with_extra_req(Bin, Sym, Node, So, Goal) :- + setup_call_cleanup(assertz(abi_resolve:symreq(Bin, Sym, Node, So, 'GLOBAL')), + Goal, + retract(abi_resolve:symreq(Bin, Sym, Node, So, 'GLOBAL'))). - check('simulated removal -> no_candidate (min 2.34 > max 2.29 squeeze)', - ( abi_range(Bin, So, Cands, drop(getenv, '2.30'), RD), - report(' range(drop)', RD), RD = no_candidate(_) )), +% =========================================================================== +% B. Version axes (Debian package axis via the frozen deb/3 comparison) +% =========================================================================== - pass_count(P), fail_count(F), - format("~n== ~w passed, ~w failed ==~n", [P, F]), - ( F =:= 0 -> true ; halt(1) ). +section_axes :- + format("~n-- B. version axes --~n", []), + check('B1 3.1~ < 3.1 (tilde sorts first) (#4)', deb_lt('3.1~', '3.1')), + check('B2 3.1~ < 3.1~rc1 (#4)', deb_lt('3.1~', '3.1~rc1')), + check('B3 epoch: 1:2.3-1 > 3.1, not 0 (#4)', deb_lt('3.1', '1:2.3-1')), + check('B4 revision: 2.35 < 2.35-0ubuntu3 < 2.35-0ubuntu3.15 (#4)', + ( deb_lt('2.35', '2.35-0ubuntu3'), deb_lt('2.35-0ubuntu3', '2.35-0ubuntu3.15') )), + check('B5 four components kept: 2.2.5 < 2.2.5.1 < 2.2.6 (#4)', + ( deb_lt('2.2.5', '2.2.5.1'), deb_lt('2.2.5.1', '2.2.6') )), + check('B6 minimum 0 (private) is below everything', deb_lt('0', '0.0.1')), + check('B7 non-deb release ids are labels that only match themselves', + ( rel_term('file:/tmp/x.so', label(_)), rel_le(label(a), label(a)), \+ rel_le(label(a), label(b)), + \+ rel_le(label(a), deb(0, [s([], 1)], [])) )), + check('B8 ELF nodes are never ordered numerically: GLIBC_2.34 vs GLIBC_2.4 is identity only (#1,#4)', + ( \+ catch(rel_lt('GLIBC_2.4', 'GLIBC_2.34'), _, fail), + rel_term('GLIBC_2.34', label('GLIBC_2.34')) )). + +deb_lt(A, B) :- + rel_term(A, DA), rel_term(B, DB), + DA = deb(_, _, _), DB = deb(_, _, _), + rel_lt(DA, DB), \+ rel_lt(DB, DA). + +% =========================================================================== +% C. Model fixtures (in-memory stores) +% =========================================================================== + +fx_clear :- abi_store_clear. +fx(Fact) :- assertz(abi_resolve:Fact). +fx_since(A, since(D, A)) :- parse_deb_version(A, D). +fx_rel(A, R) :- rel_term(A, R). +fx_releases(So, Atoms) :- forall(member(A, Atoms), ( fx_rel(A, R), fx(release(So, R, A)) )). + +section_model :- + format("~n-- C. model fixtures --~n", []), + + % C1: Astra #1 -- foo@LIB_1 required, only foo@LIB_2 exported (same soname). + fx_clear, + fx_since('1.0', S1), fx_rel('2.0-1', R2), + fx(symprov('libfoo.so.1', foo, 'LIB_2', S1)), + fx(symprov('libfoo.so.1', foo_legacy, 'LIB_1', S1)), + fx(prov_evidence('libfoo.so.1', symbols, R2, complete)), + fx(symreq(usefoo, foo, 'LIB_1', 'libfoo.so.1', 'GLOBAL')), + fx(needed(usefoo, 'libfoo.so.1')), + fx(req_evidence(usefoo, readelf, complete, usefoo)), + check('C1 foo@LIB_1 required vs foo@LIB_2 exported -> incompatible([missing(foo@LIB_1)]) (#1)', + ( abi_verdict(usefoo, 'libfoo.so.1', '2.0-1', V1), report(verdict, V1), + V1 == incompatible([missing(foo@'LIB_1')]) )), + fx(symprov('libfoo.so.1', foo, 'LIB_1', S1)), + check('C1b adding the exact foo@LIB_1 row -> compatible(exact) (#1)', + abi_verdict(usefoo, 'libfoo.so.1', '2.0-1', compatible(exact))), + + % C2: Astra #5 -- the range(1,5)-with-floor-5 bug. + fx_clear, + fx_since('1.0', Sa), fx_since('5.0', Sb), fx_rel('5.0', R5), + fx(symprov('libr.so.1', a, 'L', Sa)), + fx(symprov('libr.so.1', b, 'L', Sb)), + fx(prov_evidence('libr.so.1', symbols, R5, complete)), + fx(symreq(bin, a, 'L', 'libr.so.1', 'GLOBAL')), + fx(symreq(bin, b, 'L', 'libr.so.1', 'GLOBAL')), + fx(needed(bin, 'libr.so.1')), + fx(req_evidence(bin, readelf, complete, bin)), + fx_releases('libr.so.1', ['1.0', '2.0', '3.0', '4.0', '5.0']), + check('C2 releases 1..5, a since 1, b since 5 -> range(5.0, 5.0), NOT range(1,5) (#5)', + ( abi_range(bin, 'libr.so.1', RR), report(range, RR), RR = range('5.0', '5.0', _) )), + check('C2b verdict at 1.0 is incompatible(below_floor(b@L, 5.0)) (#5)', + abi_verdict(bin, 'libr.so.1', '1.0', incompatible([below_floor(b@'L', '5.0')]))), + fx_releases('libr.so.1', ['6.0']), + check('C2c a release newer than the evidence (6.0) is compatible(extrapolated); range(5.0, 6.0) (#5)', + ( abi_verdict(bin, 'libr.so.1', '6.0', compatible(extrapolated)), + abi_range(bin, 'libr.so.1', range('5.0', '6.0', _)) )), + check('C2d releases adding no symbols (2.0..4.0) are evaluated on the axis, each incompatible (#5)', + ( abi_range(bin, 'libr.so.1', range(_, _, P)), + memberchk('3.0'-incompatible(_), P), length(P, 6) )), + + % C3: readelf-only provider evidence (at R0): older releases are UNKNOWN. + fx_clear, + fx_rel('2.0', R20), + fx(symprov('libe.so.1', s, 'N', at(R20))), + fx(prov_evidence('libe.so.1', elf, R20, complete)), + fx(symreq(bin, s, 'N', 'libe.so.1', 'GLOBAL')), + fx(needed(bin, 'libe.so.1')), + fx(req_evidence(bin, readelf, complete, bin)), + check('C3 at(2.0) evidence: 1.0 -> unknown, 2.0 -> compatible(exact), 3.0 -> compatible(extrapolated) (#2)', + ( abi_verdict(bin, 'libe.so.1', '1.0', unknown([unknown(s@'N', evidence_release(_))])), + abi_verdict(bin, 'libe.so.1', '2.0', compatible(exact)), + abi_verdict(bin, 'libe.so.1', '3.0', compatible(extrapolated)) )), + fx_releases('libe.so.1', ['1.0', '2.0']), + check('C3b range over [1.0, 2.0] with at(2.0) evidence -> range(2.0, 2.0); 1.0 reported unknown (#2,#5)', + ( abi_range(bin, 'libe.so.1', range('2.0', '2.0', P3)), memberchk('1.0'-unknown(_), P3) )), + + % C4: Astra #2 -- incomplete evidence is never a false compat or false veto. + fx_clear, + fx(symreq(bin, s, 'N', 'libe.so.1', 'GLOBAL')), + fx(needed(bin, 'libe.so.1')), + fx(req_evidence(bin, readelf, complete, bin)), + check('C4 needed soname with NO provider evidence -> unknown([no_provider_evidence]) (#2)', + abi_verdict(bin, 'libe.so.1', '1.0', unknown([no_provider_evidence('libe.so.1')]))), + fx_clear, + fx(req_evidence(bin2, readelf, missing_file, bin2)), + fx_rel('1.0', R10), fx(prov_evidence('libe.so.1', elf, R10, complete)), + check('C4b requirement evidence missing_file -> unknown([requires_evidence(missing_file, _)]) (#2)', + abi_verdict(bin2, 'libe.so.1', '1.0', unknown([requires_evidence(missing_file, _)]))), + fx_clear, + fx(symreq(bin, plain_fn, none, none, 'GLOBAL')), + fx(needed(bin, 'liba.so.1')), fx(needed(bin, 'libb.so.1')), + fx(req_evidence(bin, readelf, complete, bin)), + fx(prov_evidence('liba.so.1', elf, R10, complete)), + check('C4c unversioned obligation, one NEEDED lib lacks evidence -> unknown, not missing (#2)', + abi_verdict(bin, 'liba.so.1', '1.0', unknown([unknown(plain_fn, no_provider_evidence('libb.so.1'))]))), + fx(prov_evidence('libb.so.1', elf, R10, complete)), + check('C4d unversioned obligation, all evidence complete, nobody exports it -> incompatible([missing(plain_fn)]) (#2)', + abi_verdict(bin, 'liba.so.1', '1.0', incompatible([missing(plain_fn)]))), + fx(symprov('libb.so.1', plain_fn, 'Base', at(R10))), + check('C4e unversioned obligation satisfied by ANY exported node of another NEEDED lib -> compatible (#2)', + abi_verdict(bin, 'liba.so.1', '1.0', compatible(exact))), + + % C5: non-numeric nodes are first-class identities. + fx_clear, + fx(symprov('libpub.so.1', pub_fn, 'PUBLIC', at(R10))), + fx(prov_evidence('libpub.so.1', elf, R10, complete)), + fx(symreq(bin, pub_fn, 'PUBLIC', 'libpub.so.1', 'GLOBAL')), + fx(needed(bin, 'libpub.so.1')), + fx(req_evidence(bin, readelf, complete, bin)), + check('C5 pub_fn@PUBLIC (no numeric bound) is matched exactly -> compatible (#1,#2)', + abi_verdict(bin, 'libpub.so.1', '1.0', compatible(exact))), + check('C5b pub_fn@PUBLIC_2 -> incompatible([missing(pub_fn@PUBLIC_2)]) (#1)', + with_extra_req(bin, pub_fn, 'PUBLIC_2', 'libpub.so.1', + abi_verdict(bin, 'libpub.so.1', '1.0', incompatible([missing(pub_fn@'PUBLIC_2')])))), + + % C6: a weak reference to an absent symbol is not a veto. + fx(symreq(bin, maybe_fn, 'PUBLIC', 'libpub.so.1', 'WEAK')), + check('C6 weak requirement of an absent symbol -> still compatible (weak_unresolved)', + ( abi_verdict(bin, 'libpub.so.1', '1.0', compatible(exact)), + req_status(bin, 'libpub.so.1', R10, none, weak_unresolved(maybe_fn@'PUBLIC')) )). + +% =========================================================================== +% D. ELF fixtures built by run_abi_verify.sh (real ingest pipeline) +% =========================================================================== + +section_elf_fixtures(Out) :- + format("~n-- D. ELF fixtures (~w/fx) --~n", [Out]), + atom_concat(Out, '/fx', FX), + atom_concat(FX, '/BUILT', Built), + ( exists_file(Built) + -> elf_fixture_checks(FX) + ; ( getenv('ABI_ALLOW_SKIP', '1') + -> skip('D* ELF fixtures not built (no gcc); ABI_ALLOW_SKIP=1') + ; check('D0 ELF fixtures were built (gcc available)', fail) + ) + ), + symbols_fixture_checks(FX). + +elf_fixture_checks(FX) :- + atom_concat(FX, '/usefoo', UseFoo), + atom_concat(FX, '/usecommon', UseCommon), + atom_concat(FX, '/usepub', UsePub), + + atom_concat(FX, '/store_foo_v2', StV2), load_abi_store(StV2), + check('D1 [ELF] usefoo vs libfoo v2 (foo@LIB_2 only, LIB_1 node present) -> incompatible([missing(foo@LIB_1)]); loader agrees (#1)', + ( abi_verdict(UseFoo, 'libfoo.so.1', '2.0-1', V), report(verdict, V), + V == incompatible([missing(foo@'LIB_1')]) )), + check('D1b [ELF] v2 store has foo@LIB_2 and foo_legacy@LIB_1 as exact identities (#1)', + ( symprov('libfoo.so.1', foo, 'LIB_2', at(_)), symprov('libfoo.so.1', foo_legacy, 'LIB_1', at(_)), + \+ symprov('libfoo.so.1', foo, 'LIB_1', _) )), + atom_concat(FX, '/store_foo_v1', StV1), load_abi_store(StV1), + check('D2 [ELF] usefoo vs libfoo v1 (foo@LIB_1) -> compatible(exact); range(1.0-1, 1.0-1) (#1,#5)', + ( abi_verdict(UseFoo, 'libfoo.so.1', '1.0-1', compatible(exact)), + abi_range(UseFoo, 'libfoo.so.1', range('1.0-1', '1.0-1', _)) )), + + atom_concat(FX, '/store_common', StC), load_abi_store(StC), + check('D3 [ELF] COMMON_1 in two libs: alpha_fn -> libalpha.so.1, beta_fn -> libbeta.so.1 (version-index join) (#3)', + ( symreq(UseCommon, alpha_fn, 'COMMON_1', 'libalpha.so.1', 'GLOBAL'), + symreq(UseCommon, beta_fn, 'COMMON_1', 'libbeta.so.1', 'GLOBAL'), + \+ symreq(UseCommon, alpha_fn, _, 'libbeta.so.1', _), + \+ symreq(UseCommon, beta_fn, _, 'libalpha.so.1', _) )), + check('D3b [ELF] both COMMON_1 providers verdict compatible(exact) (#3)', + ( abi_verdict(UseCommon, 'libalpha.so.1', '1.0-1', compatible(exact)), + abi_verdict(UseCommon, 'libbeta.so.1', '1.0-1', compatible(exact)) )), + + atom_concat(FX, '/store_pub', StP), load_abi_store(StP), + check('D4 [ELF] pub_fn@PUBLIC (non-numeric node) and plain_fn (unversioned) both ingested as obligations (#2)', + ( symreq(UsePub, pub_fn, 'PUBLIC', 'libpub.so.1', 'GLOBAL'), + symreq(UsePub, plain_fn, none, none, 'GLOBAL') )), + check('D4b [ELF] unversioned export recorded as @Base; verdicts compatible(exact) (#2)', + ( symprov('libplain.so.1', plain_fn, 'Base', at(_)), + abi_verdict(UsePub, 'libpub.so.1', '1.0-1', compatible(exact)), + abi_verdict(UsePub, 'libplain.so.1', '1.0-1', compatible(exact)) )), + atom_concat(FX, '/store_pub_nolibpub', StPn), load_abi_store(StPn), + check('D5 [ELF] libpub evidence absent -> unknown([no_provider_evidence(libpub.so.1)]), not compatible (#2)', + abi_verdict(UsePub, 'libpub.so.1', '1.0-1', unknown([no_provider_evidence('libpub.so.1')]))), + atom_concat(FX, '/store_pub_noplain', StPp), load_abi_store(StPp), + check('D5b [ELF] unversioned plain_fn with libplain evidence absent -> unknown, not missing (#2)', + abi_verdict(UsePub, 'libpub.so.1', '1.0-1', unknown([unknown(plain_fn, no_provider_evidence('libplain.so.1'))]))), + + atom_concat(FX, '/store_missing', StM), load_abi_store(StM), + atom_concat(FX, '/does-not-exist', Missing), + check('D6 [ELF] missing ELF -> req_evidence missing_file recorded; verdict unknown (#2)', + ( req_evidence(Missing, readelf, missing_file, _), + \+ symreq(Missing, _, _, _, _), + abi_verdict(Missing, 'libc.so.6', '1.0', unknown([requires_evidence(missing_file, _)])) )). -report(Label, Val) :- format("~w = ~w~n", [Label, Val]). +symbols_fixture_checks(FX) :- + atom_concat(FX, '/simple', StS), load_abi_store(StS), + check('D7 [.symbols] simple cases: 2 sonames, |/* lines, comments, private 0, tilde + epoch minimums (#7)', + ( aggregate_all(count, symprov(_, _, _, _), 5), + symprov('libsimple.so.1', simple_old, 'SIMPLE_1.0', since(deb(1, _, _), '1:0.9-2')), + symprov('libsimple.so.1', simple_new, 'SIMPLE_1.2', since(_, '1.2~rc1')), + symprov('libsimple.so.1', '_private_thing', 'SIMPLE_PRIVATE', since(_, '0')), + symprov('libsimple-extra.so.0', extra_fn, 'Base', since(_, '1.1')), + prov_evidence('libsimple.so.1', symbols, _, complete), + prov_evidence('libsimple-extra.so.0', symbols, _, complete) )), + atom_concat(FX, '/tmpl_arch_amd64', StA), load_abi_store(StA), + check('D8 [.symbols] (arch=..) processed with --arch amd64: amd64 rows kept, !amd64 dropped, (optional) kept (#7)', + ( aggregate_all(count, symprov(_, _, _, _), 4), + symprov('libtmpl.so.1', only_amd64, 'Base', _), symprov('libtmpl.so.1', bits64, 'Base', _), + symprov('libtmpl.so.1', maybe_fn, 'Base', _), symprov('libtmpl.so.1', plain_fn, 'Base', _), + \+ symprov('libtmpl.so.1', not_amd64, _, _) )), + forall(member(T, [tmpl_symver, tmpl_cxx, tmpl_arch]), + ( atomic_list_concat([FX, '/', T, '/symprov.jsonl'], F), + format(atom(Name), "D9 [.symbols] ~w rejected loudly: no store written (#7)", [T]), + check(Name, \+ exists_file(F)) )). From 0e4253afb4a51eda999239c0bf53eff69f1f2770 Mon Sep 17 00:00:00 2001 From: "John William Creighton (s243a)" Date: Sun, 13 Sep 2026 00:22:50 -0600 Subject: [PATCH 03/10] abi lane: address Sol's re-review (multi-tier bounds, default binding, atomic ingest) Sol's REQUEST-CHANGES on the redesigned symbol-level ABI lane, all points: P1a Provider bounds are tied to their evidence row (since/4 carries the evidence release; at/2 = its own) and every evidence row is aggregated per identity (ident_status/5: at-release decides, nearest below/above combined) instead of `->` committing to the first symprov/4. since(2.0) + at(1.0) at release 1.0 is compatible(exact), not below_floor. C7*, A25, C7e (a bound without its evidence row is not evidence). P1b Ingest keeps the @ vs @@ distinction from the .gnu.version hidden bit (cross-checked against the name); rows carry default|nondefault| unproven. An unversioned reference is satisfied only by Base or a default export; .symbols-only rows yield unknown; a nondefault-only export is a hard no_default_export veto. The loader's oldest-node rule (verdef index 2 binds even when hidden) is encoded and verified with the loader. D10/D10b/D10c (gcc libhid fixtures), C8*, A7b/A7c. P1c (optional) rows are exports only when --elf cross-checks them against the binary; without --elf the file is rejected. D8 inverted: the optional row absent from the ELF is NOT stored. libc6's .symbols is now ingested with --elf libc.so.6 (0 dropped, 0 disagreements). P2a symbols-dir: a file with an unresolvable block is rejected atomically (exit 3, no rows, no evidence). D11 + fixtures/batch. P2b Tag whitelist (every other tag rejects the file); --release validated by the same Debian-version gate everywhere. D9 tmpl_unknown_tag, bad_release (symbols-file and elf). P2c crosscheck.mjs fails on empty inputs / zero denominators. P2d soname_mismatch only under a declared replaces(New, Old) row (replaces.jsonl, `ingest replaces`); no stem heuristic. C9*, A19b. P3 The dotted-version comparator over node names is gone; per-name comparison is node-set equality; fixtures/crosscheck pins the case. Res. compatible(curated) for .symbols-derived presence; exact only for readelf at that release. Also: absence from complete evidence propagates down (missing for Rel =< R1) but not up (unknown for later releases); an observed present-then-absent gives unknown(dropped_between). REVIEW_NOTES.md maps each point to the fix and its fixture; README/HOWTO updated. run_abi_verify.sh: 92 passed, 0 failed, 0 skipped. Frozen resolver.pl / resolver_store.pl / debian/ untouched. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01RoXjhStCqoig6944pVNBGe --- examples/pkg_resolver/abi/README.md | 136 +++++--- examples/pkg_resolver/abi/REVIEW_NOTES.md | 207 ++++++------ examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md | 90 ++++-- examples/pkg_resolver/abi/abi_cli.pl | 2 +- examples/pkg_resolver/abi/abi_resolve.pl | 304 +++++++++++++----- examples/pkg_resolver/abi/crosscheck.mjs | 69 ++-- .../abi/fixtures/batch/mixed.symbols | 7 + .../abi/fixtures/crosscheck/elf.symprov.jsonl | 5 + .../crosscheck/elf_dropnode.symprov.jsonl | 4 + .../abi/fixtures/crosscheck/sym.symprov.jsonl | 6 + examples/pkg_resolver/abi/fixtures/hid.c | 13 + .../pkg_resolver/abi/fixtures/hid_idx2.map | 1 + .../pkg_resolver/abi/fixtures/hid_idx3.map | 2 + .../pkg_resolver/abi/fixtures/hid_plain.c | 5 + .../abi/fixtures/tmpl_arch.symbols | 1 - .../abi/fixtures/tmpl_optional.symbols | 3 + .../abi/fixtures/tmpl_unknown_tag.symbols | 3 + examples/pkg_resolver/abi/fixtures/usehid.c | 4 + examples/pkg_resolver/abi/ingest_symbols.mjs | 216 +++++++++---- examples/pkg_resolver/abi/run_abi_verify.sh | 122 ++++++- examples/pkg_resolver/abi/test_abi.pl | 244 +++++++++++--- 21 files changed, 1050 insertions(+), 394 deletions(-) create mode 100644 examples/pkg_resolver/abi/fixtures/batch/mixed.symbols create mode 100644 examples/pkg_resolver/abi/fixtures/crosscheck/elf.symprov.jsonl create mode 100644 examples/pkg_resolver/abi/fixtures/crosscheck/elf_dropnode.symprov.jsonl create mode 100644 examples/pkg_resolver/abi/fixtures/crosscheck/sym.symprov.jsonl create mode 100644 examples/pkg_resolver/abi/fixtures/hid.c create mode 100644 examples/pkg_resolver/abi/fixtures/hid_idx2.map create mode 100644 examples/pkg_resolver/abi/fixtures/hid_idx3.map create mode 100644 examples/pkg_resolver/abi/fixtures/hid_plain.c create mode 100644 examples/pkg_resolver/abi/fixtures/tmpl_optional.symbols create mode 100644 examples/pkg_resolver/abi/fixtures/tmpl_unknown_tag.symbols create mode 100644 examples/pkg_resolver/abi/fixtures/usehid.c diff --git a/examples/pkg_resolver/abi/README.md b/examples/pkg_resolver/abi/README.md index 8164358aa..3919bb835 100644 --- a/examples/pkg_resolver/abi/README.md +++ b/examples/pkg_resolver/abi/README.md @@ -12,8 +12,9 @@ and `resolver_store.pl` are not edited; the Debian-version comparison is delegated to `resolver:version_lt/2` on `deb/3` terms from `debian/deb_parse.pl`. -The lane was redesigned after the PR #4262 review; `REVIEW_NOTES.md` maps each -review point to the code and the fixture that proves it. +The lane was redesigned after the PR #4262 review and revised after Sol's +re-review; `REVIEW_NOTES.md` maps each review point to the code and the +fixture that proves it. ## The model @@ -22,60 +23,84 @@ kept strictly apart: | Axis | Values | Ordering | Used for | |------|--------|----------|----------| -| ELF version node | `GLIBC_2.34`, `LIBSELINUX_1.0`, `COMMON_1`, `PUBLIC`, `Base` | **none** — opaque labels matched by string equality | *Does release R export `Sym@Node`?* A requirement `foo@LIB_1` is satisfied only by a provider row `foo@LIB_1` on the same soname. `foo@LIB_2` is not a match (the loader agrees: `undefined symbol: foo, version LIB_1`). `Base` is dpkg's spelling of "unversioned". | +| ELF version node | `GLIBC_2.34`, `LIBSELINUX_1.0`, `COMMON_1`, `PUBLIC`, `Base` | **none** — opaque labels matched by string equality | *Does release R export `Sym@Node`?* A requirement `foo@LIB_1` is satisfied only by a provider row `foo@LIB_1` on the same soname. `foo@LIB_2` is not a match (the loader agrees: `undefined symbol: foo, version LIB_1`). `Base` is dpkg's spelling of "unversioned". An **unversioned requirement** binds, as the loader does, only to a `Base` export or a **default** (`@@`) export — never to a hidden (`@`) one. | | Debian package version | `2.34`, `3.1~`, `1:2.3-1`, `2.35-0ubuntu3.15` | `resolver:version_lt/2` over `deb(Epoch, Upstream, Revision)` (epoch, `~`, revision, any number of components) | the `.symbols` minimum-version field (a curated **lower bound**, not an introduction date) and the **release candidates**. | -Evidence is explicit and tri-state: +Evidence is explicit, and every provider bound is tied to the evidence row it +came from: - `prov_evidence(So, symbols|elf, R0, complete)` — the soname's export set is completely known at evidence release `R0`. - `req_evidence(Bin, readelf, complete | missing_file | readelf_failed | inconsistent, Detail)`. -- Provider bounds: `since(Min)` from `.symbols` (exported at every release - `>= Min`, exact up to `R0`, *extrapolated* beyond it); `at(R0)` from readelf - (exact at `R0`, extrapolated beyond, **unknown** before). +- Provider bounds: `since(Min, MinAtom, R0, Bind)` from the `.symbols` evidence + taken at `R0` (exported at `R0` and, by the curated bound, at every release + `>= Min`); `at(R0, Bind)` from readelf at `R0`. `Bind` is the + default-version binding: `default` (an unversioned reference binds to it: + `@@`, `Base`, or the oldest version node which the loader also accepts), + `nondefault` (hidden `@`), or `unproven` (a `.symbols` row not cross-checked + against the ELF with `--elf`). + +Per identity and release, **all** evidence rows are aggregated: evidence +taken *at* the release decides (readelf before `.symbols`); otherwise the +nearest evidence below (presence extrapolates upward) and the nearest +evidence above (absence propagates downward; a curated floor covers +`Rel >= Min`) are combined. A release satisfied by any evidence row is never +vetoed by another row's floor. Verdict for `(Bin, So, Rel)`: ``` -compatible(exact | extrapolated) presence — defeasible "structurally possible" -incompatible([missing(Sym@Node) | below_floor(Sym@Node, Min) | soname_mismatch(...)]) +compatible(exact | curated | extrapolated) presence — defeasible "structurally possible" + exact every requirement observed by readelf at exactly Rel + curated some requirement rests on .symbols metadata only + extrapolated some requirement rests on the monotone-export assumption +incompatible([missing(Sym@Node) | missing(Sym@Node, observed_absent(Src, R1)) + | missing(Sym, no_default_export(So, Node)) | below_floor(Sym@Node, Min) + | soname_mismatch(offered(So), needed(N))]) HARD veto — only reachable with complete, attributed evidence unknown([no_requires_evidence | requires_evidence(Status,_) | no_provider_evidence(So) - | unknown(Sym@Node, evidence_release(R0)) | ...]) -not_needed(So) + | unknown(Sym@Node, evidence_release(R0) | absent_at(Src, R1) | dropped_between(R0, Src, R1)) + | unknown(Sym, default_binding_unproven(So, Node)) | ...]) +not_needed(So) not in DT_NEEDED and not a declared replacement ``` Documented defeasible assumption: within a soname, exports do not disappear (removing one is an ABI break that requires a soname bump), so presence at -`R0` extrapolates to later releases and absence from a complete export set is -a veto for the soname. `drop(Sym, Node, At)` models a violation of that -assumption to exercise the upper bound. +`R0` extrapolates to later releases and absence from a complete export set at +`R1` is a veto for every release `<= R1` (later releases may add symbols: +unknown). `drop(Sym, Node, At)` models a violation of that assumption to +exercise the upper bound; an *observed* violation (present at `R0`, absent +at a later `R1`) makes the releases in between `unknown(dropped_between(...))`. +`soname_mismatch` needs a declared `replaces(New, Old)` relation +(`replaces.jsonl`); there is no name-stem heuristic. ## Files | File | Role | |------|------| -| `ingest_symbols.mjs` | Ingest: `.symbols` (since bounds), `readelf` provides (`at` bounds), `readelf` requires (attributed via the ELF version **index**), NEEDED, evidence, release axis. Loud failures (exit 3), never an empty success. | -| `abi_resolve.pl` | Resolver: store loading, the two axes, per-requirement status, verdicts, floor, range over the real release axis. | +| `ingest_symbols.mjs` | Ingest: `.symbols` (since bounds, optionally cross-checked against the ELF with `--elf`), `readelf` provides (`at` bounds with default-version binding), `readelf` requires (attributed via the ELF version **index**), NEEDED, evidence, release axis, declared soname succession (`replaces`). Loud, atomic failures (exit 3), never an empty or partial success. | +| `abi_resolve.pl` | Resolver: store loading, the two axes, per-identity evidence aggregation, per-requirement status, verdicts, floor, range over the real release axis. | | `abi_cli.pl` | Driver: `verdict` / `status` / `floor` / `axis` / `range`. | -| `crosscheck.mjs` | readelf-vs-`.symbols` cross-check on exact identity (+ the corrected legacy per-name figure). | +| `crosscheck.mjs` | readelf-vs-`.symbols` cross-check on exact identity + per-name node sets (no node-name parsing or ordering); fails on empty input. | | `test_abi.pl` | Assertions: real data (A), version axes (B), model fixtures (C), gcc-built ELF fixtures + `.symbols` template fixtures (D). | -| `run_abi_verify.sh` | End-to-end: build the real store, cross-check, build/ingest fixtures, run the tests. | -| `fixtures/` | C sources + version scripts for the ELF fixtures; `.symbols` fixtures (simple cases, template rejects, arch selectors). | +| `run_abi_verify.sh` | End-to-end: build the real store, cross-check (+ negatives), build/ingest fixtures with the loader as ground truth, run the tests. | +| `fixtures/` | C sources + version scripts for the ELF fixtures (incl. the hidden-version `libhid`); `.symbols` fixtures (simple cases, template rejects, arch selectors, `(optional)`, unknown tag, batch atomicity); `crosscheck/` static regression pair. | Generated stores land in `./.out` (gitignored). ## Store shape (P/2 JSONL, `[key, value]`) ``` -symprov.jsonl ["|@", ["since", ""]] # .symbols tier - ["|@", ["at", ""]] # readelf tier +symprov.jsonl ["|@", ["since", "", "", ]] # .symbols tier + ["|@", ["at", "", ]] # readelf tier + = "default" | "nondefault" | "unproven" symreq.jsonl ["|@", ["", "GLOBAL"|"WEAK"]] ["|", ["", "GLOBAL"|"WEAK"]] # unversioned reference needed.jsonl ["", ""] evidence.jsonl ["provides|", ["symbols"|"elf", "", "complete", ""]] ["requires|", ["readelf", "complete"|"missing_file"|"readelf_failed"|"inconsistent", ""]] releases.jsonl ["", ""] # the candidate axis (actual releases) +replaces.jsonl ["", ""] # declared soname succession ``` ## The three provider tiers (cheapest first) @@ -87,14 +112,23 @@ releases.jsonl ["", ""] # the cand 2. **`.symbols` control member** (`ingest_symbols.mjs symbols-file`) — every exported `sym@node` with a curated minimum package version. Already on disk under `/var/lib/dpkg/info/*.symbols`; a few KB per package, no binary - download. Binary-form files are fully supported; source-template constructs - are processed where their semantics are known (`(optional)`, `(arch=...)` - with `--arch`) and **rejected loudly** otherwise (`(symver)`, `(regex)`, - quoted C++ patterns, `#include`). + download. Binary-form files are fully supported. Source-template tags are + **whitelisted**: `(arch=...)` / `(arch-bits=...)` / `(arch-endian=...)` + with `--arch`, `(ignore-blacklist)`, and `(optional)` **only with + `--elf `** (dpkg lets an optional symbol stay in the template after + it left the binary, so the row counts only if the ELF exports it); + every other tag, `(symver)`, `(regex)`, quoted C++ patterns, `#include`, + an unknown evidence release or a non-Debian `--release` **rejects the whole + file** (exit 3, nothing partial, also in batch mode). With `--elf` the + file is cross-checked against the ELF and each row's default-version + binding is taken from it; without it rows are `unproven` for unversioned + references. 3. **`readelf`** (`ingest_symbols.mjs elf` / `requires`) — any ELF. Provides - are exact for the file's own release; requires are joined to their soname - through `.gnu.version` -> `.gnu.version_r` by index, so two libraries using - the same node name never collide. + are exact for the file's own release and carry the default-version binding + (`.gnu.version` hidden bit, cross-checked against the `@`/`@@` spelling); + requires are joined to their soname through `.gnu.version` -> + `.gnu.version_r` by index, so two libraries using the same node name never + collide. ## Verify on this machine @@ -104,32 +138,46 @@ releases.jsonl ["", ""] # the cand Real-data results (Ubuntu 22.04.5, libc6 2.35-0ubuntu3.15, libselinux1 3.3-1build2, `/bin/ls`): -- Ingest: libc6 `.symbols` -> **4827** `symprov` rows over 20 sonames - (**3006** for `libc.so.6`); libselinux1 -> 238; `/bin/ls` -> **112 versioned - + 3 unversioned (weak)** requirements, `NEEDED = [libselinux.so.1, libc.so.6]`. +- Ingest: libc6 `.symbols` cross-checked against `libc.so.6` with `--elf` + -> **4827** `symprov` rows over 20 sonames (**3006** for `libc.so.6`, 0 + optional rows dropped, 0 disagreements, **115** hidden `@` exports recorded + `nondefault`, `memcpy@GLIBC_2.2.5` at verdef index 2 recorded `default`); + libselinux1 -> 238; `/bin/ls` -> **112 versioned + 3 unversioned (weak)** + requirements, `NEEDED = [libselinux.so.1, libc.so.6]`. - Cross-check `readelf(libc.so.6)` vs `.symbols`: **3006/3006 exact - `sym@node` identities agree (100%)**. The corrected legacy per-name - comparison (earliest row on *both* sides) is **2478/2478 (100%)**; the - earlier 91.7% was an aggregation bug (last curated row vs earliest ELF row), - not a glibc 2.34 merge effect. + `sym@node` identities agree (100%)**; per-name node sets **2763/2763 + (100%)**. The earlier 91.7% was an aggregation bug (last curated row vs + earliest ELF row), not a glibc 2.34 merge effect; `fixtures/crosscheck/` + pins that case. Empty inputs fail the cross-check. - `floor /bin/ls libc.so.6` = **2.34**; `floor /bin/ls libselinux.so.1` = **3.1~** — equal to coreutils' declared Pre-Depends. - Release axis (from `apt-cache madison` + dpkg): `[2.35-0ubuntu3, 2.35-0ubuntu3.15]`; `range /bin/ls libc.so.6` = **[2.35-0ubuntu3, - 2.35-0ubuntu3.15]**, both ends `compatible(exact)`. + 2.35-0ubuntu3.15]**, both ends `compatible(curated)` (the evidence is + `.symbols` metadata; `exact` needs readelf at that release). - Hypothetical older release `2.31-0ubuntu9.9` -> `incompatible([below_floor(__libc_start_main@GLIBC_2.34, 2.34), below_floor(lstat@GLIBC_2.33, 2.33)])`; an axis extended below the floor yields min `2.34-0ubuntu3`, never the - lowest release. + lowest release. Adding readelf evidence for that release turns the verdict + into `compatible(exact)` — a curated floor never vetoes a release another + evidence row satisfies. - Hypothetical removal of `getenv@GLIBC_2.2.5` at `2.35-0ubuntu3.15` caps the range at `[2.35-0ubuntu3, 2.35-0ubuntu3]`; removing `__libc_start_main@GLIBC_2.34` at the oldest release -> `no_candidate`. -- Offering `libc.so.7` -> `incompatible([soname_mismatch(...)])`; a binary with - no requirement evidence -> `unknown([no_requires_evidence(...)])`. +- Offering `libc.so.7` -> `incompatible([soname_mismatch(...)])` because the + store declares `replaces(libc.so.7, libc.so.6)`; `libselinux.so.10` (same + stem, no declaration) -> `not_needed`; a binary with no requirement + evidence -> `unknown([no_requires_evidence(...)])`. - ELF fixtures (gcc): `foo@LIB_1` required vs `foo@LIB_2` exported under the same soname -> `incompatible([missing(foo@LIB_1)])`, and the loader agrees - (`undefined symbol: foo, version LIB_1`); `COMMON_1` in two libraries - attributed correctly; `pub_fn@PUBLIC` + unversioned `plain_fn` preserved; - missing ELF -> `unknown([requires_evidence(missing_file, _)])`. - -`test_abi.pl`: 63 checks, all passing. + (`undefined symbol: foo, version LIB_1`); `hid_fn` exported only at a + hidden `@HID_1` (verdef index 3) does **not** satisfy an unversioned + requirement -> `incompatible([missing(hid_fn, no_default_export(...))])`, + and the loader agrees (`undefined symbol: hid_fn`), while the same hidden + export at verdef index 2 binds (loader and lane agree); `COMMON_1` in two + libraries attributed correctly; `pub_fn@PUBLIC` + unversioned `plain_fn` + preserved; missing ELF -> `unknown([requires_evidence(missing_file, _)])`; + `(optional)` row absent from the ELF dropped, present rows kept. + +`test_abi.pl`: 92 checks, all passing; `run_abi_verify.sh` additionally +asserts the exit-3 rejections, batch atomicity and the cross-check negatives. diff --git a/examples/pkg_resolver/abi/REVIEW_NOTES.md b/examples/pkg_resolver/abi/REVIEW_NOTES.md index bd1beb9a9..cae76ac62 100644 --- a/examples/pkg_resolver/abi/REVIEW_NOTES.md +++ b/examples/pkg_resolver/abi/REVIEW_NOTES.md @@ -10,121 +10,111 @@ the Debian package-version axis with the ELF node axis, computed an unsatisfiable lower bound, mis-aggregated the cross-check (91.7%), and silently mis-ingested `.symbols` templates. The model and ingest were redesigned; the direction (three evidence tiers, a driver above the frozen -resolver) is unchanged. Frozen `resolver.pl` / `resolver_store.pl` / -`debian/` are untouched (`git diff origin/main -- examples/pkg_resolver/{resolver.pl,resolver_store.pl,debian}` is empty). +resolver) is unchanged. Sol's re-review of that redesign (REQUEST-CHANGES, +eight points) is addressed in the second section below. Frozen `resolver.pl` +/ `resolver_store.pl` / `debian/` are untouched (`git diff origin/main -- +examples/pkg_resolver/{resolver.pl,resolver_store.pl,debian}` is empty). Run `./run_abi_verify.sh` (needs node, readelf, swipl, gcc). Expected tail: -`== 63 passed, 0 failed, 0 skipped ==`. Check names in `test_abi.pl` carry the -review point they prove, e.g. `(#3)`. - -## Sol's checklist +`== 92 passed, 0 failed, 0 skipped ==`. Check names in `test_abi.pl` carry the +review point they prove: `(#n)` for Astra's points, `(sol-...)` for Sol's. +The shell script's own assertions (exit codes, files not written, expected +cross-check failures) abort the run with `FAIL:` before the Prolog section. + +## Sol re-review — fixes + +Each row: Sol's point -> what changed -> the fixture that fails if the fix is +reverted. Sol judged the extrapolated/monotone-export disclosure adequate; it +is unchanged. + +| Sol point | Fix (location) | Proving fixture(s) | +|-----------|----------------|--------------------| +| **P1a** `->` committed to the first `symprov/4`; `since(2.0)` + `at(1.0)` vetoed release 1.0 as `below_floor` | `abi_resolve.pl`: every bound now carries its evidence release (`since(Min, MinAtom, R0, Bind)`, `at(R0, Bind)`) and is usable only through its own complete `prov_evidence` row (`bound_evidence/4`). `ident_status/5` collects what EVERY evidence row says about the identity (`ev_says/5`) and combines them: evidence AT the release decides (readelf before `.symbols`); otherwise the nearest evidence BELOW (presence extrapolates upward) and ABOVE (absence propagates downward under monotone exports; a curated floor covers `Rel >= Min`) are combined in `combine/4`. A release satisfied by any row is never vetoed by another row's floor. Ingest (`ingest_symbols.mjs`) writes the evidence release into every `since` row. | **C7** (`since(2.0)` asserted first, `at(1.0)` second; release 1.0 -> `compatible(exact)`), C7b (1.5 extrapolated, 2.5/3.0 curated, 4.0 extrapolated), C7c (0.5 -> unknown, not a veto), C7d (range min is the observed release), C7e (a bound without its evidence row does not count), C7f (absence propagates down, not up), C7g (present-then-absent -> `unknown(dropped_between)`), **A25** (real `/bin/ls`: ELF evidence at 2.31-0ubuntu9.9 turns the below_floor veto into `compatible(exact)`), D7 (rows tied to the evidence release). | +| **P1b** `@` (non-default) vs `@@` (default) discarded; a non-default-only export satisfied an unversioned reference | Ingest `elfProvides()` records a binding per row from the `.gnu.version` hidden bit, cross-checked against the `@`/`@@` spelling (`inconsistent` -> exit 3): `default` = non-hidden **or verdef index 2** (the loader binds legacy unversioned references to the oldest node even when hidden — verified with the loader, see D10c), `nondefault` = hidden at index >= 3. `.symbols` rows are `unproven` unless cross-checked with `--elf` (then the ELF's binding is copied); `Base` always binds. Store: `["at", rel, binding]`, `["since", minver, rel, binding]`. Resolver `unversioned_status/7`: satisfied only by a `Base` or `default` export (`unversioned_in/5`); an `unproven` row yields `unknown(Sym, default_binding_unproven(So, Node))`; a `nondefault`-only export yields the hard `missing(Sym, no_default_export(So, Node))`. | **D10** (gcc `libhid.so.1` exporting `hid_fn` only at hidden `@HID_1`, verdef index 3: unversioned require -> `incompatible([missing(hid_fn, no_default_export(...))])`; `run_abi_verify.sh` runs the loader: `undefined symbol: hid_fn`), D10b (`@Base` -> compatible), **D10c** (hidden at index 2 -> recorded `default`, compatible; loader binds), **C8** (model: nondefault-only -> hard veto), C8b (adding a default node -> compatible), **C8c** (`.symbols`-only -> `unknown(default_binding_unproven)`), C8d (`Base` row -> compatible(curated)), C8e/C8f (versioned requirements unaffected), A7b/A7c (real libc6: every `libc.so.6` row ELF-proven; `memcpy@GLIBC_2.2.5` index 2 -> default, 115 hidden rows -> nondefault), D4b. | +| **P1c** `(optional)` template rows emitted as complete exports (D8 asserted the unsafe behaviour) | Ingest: an `(optional)` row is not an export fact by itself. With `--elf ` the file is cross-checked against the ELF: an optional row the ELF does not export is **dropped** (stderr notice), a non-optional row the ELF does not export or an ELF export absent from the file **rejects** the file; without `--elf` any `(optional)` row rejects the file (exit 3). Binary-form `.symbols` under `/var/lib/dpkg/info` carry no tags, so the real path is unaffected; `run_abi_verify.sh` now ingests libc6's `.symbols` with `--elf libc.so.6` (0 dropped, 0 disagreements). | **D8 inverted** (`tmpl_optional.symbols` + `--elf libtmpl.so.1` which lacks `maybe_fn`: only `plain_fn` stored, evidence complete, `\+ symprov(maybe_fn)`), **D8b** (no `--elf` -> file rejected, no store), D8d (non-optional row absent from the ELF -> rejected), `run_abi_verify.sh` asserts the exit-3 and the dropped-row notice. `tmpl_arch.symbols` no longer carries the optional row (D8c keeps the arch checks: 3 rows). | +| **P2a** batch mode skipped a block with an unknown evidence release but counted the file as success and exited 0 | `cmdSymbolsFile()` collects every problem of every block and returns `null` (nothing added to the sink) if any exists — single-file and batch alike; `symbols-dir` counts it as rejected and exits 3. | **D11** + `run_abi_verify.sh` (`fixtures/batch/mixed.symbols`: block 1 resolvable via installed `libc6`, block 2 `#PACKAGE#`: exit 3, stderr `files=0 rejected=1 symprov=0`, `symprov.jsonl` and `evidence.jsonl` empty — no `good_fn` row). | +| **P2b** only a fixed tag list rejected; unknown tags silently stripped; `--release` unvalidated | `SUPPORTED_TAGS` whitelist (`optional`, `arch`, `arch-bits`, `arch-endian`, `ignore-blacklist`); every other tag rejects the file. `validRelease()` is the single Debian-version gate: `--release` is validated up front for every command, dpkg-guessed releases and the `releases` axis pass through the same regex. | **D9 tmpl_unknown_tag** (`(frobnicate)` -> exit 3, no store), **D9 bad_release** (`--release definitely-not-a-debian-version` -> exit 3, no store; also asserted for the `elf` path), `run_abi_verify.sh` `expect_reject` for both. | +| **P2c** `crosscheck.mjs` passed `/dev/null` vs `/dev/null` (`NaN < 100` is false) | Both symbol sets and the per-name denominator must be nonzero; `fail()` exits 1 otherwise; the pass conditions are `=== 100`, never `< 100`. | `run_abi_verify.sh` (sol-P2c): `/dev/null` vs `/dev/null` and `sym` vs `/dev/null` must fail with `FAIL: empty symbol set`. | +| **P2d** `soname_mismatch` from a stem heuristic (`libfoo.so.2` vs unrelated NEEDED `libfoo.so.1-extra`) | `so_stem/2` removed. `soname_offer/3` reports `mismatch(N)` only under a declared `replaces(Offered, N)` with `needed(Bin, N)`; anything else not NEEDED is `not_needed`. New store file `replaces.jsonl` (`ingest_symbols.mjs replaces ...`); `run_abi_verify.sh` declares `libc.so.7 replaces libc.so.6` for the real store. | **C9** (offer `libfoo.so.2`, NEEDED `libfoo.so.1-extra` -> `not_needed`), C9b (still not_needed with `libfoo.so.1` NEEDED but no relation), C9c (declared relation -> `soname_mismatch`), **A19b** (real: `libselinux.so.10` -> `not_needed`), A19 (real: `libc.so.7` mismatch only because declared). | +| **P3** local dotted-version comparator over node names in the legacy per-name figure | `nodeNum()`/`cmpDotted()`/`earliest()` removed. The per-name figure is now "per-name node-SET agreement" (sets compared, nothing parsed or ordered). Real data: 3006/3006 exact, 2763/2763 per-name. | `fixtures/crosscheck/` (sol-P3): a two-node symbol, a `GLIBC_PRIVATE` node, a `Base` node, another soname to be ignored -> 5/5 and 4/4; `elf_dropnode` must fail on both checks; asserted in `run_abi_verify.sh`. | +| **Residual** `.symbols`-derived verdicts labelled `compatible(exact)` | New basis `curated`: presence resting on `.symbols` metadata (at or below its evidence release, `Min =< Rel`) is `provided(_, curated)`; `exact` is reserved for readelf at exactly the release. The verdict basis is the weakest among the requirements (`exact < curated < extrapolated`). | A12 (real: `compatible(curated)` at the `.symbols` evidence release), C1b, C7b, C8d; `exact` still on ELF fixtures (D2, D3b, D4b, D10b/c, C7). | + +Changes that fell out of the P1a restructuring (documented, not requested): + +- **Absence propagates down, not up.** Before, a missing provider row was a + veto for *every* release of the soname; now absence from complete evidence + at `R1` is `missing` for `Rel =< R1` (reported as `missing(Sym@Node)` at + `R1` itself and `missing(Sym@Node, observed_absent(Src, R1))` when + inferred) and `unknown(absent_at(Src, R1))` for `Rel > R1` — later releases + add symbols. C7f, A17. +- **Observed drop.** Present at `R0`, absent at a later `R1`: a release in + between is `unknown(dropped_between(R0, Src, R1))`, neither extrapolated + nor vetoed. C7g. +- `abi_floor/3` takes the highest minimum over all `since` rows of an + identity (several `.symbols` evidence releases may coexist). + +## Sol's checklist (first review) ### (a) Exact version-node identity end-to-end — Astra #1 - **Ingest** (`ingest_symbols.mjs`): `.symbols` rows are split at the last `@` into `(sym, node)` and stored as `"|@"` with the minimum - version *verbatim* (`["since", ""]`). readelf provides carry the - node from `.gnu.version_d` by index (`["at", ""]`). There is no - `verNum()` anymore; nothing is parsed out of a node name. + version *verbatim* (`["since", "", "", ]`). + readelf provides carry the node from `.gnu.version_d` by index + (`["at", "", ]`). Nothing is parsed out of a node name. - **Store** (`abi_resolve.pl`): `symprov(So, Sym, Node, Bound)`, `symreq(Bin, Sym, Node, So, Bind)`. Loading rejects an empty node. -- **Match** (`versioned_status/7`): `symprov(So, Sym, Node, _)` — same - soname, same symbol, same node, by unification. No fallback to `symprov(So, - Sym, _, _)` for versioned requirements. -- **Fixtures**: C1/C1b (in-memory: `foo@LIB_1` vs `foo@LIB_2` -> - `incompatible([missing(foo@LIB_1)])`, then exact row -> `compatible(exact)`); - **D1** (gcc-built `libfoo.so.1` v1/v2, real ingest; the loader is run as - ground truth and must print `undefined symbol: foo, version LIB_1`); - A17/A18 on real libc (`getenv@GLIBC_2.99` -> missing; - `pthread_setname_np@GLIBC_2.12` (real non-default node) -> provided, - `@GLIBC_2.13` -> missing); A7 (no node-less provider rows); B8 (node - labels are never ordered numerically). +- **Match** (`versioned_status/7` -> `ident_status/5`): same soname, same + symbol, same node, by unification. No fallback to `symprov(So, Sym, _, _)` + for versioned requirements. +- **Fixtures**: C1/C1b, **D1** (gcc-built `libfoo.so.1` v1/v2; the loader + must print `undefined symbol: foo, version LIB_1`), A17/A18 on real libc, + A7, B8. ### (b) verneed attribution via version INDEX — Astra #3 -- `elfTables()` parses `readelf -W -V`: the `.gnu.version` array (per-dynsym - index, hidden bit `h`), `.gnu.version_d` (index -> name) and - `.gnu.version_r` (index -> `{file, name, weak}`). `elfRequires()` joins each - UND symbol by its **index** to `(file, node)`; the `name@VER` text is only - used as a consistency assertion (mismatch -> `inconsistent` evidence, exit 3). -- Also enforced: the verneed file must be in DT_NEEDED; a symbol whose index - has no verneed entry is an ingest error, not a `?` soname. -- **Fixture D3**: `libalpha.so.1` and `libbeta.so.1` both define `COMMON_1`; - `usecommon` needs one symbol from each. Asserted: `alpha_fn@COMMON_1 -> - libalpha.so.1`, `beta_fn@COMMON_1 -> libbeta.so.1`, and the negatives. D3b: - both verdicts `compatible(exact)`. A4/A5 on `/bin/ls`: `__libc_start_main@GLIBC_2.34 - -> libc.so.6`, `freecon@LIBSELINUX_1.0 -> libselinux.so.1`. +- `elfTables()` parses `readelf -W -V`: `.gnu.version` (per-dynsym index, + hidden bit `h`), `.gnu.version_d` (index -> name) and `.gnu.version_r` + (index -> `{file, name, weak}`). `elfRequires()` joins each UND symbol by + its **index** to `(file, node)`; the `name@VER` text is only a consistency + assertion (mismatch -> `inconsistent` evidence, exit 3). +- **Fixture D3**: `libalpha.so.1` / `libbeta.so.1` both define `COMMON_1`; + attribution asserted both ways. A4/A5 on `/bin/ls`. ### (c) Two axes separate; deb parsing reused — Astra #4 - Package versions are parsed once at load by `debian/deb_parse:parse_deb_version/2` - (`rel_term/2`, `assert_symprov/2`) and ordered by the frozen - `resolver:version_lt/2` on `deb/3` (`rel_lt/2`, `rel_le/2`). No local - version arithmetic exists in the lane. `since(Deb, Atom)` keeps the original - atom so `3.1~` is reported as `3.1~`, not re-formatted. -- ELF nodes never enter `rel_term/2` as versions: a node string that is not a - Debian version becomes `label(Atom)` and only matches itself (B7/B8). -- **Fixtures**: B1–B6 (`3.1~ < 3.1`, `3.1~ < 3.1~rc1`, `1:2.3-1 > 3.1`, - `2.35 < 2.35-0ubuntu3 < 2.35-0ubuntu3.15`, `2.2.5 < 2.2.5.1 < 2.2.6`, `0` - lowest); D7 (`.symbols` fixture with `1:0.9-2` and `1.2~rc1` minimums loaded - as `deb(1, ...)` / preserved atoms); **A8–A10** on real data: the computed - floors `2.34` and `3.1~` equal coreutils' declared - `libc6 (>= 2.34), libselinux1 (>= 3.1~)` read from dpkg. -- Documented (`README.md`, `SYMBOL_ABI_HOWTO.md`, header comment of - `abi_resolve.pl`): `.symbols` minimum versions are curated lower bounds, - raisable by policy, not introduction dates; a release below one is reported - as `below_floor`, distinct from `missing`. + and ordered by the frozen `resolver:version_lt/2` on `deb/3`. No local + version arithmetic exists in the lane (the last one, in `crosscheck.mjs`, + went with Sol P3). +- **Fixtures**: B1–B8, D7, **A8–A10** (computed floors `2.34` / `3.1~` equal + coreutils' declared dependency read from dpkg). ### (d) Incomplete evidence never becomes a false veto or false compat — Astra #2 -- `evidence.jsonl` rows are emitted by every successful ingest; a missing or - unreadable ELF makes `requires` exit 3 **and** record - `["requires|", ["readelf", "missing_file", ...]]` (no empty success); - `elf` on a missing file exits 3 with nothing written. -- `abi_verdict/5` short-circuits to `unknown(...)` when: no `req_evidence`, - `req_evidence` not `complete`, or no complete `prov_evidence` for the - soname; `at(R0)` evidence yields `unknown(Sym@Node, evidence_release(R0))` - for older releases; an unversioned obligation is `unknown` while any NEEDED - object lacks evidence. `needed/2` is checked (`not_needed`, and - `soname_mismatch` is a hard veto when the offered soname differs from the - NEEDED one with the same stem). -- Non-numeric nodes and unversioned references are kept (`pub_fn@PUBLIC`, - `plain_fn`), weak references are flagged `WEAK` and classified - `weak_unresolved` (never a veto). -- **Fixtures**: C3/C3b (at-evidence: older -> unknown), C4–C4e (no provider - evidence -> unknown; `missing_file` -> unknown; unversioned with one lib - unevidenced -> unknown, with all evidenced and nobody exporting -> - `missing`, exported by another NEEDED lib -> compatible), C5/C5b, C6; - **D4–D6** through the real pipeline (`usepub` / `libpub` / `libplain`; - `store_pub_nolibpub`, `store_pub_noplain`, `store_missing`); A1–A3, A6, - A19–A22 on real data. +- Every successful ingest emits an `evidence` row; a missing/unreadable ELF + makes `requires` exit 3 **and** record an incomplete evidence row. +- `abi_verdict/5` short-circuits to `unknown(...)` without complete + requirement evidence or complete provider evidence; `at(R0)` evidence + yields `unknown(Sym@Node, evidence_release(R0))` for older releases; an + unversioned obligation is `unknown` while any NEEDED object lacks evidence + or while its only providers are `unproven`. +- **Fixtures**: C3/C3b, C4–C4e, C5/C5b, C6, C8c, **D4–D6**, A1–A3, A6, + A20–A22. ### (e) `[min, max]` satisfiable at both ends — Astra #5 -- The candidate axis is `releases.jsonl` (actual releases; on this machine - from `apt-cache madison libc6` + dpkg), never symbol-intro points. - `abi_range/5` evaluates **every** release and takes min/max from the - releases whose verdict is `compatible(_)` (`range_min_max/3`), so both ends - are compatible by construction; releases that add no symbols are still - evaluated (C2d). -- **Fixtures**: **C2** — releases 1..5, `a since 1`, `b since 5` -> - `range(5.0, 5.0)`, verdict at 1.0 `incompatible([below_floor(b@L, 5.0)])` - (the old code returned `range(1,5)`); C2c (6.0 -> `compatible(extrapolated)`, - `range(5.0, 6.0)`); C3b (unknown releases are reported, not counted); A13 - (real axis, both ends compatible); **A15** (axis extended below the floor: - min is `2.34-0ubuntu3`, not `2.31-0ubuntu9.9`); A23/A24 (hypothetical - removal caps the max / squeezes to `no_candidate`); D2 (`range(1.0-1, 1.0-1)`). +- `abi_range/5` evaluates **every** release of the actual axis and takes + min/max from the compatible ones (`range_min_max/3`). +- **Fixtures**: **C2** (`range(5.0, 5.0)`, not `range(1,5)`), C2c/C2d, C3b, + C7d, A13, **A15**, A23/A24, D2. ### (f) Cross-check aggregation fixed; false 2.34-merge claim removed — Astra #6 -- `crosscheck.mjs` compares the two tiers on **exact `sym@node` sets**: - `.symbols=3006 readelf=3006 shared=3006`, 100%. It also reproduces the - legacy per-name figure with earliest-row aggregation on *both* sides: - 2478/2478 (100%) (the review's 2443/2443 used the old script's symbol - filter; either way it is 100%). Both must be 100% or the script fails. -- `README.md` and `SYMBOL_ABI_HOWTO.md` no longer claim a glibc 2.34 - pthread/rt-merge divergence; they state the 91.7% was the last-row-vs- - earliest-row bug. +- `crosscheck.mjs`: exact `sym@node` sets 3006/3006 and per-name node sets + 2763/2763 on Ubuntu 22.04 libc6; `fixtures/crosscheck/` pins the two-node + case that produced the old 91.7%. Docs no longer claim a glibc 2.34 merge + divergence. ### (g) Frozen files untouched @@ -135,27 +125,26 @@ empty. The lane only *imports* `resolver:version_lt/2` and ### Astra #7 — `.symbols` templates handled or rejected loudly -- `parseSymbolsFile()` processes `(optional)` (kept), `(ignore-blacklist)` - (kept), `(arch=…)`, `(arch-bits=…)`, `(arch-endian=…)` (row kept iff it - selects `--arch`; **rejected** without `--arch`), and rejects `(symver)`, - `(regex)`, `(c++…)`, quoted pattern rows, `#include`, malformed rows and - non-Debian minimum versions — the whole file is refused (exit 3) with line - numbers; nothing partial is written. `#PACKAGE#` headers are accepted but - then `--release` is mandatory (the evidence release cannot be looked up). -- The simple cases keep working: multiple sonames, `|` alt-dep lines, `*` - meta fields, `#` comments, `#MINVER#`, private minimum `0`, dep-id column. -- **Fixtures**: D7 (`fixtures/simple.symbols`), D8 - (`tmpl_arch.symbols` with `--arch amd64`: `!amd64` row dropped, `(optional)` - kept), D9 x3 (`tmpl_symver`, `tmpl_cxx`, `tmpl_arch` without `--arch` -> - no store written; `run_abi_verify.sh` also asserts the non-zero exit). +- Whitelisted tags are processed (`(arch=…)`, `(arch-bits=…)`, + `(arch-endian=…)` with `--arch`; `(ignore-blacklist)`; `(optional)` only + with `--elf`), everything else rejects the whole file with line numbers; + nothing partial is written, also in batch mode. `#PACKAGE#` headers make + `--release` mandatory. +- **Fixtures**: D7, D8/D8b/D8c/D8d, D9 x5, D11. ## Things worth a second look -- `unversioned_status/7` treats an unversioned reference as satisfied by *any* - exported node of the symbol in *any* NEEDED object (the loader's default- - version rule is not modelled). Documented in the HOWTO "Limits". -- `bound_holds(since(Min,_), R0, Rel, extrapolated)` for `Rel > R0` and - `at(R0)` for `Rel > R0` both rely on the in-soname monotone-export - assumption; the `Basis` value makes that visible to callers. +- `default` for a hidden export at verdef index 2 encodes glibc's legacy + rule for unversioned references (`dl-lookup.c`: index < 3 is accepted + before the hidden test; D10c and `run_abi_verify.sh` verify it with the + loader). `dlsym()` lookups (`DL_LOOKUP_RETURN_NEWEST`) do not use that rule; + the lane models link-time/`DT_NEEDED` references only. +- With `--elf`, the ELF is used to prove bindings and to cross-check the + `.symbols` rows, but no `at()` rows are emitted from it, so the verdict + basis stays `curated`; ingest the ELF with `elf` (as `run_abi_verify.sh` + does for the fixture stores) to get `exact`. +- `unknown(dropped_between(R0, Src, R1))` and `unknown(absent_at(Src, R1))` + are new reasons callers may want to distinguish from the evidence-gap + reasons. - `abi_floor/3` fails (rather than guessing) when a requirement has no `since()` row — readelf-only provider evidence has no package floor. diff --git a/examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md b/examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md index 452510c08..3bc699398 100644 --- a/examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md +++ b/examples/pkg_resolver/abi/SYMBOL_ABI_HOWTO.md @@ -16,8 +16,9 @@ so we can compute the real `[min, max]` compatible release range for a binary. plus any unversioned references. - **Compatible(bin, release)** ⟺ every NEEDED soname is offered under its exact name AND every required `sym@node` is exported by that soname at that - release (unversioned references: exported under any node by any NEEDED - object). + release (unversioned references: exported as `Base` or as the **default** + version — `@@`, or the oldest version node — by some NEEDED object; a + hidden `@` export does not count, exactly as in the loader). Two axes, never mixed: @@ -52,11 +53,21 @@ Two axes, never mixed: The minimum version is a **lower bound** a dependent must declare, *not* a ground-truth introduction date: Debian policy lets a maintainer raise it after a compatible behaviour change. The lane therefore stores it as - `since(Min)` and reports a release below it as `below_floor` (the same + `since(Min, ..., R0, Bind)` tied to the evidence release `R0` the file was + taken at, and reports a release below it as `below_floor` (the same conservative floor `dpkg-shlibdeps` emits), never as proof the symbol was - absent. + absent — and never when another evidence row (e.g. readelf at that + release) shows the symbol present. A `.symbols` file says nothing about + `@` vs `@@`, so its rows are `unproven` for unversioned references unless + the file is cross-checked against the library with `--elf lib.so` + (`run_abi_verify.sh` does this for libc6: 3006 rows, 0 disagreements). + Source-template tags are whitelisted (`(arch…)` with `--arch`, + `(ignore-blacklist)`, `(optional)` only with `--elf`); anything else, or a + block whose evidence release is unknown, rejects the whole file (exit 3), + also in `symbols-dir` batch mode. 3. **`readelf`** on the ELF itself — exact for that file's release - (`at(R0)`), unknown for older releases, extrapolated for newer ones. + (`at(R0, Bind)`, with the default-version binding from `.gnu.version`), + unknown for older releases, extrapolated for newer ones. ## Worked examples (real output, Ubuntu 22.04.5, after `./run_abi_verify.sh`) @@ -92,8 +103,8 @@ $ ... axis libc.so.6 axis libc.so.6: [2.35-0ubuntu3,2.35-0ubuntu3.15] # apt-cache madison + dpkg $ ... range /bin/ls libc.so.6 range /bin/ls libc.so.6: [2.35-0ubuntu3, 2.35-0ubuntu3.15] - 2.35-0ubuntu3: compatible(exact) - 2.35-0ubuntu3.15: compatible(exact) + 2.35-0ubuntu3: compatible(curated) + 2.35-0ubuntu3.15: compatible(curated) $ ... verdict /bin/ls libc.so.6 2.31-0ubuntu9.9 verdict /bin/ls libc.so.6 2.31-0ubuntu9.9: incompatible([below_floor('__libc_start_main'@'GLIBC_2.34','2.34'),below_floor(lstat@'GLIBC_2.33','2.33')]) ``` @@ -101,6 +112,12 @@ The candidates are releases, not symbol-introduction points; `range/3` reports a `[min, max]` whose ends both carry `compatible` verdicts. With an axis extended below the floor (`2.31-0ubuntu9.9, 2.34-0ubuntu3, ...`) the min is `2.34-0ubuntu3` — the first release that actually satisfies the requirements. +The basis is `curated` because the only evidence is the `.symbols` file; +`exact` is reserved for readelf observations at that very release. Bounds +from several evidence rows are aggregated, never taken first-match: if +readelf evidence for `2.31-0ubuntu9.9` were ingested and showed every +required identity, that release would be `compatible(exact)` despite the +curated floor of 2.34 (test A25 does exactly that). ### 4. Exact node identity (the case the old model got wrong) @@ -143,13 +160,36 @@ Likewise a NEEDED soname with no provider evidence yields ``` exact sym@node identity: .symbols=3006 readelf=3006 shared=3006 only-readelf=0 only-.symbols=0 identity agreement: 3006/3006 (100.0%) - legacy per-name earliest-row comparison (corrected aggregation): 2478/2478 (100.0%) + per-name node-set agreement: 2763/2763 (100.0%) ``` The earlier "91.7% agreement, explained by the glibc 2.34 pthread/rt merge" was an aggregation bug: the last curated row of a symbol was compared with the earliest ELF node of that symbol, so any symbol with two nodes (`pthread_setname_np@GLIBC_2.12` + `@GLIBC_2.34`) "disagreed". Compared -consistently, everything agrees; the merge explanation was false. +consistently, everything agrees; the merge explanation was false. The +per-name figure now compares the *set* of nodes per name — nothing is parsed +out of a node name and nothing is ordered — and `fixtures/crosscheck/` pins +the two-node case (plus a `GLIBC_PRIVATE` node and a `Base` node) as a static +regression. Two empty inputs fail the check instead of passing on `NaN`. + +### 8. An unversioned reference vs a non-default-only export + +Fixture `libhid.so.1`: `hid_fn` is exported only as hidden `hid_fn@HID_1` +(`.symver` with a single `@`); `usehid` was linked against an unversioned +build and so references `hid_fn` without a version. +``` +$ LD_LIBRARY_PATH=.out/fx/hid_idx3 .out/fx/usehid +usehid: symbol lookup error: usehid: undefined symbol: hid_fn +$ grep hid_fn .out/fx/store_hid_idx3/symprov.jsonl +["libhid.so.1|hid_fn@HID_1",["at","1.0-1","nondefault"]] +$ ... .out/fx/store_hid_idx3 status .out/fx/usehid libhid.so.1 1.0-1 + missing(hid_fn,no_default_export('libhid.so.1','HID_1')) +``` +The same hidden export at verdef index 2 (the oldest node, `hid_idx2`) is +bound by the loader for legacy unversioned references, runs, and is recorded +`default`. A `.symbols`-only store cannot tell `@` from `@@`, so there the +answer is `unknown([unknown(hid_fn, default_binding_unproven(...))])`, never +`compatible`. ## Epistemics @@ -157,25 +197,37 @@ consistently, everything agrees; the merge explanation was false. when requirement evidence is complete and provider evidence for that soname is complete and attributed by index. Otherwise the answer is `unknown`. - **Presence = defeasible "structurally possible"** (`compatible(exact | - extrapolated)`) — necessary, not sufficient; semantics are unverified. It - never outranks a declared or tested dependency; it widens the candidate set - with a low-confidence maybe. + curated | extrapolated)`) — necessary, not sufficient; semantics are + unverified. It never outranks a declared or tested dependency; it widens + the candidate set with a low-confidence maybe. +- **`curated`** means the presence rests on `.symbols` metadata (a curated + export list), not on a direct readelf observation at that release; it is + reported distinctly from `exact`. - **`below_floor`** is a curated floor, not proof of absence, and is reported distinctly so a caller can choose to treat it as declared-dependency - strength rather than ELF-hard strength. + strength rather than ELF-hard strength. It never overrides another evidence + row that shows the identity present at the release. - **`extrapolated`** relies on the in-soname monotone-export assumption and is - reported distinctly from `exact`. + reported distinctly from `exact`. Absence propagates the other way: absent + from complete evidence at `R1` means absent at every `Rel <= R1` + (`missing(..., observed_absent(Src, R1))`), but says nothing about later + releases (`unknown(..., absent_at(Src, R1))`). Present at `R0` and absent + at a later `R1` is an observed drop: the releases in between are + `unknown(..., dropped_between(R0, Src, R1))`. Confidence order: `tested > declared repo dep > ELF hard veto / ELF maybe > soname default`. ## Limits -- Hidden (`@`, non-default) versions are matched like default (`@@`) ones, - which is what the loader does for an exact versioned reference; the "which - node does an *unversioned* reference bind to" rule is simplified to "any - exported node". +- The unversioned-reference rule follows `ld.so` for `DT_NEEDED`-driven + binding: `Base`, the `@@` default, or the oldest version node (verdef + index 2, accepted by glibc for legacy binaries even when hidden). `dlsym()` + lookups use a different rule and are not modelled. - `.symbols` source-template constructs that need the binary to expand - (`(symver)`, `(regex)`, quoted C++ patterns) are rejected, not interpreted. + (`(symver)`, `(regex)`, quoted C++ patterns) and any unknown tag are + rejected, not interpreted; `(optional)` rows need `--elf`. +- `soname_mismatch` needs a declared `replaces(New, Old)` row; the lane does + not guess that `libfoo.so.2` succeeds `libfoo.so.1` from the name. - The release axis is an input (from `apt-cache madison`, dpkg, a snapshot store); the lane never invents candidates from symbol data. diff --git a/examples/pkg_resolver/abi/abi_cli.pl b/examples/pkg_resolver/abi/abi_cli.pl index 5b7b672b0..ff50cf29d 100644 --- a/examples/pkg_resolver/abi/abi_cli.pl +++ b/examples/pkg_resolver/abi/abi_cli.pl @@ -9,7 +9,7 @@ % % Commands: % verdict [DropSym DropNode DropAt] -% compatible(exact|extrapolated) | incompatible([...]) | unknown([...]) | not_needed(_) +% compatible(exact|curated|extrapolated) | incompatible([...]) | unknown([...]) | not_needed(_) % status one line per requirement (provided / missing / ...) % floor curated lower bound implied by `.symbols` (= dpkg-shlibdeps' dep) % axis the ingested release candidates, ascending diff --git a/examples/pkg_resolver/abi/abi_resolve.pl b/examples/pkg_resolver/abi/abi_resolve.pl index d9c38abe2..5a2cae07c 100644 --- a/examples/pkg_resolver/abi/abi_resolve.pl +++ b/examples/pkg_resolver/abi/abi_resolve.pl @@ -3,7 +3,8 @@ % Copyright (c) 2026 John William Creighton (@s243a) % % abi_resolve.pl -- symbol-level ABI-compatibility resolver (redesigned after -% the PR #4262 review; see REVIEW_NOTES.md for the point-by-point map). +% the PR #4262 review, revised after Sol's re-review; see REVIEW_NOTES.md for +% the point-by-point map). % % MODEL % Two independent axes: @@ -11,42 +12,57 @@ % PUBLIC, Base) is an opaque label. A requirement `Sym@Node` on soname So % is satisfied only by a provider row with the SAME (So, Sym, Node) -- % string equality, never numeric ordering, never a bare-name fallback. -% `Base` = dpkg's spelling of "unversioned"; an unversioned REQUIREMENT -% (no node) is satisfied by any exported node of that symbol. +% `Base` = dpkg's spelling of "unversioned". An unversioned REQUIREMENT +% (no node) binds, as the loader does, only to a `Base` export or to a +% DEFAULT export (`@@`, or the oldest version node); a provider row whose +% default binding is unproven (a `.symbols` row not cross-checked against +% the ELF) yields unknown, never compatible. % * Debian package-version axis: the `.symbols` minimum-version and the % release candidates are deb/3 terms produced by debian/deb_parse.pl and % ordered by the frozen resolver:version_lt/2 (epoch, ~, revision all % handled there). A non-deb release id is kept as label(Atom) and only % ever matches itself. -% Evidence is explicit: +% Evidence is explicit and every provider bound is tied to the evidence row +% it rests on (Sol P1a): % prov_evidence(So, Src, R0, complete) -- So's export set is known % completely at evidence release R0 (Src = symbols | elf). +% symprov(So, Sym, Node, since(Min, MinAtom, R0, Bind)) -- from the +% `.symbols` evidence at R0: exported at R0 and, by the curated lower +% bound, at every release >= Min. Min is NOT a ground-truth introduction +% date (Debian policy lets it be raised); R < Min is `below_floor` (the +% conservative floor dpkg-shlibdeps emits) unless direct evidence says +% otherwise. +% symprov(So, Sym, Node, at(R0, Bind)) -- from readelf at R0. +% Bind = default | nondefault | unproven (default-version binding) % req_evidence(Bin, Src, Status, Detail) -- Bin's requirement set is % complete, or why not (missing_file / readelf_failed / inconsistent). -% Provider bounds: -% since(Min, MinAtom) -- from `.symbols`: exported at R0 and, by the -% curated lower bound, at every release >= Min. Min is NOT a ground-truth -% introduction date (Debian policy lets it be raised); R < Min is treated -% as `below_floor` (the same conservative floor dpkg-shlibdeps emits). -% at(R0) -- from readelf: exported at exactly R0. +% Per-identity status at release Rel aggregates EVERY complete evidence row +% of the soname (ident_status/5): evidence AT Rel decides directly; else the +% nearest evidence BELOW Rel (presence extrapolates upward) and the nearest +% evidence ABOVE Rel (absence propagates downward, a curated floor covers +% Rel >= Min) are combined. A release satisfied by ANY evidence row is +% never vetoed by another row's bound. % Extrapolation (defeasible, documented): within a soname, exports do not % disappear (removing one is an ABI break that requires a soname bump), so -% presence at R0 is extrapolated to R > R0 with basis `extrapolated`, and -% absence from a complete export set is a veto for every release of the -% soname. Presence never becomes a guarantee: compatible(_) is defeasible. -% For at(R0) evidence, R < R0 is UNKNOWN (readelf says nothing about older -% releases). A hypothetical drop(Sym, Node, At) models an in-soname removal -% (violating the assumption) to exercise the upper bound. +% presence at R0 extends to R > R0 with basis `extrapolated`, and absence +% from a complete export set at R1 is a veto for R =< R1. Absence at R1 says +% nothing about R > R1 (later releases add symbols): unknown. Presence never +% becomes a guarantee: compatible(_) is defeasible. A hypothetical +% drop(Sym, Node, At) models an in-soname removal to exercise the upper bound. % % VERDICTS abi_verdict(Bin, So, Rel, Verdict): -% compatible(exact | extrapolated) -% incompatible([missing(Sym@Node) | below_floor(Sym@Node, MinAtom) +% compatible(exact | curated | extrapolated) +% exact -- every requirement observed by readelf at exactly Rel +% curated -- at least one rests on `.symbols` metadata only (Sol residual) +% extrapolated -- at least one rests on the monotone-export assumption +% incompatible([missing(Sym@Node) | missing(Sym@Node, Why) | below_floor(Sym@Node, MinAtom) % | soname_mismatch(offered(So), needed(N)) ...]) -- HARD veto, % only reachable when requirement AND provider evidence are complete % unknown([no_requires_evidence(Bin) | requires_evidence(Status, Detail) -% | no_provider_evidence(So) | unknown(Sym@Node, evidence_release(R0)) -% | unknown(Sym, no_provider_evidence(S)) ...]) -% not_needed(So) -- Bin has no DT_NEEDED entry for So (and no stem clash) +% | no_provider_evidence(So) | unknown(Sym@Node, Why) +% | unknown(Sym, Why) ...]) +% not_needed(So) -- Bin has no DT_NEEDED entry for So and So is not a +% declared replacement (replaces/2) of a NEEDED soname % % RANGE abi_range(Bin, So, Releases, Result): every release in the ACTUAL % candidate axis is evaluated; range(Min, Max, Pairs) has compatible verdicts @@ -61,12 +77,14 @@ symprov/4, symreq/5, needed/2, + replaces/2, prov_evidence/4, req_evidence/4, release/3, rel_term/2, rel_le/2, rel_lt/2, + ident_status/5, provides_at/5, req_status/5, abi_verdict/4, @@ -88,9 +106,10 @@ :- use_module(library(lists)). :- use_module(library(apply)). -:- dynamic symprov/4. % symprov(SoName, Sym, Node, Bound) Bound = since(Deb, Atom) | at(Rel) +:- dynamic symprov/4. % symprov(SoName, Sym, Node, Bound) Bound = since(Deb, Atom, R0, Bind) | at(R0, Bind) :- dynamic symreq/5. % symreq(Binary, Sym, Node, SoName, Bind) Node/SoName = none if unversioned :- dynamic needed/2. % needed(Binary, SoName) +:- dynamic replaces/2. % replaces(NewSoName, OldSoName) declared soname succession :- dynamic prov_evidence/4. % prov_evidence(SoName, Src, Rel, Status) :- dynamic req_evidence/4. % req_evidence(Binary, Src, Status, Detail) :- dynamic release/3. % release(SoName, Rel, Atom) @@ -103,6 +122,7 @@ retractall(symprov(_, _, _, _)), retractall(symreq(_, _, _, _, _)), retractall(needed(_, _)), + retractall(replaces(_, _)), retractall(prov_evidence(_, _, _, _)), retractall(req_evidence(_, _, _, _)), retractall(release(_, _, _)). @@ -112,6 +132,7 @@ load_rows(Dir, 'symprov.jsonl', assert_symprov), load_rows(Dir, 'symreq.jsonl', assert_symreq), load_rows(Dir, 'needed.jsonl', assert_needed), + load_rows(Dir, 'replaces.jsonl', assert_replaces), load_rows(Dir, 'evidence.jsonl', assert_evidence), load_rows(Dir, 'releases.jsonl', assert_release). @@ -141,18 +162,32 @@ load_row_lines(S, Path, N1, Handler) ). -% "|@" -> since(Deb, Atom) | at(Rel) -assert_symprov(K, [Kind, V]) :- +% "|@" -> +% ["since", MinVer, EvidenceRelease, Bind] -> since(Deb, MinVer, R0, Bind) +% ["at", EvidenceRelease, Bind] -> at(R0, Bind) +% The evidence release is part of the row so a bound is tied to the evidence +% it came from even when several evidence rows exist for one soname. +assert_symprov(K, [Kind | V]) :- split_first(K, '|', So, Ident), split_last(Ident, '@', Sym, Node), Node \== '', ( Kind == since - -> parse_deb_version(V, Deb), Bound = since(Deb, V) + -> V = [Min, EvRel, Bind0], + parse_deb_version(Min, Deb), rel_term(EvRel, R0), + binding(Node, Bind0, Bind), + Bound = since(Deb, Min, R0, Bind) ; Kind == at - -> rel_term(V, Rel), Bound = at(Rel) + -> V = [EvRel, Bind0], + rel_term(EvRel, R0), + binding(Node, Bind0, Bind), + Bound = at(R0, Bind) ), assertz(symprov(So, Sym, Node, Bound)). +% A `Base` (unversioned) export always binds an unversioned reference. +binding('Base', _, default) :- !. +binding(_, Bind, Bind) :- memberchk(Bind, [default, nondefault, unproven]). + % "|[@]" -> symreq(Bin, Sym, Node, SoName, Bind) assert_symreq(K, [So0, Bind]) :- split_first(K, '|', Bin, Ident), @@ -167,10 +202,15 @@ atom(So), assertz(needed(Bin, So)). +assert_replaces(New, Old) :- + atom(Old), + assertz(replaces(New, Old)). + assert_evidence(K, V) :- split_first(K, '|', Kind, Subject), ( Kind == provides -> V = [Src, RelAtom, Status, _Source], + memberchk(Src, [symbols, elf]), rel_term(RelAtom, Rel), assertz(prov_evidence(Subject, Src, Rel, Status)) ; Kind == requires @@ -231,24 +271,102 @@ pairs_values(Pairs, Atoms). % --------------------------------------------------------------------------- -% Provision at a release +% Evidence rows and what each says about one identity (Sol P1a) % --------------------------------------------------------------------------- -% provides_at(So, Sym, Node, Rel, Basis): So exports exactly Sym@Node at Rel; -% Basis = exact (inside the evidence) | extrapolated (beyond the evidence -% release, under the in-soname monotone-export assumption). +% bound_evidence(So, Bound, Src, R0): the complete evidence row a bound rests +% on. A bound whose evidence row is missing/incomplete is not usable. +bound_evidence(So, since(_, _, R0, _), symbols, R0) :- prov_evidence(So, symbols, R0, complete). +bound_evidence(So, at(R0, _), elf, R0) :- prov_evidence(So, elf, R0, complete). + +% observed(So, Sym, Node, Src, R1, Bound): the identity is in the Src +% evidence taken at R1. +observed(So, Sym, Node, symbols, R1, B) :- B = since(_, _, R1, _), symprov(So, Sym, Node, B). +observed(So, Sym, Node, elf, R1, B) :- B = at(R1, _), symprov(So, Sym, Node, B). + +% ev_says(So, Sym, Node, R1, Says): for every complete evidence row (Src, R1) +% of So, whether Sym@Node is present in it (and under which bound) or absent. +ev_says(So, Sym, Node, R1, Says) :- + prov_evidence(So, Src, R1, complete), + ( observed(So, Sym, Node, Src, R1, Bound) + -> Says = present(Src, Bound) + ; Says = absent(Src) + ). + +% ident_status(+So, +Sym, +Node, +Rel, -Status): the status of the exact +% identity Sym@Node on So at Rel, aggregated over ALL complete evidence rows. +% provided(exact | curated | extrapolated) +% missing(Why) absent from a complete export set at/above Rel +% below_floor(MinAtom) only a curated floor above Rel covers it, and Rel < Min +% unknown(Why) no evidence row speaks about Rel +% Rule: evidence AT Rel decides (readelf before .symbols). Otherwise the +% nearest evidence BELOW Rel (last known state) and the nearest evidence ABOVE +% Rel are combined: presence below extrapolates upward unless the row above +% observed absence (then the identity was dropped somewhere in between: +% unknown, not a false compat and not a false veto); absence above propagates +% downward (monotone exports); a curated floor above covers Rel >= Min. +ident_status(So, Sym, Node, Rel, Status) :- + findall(R1-Says, ev_says(So, Sym, Node, R1, Says), Rows), + Rows \== [], + ( member(Rel1-_, Rows), Rel1 == Rel + -> says_at(Rows, Rel, Says), + says_status(Says, Rel, Status) + ; nearest_below(Rows, Rel, Below), + nearest_above(Rows, Rel, Above), + combine(Below, Above, Rel, Status) + ). + +% says_at(Rows, R, Says): what the evidence taken at exactly R says; when both +% tiers were taken at R, readelf (direct observation) decides. +says_at(Rows, R, Says) :- + ( member(R1-present(elf, B), Rows), R1 == R -> Says = present(elf, B) + ; member(R1-absent(elf), Rows), R1 == R -> Says = absent(elf) + ; member(R1-present(symbols, B), Rows), R1 == R -> Says = present(symbols, B) + ; Says = absent(symbols) + ). + +says_status(present(elf, _), _, provided(exact)). +says_status(present(symbols, since(Min, MinAtom, _, _)), Rel, Status) :- + ( rel_le(Min, Rel) -> Status = provided(curated) ; Status = below_floor(MinAtom) ). +says_status(absent(Src), Rel, missing(observed_absent(Src, Rel))). + +% Distinct evidence releases strictly below / above Rel; the nearest one wins. +nearest_below(Rows, Rel, Below) :- + findall(R-R, ( member(R-_, Rows), rel_lt(R, Rel) ), Bs0), + predsort(rel_cmp, Bs0, Bs), + ( Bs == [] -> Below = none + ; last(Bs, R0-_), says_at(Rows, R0, Says), Below = ev(R0, Says) + ). + +nearest_above(Rows, Rel, Above) :- + findall(R-R, ( member(R-_, Rows), rel_lt(Rel, R) ), As0), + predsort(rel_cmp, As0, As), + ( As == [] -> Above = none + ; As = [R1-_|_], says_at(Rows, R1, Says), Above = ev(R1, Says) + ). + +% combine(Below, Above, Rel, Status) +combine(ev(R0, present(_, _)), ev(R1, absent(Src)), _, unknown(dropped_between(R0, Src, R1))) :- !. +combine(ev(_, present(_, _)), ev(_, present(symbols, since(Min, _, _, _))), Rel, provided(curated)) :- + rel_le(Min, Rel), !. +combine(ev(_, present(_, _)), _, _, provided(extrapolated)) :- !. +combine(_, ev(R1, absent(Src)), _, missing(observed_absent(Src, R1))) :- !. +combine(_, ev(_, present(symbols, since(Min, MinAtom, _, _))), Rel, Status) :- !, + ( rel_le(Min, Rel) -> Status = provided(curated) ; Status = below_floor(MinAtom) ). +combine(_, ev(R1, present(elf, _)), _, unknown(evidence_release(R1))) :- !. +combine(ev(R0, absent(Src)), none, _, unknown(absent_at(Src, R0))) :- !. +combine(none, none, _, unknown(no_evidence)). + +% provides_at(So, Sym, Node, Rel, Basis): So exports exactly Sym@Node at Rel. provides_at(So, Sym, Node, Rel, Basis) :- + ident_status(So, Sym, Node, Rel, provided(Basis)). + +% node_binding(So, Sym, Node, Bind): the default-version binding recorded for +% an export (default: an unversioned reference binds to it; nondefault: it +% does not; unproven: `.symbols` row not cross-checked against the ELF). +node_binding(So, Sym, Node, Bind) :- symprov(So, Sym, Node, Bound), - prov_evidence(So, _, R0, complete), - bound_holds(Bound, R0, Rel, Basis). - -bound_holds(since(Min, _), R0, Rel, Basis) :- - rel_le(Min, Rel), - ( rel_le(Rel, R0) -> Basis = exact ; Basis = extrapolated ). -bound_holds(at(R0), _, Rel, Basis) :- - ( Rel == R0 -> Basis = exact - ; rel_lt(R0, Rel) -> Basis = extrapolated - ). + ( Bound = since(_, _, _, B) -> Bind = B ; Bound = at(_, B) -> Bind = B ). hyp_dropped(drop(Sym, Node, At), Sym, Node, Rel) :- rel_term(At, AtRel), @@ -270,43 +388,80 @@ symreq(Bin, Sym, none, none, Bind), unversioned_status(Bin, So, Sym, Bind, Rel, Hyp, Status). +% missing(Sym@Node) = absent from the complete export set observed AT Rel; +% missing(Sym@Node, observed_absent(Src, R1)) = absent at a LATER release R1, +% hence absent at Rel under monotone exports (the inference is visible). versioned_status(So, Sym, Node, Bind, Rel, Hyp, Status) :- ( hyp_dropped(Hyp, Sym, Node, Rel) -> Status = missing(Sym@Node, hypothetical_drop) - ; symprov(So, Sym, Node, Bound) - -> prov_evidence(So, _, R0, complete), - ( bound_holds(Bound, R0, Rel, Basis) - -> Status = provided(Sym@Node, Basis) - ; Bound = since(_, MinAtom) - -> Status = below_floor(Sym@Node, MinAtom) - ; Status = unknown(Sym@Node, evidence_release(R0)) + ; ident_status(So, Sym, Node, Rel, S) + -> ( S = provided(Basis) -> Status = provided(Sym@Node, Basis) + ; S = below_floor(MinAtom) -> Status = below_floor(Sym@Node, MinAtom) + ; S = missing(_), Bind == 'WEAK' -> Status = weak_unresolved(Sym@Node) + ; S = missing(observed_absent(_, R1)), R1 == Rel -> Status = missing(Sym@Node) + ; S = missing(Why) -> Status = missing(Sym@Node, Why) + ; S = unknown(Why) -> Status = unknown(Sym@Node, Why) ) - ; Bind == 'WEAK' - -> Status = weak_unresolved(Sym@Node) - ; Status = missing(Sym@Node) + ; Status = unknown(Sym@Node, no_provider_evidence(So)) ). -% An unversioned reference binds to any exported node of Sym in any NEEDED -% object. Against the queried So we evaluate at Rel; against the other NEEDED -% objects at their own evidence release. Complete absence is a veto only when -% every NEEDED object has complete provider evidence. +% An unversioned reference binds (Sol P1b) to a `Base` export or to a DEFAULT +% export of Sym in any NEEDED object -- never to a non-default (`@`) one. A +% provider row whose binding is unproven (a `.symbols` row not cross-checked +% against the ELF) can only make the answer unknown. Against the queried So +% we evaluate at Rel; against the other NEEDED objects at their own evidence +% release. Complete absence is a veto only when every NEEDED object has +% complete provider evidence. unversioned_status(Bin, So, Sym, Bind, Rel, Hyp, Status) :- ( \+ hyp_dropped_any(Hyp, Sym), - ( provides_at(So, Sym, _, Rel, Basis) - -> Status = provided(Sym, any_node(So, Basis)) + ( unversioned_in(So, Sym, Rel, Node, Basis) + -> Status = provided(Sym, default_node(So, Node, Basis)) ; needed(Bin, S), S \== So, prov_evidence(S, _, R0, complete), - provides_at(S, Sym, _, R0, _) - -> Status = provided(Sym, any_node(S, exact)) + unversioned_in(S, Sym, R0, Node, Basis) + -> Status = provided(Sym, default_node(S, Node, Basis)) ; fail ) -> true ; Bind == 'WEAK' -> Status = weak_unresolved(Sym) % a weak ref never vetoes, so missing evidence is moot + ; unversioned_unproven(Bin, So, Sym, Rel, S1, N1) + -> Status = unknown(Sym, default_binding_unproven(S1, N1)) ; needed(Bin, S), \+ prov_evidence(S, _, _, complete) -> Status = unknown(Sym, no_provider_evidence(S)) + ; unversioned_unknown(Bin, So, Sym, Rel, _, Why) + -> Status = unknown(Sym, Why) + ; unversioned_nondefault(Bin, So, Sym, Rel, S3, N3) + -> Status = missing(Sym, no_default_export(S3, N3)) ; Status = missing(Sym) ). +% A default-bound export of Sym in S provided at Rel (any node). +unversioned_in(S, Sym, Rel, Node, Basis) :- + node_binding(S, Sym, Node, default), + ident_status(S, Sym, Node, Rel, provided(Basis)). + +% Some export of Sym that is provided at the relevant release but whose +% default binding is unproven. +unversioned_unproven(Bin, So, Sym, Rel, S, Node) :- + needed_at(Bin, So, Rel, S, R), + node_binding(S, Sym, Node, unproven), + ident_status(S, Sym, Node, R, provided(_)). + +unversioned_unknown(Bin, So, Sym, Rel, S, Why) :- + needed_at(Bin, So, Rel, S, R), + node_binding(S, Sym, Node, B), B \== nondefault, + ident_status(S, Sym, Node, R, unknown(Why)). + +unversioned_nondefault(Bin, So, Sym, Rel, S, Node) :- + needed_at(Bin, So, Rel, S, R), + node_binding(S, Sym, Node, nondefault), + ident_status(S, Sym, Node, R, provided(_)). + +% needed_at(Bin, So, Rel, S, R): the queried So at Rel, other NEEDED objects +% at their own evidence release(s). +needed_at(_, So, Rel, So, Rel). +needed_at(Bin, So, _, S, R) :- needed(Bin, S), S \== So, prov_evidence(S, _, R, complete). + hyp_dropped_any(drop(Sym, _, _), Sym). % --------------------------------------------------------------------------- @@ -332,6 +487,8 @@ aggregate_statuses(Ss, Verdict) ). +% The verdict basis is the WEAKEST basis among the provided requirements: +% exact < curated < extrapolated. aggregate_statuses(Ss, Verdict) :- include(hard_veto, Ss, Hard), include(is_unknown, Ss, Unk), @@ -339,13 +496,18 @@ -> Verdict = incompatible(Hard) ; Unk \== [] -> Verdict = unknown(Unk) - ; memberchk(provided(_, extrapolated), Ss) - -> Verdict = compatible(extrapolated) - ; memberchk(provided(_, any_node(_, extrapolated)), Ss) + ; has_basis(Ss, extrapolated) -> Verdict = compatible(extrapolated) + ; has_basis(Ss, curated) + -> Verdict = compatible(curated) ; Verdict = compatible(exact) ). +has_basis(Ss, Basis) :- + ( memberchk(provided(_, Basis), Ss) -> true + ; memberchk(provided(_, default_node(_, _, Basis)), Ss) + ). + hard_veto(missing(_)). hard_veto(missing(_, _)). hard_veto(below_floor(_, _)). @@ -353,22 +515,18 @@ % soname_offer(Bin, So, Offer): needed | mismatch(NeededSoName) | not_needed. % Offering libfoo.so.2 to a binary whose DT_NEEDED says libfoo.so.1 is a hard -% veto: the loader matches DT_NEEDED by exact soname string. +% veto ONLY under a declared succession relation replaces(libfoo.so.2, +% libfoo.so.1) (the loader matches DT_NEEDED by exact soname string). Without +% that declaration a name that is not NEEDED is simply not_needed -- no stem +% heuristic (Sol P2d: libfoo.so.2 vs an unrelated NEEDED libfoo.so.1-extra). soname_offer(Bin, So, Offer) :- ( needed(Bin, So) -> Offer = needed - ; so_stem(So, Stem), - needed(Bin, N), so_stem(N, Stem) + ; replaces(So, N), needed(Bin, N) -> Offer = mismatch(N) ; Offer = not_needed ). -so_stem(So, Stem) :- - ( sub_atom(So, B, _, _, '.so') - -> sub_atom(So, 0, B, _, Stem) - ; Stem = So - ). - % --------------------------------------------------------------------------- % Floor: the curated lower bound implied by `.symbols` (= dpkg-shlibdeps' dep) % --------------------------------------------------------------------------- @@ -383,8 +541,10 @@ maplist(req_floor(So), Reqs, Mins), max_deb(Mins, _-Floor). -req_floor(So, Sym-Node, Deb-Atom) :- - symprov(So, Sym, Node, since(Deb, Atom)). +req_floor(So, Sym-Node, Max) :- + findall(Deb-Atom, symprov(So, Sym, Node, since(Deb, Atom, _, _)), Ms), + Ms \== [], + max_deb(Ms, Max). max_deb([M|Ms], Max) :- foldl(max_deb_1, Ms, M, Max). max_deb_1(D-A, D0-A0, Out) :- ( rel_lt(D0, D) -> Out = D-A ; Out = D0-A0 ). diff --git a/examples/pkg_resolver/abi/crosscheck.mjs b/examples/pkg_resolver/abi/crosscheck.mjs index 2307bbd04..b9a41ec8d 100644 --- a/examples/pkg_resolver/abi/crosscheck.mjs +++ b/examples/pkg_resolver/abi/crosscheck.mjs @@ -3,40 +3,45 @@ // Copyright (c) 2026 John William Creighton (@s243a) // // crosscheck.mjs -- compare the two provider tiers for one soname: -// readelf(lib.so) -> symprov rows ["so|sym@node", ["at", rel]] -// .symbols file -> symprov rows ["so|sym@node", ["since", minver]] +// readelf(lib.so) -> symprov rows ["so|sym@node", ["at", rel, binding]] +// .symbols file -> symprov rows ["so|sym@node", ["since", minver, rel, binding]] // // (1) EXACT IDENTITY: the sets of sym@node must be equal. This is the check // the model actually relies on (a requirement matches a provider by the // exact triple), and on Ubuntu 22.04 libc6 it is 3006/3006. -// (2) LEGACY PER-NAME COMPARISON, CORRECTED: the original script compared the -// LAST curated row of a symbol against the EARLIEST ELF node of that -// symbol, so every symbol with two nodes (e.g. pthread_setname_np@GLIBC_2.12 -// + @GLIBC_2.34 after the 2.34 libpthread merge) "disagreed" (91.7%). With -// the same earliest-row aggregation on both sides the figure is 100%; the -// "glibc 2.34 merge divergence" explanation was an artifact. Note this -// comparison mixes axes (node label number vs package version) and is kept -// only as the corrected regression figure. +// (2) PER-NAME COMPARISON: for every symbol NAME the SET of nodes must agree +// on both sides. This is the corrected form of the legacy per-name figure +// (the original script compared the LAST curated row of a symbol against +// the EARLIEST ELF node, so every two-node symbol -- e.g. +// pthread_setname_np@GLIBC_2.12 + @GLIBC_2.34 -- "disagreed": 91.7%). +// Node names are opaque labels: nothing is parsed out of them and nothing +// is ordered (Sol P3 removed the local dotted-version comparator); the +// fixture fixtures/crosscheck/ keeps the regression pinned. +// Both figures must be 100% AND every denominator must be nonzero: two empty +// inputs (e.g. /dev/null vs /dev/null) FAIL instead of passing on NaN (Sol P2c). // // usage: crosscheck.mjs import { readFileSync } from "node:fs"; const [symFile, elfFile, so] = process.argv.slice(2); +if (!symFile || !elfFile || !so) { console.error("usage: crosscheck.mjs "); process.exit(2); } function loadRows(file, tag) { - const rows = []; // {sym, node, v} + const rows = []; // {sym, node} for (const l of readFileSync(file, "utf8").split("\n")) { if (!l) continue; const [k, v] = JSON.parse(l); if (!k.startsWith(so + "|") || v[0] !== tag) continue; const ident = k.slice(so.length + 1); const at = ident.lastIndexOf("@"); - rows.push({ sym: ident.slice(0, at), node: ident.slice(at + 1), v: v[1] }); + rows.push({ sym: ident.slice(0, at), node: ident.slice(at + 1) }); } return rows; } +function fail(msg) { console.error(` FAIL: ${msg}`); process.exit(1); } + const S = loadRows(symFile, "since"), E = loadRows(elfFile, "at"); const sKeys = new Set(S.map((r) => `${r.sym}@${r.node}`)); const eKeys = new Set(E.map((r) => `${r.sym}@${r.node}`)); @@ -46,28 +51,30 @@ for (const k of sKeys) if (!eKeys.has(k)) onlyS.push(k); console.log(` exact sym@node identity: .symbols=${sKeys.size} readelf=${eKeys.size} shared=${both} only-readelf=${onlyE.length} only-.symbols=${onlyS.length}`); if (onlyE.length) console.log(` only-readelf sample: ${onlyE.slice(0, 5).join(", ")}`); if (onlyS.length) console.log(` only-.symbols sample: ${onlyS.slice(0, 5).join(", ")}`); -const identityPct = (100 * both) / Math.max(sKeys.size, eKeys.size); -console.log(` identity agreement: ${both}/${Math.max(sKeys.size, eKeys.size)} (${identityPct.toFixed(1)}%)`); +if (sKeys.size === 0 || eKeys.size === 0) fail(`empty symbol set for ${so} (.symbols=${sKeys.size} readelf=${eKeys.size}); nothing to compare`); +const denom = Math.max(sKeys.size, eKeys.size); +const identityPct = (100 * both) / denom; +console.log(` identity agreement: ${both}/${denom} (${identityPct.toFixed(1)}%)`); -// Legacy per-name figure, corrected: EARLIEST row on BOTH sides. -function nodeNum(node) { const m = node.match(/_([0-9][0-9.]*)$/); return m ? m[1] : null; } -function cmpDotted(a, b) { - const pa = a.split("."), pb = b.split("."); - for (let i = 0; i < Math.max(pa.length, pb.length); i++) { const x = +(pa[i] || 0), y = +(pb[i] || 0); if (x !== y) return x - y; } - return 0; -} -function earliest(rows, pick) { +// Per-name node-set agreement (no ordering, no parsing of node names). +function nodeSets(rows) { const m = new Map(); - for (const r of rows) { const n = pick(r); if (n === null) continue; if (!m.has(r.sym) || cmpDotted(n, m.get(r.sym)) < 0) m.set(r.sym, n); } + for (const r of rows) { if (!m.has(r.sym)) m.set(r.sym, new Set()); m.get(r.sym).add(r.node); } return m; } -const sE = earliest(S, (r) => nodeNum(r.node)); // earliest numeric node per name (curated rows) -const eE = earliest(E, (r) => nodeNum(r.node)); // earliest numeric node per name (readelf rows) -let shared = 0, agree = 0; const dis = []; -for (const [sym, n] of eE) { if (!sE.has(sym)) continue; shared++; if (sE.get(sym) === n) agree++; else if (dis.length < 5) dis.push(`${sym} (.symbols=${sE.get(sym)} readelf=${n})`); } -const pct = (100 * agree) / shared; -console.log(` legacy per-name earliest-row comparison (corrected aggregation): ${agree}/${shared} (${pct.toFixed(1)}%)`); +const sN = nodeSets(S), eN = nodeSets(E); +const names = new Set([...sN.keys(), ...eN.keys()]); +let agree = 0; const dis = []; +for (const sym of names) { + const a = sN.get(sym) || new Set(), b = eN.get(sym) || new Set(); + const same = a.size === b.size && [...a].every((n) => b.has(n)); + if (same) agree++; + else if (dis.length < 5) dis.push(`${sym} (.symbols={${[...a].join(",")}} readelf={${[...b].join(",")}})`); +} +if (names.size === 0) fail("no symbol names to compare"); +const pct = (100 * agree) / names.size; +console.log(` per-name node-set agreement: ${agree}/${names.size} (${pct.toFixed(1)}%)`); if (dis.length) console.log(` disagreements: ${dis.join(", ")}`); -if (identityPct < 100) { console.error(" FAIL: exact identity sets differ"); process.exit(1); } -if (pct < 100) { console.error(" FAIL: corrected per-name comparison below 100%"); process.exit(1); } +if (!(identityPct === 100)) fail("exact identity sets differ"); +if (!(pct === 100)) fail("per-name node sets differ"); console.log(" PASS: readelf and .symbols agree on every exact sym@node identity"); diff --git a/examples/pkg_resolver/abi/fixtures/batch/mixed.symbols b/examples/pkg_resolver/abi/fixtures/batch/mixed.symbols new file mode 100644 index 000000000..f35a5fa43 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/batch/mixed.symbols @@ -0,0 +1,7 @@ +# Batch-mode atomicity fixture (Sol P2a): the first block's package (libc6) is +# installed so its evidence release is known; the second block's is not +# (#PACKAGE#, no --release). The WHOLE file must be rejected: no good_fn row. +libbatchgood.so.1 libc6 #MINVER# + good_fn@Base 1.0 +libbatchbad.so.1 #PACKAGE# #MINVER# + bad_fn@Base 1.0 diff --git a/examples/pkg_resolver/abi/fixtures/crosscheck/elf.symprov.jsonl b/examples/pkg_resolver/abi/fixtures/crosscheck/elf.symprov.jsonl new file mode 100644 index 000000000..7d63345d3 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/crosscheck/elf.symprov.jsonl @@ -0,0 +1,5 @@ +["libx.so.1|pthread_setname_np@GLIBC_2.34",["at","2.35-1","default"]] +["libx.so.1|pthread_setname_np@GLIBC_2.12",["at","2.35-1","nondefault"]] +["libx.so.1|getenv@GLIBC_2.2.5",["at","2.35-1","default"]] +["libx.so.1|__libc_enable_secure@GLIBC_PRIVATE",["at","2.35-1","default"]] +["libx.so.1|plain_fn@Base",["at","2.35-1","default"]] diff --git a/examples/pkg_resolver/abi/fixtures/crosscheck/elf_dropnode.symprov.jsonl b/examples/pkg_resolver/abi/fixtures/crosscheck/elf_dropnode.symprov.jsonl new file mode 100644 index 000000000..e33e5255c --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/crosscheck/elf_dropnode.symprov.jsonl @@ -0,0 +1,4 @@ +["libx.so.1|pthread_setname_np@GLIBC_2.34",["at","2.35-1","default"]] +["libx.so.1|getenv@GLIBC_2.2.5",["at","2.35-1","default"]] +["libx.so.1|__libc_enable_secure@GLIBC_PRIVATE",["at","2.35-1","default"]] +["libx.so.1|plain_fn@Base",["at","2.35-1","default"]] diff --git a/examples/pkg_resolver/abi/fixtures/crosscheck/sym.symprov.jsonl b/examples/pkg_resolver/abi/fixtures/crosscheck/sym.symprov.jsonl new file mode 100644 index 000000000..185ebbb67 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/crosscheck/sym.symprov.jsonl @@ -0,0 +1,6 @@ +["libx.so.1|pthread_setname_np@GLIBC_2.12",["since","2.12","2.35-1","unproven"]] +["libx.so.1|pthread_setname_np@GLIBC_2.34",["since","2.34","2.35-1","unproven"]] +["libx.so.1|getenv@GLIBC_2.2.5",["since","2.2.5","2.35-1","unproven"]] +["libx.so.1|__libc_enable_secure@GLIBC_PRIVATE",["since","0","2.35-1","unproven"]] +["libx.so.1|plain_fn@Base",["since","1.0","2.35-1","default"]] +["libother.so.9|not_this_soname@OTHER_1",["since","1.0","2.35-1","unproven"]] diff --git a/examples/pkg_resolver/abi/fixtures/hid.c b/examples/pkg_resolver/abi/fixtures/hid.c new file mode 100644 index 000000000..2a9b17f8e --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/hid.c @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: MIT OR Apache-2.0 */ +/* libhid fixture (Sol P1b): hid_fn is exported ONLY at a NON-DEFAULT version + * node (`.symver` with a single `@` = hidden, readelf shows hid_fn@HID_1, not + * @@). Built twice under soname libhid.so.1: + * hid_idx3.map: HID_0 first (verdef index 2), HID_1 second (index 3) -- the + * loader refuses an unversioned reference: "undefined symbol: hid_fn" + * hid_idx2.map: HID_1 is the only node (verdef index 2) -- the loader binds a + * legacy unversioned reference to the oldest node even though it + * is hidden (glibc dl-lookup: index < 3 is accepted before the + * hidden test). Ingest records index 2 as "default" for that reason. */ +int other_fn(void) { return 9; } +int hid_fn_impl(void) { return 5; } +__asm__(".symver hid_fn_impl, hid_fn@HID_1"); diff --git a/examples/pkg_resolver/abi/fixtures/hid_idx2.map b/examples/pkg_resolver/abi/fixtures/hid_idx2.map new file mode 100644 index 000000000..fcb12d734 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/hid_idx2.map @@ -0,0 +1 @@ +HID_1 { global: hid_fn; other_fn; local: *; }; diff --git a/examples/pkg_resolver/abi/fixtures/hid_idx3.map b/examples/pkg_resolver/abi/fixtures/hid_idx3.map new file mode 100644 index 000000000..58f4e65a5 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/hid_idx3.map @@ -0,0 +1,2 @@ +HID_0 { global: other_fn; local: *; }; +HID_1 { global: hid_fn; } HID_0; diff --git a/examples/pkg_resolver/abi/fixtures/hid_plain.c b/examples/pkg_resolver/abi/fixtures/hid_plain.c new file mode 100644 index 000000000..9cb851ad0 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/hid_plain.c @@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: MIT OR Apache-2.0 */ +/* Unversioned build of libhid.so.1 (hid_fn@Base): usehid is linked against + * this one, so its reference to hid_fn is UNVERSIONED. */ +int other_fn(void) { return 9; } +int hid_fn(void) { return 5; } diff --git a/examples/pkg_resolver/abi/fixtures/tmpl_arch.symbols b/examples/pkg_resolver/abi/fixtures/tmpl_arch.symbols index d91aa4e2b..791ca6ed9 100644 --- a/examples/pkg_resolver/abi/fixtures/tmpl_arch.symbols +++ b/examples/pkg_resolver/abi/fixtures/tmpl_arch.symbols @@ -2,5 +2,4 @@ libtmpl.so.1 #PACKAGE# #MINVER# (arch=amd64)only_amd64@Base 1.0 (arch=!amd64)not_amd64@Base 1.0 (arch-bits=64)bits64@Base 1.0 - (optional)maybe_fn@Base 1.0 plain_fn@Base 1.0 diff --git a/examples/pkg_resolver/abi/fixtures/tmpl_optional.symbols b/examples/pkg_resolver/abi/fixtures/tmpl_optional.symbols new file mode 100644 index 000000000..07bc62fd0 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/tmpl_optional.symbols @@ -0,0 +1,3 @@ +libtmpl.so.1 #PACKAGE# #MINVER# + (optional)maybe_fn@Base 1.0 + plain_fn@Base 1.0 diff --git a/examples/pkg_resolver/abi/fixtures/tmpl_unknown_tag.symbols b/examples/pkg_resolver/abi/fixtures/tmpl_unknown_tag.symbols new file mode 100644 index 000000000..45c503b67 --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/tmpl_unknown_tag.symbols @@ -0,0 +1,3 @@ +libtmpl.so.1 #PACKAGE# #MINVER# + (frobnicate)mystery_fn@Base 1.0 + plain_fn@Base 1.0 diff --git a/examples/pkg_resolver/abi/fixtures/usehid.c b/examples/pkg_resolver/abi/fixtures/usehid.c new file mode 100644 index 000000000..bce8b199c --- /dev/null +++ b/examples/pkg_resolver/abi/fixtures/usehid.c @@ -0,0 +1,4 @@ +/* SPDX-License-Identifier: MIT OR Apache-2.0 */ +/* Unversioned reference to hid_fn from libhid.so.1. */ +int hid_fn(void); +int main(void) { return hid_fn() == 5 ? 0 : 1; } diff --git a/examples/pkg_resolver/abi/ingest_symbols.mjs b/examples/pkg_resolver/abi/ingest_symbols.mjs index e4454632f..9d6ce5b66 100644 --- a/examples/pkg_resolver/abi/ingest_symbols.mjs +++ b/examples/pkg_resolver/abi/ingest_symbols.mjs @@ -25,27 +25,44 @@ // * Evidence completeness is explicit: every successful ingest emits an // `evidence` row; a missing / unreadable ELF is a loud failure (exit 3) // that -- when --out is given -- records a failure evidence row instead -// of an empty "success". +// of an empty "success". A `.symbols` file is ingested atomically: one +// unsupported row, one unknown tag, one block whose evidence release is +// unknown, or a bad --release rejects the WHOLE file (also in batch +// mode), nothing partial is written, and the run exits 3 (Sol P2). +// * Default-version binding is retained (Sol P1b): every provider row says +// whether an UNVERSIONED reference binds to it. From readelf that is the +// `.gnu.version` hidden bit (`@` = hidden = "nondefault", `@@` = +// "default"); the loader additionally binds legacy unversioned references +// to the OLDEST version node (verdef index 2) even when hidden, so index 2 +// is recorded as "default" too (verified with the loader, fixture D10c). +// A `.symbols` file carries no `@@` information: its rows are "unproven" +// unless cross-checked against the ELF with --elf, and the resolver then +// answers unknown -- never compatible -- for an unversioned reference. +// `Base` (unversioned export) always binds. // // Store rows (all JSON arrays `[key, value]`): -// symprov.jsonl ["|@", ["since", ""]] (.symbols) -// ["|@", ["at", ""]] (readelf) +// symprov.jsonl ["|@", ["since", "", "", ]] (.symbols) +// ["|@", ["at", "", ]] (readelf) +// = "default" | "nondefault" | "unproven" // symreq.jsonl ["|@", ["", "GLOBAL"|"WEAK"]] // ["|", ["", "GLOBAL"|"WEAK"]] (unversioned) // needed.jsonl ["", ""] // evidence.jsonl ["provides|", ["symbols"|"elf", "", "complete", ""]] // ["requires|", ["readelf", "complete"|"missing_file"|"readelf_failed"|"inconsistent", ""]] // releases.jsonl ["", ""] (candidate axis) +// replaces.jsonl ["", ""] (declared soname succession) // // Usage: -// node ingest_symbols.mjs symbols-file [--release V] [--arch A] [--out DIR] [--append] [--stdout] -// node ingest_symbols.mjs symbols-dir [--arch A] --out DIR +// node ingest_symbols.mjs symbols-file [--release V] [--arch A] [--elf lib.so] [--out DIR] [--append] [--stdout] +// node ingest_symbols.mjs symbols-dir [--release V] [--arch A] --out DIR // node ingest_symbols.mjs elf [--release V] [--out DIR] [--append] [--stdout] // node ingest_symbols.mjs requires [--out DIR] [--append] [--stdout] // node ingest_symbols.mjs releases ... [--out DIR] [--append] [--stdout] +// node ingest_symbols.mjs replaces ... [--out DIR] [--append] [--stdout] // // Exit codes: 0 ok; 2 usage; 3 evidence failure (missing file, readelf failure, -// unsupported .symbols template, unknown evidence release). +// unsupported/unknown .symbols template construct, unknown or invalid evidence +// release, (optional) row without ELF cross-check, ELF/.symbols disagreement). import { execFileSync } from "node:child_process"; import { readFileSync, existsSync, appendFileSync, writeFileSync, mkdirSync, readdirSync } from "node:fs"; @@ -106,20 +123,23 @@ const DEB_VERSION_RE = /^(?:\d+:)?\d[A-Za-z0-9.+~:-]*$/; // @ [] symbol row (indented) // // SOURCE-TEMPLATE syntax (debian/*.symbols in source packages) differs and is -// only partially supportable without the binary at hand. We process the -// semantics we can and REJECT the rest loudly (never silently mis-ingest): -// (optional) processed: the row is kept (the tag only relaxes -// dpkg-gensymbols' diff, it does not change the ABI fact) +// only partially supportable without the binary at hand. Tags are WHITELISTED: +// the ones below are processed, EVERY other tag rejects the file (Sol P2). +// (optional) dpkg lets an optional symbol stay in the template +// after it disappeared from the binary, so the row is +// NOT an export fact by itself (Sol P1c). It is kept only +// when --elf is given and the ELF exports that exact +// sym@node; an optional row absent from the ELF is +// dropped (reported on stderr); without --elf the file +// is rejected. // (arch=..)/(arch-bits=..)/(arch-endian=..) // processed when --arch is given (row kept iff it // selects the arch; arch-bits/endian derived from it); // rejected otherwise -// (symver) rejected: `(symver)NODE minver` expands to "every -// symbol under NODE", which needs the binary to expand -// (regex) rejected: pattern rows need the binary to expand -// (c++) / (c++11) ... rejected: demangled C++ patterns (quoted) are not -// ELF symbol identities // (ignore-blacklist) processed (ignored; does not affect identity) +// (symver), (regex), (c++...), any unknown tag +// rejected: symver/regex/c++ need the binary to expand; +// an unknown tag has unknown semantics // #include "file" rejected (template include) // #PACKAGE# accepted in the header; the package name is then // unknown so --release becomes mandatory @@ -164,12 +184,13 @@ function parseTags(line) { return { tags, rest }; } -const UNSUPPORTED_TAGS = ["symver", "regex", "c++", "c++11", "c++14", "c++17", "c++20"]; +// Whitelist (Sol P2): every tag not listed here rejects the file. +const SUPPORTED_TAGS = new Set(["optional", "arch", "arch-bits", "arch-endian", "ignore-blacklist"]); function parseSymbolsFile(path, { arch = null } = {}) { if (!existsSync(path)) die(`symbols file not found: ${path}`); const text = readFileSync(path, "utf8"); - const blocks = []; // {soname, package, rows: [{sym, node, minver}]} + const blocks = []; // {soname, package, rows: [{sym, node, minver, optional}]} const errors = []; // unsupported template constructs (line numbers) let cur = null; let lineNo = 0; @@ -194,11 +215,9 @@ function parseSymbolsFile(path, { arch = null } = {}) { if (line[0] === "|" || line[0] === "*" || line[0] === "#") continue; const { tags, rest } = parseTags(line); line = rest.trimStart(); - // Reject template-only semantics loudly. - for (const u of UNSUPPORTED_TAGS) { - if (tags.has(u)) { errors.push(`${lineNo}: unsupported template tag (${u}): ${raw.trim()}`); tags.clear(); line = null; break; } - } - if (line === null) continue; + // Whitelist: reject every tag whose semantics we do not implement. + const unknown = [...tags.keys()].filter((t) => !SUPPORTED_TAGS.has(t)); + if (unknown.length) { errors.push(`${lineNo}: unsupported template tag (${unknown.join("|")}): ${raw.trim()}`); continue; } if (line[0] === '"') { errors.push(`${lineNo}: quoted (pattern) symbol needs the binary to expand: ${raw.trim()}`); continue; } // Architecture selectors. let archOk = true; @@ -221,7 +240,7 @@ function parseSymbolsFile(path, { arch = null } = {}) { const sym = ident.slice(0, at), node = ident.slice(at + 1); if (!node) { errors.push(`${lineNo}: empty version node: ${raw.trim()}`); continue; } if (!DEB_VERSION_RE.test(minver)) { errors.push(`${lineNo}: minimum-version is not a Debian version: ${raw.trim()}`); continue; } - cur.rows.push({ sym, node, minver }); + cur.rows.push({ sym, node, minver, optional: tags.has("optional"), lineNo }); } return { blocks, errors }; } @@ -245,17 +264,17 @@ function elfTables(file) { const dynOut = readelf(["-W", "-d"], file); if (symOut === null || verOut === null || dynOut === null) return null; - const syms = []; // {idx, bind, ndx, name, verName (from name@VER), hiddenFromName} + const syms = []; // {idx, bind, ndx, name, verName (from name@VER), defaultFromName (@@)} for (const raw of symOut.split("\n")) { const m = raw.match(/^\s*(\d+):\s+\S+\s+\S+\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)?(?:\s+\((\d+)\))?\s*$/); if (!m) continue; const [, idx, , bind, , ndx, name0] = m; if (!name0) continue; - let name = name0, verName = null; + let name = name0, verName = null, defaultFromName = null; const dd = name0.indexOf("@@"), d = name0.indexOf("@"); - if (dd >= 0) { name = name0.slice(0, dd); verName = name0.slice(dd + 2); } - else if (d > 0) { name = name0.slice(0, d); verName = name0.slice(d + 1); } - syms.push({ idx: +idx, bind, ndx, name, verName }); + if (dd >= 0) { name = name0.slice(0, dd); verName = name0.slice(dd + 2); defaultFromName = true; } + else if (d > 0) { name = name0.slice(0, d); verName = name0.slice(d + 1); defaultFromName = false; } + syms.push({ idx: +idx, bind, ndx, name, verName, defaultFromName }); } // .gnu.version: index -> {ver, hidden} @@ -296,14 +315,24 @@ function elfTables(file) { return { syms, versym, verdef, verneed, soname, needed, hasVersioning: versym.size > 0 }; } -// Provides: defined dynamic symbols with their exact version node. +// Provides: defined dynamic symbols with their exact version node and their +// default-version binding (Sol P1b): +// binding = "default" an unversioned reference binds to it: the `@@` +// default (versym hidden bit clear), or the oldest +// version node (verdef index 2), which the loader +// accepts for legacy unversioned references even +// when hidden (glibc dl-lookup: index < 3 is taken +// before the hidden test; fixture D10c proves it) +// = "nondefault" hidden (`@`) at verdef index >= 3: an unversioned +// reference does NOT bind to it (fixture D10) +// `Base` is always "default". function elfProvides(t) { - const rows = []; // {sym, node} + const rows = []; // {sym, node, binding} const problems = []; for (const s of t.syms) { if (s.ndx === "UND" || s.ndx === "Ndx") continue; if (s.bind !== "GLOBAL" && s.bind !== "WEAK") continue; // LOCAL never exported - let node; + let node, binding = "default"; if (!t.hasVersioning) node = "Base"; else { const v = t.versym.get(s.idx); @@ -314,13 +343,31 @@ function elfProvides(t) { node = t.verdef.get(v.ver); if (!node) { problems.push(`dynsym ${s.idx} (${s.name}) versym ${v.ver} has no verdef entry`); continue; } if (s.verName && s.verName !== node) problems.push(`dynsym ${s.idx}: name says @${s.verName} but verdef index says ${node}`); + if (s.defaultFromName !== null && s.defaultFromName === v.hidden) + problems.push(`dynsym ${s.idx} (${s.name}@${node}): name says ${s.defaultFromName ? "@@ default" : "@ hidden"} but versym hidden bit says ${v.hidden ? "hidden" : "default"}`); + binding = (!v.hidden || v.ver === 2) ? "default" : "nondefault"; } } - rows.push({ sym: s.name, node }); + rows.push({ sym: s.name, node, binding }); } return { rows, problems }; } +// Exact export map of an ELF: "sym@node" -> binding. Used to cross-check a +// `.symbols` file against the binary it describes (Sol P1c). +function elfExportMap(path, what) { + const t = elfTables(path); + if (!t) die(`${what}: --elf ${path}: cannot read (missing file or readelf failure)`); + const { rows, problems } = elfProvides(t); + if (problems.length) die(`${what}: --elf ${path}: inconsistent version tables:\n ${problems.slice(0, 10).join("\n ")}`); + const map = new Map(); + for (const { sym, node, binding } of rows) { + const k = `${sym}@${node}`; + if (!map.has(k) || binding === "default") map.set(k, binding); + } + return { soname: t.soname || path, map }; +} + // Requires: undefined dynamic symbols, each attributed to (file, node) via its // version INDEX. Never keyed by node name. function elfRequires(t) { @@ -347,7 +394,7 @@ function elfRequires(t) { // --------------------------------------------------------------------------- // Output sink. // --------------------------------------------------------------------------- -const STORES = ["symprov", "symreq", "needed", "evidence", "releases"]; +const STORES = ["symprov", "symreq", "needed", "evidence", "releases", "replaces"]; function makeSink(outDir, toStdout) { const buffers = Object.fromEntries(STORES.map((s) => [s, []])); @@ -370,7 +417,7 @@ function makeSink(outDir, toStdout) { } function parseArgs(rest) { - const opts = { out: null, stdout: false, append: false, release: null, arch: null, positional: [] }; + const opts = { out: null, stdout: false, append: false, release: null, arch: null, elf: null, positional: [] }; for (let i = 0; i < rest.length; i++) { const a = rest[i]; if (a === "--out") opts.out = rest[++i]; @@ -378,70 +425,99 @@ function parseArgs(rest) { else if (a === "--append") opts.append = true; else if (a === "--release") opts.release = rest[++i]; else if (a === "--arch") opts.arch = rest[++i]; + else if (a === "--elf") opts.elf = rest[++i]; else opts.positional.push(a); } if (!opts.out && !opts.stdout) opts.stdout = true; return opts; } -function requireRelease(opts, guess, what) { - const rel = opts.release || guess; +// One Debian-version gate for every evidence release, whichever command +// supplied it (Sol P2): --release, dpkg-query, or the `releases` axis. +function validRelease(rel, what) { if (!rel) die(`${what}: evidence release unknown (not owned by an installed package); pass --release `); - if (!DEB_VERSION_RE.test(rel)) die(`${what}: --release '${rel}' is not a Debian version`); + if (!DEB_VERSION_RE.test(rel)) die(`${what}: release '${rel}' is not a Debian version`); return rel; } +function requireRelease(opts, guess, what) { + return validRelease(opts.release || guess, what); +} + // --------------------------------------------------------------------------- // Commands. // --------------------------------------------------------------------------- -function cmdSymbolsFile(opts, path, sink, { batch = false } = {}) { +// A `.symbols` file is ingested ATOMICALLY: any error in any block returns +// null (nothing is added to the sink), in single-file and batch mode alike. +function cmdSymbolsFile(opts, path, sink, elf = null) { const { blocks, errors } = parseSymbolsFile(path, { arch: opts.arch }); - if (errors.length) { - process.stderr.write(`ingest_symbols: ${path}: ${errors.length} unsupported/malformed row(s) -- rejecting the file:\n`); - for (const e of errors.slice(0, 20)) process.stderr.write(` ${e}\n`); - if (errors.length > 20) process.stderr.write(` ... ${errors.length - 20} more\n`); - return null; - } - if (!blocks.length) { process.stderr.write(`ingest_symbols: ${path}: no soname blocks\n`); return null; } - let n = 0; + if (!blocks.length && !errors.length) errors.push("no soname blocks"); + const rows = []; // deferred until the whole file is clean const sonames = []; + let dropped = 0; for (const b of blocks) { const guess = b.package ? dpkgVersion(b.package) : null; const rel = opts.release || guess; - if (!rel) { - process.stderr.write(`ingest_symbols: ${path}: ${b.soname}: evidence release unknown (package ${b.package || "#PACKAGE#"} not installed); pass --release\n`); - if (!batch) return null; else continue; + if (!rel) { errors.push(`${b.soname}: evidence release unknown (package ${b.package || "#PACKAGE#"} not installed); pass --release`); continue; } + if (!DEB_VERSION_RE.test(rel)) { errors.push(`${b.soname}: release '${rel}' is not a Debian version`); continue; } + const xcheck = elf && elf.soname === b.soname ? elf.map : null; + const seen = new Set(); + for (const { sym, node, minver, optional, lineNo } of b.rows) { + const k = `${sym}@${node}`; + if (xcheck) { + const binding = xcheck.get(k); + if (binding === undefined) { + if (optional) { dropped++; process.stderr.write(`ingest_symbols: ${path}:${lineNo}: (optional) ${k} not exported by ${opts.elf}; row dropped\n`); continue; } + errors.push(`${lineNo}: ${k} is in the .symbols file but not exported by --elf ${opts.elf}`); + continue; + } + seen.add(k); + rows.push([`${b.soname}|${k}`, ["since", minver, rel, binding]]); + } else { + if (optional) { errors.push(`${lineNo}: (optional) ${k} needs an ELF cross-check (--elf ) to count as an export`); continue; } + rows.push([`${b.soname}|${k}`, ["since", minver, rel, node === "Base" ? "default" : "unproven"]]); + } } - for (const { sym, node, minver } of b.rows) sink.add("symprov", `${b.soname}|${sym}@${node}`, ["since", minver]); - sink.add("evidence", `provides|${b.soname}`, ["symbols", rel, "complete", path]); - n += b.rows.length; + if (xcheck) for (const k of xcheck.keys()) if (!seen.has(k)) errors.push(`${b.soname}: ${k} is exported by --elf ${opts.elf} but absent from the .symbols file (evidence would not be complete)`); + rows.push([`provides|${b.soname}`, ["symbols", rel, "complete", path], "evidence"]); sonames.push(b.soname); } - return { n, sonames }; + if (errors.length) { + process.stderr.write(`ingest_symbols: ${path}: ${errors.length} problem(s) -- rejecting the whole file, nothing written:\n`); + for (const e of errors.slice(0, 20)) process.stderr.write(` ${e}\n`); + if (errors.length > 20) process.stderr.write(` ... ${errors.length - 20} more\n`); + return null; + } + let n = 0; + for (const [k, v, store] of rows) { sink.add(store || "symprov", k, v); if (!store) n++; } + return { n, sonames, dropped }; } const [cmd, ...rest] = process.argv.slice(2); const opts = parseArgs(rest); +if (opts.release !== null) validRelease(opts.release, cmd); if (cmd === "symbols-file") { const path = opts.positional[0]; if (!path) die("symbols-file: missing path", 2); + const elf = opts.elf ? elfExportMap(opts.elf, `symbols-file ${path}`) : null; const sink = makeSink(opts.out, opts.stdout); - const r = cmdSymbolsFile(opts, path, sink); + const r = cmdSymbolsFile(opts, path, sink, elf); if (!r) process.exit(EXIT_EVIDENCE); sink.flush(opts.append); - process.stderr.write(`symbols-file ${path}: symprov=${r.n} sonames=${r.sonames.length} [${r.sonames.slice(0, 6).join(", ")}${r.sonames.length > 6 ? ", ..." : ""}]\n`); + process.stderr.write(`symbols-file ${path}: symprov=${r.n} sonames=${r.sonames.length} [${r.sonames.slice(0, 6).join(", ")}${r.sonames.length > 6 ? ", ..." : ""}]${elf ? ` cross-checked=${elf.soname} optional-dropped=${r.dropped}` : ""}\n`); } else if (cmd === "symbols-dir") { const dir = opts.positional[0]; if (!dir || !opts.out) die("symbols-dir: needs and --out DIR", 2); + if (opts.elf) die("symbols-dir: --elf applies to a single file; use symbols-file", 2); const files = readdirSync(dir).filter((f) => f.endsWith(".symbols")).map((f) => join(dir, f)); mkdirSync(opts.out, { recursive: true }); for (const s of ["symprov", "evidence"]) writeFileSync(join(opts.out, s + ".jsonl"), ""); let total = 0, ok = 0, rejected = 0; for (const f of files) { const sink = makeSink(opts.out, false); - const r = cmdSymbolsFile(opts, f, sink, { batch: true }); - if (!r) { rejected++; continue; } + const r = cmdSymbolsFile(opts, f, sink); + if (!r) { rejected++; continue; } // atomic: no block of a rejected file is written sink.flush(true); total += r.n; ok++; } @@ -458,16 +534,16 @@ if (cmd === "symbols-file") { const { rows, problems } = elfProvides(t); if (problems.length) die(`elf ${path}: inconsistent version tables:\n ${problems.slice(0, 10).join("\n ")}`); const sink = makeSink(opts.out, opts.stdout); - const seen = new Set(); - for (const { sym, node } of rows) { + const seen = new Map(); // one identity per sym@node; "default" wins if both appear + for (const { sym, node, binding } of rows) { const k = `${so}|${sym}@${node}`; - if (seen.has(k)) continue; // @ and @@ of the same node are one identity - seen.add(k); - sink.add("symprov", k, ["at", rel]); + if (!seen.has(k) || binding === "default") seen.set(k, binding); } + let nd = 0; + for (const [k, binding] of seen) { sink.add("symprov", k, ["at", rel, binding]); if (binding === "nondefault") nd++; } sink.add("evidence", `provides|${so}`, ["elf", rel, "complete", path]); sink.flush(opts.append); - process.stderr.write(`elf ${path}: soname=${so} release=${rel} symprov=${seen.size}\n`); + process.stderr.write(`elf ${path}: soname=${so} release=${rel} symprov=${seen.size} (nondefault=${nd})\n`); } else if (cmd === "requires") { const path = opts.positional[0]; if (!path) die("requires: missing path", 2); @@ -497,12 +573,22 @@ if (cmd === "symbols-file") { } else if (cmd === "releases") { const [so, ...vers] = opts.positional; if (!so || !vers.length) die("releases: needs ...", 2); - for (const v of vers) if (!DEB_VERSION_RE.test(v)) die(`releases: '${v}' is not a Debian version`); + for (const v of vers) validRelease(v, `releases ${so}`); const sink = makeSink(opts.out, opts.stdout); for (const v of vers) sink.add("releases", so, v); sink.flush(opts.append); process.stderr.write(`releases ${so}: ${vers.length} candidate(s)\n`); +} else if (cmd === "replaces") { + // Declared soname succession (Sol P2d): offering to a binary whose + // DT_NEEDED names is a soname_mismatch only under this relation; any + // other name that is not NEEDED is simply not_needed (no stem heuristic). + const [so, ...olds] = opts.positional; + if (!so || !olds.length) die("replaces: needs ...", 2); + const sink = makeSink(opts.out, opts.stdout); + for (const o of olds) sink.add("replaces", so, o); + sink.flush(opts.append); + process.stderr.write(`replaces ${so}: ${olds.join(", ")}\n`); } else { - process.stderr.write("usage: ingest_symbols.mjs symbols-file|symbols-dir|elf|requires|releases [--release V] [--arch A] [--out DIR] [--append] [--stdout]\n"); + process.stderr.write("usage: ingest_symbols.mjs symbols-file|symbols-dir|elf|requires|releases|replaces [--release V] [--arch A] [--elf lib.so] [--out DIR] [--append] [--stdout]\n"); process.exit(2); } diff --git a/examples/pkg_resolver/abi/run_abi_verify.sh b/examples/pkg_resolver/abi/run_abi_verify.sh index cf87b05f0..656ab2daa 100755 --- a/examples/pkg_resolver/abi/run_abi_verify.sh +++ b/examples/pkg_resolver/abi/run_abi_verify.sh @@ -5,15 +5,20 @@ # run_abi_verify.sh -- build the symbol-level ABI store from THIS machine's # real Ubuntu/Debian data plus the review fixtures, then verify. # -# 1. Real data: libc6 + libselinux1 `.symbols` -> symprov (since bounds); -# /bin/ls -> symreq (attributed via the ELF version index) + NEEDED; -# the release axis from `apt-cache madison` + dpkg (real candidates). +# 1. Real data: libc6 + libselinux1 `.symbols` -> symprov (since bounds; +# libc6 cross-checked against libc.so.6 with --elf so default-version +# binding is proven); /bin/ls -> symreq (attributed via the ELF version +# index) + NEEDED; the release axis from `apt-cache madison` + dpkg (real +# candidates); a declared soname succession (libc.so.7 replaces libc.so.6). # 2. Cross-check: readelf(libc.so.6) vs `.symbols` on EXACT sym@node -# identity (must be 100%), plus the corrected per-name earliest-row -# comparison the old script got wrong (must also be 100%). +# identity (must be 100%) and per-name node sets (must be 100%); the +# empty-input and dropped-node negatives must FAIL (Sol P2c/P3). # 3. ELF fixtures (gcc): foo@LIB_1 vs foo@LIB_2 under one soname (with the # loader as ground truth), COMMON_1 cross-attribution, PUBLIC + -# unversioned obligations, a missing ELF, template `.symbols` rejects. +# unversioned obligations, a missing ELF, non-default-only exports (Sol +# P1b, loader as ground truth), `.symbols` template accept/reject cases +# including (optional) with and without an ELF cross-check (Sol P1c), +# unknown tags, bad --release, and batch-mode atomicity (Sol P2). # 4. test_abi.pl: the Prolog assertions over all of the above. # # Everything lands in ./.out (gitignored). Requires: node, readelf, swipl; @@ -30,6 +35,17 @@ BINARY="${ABI_BINARY:-/bin/ls}" LIBSO="${ABI_LIBSO:-/lib/x86_64-linux-gnu/libc.so.6}" ARCH="$(dpkg --print-architecture 2>/dev/null || echo amd64)" +fail() { echo "FAIL: $*" >&2; exit 1; } + +# expect_reject