Skip to content

Solaris 9 SPARC: seed-based CI, and raw device enumeration - #81

Merged
danifunker merged 14 commits into
mainfrom
sol9-path-fix
Sep 10, 2026
Merged

danifunker merged 14 commits into
mainfrom
sol9-path-fix

Conversation

@danifunker

@danifunker danifunker commented Sep 9, 2026

Copy link
Copy Markdown
Owner

Two related pieces of Solaris 9 SPARC work. The release pipeline leg had never run end to
end, and rb-cli could not see a local disk at all.

Part 1 — the release pipeline builds from a seed, not a container

CI no longer builds a toolchain or uses Docker. SOL9_SPARC64_SYSROOT_URL holds a seed
the cross toolchain, its sysroot, mrustc and both prebuilt standard libraries, 47 MB — which
the job unpacks at / before running scripts/build-sol9.sh.

Before After
CI artifact 7 GB image 47 MB seed
CI time ~100 min ~20 min
External deps at build time Debian base, apt, ftp.gnu.org ×2 one download
Dockerfiles on the release path 3 0

The container rebuilt, every run, an environment that already exists on a developer machine.
It was also a liability: Debian 11 reached EOL mid-development and broke the build twice in
one evening — an expired Release file, then its security packages 404ing out of the pool,
with archive.debian.org carrying no bullseye-security to fall back on.

Two scripts replace the three Dockerfiles: scripts/build-sol9-toolchain.sh builds binutils
and GCC 4.9.4 natively (GitHub's ubuntu-latest and a current desktop are both Ubuntu 24.04
/ GCC 13.3, which 4.9.4 builds under), and scripts/pack-sol9-seed.sh packs the seed.

Verified in a bare ubuntu:24.04 container — eight apt packages plus the seed — and then
in the real pipeline, whose artifact I downloaded and ran on a Sun Blade 2500: it produced a
zstd backup that restored byte-identically with fsck clean.

Part 2 — rb-cli can see local disks

Solaris fell into the catch-all arms of src/os/mod.rs, so show devices listed nothing.
There is now a src/os/solaris.rs beside linux.rs and macos.rs — it has to live there,
because rb-cli-sol9 carries no Rust source of its own and compiles ../src/lib.rs.

Enumeration deliberately opens no device. A wedged driver blocks in open(2) and ignores
O_NONBLOCK — measured on the Blade, where a USB stick behind scsa2usb left processes that
kill -9 could not touch. Solaris' own tools have the same flaw: format(1M) and
rmformat(1) both hang on that disk, and Oracle warns never to point format at a USB
drive. The first version of this module did exactly that, and hung.

So it reads what the drivers publish — the same source iostat -E uses, and the only one
that keeps working: readlink into /devices, /etc/path_to_inst for the driver instance,
then libkstat for Size, Vendor and Product. A device that has stopped answering
reports zero and is skipped. libdiskmgt, which format and the installer use, is 32-bit
only on Solaris 9
and unavailable to a sparcv9 binary.

Only kstat_named_t is modelled; kstat_t stays opaque because kstat_data_lookup returns
a pointer into the library's own buffer, which removes most of the FFI risk.

The whole-disk node differs by architecture and both are probed — SPARC has only SMI slices
and calls slice 2 the whole disk, x86 wraps them in an fdisk table and calls it p0. That is
commented and unit-tested in both spellings, because Oracle's USB docs show p0 throughout
(those examples are x86) and someone would otherwise "fix" the SPARC path to match.

Verified on hardware with the wedged stick still attached — both healthy disks listed
with correct sizes and inquiry strings, the bad one skipped, in 112 ms:

"name": "c0t0d0",  "media_name": "QUANTUM BlueSCSI UltraW",   "bus_protocol": "sd"
"name": "c0t6d0",  "media_name": "BlueSCSI CD-ROM CDU-55S",   "bus_protocol": "sd"

Six latent bugs this flushed out

Each would have fired the first time the pipeline ran for real:

  1. SOL9_LIBGCC unset in the container — defaulted to a path that does not exist there, so dist would have refused to package.
  2. mksysroot.sh disk mode had never workedrb-cli get -r lays a directory source out under the destination, producing usr/include/include/stdio.h.
  3. That mode left every symlink as a text file, so -lc would find a 12-byte libc.so. Now replayed — 2,400 links on a real disk image, all seven link-time libraries resolving.
  4. MRUSTC_TARGET_VER unset in the container path, so mrustc fell back to 1.29 mode and died parsing 1.74's asm!.
  5. The cross toolchain was never on PATH for sol9libs, so mrustc used the host x86 gcc and failed with bad value 'v9' for '-mtune='.
  6. libc was declared for linux and macos only. Solaris built anyway because rb-cli-sol9 declares it separately, but cargo check --target sparcv9-sun-solaris could not type-check the module — that check is now 24 seconds instead of a 20-minute mrustc round trip.

Known limitations

Tracked in docs/solaris9-open-items.md, which separates known-wrong from untested. The
headlines:

  • USB mass storage is beta. scsa2usb wedges on devices that do not implement the full SCSI command set. scripts/solaris-usb-unblock.sh diagnoses it without touching a disk and prints Oracle's remedy, applying it only with --apply.
  • The write path still bails for Solaris. Enumeration and read are proven; writing needs a device that is not the root disk.
  • A CD-ROM reports removable: no. A failed DKIOCREMOVABLE is treated as "not removable" — known-wrong rather than verified.
  • Solaris 10 rests on an untested assumption — forward binary compatibility should make the Solaris 9 artifact run there, but nobody has tried it.

Also simplifies a boolean in src/fs/sfs.rs that a newer clippy flags as nonminimal_bool,
which was failing the pre-commit hook for every commit in the repo.

🤖 Generated with Claude Code

@danifunker danifunker changed the title Solaris 9 SPARC in the release pipeline, built from a seed Solaris 9 SPARC: seed-based CI, and raw device enumeration Sep 9, 2026
danifunker and others added 12 commits September 10, 2026 03:54
mrustc spawns `$SOL9_TARGET-gcc` by name from the target spec, so the stdlib
build needs ~/sol9-toolchain/opt/bin on PATH. The script never added it, so the
stage only worked inside docker/sol9.Dockerfile, where the toolchain is already
there. On a local toolchain it silently fell through to the host x86 gcc and
died with "bad value 'v9' for '-mtune=' switch" -- an error that reads like a
broken target spec rather than a missing PATH entry.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rename the CI secret to SOL9_SPARC64_SYSROOT_URL. Solaris also shipped on x86
(rusty-backup already parses Solaris-x86 VTOCs), so an arch-less name would have
to be renamed the moment a second Solaris target appears. Nothing is orphaned:
the old name was never created as a repo secret.

Three fixes found while exercising the path end to end:

* The build container never set SOL9_LIBGCC, whose default lives under
  $SOL9_TOOLCHAIN -- a path that does not exist in the image. The `dist` stage
  refuses to package without it, so the pipeline job would have died the first
  time it ran with the secret set. It never did run, which is why this stayed
  hidden.

* `mksysroot.sh disk` passed the directory itself as the destination, but
  `rb-cli get -r` lays a directory source out *under* the destination, so the
  result was usr/include/include/stdio.h and the trailing sanity check failed.
  Disk mode has never worked; only the media path was exercised.

* The same mode left every symlink as the text file rb-cli writes in place of
  one, so `-lc` would find a 12-byte libc.so and fail far from the cause. Parse
  the paths rb-cli reports and recreate the links. Verified against a real
  Solaris disk image: 2400 links recreated, and libc/libm/libsocket/libnsl/
  librt/libdl/libpthread all resolve.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every other platform leg is a matrix over `arch` and uploads
`rb-cli-<os>-<arch>-<version>.tar.gz`; the Solaris job was the one exception,
producing a bare `rb-cli-sol9` artifact with no arch and no version. Bring it
into line: one matrix row, `arch: sparc64`, artifact
`rb-cli-solaris9-sparc64-<version>.tar.gz`.

The row carries the mrustc triple and threads it into the container as a build
arg and an environment variable, so the matrix value is what actually selects
the target rather than the script's default. Solaris also shipped on x86 and
the sysroot secret is already named per-arch, so a second target is a row here
rather than another job.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two fixes to the first real run of this job.

The toolchain no longer has to be built in CI. The image already contains the
sysroot, so it is exactly as unpublishable and exactly as hostable as the
sysroot is; hosting it turns a ~40 minute GCC 4.9.4 build into a 262 MB
download. Set SOL9_SPARC64_TOOLCHAIN_URL to a `docker save mrustc-sol9-cross |
gzip` and the job fetches it; building from the sysroot stays the fallback when
that secret is absent, and the run cache still short-circuits both. With the
toolchain hosted the sysroot secret is optional, so the gate now accepts either.

The sysroot layout check named `usr/include/stdio.h` as an exact tar member,
which rejects a perfectly good `./usr/...` tarball -- what `tar -c .` produces,
as against `tar -C / -c usr`. Match both, and on failure print the tarball's
top-level entries so a wrong upload identifies itself instead of costing a
round trip.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
bullseye reached EOL and its security suite is mid-migration: deb.debian.org
still serves the Release file but the packages behind it 404, and
archive.debian.org has no bullseye-security to fall back to. The base image
already carries deb11u14 packages that archive's main (deb11u11) cannot
satisfy, so pointing at the archive gives unmet dependencies instead. There is
no version of that pin left to hold, so move to debian:12-slim.

The Dockerfile always said the pin was for reproducibility rather than
necessity -- GCC 4.9.4 is verified building under host GCC 13.3, and Debian 12
gives 12.2, inside that range.

Also stop `tar | grep -q` killing tar with SIGPIPE, which printed "tar: stdout:
write error" directly above the real failure and read like the cause.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SOL9_SPARC64_SYSROOT_URL now accepts either a sysroot or a `docker save` of the
toolchain image, and the job sniffs the download to decide: manifest.json means
load it, usr/include/stdio.h means build from it, anything else is rejected with
the tarball's top-level entries so a wrong upload names itself.

Both artifacts carry Sun's headers, so both are equally unpublishable and
equally hostable -- which made a second secret pure bookkeeping, and the wrong
file has already been uploaded to the wrong one twice today. A saved image also
skips the ~40 minute GCC build entirely.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI was reconstructing, on every run, an environment that already exists on a
developer machine: ~40 minutes building GCC 4.9.4 from source in a Debian image,
then ~40 more rebuilding mrustc and both standard libraries. It now downloads a
seed -- the cross toolchain, its sysroot, mrustc and the two prebuilt stdlibs --
unpacks it at /, and runs scripts/build-sol9.sh. 47 MB against 7 GB for the
equivalent image, and about 20 minutes against 100.

The container was also a liability. Debian 11 reached EOL mid-flight and broke
the build twice in one evening, first through an expired Release file and then
through its security packages 404ing out of the pool, with archive.debian.org
carrying no bullseye-security to fall back on. None of that touches a seed.

Two new scripts replace the three Dockerfiles:

* build-sol9-toolchain.sh builds binutils and GCC 4.9.4 natively. GitHub's
  ubuntu-latest and a current desktop are both Ubuntu 24.04 with GCC 13.3, and
  4.9.4 builds under that with -std=gnu++98 -fpermissive -- so the toolchain is
  built on the OS that will run it. (A 4.9.4 built on Debian 12 also runs
  unmodified on 24.04; glibc is forward compatible.)
* pack-sol9-seed.sh packs the seed. It keeps *.rlib.hir: the .rlib is a 0-byte
  marker and .hir is where mrustc keeps the crate metadata, so stripping it as
  an intermediate yields a seed where every dependent crate dies with "Unable to
  deserialise crate metadata". Dropping the emitted *.rlib.c is the size win.

mksysroot.sh moves to scripts/mksysroot-solaris.sh -- extracting a sysroot was
never docker-specific.

The job also now takes a rust toolchain the way the rest of the workflow does:
Ubuntu's packaged cargo is too old for `cargo vendor` here, since a crate in the
graph needs edition2024.

Verified end to end in a bare ubuntu:24.04 container -- eight apt packages plus
the seed -- producing a 16 MB bundle whose rb-cli is SPARC V9 with only the
three weak GCC/GCJ hooks unresolved, matching the binary this machine builds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Solaris fell into the catch-all arms of src/os/mod.rs, so `show devices` listed
nothing and device writes were refused outright. There is now a src/os/solaris.rs
beside linux.rs and macos.rs -- it has to live there, because rb-cli-sol9 carries
no Rust source of its own and compiles ../src/lib.rs.

Enumeration deliberately opens no device. A wedged driver blocks in open(2) and
ignores O_NONBLOCK: measured on a Sun Blade 2500 where a USB stick behind
scsa2usb left processes that kill -9 could not touch. Solaris' own tools have the
same flaw -- format(1M) and rmformat(1) both hang on that disk, and Oracle warns
never to point format at a USB drive -- so listing by opening is a hang waiting
for one bad device. The first version of this module did exactly that and hung.

Instead the module reads what the drivers publish, which is what iostat -E does
and the only source that keeps working: readlink /dev/rdsk/<disk> into /devices,
/etc/path_to_inst for the driver instance, then libkstat for Size, Vendor and
Product. A device that has stopped answering reports zero and is skipped.
libdiskmgt, which format and the installer use, is 32-bit only on Solaris 9 and
so unavailable to a sparcv9 binary.

Only kstat_named_t is modelled; kstat_t stays opaque because kstat_data_lookup
returns a pointer into the library's own buffer, which removes most of the FFI
risk. libc alone otherwise: rb-cli-sol9 drops nix, which linux.rs and macos.rs
both use.

The whole-disk node differs by architecture and both are probed: SPARC has only
SMI slices and calls slice 2 the whole disk, while x86 wraps them in an fdisk
table and calls it p0. Oracle's USB documentation shows p0 throughout because
those examples are x86, so this is commented and unit-tested in both spellings
to stop someone "fixing" it to match.

Cargo.toml now names libc for solaris as well. rb-cli-sol9 always declared it,
so the target built anyway, but without it `cargo check --target
sparcv9-sun-solaris` cannot type-check this file -- which turns a 20-minute
mrustc round trip into 24 seconds.

Verified on the hardware with the wedged stick still attached: both healthy
disks listed with correct sizes and inquiry strings, the bad one skipped, in
112 ms.

USB mass storage is documented as beta, in docs/solaris-raw-devices.md and in
the shipped bundle's README. scripts/solaris-usb-unblock.sh reports the state
without touching a disk and prints Oracle's scsa2usb remedy, applying it only
with --apply.

Also simplifies a boolean in src/fs/sfs.rs that a newer clippy flags as
nonminimal, which was failing the pre-commit hook for every commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collects the known-incomplete and known-wrong parts in one place rather than
leaving them spread across commit messages: raw-device writing still bails,
free space always reports zero, a CD-ROM's removable flag is wrong because a
failed DKIOCREMOVABLE is read as "not removable", USB is unproven beyond one
stick that wedges, and Solaris 10 rests on a forward-compatibility assumption
nobody has tested.

Says outright which items are untested rather than incomplete, so the two are
not confused later.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s root

The Solaris job failed on one run and passed on a re-run of the same commit and
seed. minicargo can schedule a crate's build script twice at once -- its own
progress line showed `serde_core (build)` listed twice -- and the second worker
then execs the binary while the first is still linking it, before the linker has
set the executable bit. The error is "Unable to run process ... Permission
denied", which reads like a missing chmod and is really a window where the mode
is not set yet.

The job now retries the sol9 stage once. That works because the build is
incremental, so the retry resumes from the crates already finished, but it is a
mitigation: the race is upstream in minicargo and is tracked in
docs/solaris9-open-items.md rather than being quietly papered over.

Separately, pack-sol9-seed.sh stored the packing user's uid, so unpacking at /
handed /opt/mrustc to whoever holds uid 1000 on the target machine. It stores
root ownership now, which is what a tarball meant for `tar -C /` should do.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…estore

Restoring to a device on macOS ended with "syncing the target: Inappropriate
ioctl for device (os error 25)" after the data was already written. macOS
implements File::sync_all as fcntl(F_FULLFSYNC), and a raw /dev/rdiskN answers
ENOTTY -- so a restore that had completely succeeded reported failure at the
last step.

SectorAlignedWriter::sync_all now goes through sync_committed, which falls back
to fsync on ENOTTY and only reports success if that is refused the same way.
Writes to a raw device are unbuffered, so there is nothing left to push. A
regular file always accepts fsync, so this cannot silently skip flushing one --
covered by a test, because that is the property worth protecting.

Both writer types share the seam, so all four device-target call sites (restore,
restore::single, physical_write_runner, provision_runner) are fixed at once.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit inserted sync_committed between
`#[cfg(not(target_os = "windows"))]` and the struct that attribute guarded. The
cfg then applied to the helper -- so it vanished on Windows -- and left the
non-Windows SectorAlignedWriter ungated, colliding with the Windows one:
"the name `SectorAlignedWriter` is defined multiple times". Linux, macOS and
Solaris all built; every Windows leg failed.

Moving the helper above the doc comment that introduces the pair fixes it.
Verified with `cargo check --target {x86_64,i686}-pc-windows-msvc`, which is
what should have been run before pushing the previous commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
danifunker and others added 2 commits September 10, 2026 04:59
The job was continue-on-error from when it built GCC 4.9.4 from source, where a
mirror hiccup could redden a release for reasons unrelated to the code. It now
downloads a prebuilt seed, and a missing secret already skips the job -- so the
flag no longer guarded against anything except telling us the truth. A failure
here means the Solaris build is broken, and the release job waits on it.

One consequence worth naming: minicargo's build-script race can now block a
release rather than warn. The stage already retries once, which has absorbed it
so far; if it recurs past that, the lever is lowering JOBS from 4 to remove the
concurrency, at a cost in build time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SOL9_OUT defaults to $MRUSTC_DIR/output-rb-sol9, which in CI is inside the seed
unpacked at /opt/mrustc. The seed is extracted as root and carries its packer's
uid, so whether the job user can write there is an accident of which uid the
runner happens to use. Reproduced in a container as uid 1001:

    mkdir: cannot create directory '/opt/mrustc/output-rb-sol9': Permission denied

and with SOL9_OUT pointing at a writable directory the same check passes. The
seed's tree is input and should be read-only; only these two directories are
ever written, so they move to the workspace.

Whether this is also what failed run #668 is not established. That job died
executing a build script under the same tree with EACCES, which fits, but the
runs that succeeded fit a runner uid matching the seed's. The earlier "minicargo
build-script race" reading was inferred from a single passing re-run and a
duplicate progress line, which is thin evidence; the retry that appeared to fix
it more likely just resumed from state that already existed. The retry stays for
now, but the race is no longer claimed as diagnosed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@danifunker
danifunker merged commit a49eef4 into main Sep 10, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants