Keep what a recomputation needs, and say which Julia and BLAS ran (0.8.7) - #58
Merged
Merged
Conversation
…8.7) A revision was shown on 2026-09-25 not to be rebuildable from itself: with an empty depot, an empty HOME and no network, a restore needed git-URL packages that only an SSH remote could give back, and `julia` started another version than the one recorded. The same day, one point recomputed at BLAS thread counts 1, 8 and 18 gave three different last bits, and the count was nowhere in the observation. - A run-start observation keeps every package the process loaded from a depot as a `depot` root, whose head is the tree the Manifest pins. A render does not by default: its plotting stack is not what a recomputation needs. - Contents are kept for any file up to `materialize_limit` (1 MiB), not only .jl and .toml, which still are at any size. An extension list dropped the tables and scripts a package reads, without a word. - The study is the active environment's directory when the config lies in it, so a repository of many studies captures one; otherwise it is the config's repository, as before. - The vault's own output is never source, even inside the study. - `julia` records the binary's digest, bindir, platform, CPU, BLAS's thread count and its libraries. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
|
📚 Docs preview: https://qatlashub.github.io/DataVault.jl/previews/PR58/ (updates on each push to this PR) |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
… ran Found by recomputing a real study from its revision with an empty depot: JLD2 loads Zstd_jll, whose library lives in `artifacts/<tree>`, which no package tree holds; the library's `libzstd.so -> libzstd.so.1` links had no content to lay out; and `julia compute.jl` does not appear in `Base._included_files`, so nothing said which script to run. - Each artifact a loaded depot package's Artifacts.toml selects for this platform, where installed, is an `artifact:<name>:<tree>` root. - A symlink's target is kept as its blob. - Depot and artifact roots are kept whole, up to the hash limit, not only up to the 1 MiB meant for a study's own files. - `program` records `abspath(PROGRAM_FILE)`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
From the review of this PR. `_artifact_roots` caught a TOML or `artifact_meta` failure and moved on, leaving the observation to look complete while a JLL's library was missing from it. It is now a note, and the inventory is then not complete, as a root that cannot be listed already was. Tests for the paths codecov found untested: an unreadable Artifacts.toml, a lazy artifact never fetched, a depot package kept whole under a 1-byte materialize limit, and `program` holding the script's path. The docstring now says what is kept for depot and artifact roots and for symlinks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A revision turned out not to be rebuildable from its own content (measured 2026-09-25,
Vault/infra/REGISTRY-REVIEW-ROUND3.md§8.1). This PR changes whatobserve_sourceskeeps so that it can be. The other half is QAtlasHub/Archeion.jl#78 (restore/verify).Roots
config: the active environment's directory when the config lies inside it. Otherwise, as before, the git toplevel or the config's directory.dirtyis read for that subtree.depot(run-start only by default,depot_packages = (phase == "run-start")): every package the process loaded from a depot. Itsheadis the tree hash the Manifest pins.artifact: for each loaded depot package, the artifacts itsArtifacts.tomlselects for this platform, where installed. Each is namedartifact:<name>:<tree>, withhead= tree. If an artifact cannot be resolved, a note is written andinventory_complete = false.vault.outdiris excluded from every root, and the walk does not enter it.Contents
.jl/.tomlfiles are kept at any size, as before.materialize_limit(default 1 MiB, recorded instate.toml).hash_limit, because a library is routinely larger than 1 MiB.The observation record
juliagainsbindir,executable_sha256,platform,cpu_name,blas_threadsandblas_libraries. BLAS's thread count changes the last bits: counts 1, 8 and 18 each gave a different result, and 18 matched bitwise.programrecordsabspath(PROGRAM_FILE), becausejulia <file>is not amongmain_files.New stdlib dependencies:
LinearAlgebraandArtifacts.files.tsvkeeps its six columns with the digest last.Cost
A run-start observation now reads and hashes the loaded packages. With 21 packages that took about 0.5–0.7 s per process. That broke a SweepRunner test which assumed
run!starts within 50 ms; it is fixed in QAtlasHub/SweepRunner.jl#57. Caching per-(uuid, tree) to avoid re-hashing on every worker is left for a follow-up.Checked
Locally, only
test/vault/test_observe.jlwas run: 15 testsets pass. The full suite is left to CI. End to end, a real 5-point WilsonNRG study recorded with this was recomputed byArcheion.verifyfrom its revision alone, with an empty depot, an empty HOME and an unshared network. All 5 result files were byte-identical.🤖 Generated with Claude Code