M2/E4/#14 — Tier 2 manifest schema - #17
Merged
Merged
Conversation
Canonical: Milestone M2 · Epic E4 · Issue #14. Deps: #11 (complete). Eligibility recomputed by parsing origin/main:docs/tis/TIS-v1.0.md §15 — complete {1..13}, eligible {#14 order 14, #15 order 15}; lowest Order selects #14. Completes Epic E4. The twelfth contract, and the one git actually carries: a Tier 2 manifest is what the repository holds in place of ~600 MB it deliberately does not (ADR-0023, E6 §19). ALMOST EVERYTHING THIS SCHEMA DOES IS A REFUSAL ADR-0023 defines three tiers with three different homes, and each must forbid the others. Expressed as prose, "a Tier 0 manifest must not carry a deposition" is a sentence someone has to read. Expressed as `if tier = 0 then not deposition`, a manifest that would redistribute another organisation's raw archive CANNOT BE WRITTEN. tier 0 retrieval descriptor required; deposition and path impossible (STD-23, E6 §11(iii) — identified, never stored, never redistributed) tier 1 deposition required; retrieval and path impossible; retention must be permanent, because a prunable Tier 1 release makes a DOI resolve to nothing tier 2 path required, and constrained so it cannot leave the repository root RETENTION IS UNREPRESENTABLE WHEN IT WOULD BE WRONG E6 §11(iv): an artifact referenced by an Evidence Binding is NEVER pruned. So `prunable` with a non-empty `referenced_by` is refused by the contract rather than discouraged in a comment — it is the one combination that would let a published claim lose the bytes underneath it. A `permanent` entry likewise cannot carry a `prune_after`: permitting both would let a manifest state a contradiction that a retention run would then arbitrate silently, the same way every time. DOI IS REQUIRED AS A FIELD AND NULLABLE IN VALUE ADR-0023's fallback degrades citability, and says the degradation is published rather than hidden. Optional would make "no DOI" and "nobody recorded whether there is one" the same document — the ingest_time reasoning of ADR-0022 applied to citability. FEATURE-LEVEL INTEGRATION — THE DIGEST IS THE SAME DIGEST tests/integration/test_manifest_with_platform.py checks the one thing no schema can: that the digest in a manifest is the digest of the bytes it claims to describe. It needs four components at once — the kernel mints it (#10, the only thing permitted to), the contract validates the release (#11), the domain holds the digest without being able to compute one (#12), and the import rules say who may do all of it (#13). Every guarantee downstream — citation, re-acquisition, supersession, evidence binding — rests on that equality, and nothing had ever checked it end to end. Also included: a manifest that is schema-valid and FALSE, pointing at bytes the store has never seen, and the demonstration that the store refuses it while the schema cannot. A FINDING ABOUT MY OWN TEST, NOT THE SCHEMA `tier: 1.0` validates, and that is correct. JSON has one number type and JSON Schema compares enum members by value, so 1.0 and 1 are the same number. The first version of that test asserted rejection and was wrong; it now asserts acceptance and records why. NO PRODUCER, NO PARSER, NO REGISTRY, NO INGEST The schema and its tests. `contexts/curation/manifest` is E6/#20's; registry/*/*.json instances are E6/#21, E7/#23, E8/#25-#26, E9/#27's; the retention gate is E6/#22's. The manifest gets no domain entity, deliberately — row 14's Unit column reads "manifest validity", a schema test, and an entity would be a producer. Two existing assertions updated because #14 changes the contract set: the expected count 11 -> 12, and manifest added to the documented no-domain-entity exclusions. VERIFICATION unit (kernel + domain) 322 passed tests/architecture 562 passed (+102) tests/integration 71 passed (+8) research/tests/v2 188 passed contracts gate: 12 schemas, 11 compared, every change additive links / imports gates clean CI: no workflow change. Both new files land in jobs that already run. Still eight. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rexy-5097
added a commit
that referenced
this pull request
Jul 30, 2026
Canonical: Milestone M3 · Epic E5 · Issue #15. Deps: #11 (complete). Eligibility recomputed by parsing origin/main:docs/tis/TIS-v1.0.md §15 — complete {1..14}, eligible {#15, #22, #23, #25, #26, #27, #39}; lowest Order selects #15. First issue of Milestone M3, and the first code in a bounded context. WHAT AN ACQUISITION CHANNEL MUST PROVIDE, AND MAY NEVER LET OUT SourceAdapter descriptor() -> SourceDescriptor acquire(selector) -> RawArtifact + AcquisitionProvenance verify_conformance what an adapter must pass before its data enters the platform boundary.stamp the single sanctioned clock read in the system Credential a secret that cannot be printed or serialised assert_credential_free refuse to let one cross the boundary No channel is implemented. The ISSDC-PRADAN adapter is #16; the SoLEXS and HEL1OS parsers are #17 and #18; the bitemporal write path is #19. A PROTOCOL, NOT A FRAMEWORK ADR-0003 is explicit that it does NOT authorise "a source-plugin registry, a dispatch layer, or a configurable adapter framework", and ADR-0025 lists generic base classes in the same category. So: a typing.Protocol that is inherited from by nothing, constructs nothing and dispatches nothing, plus a checker taking an adapter the caller already holds. A test asserts contract.py never grows register/REGISTRY/get_adapter/dispatch. Under ADR-0025's test this is a free seam — stating what a channel must provide would still be right with exactly one source forever, because the parser downstream has to trust its input regardless of which channel produced it. THREE RULES, EACH MADE STRUCTURAL RATHER THAN REMEMBERED Credentials never leave (STD-19, E5 §13). Credential redacts itself in repr, str AND format, closing the routes secrets actually escape by. The integration test writes to a real log file through six of them and greps the file — not the object, the file. A stronger result than expected fell out: SourceDescriptor has no field a credential could occupy. Every field is a typed domain value; there is no dict, no Any, no extras. A secret cannot be placed in a conforming descriptor at all, so the runtime scan is the backstop for the duck-typed case rather than the primary defence. The test that assumed otherwise was corrected to assert what actually holds. Tier 0 is referenced, never redistributed (ADR-0023, STD-23, E5 §11(iv), §12). RawArtifact has a digest, a size, a way to re-acquire, and NO field that could hold the bytes — a test pins the field set. An artifact that can hold a whole-day archive will hold one, and nobody would see it in review; the object would just be larger. Both times, the second never fabricated (ADR-0004, ADR-0022, E5 §11(i)(ii)). boundary.stamp is the exception TIS §0.4 grants, in one named module so that `grep -rn 'ingest.boundary'` enumerates every clock read in the repository. Nothing here takes a row and returns a time for it: __all__ is exactly {stamp, stamp_from_epoch, monotonic}, asserted, and E5 §11(ii) forbids the rest. THE #13 MECHANISM FIRED, EXACTLY AS BUILT Adding the first module to contexts/ingest turned the gate red on its own: contexts.ingest: declared unpopulated but contains 7 module(s). populated flipped False -> True. That is the staleness check M2/E4/#13 shipped doing the one job it was written for, one issue later, without anyone remembering it existed. FEATURE-LEVEL INTEGRATION — THE FIRST REAL DATA PATH tests/integration/test_ingest_contract_with_platform.py follows scientific data from a channel to a contract-valid, provenance-linked Observation across all six components: kernel mints the digest, ingest stamps the time, domain builds the row, contracts validate it, the manifest records where the Tier 0 bytes stay, and the import rules say who may do it. Both bitemporal paths are followed, because the difference between them is the whole of ADR-0022: a new acquisition carries both times; a historical row carries valid_time and a null ingest_time, has no acquisition to describe, and nothing anywhere would fill it. A fabricated one passes the schema and fails the invariant — the division of labour between the layers, on the case that motivated the ADR. VERIFICATION unit (kernel + domain + contexts) 372 passed (+50) tests/architecture 562 passed tests/integration 97 passed (+26) research/tests/v2 188 passed links / imports / contracts gates clean imports gate: 37 modules, 168 imports, 9 policies CI: `contexts` added to the unit job, named as a root so a new context's tests run the day they are written. Reported as EPD-007. No new job, no new context; still eight. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Rexy-5097
added a commit
that referenced
this pull request
Jul 30, 2026
Canonical: Milestone M3 · Epic E5 · Issue #16. Deps: #15 (complete, 1d65c63). Eligibility recomputed by parsing origin/main:docs/tis/TIS-v1.0.md §15 — complete {1..15}, eligible {#16, #17, #22, #23, #25, #26, #27, #39}; lowest Order selects #16. The first channel. Aditya-L1 SoLEXS daily archives, registered with latency class ~33d and granularity daily-archive (E5 §18). WHAT ACQUISITION ACTUALLY IS FOR THIS CHANNEL PRADAN is not an API. ISSDC's own published download script states the procedure — "Login to Pradan in your browser, select data of your interest and download script for the session" — and warns that "There are session download limits, request rate limit and session timeouts in place... Violations may lead to blocking." So the network step is performed by a person, outside this system, and what it leaves is an extracted archive tree. `acquire` is the step that brings that tree into the platform: locate the day, have the kernel digest every product, stamp the moment the system learned of it, return provenance for all of it. ADR-0023 already describes the result correctly — Tier 0, referenced not stored, an evictable cache that is never the system of record. NO HTTP IS PERFORMED, AND NONE IS SIMULATED Writing a fetch against a portal that requires an interactive login would mean inventing a transport this channel does not have and inventing responses to go with it. When a day's products are not on disk the adapter raises UnavailableResource — the true statement, that the data has not been retrieved. A test asserts the module imports nothing that could make a request. No retry loop and no timeout either. E5 §15 makes retry policy adapter-local, and there is no network call here to retry; a loop around a filesystem read would be fabricated behaviour. THIS ADAPTER HOLDS NO CREDENTIAL It cannot leak a secret because it has none — authentication is a browser login that happens before and outside this code. Its descriptor records requires_credentials=True because RE-ACQUIRING needs a session, which is what a future reader must know (ADR-0023). `test_the_adapter_holds_no_credential` asserts vars() is exactly {archive_root, _store}. That is a stronger form of STD-19 than confinement: there is nothing here to confine. THE STD-12 TRAP, FOUND IN THIS ARCHIVE FOR A SECOND REASON E5 §16 requires real-data guards to test for the FITS products, not for a directory, citing the legacy isdir guard that silently disabled 188 tests. The same trap exists here for a different reason, verified: the five v1.1 archives carry a TRACKED quicklook PNG, so AL1_SLX_L1_20241001_v1.1/ exists in every clean checkout while its 3.5 GB of .gz products do not. Guards test for .gz products. `products()` raises UnavailableResource naming STD-12 when a directory is present and empty, and a test reproduces exactly that condition. ARCHIVE IDENTITY IS A ROLLUP, NOT AN OPAQUE DIGEST One line per product — digest, size, relative path — sorted, digested by the kernel. Same reasoning as DatasetRelease rolling per-table digests to a release digest (ADR-0006): a changed byte anywhere changes the archive identity AND stays locatable to the product that changed. A digest over concatenated content would detect the change and lose its address. Sorted because the identity is computed over the sequence: filesystem ordering must not reach it, or two machines produce two identities for the same bytes. AppleDouble sidecars and the v1.1 quicklook PNG are excluded — neither is what ISSDC published, and including either would make the digest depend on which computer looked at the data. The product bytes are NOT registered with the store. Only the listing is. Registering them would copy Tier 0 into the store, which ADR-0023 forbids. VERIFIED AGAINST THE REAL CORPUS 2024-05-14, the X8.7 flare SALVAGE-001 records v1 reproducing to the minute from GOES — now the Aditya-L1 side of the same day, entering the platform with provenance: 4 products, 13,302,792 bytes SDD1 gti only; SDD2 gti + lc + pi exactly SPEC-parsers r6 §1.1 436 archives, 436 unique dates re-measured, matching the spec Both real-corpus tests skip rather than fail where the archive is absent (E5 §17). VERIFICATION unit (kernel + domain + contexts) 405 passed (+33) tests/architecture 562 passed tests/integration 114 passed (+17) research/tests/v2 188 passed links / imports / contracts gates clean CI: no workflow change. `contexts` is already a unit-job root and tests/integration already runs. Still eight contexts. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Rexy-5097
added a commit
that referenced
this pull request
Jul 30, 2026
Canonical: Milestone M3 · Epic E5 · Issue #17. Deps: #12, #15 (both complete). Eligibility recomputed by parsing origin/main:docs/tis/TIS-v1.0.md §15 — complete {1..16}, eligible {#17, #22, #23, #25, #26, #27, #39}; lowest Order selects #17. Three parsers, one per SoLEXS product, each implementing its section of SPEC-parsers@r6. That document is a CONTRACT: "a deviation requires a logged amendment, not a code change." lc §2.1 total-band counts per second. NaN is data, not an error. pi §2.2 340-channel PI spectra per second. Ordinal channels; keV is prohibited. gti §2.3 inclusive second-marks; Σ(STOP−START+1) == EXPOSURE, exactly. NO DEFAULT ON A MISSING KEY — THE RULE THE WHOLE SPEC OPENS WITH §5: "No default-on-missing-key (header.get(K, default) is banned for any physically meaningful key — that idiom is the direct cause of v1's thirty-sprint failure)." The concrete failure was F-05: v1 defaulted MJDREF to 58484 where the archive declares 40587, a ~49-YEAR timestamp error nothing detected. So `_fits.py` has no `get`. Every lookup raises. Five tests delete a keyword and require the parse to abort. NaN IS DATA. ZERO IS A MEASUREMENT. THEY NEVER MEET. §2.1 r2, binding: SoLEXS COUNTS uses NaN as the missing-data sentinel; NaN values pass through unchanged, are never imputed, never converted to zero, never removed; zero remains a valid physical count. Mapped onto the domain exactly: NaN -> None (observed to be absent, ADR-0017/L-07), 0.0 -> 0.0. The property is quantified over every position and both directions, because collapsing the two is the one error that would be undetectable in every downstream artifact. No interpolate/smooth/fill/impute/resample/repair function exists in any of the three modules, and a test asserts none appears. F-19 is checked AFTER the NaN branch, which is what makes §2.1's note true that the rule is "inherently NaN-safe (NaN < 0 is False)". THE keV PROHIBITION, ENFORCED BY ABSENCE §2.2: no RMF/ARF exists anywhere in the archive, so PI-channel -> keV is impossible from archive contents alone, and "no v2 artifact may state a SoLEXS energy in keV until a response file is acquired". There is no keV field, no conversion, no calibration constant — a test scans the module and its three classes for any of them. A comment is not a constraint; an absent field is. EXACT MEANS EXACT F-09 requires Σ(STOP−START+1) == EXPOSURE with tolerance 0 s. §5 says why: "the relation is definitional, and a tolerance would re-admit the ambiguity that produced CONTRADICTION-001". A one-second discrepancy — precisely what the superseded exclusive reading produces — aborts. The convention is verified for 2024-05-14 SDD2 only (§8 A-8) and is NOT widened here; §2.3 requires any deviation to terminate validation. VERIFIED AGAINST THE REAL ARCHIVE — EVERY OBSERVED VALUE REPRODUCED Re-measured from the 2024-05-14 SDD2 products, the X8.7 flare day: 86,400 rows §2.1 NAXIS2 5 NaN at offsets [0, 5, 30072, 30078, 83951] §2.1 OBSERVED, exactly 86,395 finite §2.1 == EXPOSURE 5 GTI intervals, Σ(STOP−START+1) = 86395.0 §2.3, zero error GTI-excluded set == NaN set §2.1 OBSERVED, this day SDD1 GTI NAXIS2 = 0 -> detector_active=False F-12, the one non-fatal rule TSTART 1715644800.0 = 2024-05-14T00:00:00Z §2.1 epoch anchor DETCHANS 340, CHANTYPE PI, channel map 0..339 §2.2 .pi TSTART[0] == .lc TSTART V-PI-3 / F-06 These are the specification's own OBSERVED claims, independently reproduced by this code. A drift in either the parser or the archive is now a red test rather than a quiet disagreement between a document and reality. astropy GRANTED TO INGEST — THE LINE #13 PREDICTED Reading FITS needs a FITS reader. M2/E4/#13 declined to pre-grant third-party roots and said why: "a context that later needs `astropy` should add one reviewable line rather than inherit a blanket permission nobody voted for (STD-11)." Three existing tests pinned `contexts.ingest.allow == {contracts, domain, kernel}` and went red — the mechanism working. They are updated to be STRICTER, not looser: internal grants must still equal ADR-0026's exactly, and a new test enumerates every third-party grant and refuses any that names a governed root of this repository. SCOPE .hk (§2.4, v1.1 only, 5 of 436 days) is NOT implemented: TIS E5 §5 lists this epic's SoLEXS modules as parsers/solexs/{lc,pi,gti} and assigns no .hk module to any issue. Reported as an Engineering Plan Defect rather than filled by guessing at scope. No aggregation to the 1-minute grid — §3's T1/T2 contract is the write path's (#19). No `NaN => GTI-excluded` evaluation — §2.1 places it at the day-assembly layer explicitly, "not inside the single-file .lc parser". No HEL1OS, no curation, no registry. VERIFICATION unit (kernel + domain + contexts) 470 passed (+65) tests/architecture 563 passed (+1) tests/integration 128 passed (+14) research/tests/v2 188 passed links / imports / contracts gates clean imports gate: 46 modules, 211 imports, 9 policies Eight real-corpus tests run where the archive is present and skip where it is not (STD-12, E5 §17). The guard tests for .gz PRODUCTS, never a directory: five v1.1 archive directories exist in a clean checkout because their quicklook PNGs are tracked. CI: numpy + astropy added to the unit and integration jobs. No new job, no new context; still eight. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Canonical identifier: Milestone M2 · Epic E4 · Issue #14 — Tier 2 manifest schema
Dependencies: #11 ✔ (merged as
6c26b33)Selection: eligibility recomputed by parsing
origin/main:docs/tis/TIS-v1.0.md§15 — complete{1…13}, eligible{#14 Order 14, #15 Order 15}; lowest Order selects #14.This completes Epic E4.
What this adds
contracts/manifest.schema.json— the twelfth contract, and the one git actually carries. A Tier 2 manifest is what the repository holds in place of the ~600 MB it deliberately does not (ADR-0023, E6 §19).Almost everything this schema does is a refusal
ADR-0023 defines three tiers with three different homes, and each must forbid the others. Expressed as prose, "a Tier 0 manifest must not carry a deposition" is a sentence someone has to read. Expressed as
if tier = 0 then not deposition, a manifest that would redistribute another organisation's raw archive cannot be written.retrievaldescriptordeposition,path— STD-23, E6 §11(iii): identified, never stored, never redistributeddeposition, andretention: permanentretrieval,path— a prunable Tier 1 release makes a DOI resolve to nothingpath, constrained so it cannot leave the repo rootretrieval,depositionRetention is unrepresentable when it would be wrong
E6 §11(iv): an artifact referenced by an Evidence Binding is never pruned. So
prunablewith a non-emptyreferenced_byis refused by the contract rather than discouraged in a comment — it is the one combination that would let a published claim lose the bytes underneath it.A
permanententry likewise cannot carry aprune_after. Permitting both would let a manifest state a contradiction that a retention run would then arbitrate silently, the same way every time.doiis required as a field and nullable in valueADR-0023's fallback degrades citability and says the degradation is published rather than hidden. Optional would make "no DOI" and "nobody recorded whether there is one" the same document — the
ingest_timereasoning of ADR-0022 applied to citability.Feature-level integration — the digest is the same digest
tests/integration/test_manifest_with_platform.pychecks the one thing no schema can: that the digest written into a manifest is the digest of the bytes it claims to describe. That needs four components at once —Every guarantee downstream — citation, re-acquisition, supersession, evidence binding — rests on that equality, and nothing had ever checked it end to end. Also included: a manifest that is schema-valid and false, pointing at bytes the store has never seen, with the demonstration that the store refuses it while the schema cannot.
A finding about my own test, not the schema
tier: 1.0validates, and that is correct. JSON has one number type and JSON Schema comparesenummembers by value, so1.0and1are the same number — a schema cannot distinguish them, so the alternative to accepting1.0is not rejecting it but pretending to. The first version of that test asserted rejection and was wrong; it now asserts acceptance and records why.Scope — no producer, no parser, no registry, no ingest
The schema and its tests.
contexts/curation/manifestis E6/#20's;registry/*/*.jsoninstances are E6/#21, E7/#23, E8/#25–#26, E9/#27's; the retention gate is E6/#22's.The manifest gets no domain entity, deliberately: row 14's Unit column reads "manifest validity" — a schema test, not entity round-trip — and an entity would be a producer.
Two existing assertions updated, because #14 changes the contract set: expected count 11 → 12, and
manifestadded to the documented no-domain-entity exclusions with its reason.Verification
CI impact
No workflow change. Both new files land in jobs that already run. Still eight required contexts.
🤖 Generated with Claude Code