M3/E5/#16 — ISSDC-PRADAN adapter - #19
Merged
Merged
Conversation
Canonical: Milestone M3 · Epic E5 · Issue #16. Deps: #15 (complete, 1d65c63). Eligibility recomputed by parsing origin/main:docs/tis/TIS-v1.0.md §15 — complete {1..15}, eligible {#16, #17, #22, #23, #25, #26, #27, #39}; lowest Order selects #16. The first channel. Aditya-L1 SoLEXS daily archives, registered with latency class ~33d and granularity daily-archive (E5 §18). WHAT ACQUISITION ACTUALLY IS FOR THIS CHANNEL PRADAN is not an API. ISSDC's own published download script states the procedure — "Login to Pradan in your browser, select data of your interest and download script for the session" — and warns that "There are session download limits, request rate limit and session timeouts in place... Violations may lead to blocking." So the network step is performed by a person, outside this system, and what it leaves is an extracted archive tree. `acquire` is the step that brings that tree into the platform: locate the day, have the kernel digest every product, stamp the moment the system learned of it, return provenance for all of it. ADR-0023 already describes the result correctly — Tier 0, referenced not stored, an evictable cache that is never the system of record. NO HTTP IS PERFORMED, AND NONE IS SIMULATED Writing a fetch against a portal that requires an interactive login would mean inventing a transport this channel does not have and inventing responses to go with it. When a day's products are not on disk the adapter raises UnavailableResource — the true statement, that the data has not been retrieved. A test asserts the module imports nothing that could make a request. No retry loop and no timeout either. E5 §15 makes retry policy adapter-local, and there is no network call here to retry; a loop around a filesystem read would be fabricated behaviour. THIS ADAPTER HOLDS NO CREDENTIAL It cannot leak a secret because it has none — authentication is a browser login that happens before and outside this code. Its descriptor records requires_credentials=True because RE-ACQUIRING needs a session, which is what a future reader must know (ADR-0023). `test_the_adapter_holds_no_credential` asserts vars() is exactly {archive_root, _store}. That is a stronger form of STD-19 than confinement: there is nothing here to confine. THE STD-12 TRAP, FOUND IN THIS ARCHIVE FOR A SECOND REASON E5 §16 requires real-data guards to test for the FITS products, not for a directory, citing the legacy isdir guard that silently disabled 188 tests. The same trap exists here for a different reason, verified: the five v1.1 archives carry a TRACKED quicklook PNG, so AL1_SLX_L1_20241001_v1.1/ exists in every clean checkout while its 3.5 GB of .gz products do not. Guards test for .gz products. `products()` raises UnavailableResource naming STD-12 when a directory is present and empty, and a test reproduces exactly that condition. ARCHIVE IDENTITY IS A ROLLUP, NOT AN OPAQUE DIGEST One line per product — digest, size, relative path — sorted, digested by the kernel. Same reasoning as DatasetRelease rolling per-table digests to a release digest (ADR-0006): a changed byte anywhere changes the archive identity AND stays locatable to the product that changed. A digest over concatenated content would detect the change and lose its address. Sorted because the identity is computed over the sequence: filesystem ordering must not reach it, or two machines produce two identities for the same bytes. AppleDouble sidecars and the v1.1 quicklook PNG are excluded — neither is what ISSDC published, and including either would make the digest depend on which computer looked at the data. The product bytes are NOT registered with the store. Only the listing is. Registering them would copy Tier 0 into the store, which ADR-0023 forbids. VERIFIED AGAINST THE REAL CORPUS 2024-05-14, the X8.7 flare SALVAGE-001 records v1 reproducing to the minute from GOES — now the Aditya-L1 side of the same day, entering the platform with provenance: 4 products, 13,302,792 bytes SDD1 gti only; SDD2 gti + lc + pi exactly SPEC-parsers r6 §1.1 436 archives, 436 unique dates re-measured, matching the spec Both real-corpus tests skip rather than fail where the archive is absent (E5 §17). VERIFICATION unit (kernel + domain + contexts) 405 passed (+33) tests/architecture 562 passed tests/integration 114 passed (+17) research/tests/v2 188 passed links / imports / contracts gates clean CI: no workflow change. `contexts` is already a unit-job root and tests/integration already runs. Still eight contexts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rexy-5097
added a commit
that referenced
this pull request
Jul 30, 2026
Canonical: Milestone M3 · Epic E5 · Issue #17. Deps: #12, #15 (both complete). Eligibility recomputed by parsing origin/main:docs/tis/TIS-v1.0.md §15 — complete {1..16}, eligible {#17, #22, #23, #25, #26, #27, #39}; lowest Order selects #17. Three parsers, one per SoLEXS product, each implementing its section of SPEC-parsers@r6. That document is a CONTRACT: "a deviation requires a logged amendment, not a code change." lc §2.1 total-band counts per second. NaN is data, not an error. pi §2.2 340-channel PI spectra per second. Ordinal channels; keV is prohibited. gti §2.3 inclusive second-marks; Σ(STOP−START+1) == EXPOSURE, exactly. NO DEFAULT ON A MISSING KEY — THE RULE THE WHOLE SPEC OPENS WITH §5: "No default-on-missing-key (header.get(K, default) is banned for any physically meaningful key — that idiom is the direct cause of v1's thirty-sprint failure)." The concrete failure was F-05: v1 defaulted MJDREF to 58484 where the archive declares 40587, a ~49-YEAR timestamp error nothing detected. So `_fits.py` has no `get`. Every lookup raises. Five tests delete a keyword and require the parse to abort. NaN IS DATA. ZERO IS A MEASUREMENT. THEY NEVER MEET. §2.1 r2, binding: SoLEXS COUNTS uses NaN as the missing-data sentinel; NaN values pass through unchanged, are never imputed, never converted to zero, never removed; zero remains a valid physical count. Mapped onto the domain exactly: NaN -> None (observed to be absent, ADR-0017/L-07), 0.0 -> 0.0. The property is quantified over every position and both directions, because collapsing the two is the one error that would be undetectable in every downstream artifact. No interpolate/smooth/fill/impute/resample/repair function exists in any of the three modules, and a test asserts none appears. F-19 is checked AFTER the NaN branch, which is what makes §2.1's note true that the rule is "inherently NaN-safe (NaN < 0 is False)". THE keV PROHIBITION, ENFORCED BY ABSENCE §2.2: no RMF/ARF exists anywhere in the archive, so PI-channel -> keV is impossible from archive contents alone, and "no v2 artifact may state a SoLEXS energy in keV until a response file is acquired". There is no keV field, no conversion, no calibration constant — a test scans the module and its three classes for any of them. A comment is not a constraint; an absent field is. EXACT MEANS EXACT F-09 requires Σ(STOP−START+1) == EXPOSURE with tolerance 0 s. §5 says why: "the relation is definitional, and a tolerance would re-admit the ambiguity that produced CONTRADICTION-001". A one-second discrepancy — precisely what the superseded exclusive reading produces — aborts. The convention is verified for 2024-05-14 SDD2 only (§8 A-8) and is NOT widened here; §2.3 requires any deviation to terminate validation. VERIFIED AGAINST THE REAL ARCHIVE — EVERY OBSERVED VALUE REPRODUCED Re-measured from the 2024-05-14 SDD2 products, the X8.7 flare day: 86,400 rows §2.1 NAXIS2 5 NaN at offsets [0, 5, 30072, 30078, 83951] §2.1 OBSERVED, exactly 86,395 finite §2.1 == EXPOSURE 5 GTI intervals, Σ(STOP−START+1) = 86395.0 §2.3, zero error GTI-excluded set == NaN set §2.1 OBSERVED, this day SDD1 GTI NAXIS2 = 0 -> detector_active=False F-12, the one non-fatal rule TSTART 1715644800.0 = 2024-05-14T00:00:00Z §2.1 epoch anchor DETCHANS 340, CHANTYPE PI, channel map 0..339 §2.2 .pi TSTART[0] == .lc TSTART V-PI-3 / F-06 These are the specification's own OBSERVED claims, independently reproduced by this code. A drift in either the parser or the archive is now a red test rather than a quiet disagreement between a document and reality. astropy GRANTED TO INGEST — THE LINE #13 PREDICTED Reading FITS needs a FITS reader. M2/E4/#13 declined to pre-grant third-party roots and said why: "a context that later needs `astropy` should add one reviewable line rather than inherit a blanket permission nobody voted for (STD-11)." Three existing tests pinned `contexts.ingest.allow == {contracts, domain, kernel}` and went red — the mechanism working. They are updated to be STRICTER, not looser: internal grants must still equal ADR-0026's exactly, and a new test enumerates every third-party grant and refuses any that names a governed root of this repository. SCOPE .hk (§2.4, v1.1 only, 5 of 436 days) is NOT implemented: TIS E5 §5 lists this epic's SoLEXS modules as parsers/solexs/{lc,pi,gti} and assigns no .hk module to any issue. Reported as an Engineering Plan Defect rather than filled by guessing at scope. No aggregation to the 1-minute grid — §3's T1/T2 contract is the write path's (#19). No `NaN => GTI-excluded` evaluation — §2.1 places it at the day-assembly layer explicitly, "not inside the single-file .lc parser". No HEL1OS, no curation, no registry. VERIFICATION unit (kernel + domain + contexts) 470 passed (+65) tests/architecture 563 passed (+1) tests/integration 128 passed (+14) research/tests/v2 188 passed links / imports / contracts gates clean imports gate: 46 modules, 211 imports, 9 policies Eight real-corpus tests run where the archive is present and skip where it is not (STD-12, E5 §17). The guard tests for .gz PRODUCTS, never a directory: five v1.1 archive directories exist in a clean checkout because their quicklook PNGs are tracked. CI: numpy + astropy added to the unit and integration jobs. No new job, no new context; still eight. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Canonical identifier: Milestone M3 · Epic E5 · Issue #16 — ISSDC-PRADAN adapter
Dependencies: #15 ✔ (
1d65c63)Selection: eligibility from
origin/main:docs/tis/TIS-v1.0.md§15 — complete{1…15}, eligible{#16, #17, #22, #23, #25, #26, #27, #39}; lowest Order selects #16.The first acquisition channel. Aditya-L1 SoLEXS daily archives, registered with latency class
~33dand granularitydaily-archive(E5 §18).What acquisition actually is for this channel
PRADAN is not an API. ISSDC's own published download script states the procedure — "Login to Pradan in your browser, select data of your interest and download script for the session" — and warns that "There are session download limits, request rate limit and session timeouts in place… Violations may lead to blocking."
So the network step is performed by a person, outside this system, and what it leaves behind is an extracted archive tree.
acquireis the step that brings that tree into the platform: locate the day, have the kernel digest every product, stamp the moment the system learned of it, return provenance for all of it. ADR-0023 already describes the result correctly — Tier 0, referenced not stored, an evictable cache that is never the system of record.No HTTP is performed, and none is simulated. Writing a fetch against a portal that requires an interactive login would mean inventing a transport this channel does not have, and inventing responses to go with it. When a day's products are not on disk the adapter raises
UnavailableResource— the true statement, that the data has not been retrieved. A test asserts the module imports nothing that could make a request.No retry loop and no timeout. E5 §15 makes retry policy adapter-local, and there is no network call here to retry; a loop around a filesystem read would be fabricated behaviour.
This adapter holds no credential
It cannot leak a secret because it has none — authentication is a browser login that happens before and outside this code. Its descriptor records
requires_credentials=Truebecause re-acquiring needs a session, which is what a future reader must know (ADR-0023).test_the_adapter_holds_no_credentialassertsvars()is exactly{archive_root, _store}. That is a stronger form of STD-19 than confinement: there is nothing here to confine.The STD-12 trap, found in this archive for a second reason
E5 §16 requires real-data guards to test for the products, not a directory, citing the legacy
isdirguard that silently disabled 188 tests. The same trap exists here for a different reason, verified: the five v1.1 archives carry a tracked quicklook PNG, soAL1_SLX_L1_20241001_v1.1/exists in every clean checkout while its.gzproducts do not.Guards test for
.gzproducts.products()raisesUnavailableResourcenaming STD-12 when a directory is present and empty, and a test reproduces exactly that condition.Archive identity is a rollup, not an opaque digest
One line per product — digest, size, relative path — sorted, digested by the kernel. Same reasoning as
DatasetReleaserolling per-table digests to a release digest (ADR-0006): a changed byte anywhere changes the archive identity and stays locatable to the product that changed. A digest over concatenated content would detect the change and lose its address.Sorted, because the identity is computed over the sequence — filesystem ordering must not reach it, or two machines produce two identities for the same bytes. AppleDouble sidecars and the v1.1 quicklook PNG are excluded: neither is what ISSDC published, and including either would make the digest depend on which computer looked at the data.
The product bytes are not registered with the store. Only the listing is. Registering them would copy Tier 0 into the store, which ADR-0023 forbids.
Verified against the real corpus
2024-05-14 — the X8.7 flare SALVAGE-001 records v1 reproducing to the minute from GOES, now the Aditya-L1 side of the same day entering the platform with provenance:
Both real-corpus tests skip rather than fail where the archive is absent (E5 §17) — demonstrated in the clean export below.
Feature-level integration
Every acquisition in the integration suite goes through #15's
verify_conformancerather than callingacquireand inspecting the result — the difference between an adapter that happens to produce the right shape and one the platform has accepted. A near-identical adapter breaking exactly one rule is refused by the same call that admits this one.Across all six components: #10 every digest re-derived with the kernel's own functions · #11 the Observation validates against its contract · #12 the domain holds the digest and both times · #13 the shipped import rule binds, with a deliberate cross-context violation · #14 the acquisition becomes a valid Tier 0 manifest, and cannot be recorded as deposited · #15 conformance is what admits it.
Verification
⚠ Pre-existing security finding — reported, not repaired
Investigating how PRADAN acquisition actually works surfaced live-format ISSDC session cookies committed to this public repository, in 8 tracked files on
origin/main(research/legacy/*.sh,research/data_pipeline/config.yaml,hel1os_downloader.py,downloader/recovery.py, and two report files), and also at tagv1-surya-final.STD-19: "No secret is committed." This predates #16 and is outside its ownership, so per the defect protocol it is reported rather than silently repaired — and deletion from
HEADwould not help, since the values remain in history and at the tag. The only effective remedy is invalidating the sessions at ISSDC. Details in the completion report.CI impact
No workflow change.
contextsis already a unit-job root (added by #15) andtests/integrationalready runs. Still eight required contexts.🤖 Generated with Claude Code