Skip to content

M3/E5/#16 — ISSDC-PRADAN adapter - #19

Merged
Rexy-5097 merged 1 commit into
mainfrom
feat/m3-e5-issue-16-issdc-adapter
Jul 30, 2026
Merged

M3/E5/#16 — ISSDC-PRADAN adapter#19
Rexy-5097 merged 1 commit into
mainfrom
feat/m3-e5-issue-16-issdc-adapter

Conversation

@Rexy-5097

Copy link
Copy Markdown
Owner

Canonical identifier: Milestone M3 · Epic E5 · Issue #16 — ISSDC-PRADAN adapter
Dependencies: #15 ✔ (1d65c63)
Selection: eligibility from origin/main:docs/tis/TIS-v1.0.md §15 — complete {1…15}, eligible {#16, #17, #22, #23, #25, #26, #27, #39}; lowest Order selects #16.

The first acquisition channel. Aditya-L1 SoLEXS daily archives, registered with latency class ~33d and granularity daily-archive (E5 §18).

What acquisition actually is for this channel

PRADAN is not an API. ISSDC's own published download script states the procedure — "Login to Pradan in your browser, select data of your interest and download script for the session" — and warns that "There are session download limits, request rate limit and session timeouts in place… Violations may lead to blocking."

So the network step is performed by a person, outside this system, and what it leaves behind is an extracted archive tree. acquire is the step that brings that tree into the platform: locate the day, have the kernel digest every product, stamp the moment the system learned of it, return provenance for all of it. ADR-0023 already describes the result correctly — Tier 0, referenced not stored, an evictable cache that is never the system of record.

No HTTP is performed, and none is simulated. Writing a fetch against a portal that requires an interactive login would mean inventing a transport this channel does not have, and inventing responses to go with it. When a day's products are not on disk the adapter raises UnavailableResource — the true statement, that the data has not been retrieved. A test asserts the module imports nothing that could make a request.

No retry loop and no timeout. E5 §15 makes retry policy adapter-local, and there is no network call here to retry; a loop around a filesystem read would be fabricated behaviour.

This adapter holds no credential

It cannot leak a secret because it has none — authentication is a browser login that happens before and outside this code. Its descriptor records requires_credentials=True because re-acquiring needs a session, which is what a future reader must know (ADR-0023). test_the_adapter_holds_no_credential asserts vars() is exactly {archive_root, _store}. That is a stronger form of STD-19 than confinement: there is nothing here to confine.

The STD-12 trap, found in this archive for a second reason

E5 §16 requires real-data guards to test for the products, not a directory, citing the legacy isdir guard that silently disabled 188 tests. The same trap exists here for a different reason, verified: the five v1.1 archives carry a tracked quicklook PNG, so AL1_SLX_L1_20241001_v1.1/ exists in every clean checkout while its .gz products do not.

Guards test for .gz products. products() raises UnavailableResource naming STD-12 when a directory is present and empty, and a test reproduces exactly that condition.

Archive identity is a rollup, not an opaque digest

One line per product — digest, size, relative path — sorted, digested by the kernel. Same reasoning as DatasetRelease rolling per-table digests to a release digest (ADR-0006): a changed byte anywhere changes the archive identity and stays locatable to the product that changed. A digest over concatenated content would detect the change and lose its address.

Sorted, because the identity is computed over the sequence — filesystem ordering must not reach it, or two machines produce two identities for the same bytes. AppleDouble sidecars and the v1.1 quicklook PNG are excluded: neither is what ISSDC published, and including either would make the digest depend on which computer looked at the data.

The product bytes are not registered with the store. Only the listing is. Registering them would copy Tier 0 into the store, which ADR-0023 forbids.

Verified against the real corpus

2024-05-14 — the X8.7 flare SALVAGE-001 records v1 reproducing to the minute from GOES, now the Aditya-L1 side of the same day entering the platform with provenance:

4 products, 13,302,792 bytes
SDD1 gti only; SDD2 gti + lc + pi     exactly SPEC-parsers r6 §1.1
436 archives, 436 unique dates         re-measured, matching the spec

Both real-corpus tests skip rather than fail where the archive is absent (E5 §17) — demonstrated in the clean export below.

Feature-level integration

Every acquisition in the integration suite goes through #15's verify_conformance rather than calling acquire and inspecting the result — the difference between an adapter that happens to produce the right shape and one the platform has accepted. A near-identical adapter breaking exactly one rule is refused by the same call that admits this one.

Across all six components: #10 every digest re-derived with the kernel's own functions · #11 the Observation validates against its contract · #12 the domain holds the digest and both times · #13 the shipped import rule binds, with a deliberate cross-context violation · #14 the acquisition becomes a valid Tier 0 manifest, and cannot be recorded as deposited · #15 conformance is what admits it.

Verification

unit (kernel + domain + contexts)  405 passed   (+33)
tests/architecture                 562 passed
tests/integration                  114 passed   (+17)
research/tests/v2                  188 passed
links / imports / contracts gates  clean

clean export (git archive HEAD, archive absent):
  unit 403 passed, 2 skipped · integration 88 passed, 2 skipped
  + 24 pre-existing errors (IMPL-012)
  → real-data tests SKIP, never fail: E5 §17 satisfied

⚠ Pre-existing security finding — reported, not repaired

Investigating how PRADAN acquisition actually works surfaced live-format ISSDC session cookies committed to this public repository, in 8 tracked files on origin/main (research/legacy/*.sh, research/data_pipeline/config.yaml, hel1os_downloader.py, downloader/recovery.py, and two report files), and also at tag v1-surya-final.

STD-19: "No secret is committed." This predates #16 and is outside its ownership, so per the defect protocol it is reported rather than silently repaired — and deletion from HEAD would not help, since the values remain in history and at the tag. The only effective remedy is invalidating the sessions at ISSDC. Details in the completion report.

CI impact

No workflow change. contexts is already a unit-job root (added by #15) and tests/integration already runs. Still eight required contexts.

🤖 Generated with Claude Code

Canonical: Milestone M3 · Epic E5 · Issue #16. Deps: #15 (complete, 1d65c63).
Eligibility recomputed by parsing origin/main:docs/tis/TIS-v1.0.md §15 — complete {1..15},
eligible {#16, #17, #22, #23, #25, #26, #27, #39}; lowest Order selects #16.

The first channel. Aditya-L1 SoLEXS daily archives, registered with latency class ~33d and
granularity daily-archive (E5 §18).

WHAT ACQUISITION ACTUALLY IS FOR THIS CHANNEL

PRADAN is not an API. ISSDC's own published download script states the procedure —
"Login to Pradan in your browser, select data of your interest and download script for the
session" — and warns that "There are session download limits, request rate limit and
session timeouts in place... Violations may lead to blocking."

So the network step is performed by a person, outside this system, and what it leaves is an
extracted archive tree. `acquire` is the step that brings that tree into the platform:
locate the day, have the kernel digest every product, stamp the moment the system learned
of it, return provenance for all of it. ADR-0023 already describes the result correctly —
Tier 0, referenced not stored, an evictable cache that is never the system of record.

NO HTTP IS PERFORMED, AND NONE IS SIMULATED

Writing a fetch against a portal that requires an interactive login would mean inventing a
transport this channel does not have and inventing responses to go with it. When a day's
products are not on disk the adapter raises UnavailableResource — the true statement, that
the data has not been retrieved. A test asserts the module imports nothing that could make
a request.

No retry loop and no timeout either. E5 §15 makes retry policy adapter-local, and there is
no network call here to retry; a loop around a filesystem read would be fabricated
behaviour.

THIS ADAPTER HOLDS NO CREDENTIAL

It cannot leak a secret because it has none — authentication is a browser login that
happens before and outside this code. Its descriptor records requires_credentials=True
because RE-ACQUIRING needs a session, which is what a future reader must know (ADR-0023).
`test_the_adapter_holds_no_credential` asserts vars() is exactly {archive_root, _store}.
That is a stronger form of STD-19 than confinement: there is nothing here to confine.

THE STD-12 TRAP, FOUND IN THIS ARCHIVE FOR A SECOND REASON

E5 §16 requires real-data guards to test for the FITS products, not for a directory,
citing the legacy isdir guard that silently disabled 188 tests. The same trap exists here
for a different reason, verified: the five v1.1 archives carry a TRACKED quicklook PNG, so
AL1_SLX_L1_20241001_v1.1/ exists in every clean checkout while its 3.5 GB of .gz products
do not. Guards test for .gz products. `products()` raises UnavailableResource naming STD-12
when a directory is present and empty, and a test reproduces exactly that condition.

ARCHIVE IDENTITY IS A ROLLUP, NOT AN OPAQUE DIGEST

One line per product — digest, size, relative path — sorted, digested by the kernel. Same
reasoning as DatasetRelease rolling per-table digests to a release digest (ADR-0006): a
changed byte anywhere changes the archive identity AND stays locatable to the product that
changed. A digest over concatenated content would detect the change and lose its address.

Sorted because the identity is computed over the sequence: filesystem ordering must not
reach it, or two machines produce two identities for the same bytes. AppleDouble sidecars
and the v1.1 quicklook PNG are excluded — neither is what ISSDC published, and including
either would make the digest depend on which computer looked at the data.

The product bytes are NOT registered with the store. Only the listing is. Registering them
would copy Tier 0 into the store, which ADR-0023 forbids.

VERIFIED AGAINST THE REAL CORPUS

2024-05-14, the X8.7 flare SALVAGE-001 records v1 reproducing to the minute from GOES —
now the Aditya-L1 side of the same day, entering the platform with provenance:

  4 products, 13,302,792 bytes
  SDD1 gti only; SDD2 gti + lc + pi        exactly SPEC-parsers r6 §1.1
  436 archives, 436 unique dates            re-measured, matching the spec

Both real-corpus tests skip rather than fail where the archive is absent (E5 §17).

VERIFICATION

  unit (kernel + domain + contexts)  405 passed  (+33)
  tests/architecture                 562 passed
  tests/integration                  114 passed  (+17)
  research/tests/v2                  188 passed
  links / imports / contracts gates  clean

CI: no workflow change. `contexts` is already a unit-job root and tests/integration already
runs. Still eight contexts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Rexy-5097
Rexy-5097 merged commit 40c732d into main Jul 30, 2026
8 checks passed
@Rexy-5097
Rexy-5097 deleted the feat/m3-e5-issue-16-issdc-adapter branch July 30, 2026 13:14
Rexy-5097 added a commit that referenced this pull request Jul 30, 2026
Canonical: Milestone M3 · Epic E5 · Issue #17. Deps: #12, #15 (both complete).
Eligibility recomputed by parsing origin/main:docs/tis/TIS-v1.0.md §15 — complete {1..16},
eligible {#17, #22, #23, #25, #26, #27, #39}; lowest Order selects #17.

Three parsers, one per SoLEXS product, each implementing its section of SPEC-parsers@r6.
That document is a CONTRACT: "a deviation requires a logged amendment, not a code change."

  lc   §2.1  total-band counts per second. NaN is data, not an error.
  pi   §2.2  340-channel PI spectra per second. Ordinal channels; keV is prohibited.
  gti  §2.3  inclusive second-marks; Σ(STOP−START+1) == EXPOSURE, exactly.

NO DEFAULT ON A MISSING KEY — THE RULE THE WHOLE SPEC OPENS WITH

§5: "No default-on-missing-key (header.get(K, default) is banned for any physically
meaningful key — that idiom is the direct cause of v1's thirty-sprint failure)."

The concrete failure was F-05: v1 defaulted MJDREF to 58484 where the archive declares
40587, a ~49-YEAR timestamp error nothing detected. So `_fits.py` has no `get`. Every
lookup raises. Five tests delete a keyword and require the parse to abort.

NaN IS DATA. ZERO IS A MEASUREMENT. THEY NEVER MEET.

§2.1 r2, binding: SoLEXS COUNTS uses NaN as the missing-data sentinel; NaN values pass
through unchanged, are never imputed, never converted to zero, never removed; zero
remains a valid physical count.

Mapped onto the domain exactly: NaN -> None (observed to be absent, ADR-0017/L-07),
0.0 -> 0.0. The property is quantified over every position and both directions, because
collapsing the two is the one error that would be undetectable in every downstream
artifact. No interpolate/smooth/fill/impute/resample/repair function exists in any of the
three modules, and a test asserts none appears.

F-19 is checked AFTER the NaN branch, which is what makes §2.1's note true that the rule
is "inherently NaN-safe (NaN < 0 is False)".

THE keV PROHIBITION, ENFORCED BY ABSENCE

§2.2: no RMF/ARF exists anywhere in the archive, so PI-channel -> keV is impossible from
archive contents alone, and "no v2 artifact may state a SoLEXS energy in keV until a
response file is acquired". There is no keV field, no conversion, no calibration constant
— a test scans the module and its three classes for any of them. A comment is not a
constraint; an absent field is.

EXACT MEANS EXACT

F-09 requires Σ(STOP−START+1) == EXPOSURE with tolerance 0 s. §5 says why: "the relation
is definitional, and a tolerance would re-admit the ambiguity that produced
CONTRADICTION-001". A one-second discrepancy — precisely what the superseded exclusive
reading produces — aborts. The convention is verified for 2024-05-14 SDD2 only (§8 A-8)
and is NOT widened here; §2.3 requires any deviation to terminate validation.

VERIFIED AGAINST THE REAL ARCHIVE — EVERY OBSERVED VALUE REPRODUCED

Re-measured from the 2024-05-14 SDD2 products, the X8.7 flare day:

  86,400 rows                                      §2.1 NAXIS2
  5 NaN at offsets [0, 5, 30072, 30078, 83951]     §2.1 OBSERVED, exactly
  86,395 finite                                    §2.1 == EXPOSURE
  5 GTI intervals, Σ(STOP−START+1) = 86395.0       §2.3, zero error
  GTI-excluded set == NaN set                      §2.1 OBSERVED, this day
  SDD1 GTI NAXIS2 = 0 -> detector_active=False     F-12, the one non-fatal rule
  TSTART 1715644800.0 = 2024-05-14T00:00:00Z       §2.1 epoch anchor
  DETCHANS 340, CHANTYPE PI, channel map 0..339    §2.2
  .pi TSTART[0] == .lc TSTART                      V-PI-3 / F-06

These are the specification's own OBSERVED claims, independently reproduced by this code.
A drift in either the parser or the archive is now a red test rather than a quiet
disagreement between a document and reality.

astropy GRANTED TO INGEST — THE LINE #13 PREDICTED

Reading FITS needs a FITS reader. M2/E4/#13 declined to pre-grant third-party roots and
said why: "a context that later needs `astropy` should add one reviewable line rather
than inherit a blanket permission nobody voted for (STD-11)."

Three existing tests pinned `contexts.ingest.allow == {contracts, domain, kernel}` and
went red — the mechanism working. They are updated to be STRICTER, not looser: internal
grants must still equal ADR-0026's exactly, and a new test enumerates every third-party
grant and refuses any that names a governed root of this repository.

SCOPE

.hk (§2.4, v1.1 only, 5 of 436 days) is NOT implemented: TIS E5 §5 lists this epic's
SoLEXS modules as parsers/solexs/{lc,pi,gti} and assigns no .hk module to any issue.
Reported as an Engineering Plan Defect rather than filled by guessing at scope.

No aggregation to the 1-minute grid — §3's T1/T2 contract is the write path's (#19). No
`NaN => GTI-excluded` evaluation — §2.1 places it at the day-assembly layer explicitly,
"not inside the single-file .lc parser". No HEL1OS, no curation, no registry.

VERIFICATION

  unit (kernel + domain + contexts)  470 passed  (+65)
  tests/architecture                 563 passed  (+1)
  tests/integration                  128 passed  (+14)
  research/tests/v2                  188 passed
  links / imports / contracts gates  clean
  imports gate: 46 modules, 211 imports, 9 policies

Eight real-corpus tests run where the archive is present and skip where it is not
(STD-12, E5 §17). The guard tests for .gz PRODUCTS, never a directory: five v1.1 archive
directories exist in a clean checkout because their quicklook PNGs are tracked.

CI: numpy + astropy added to the unit and integration jobs. No new job, no new context;
still eight.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant