Dev into dev-backend#308
Merged
Merged
Conversation
* Add mixed-pol schema dtypes, warnings module, and upgrade helpers Implements Phase 1 of obsdata_mixedpol_plan_v2.md (schema layer only; behavior unchanged for circular-feed inputs). - ehtim/warnings.py: MixedPolConventionWarning, MixedPolClosureSkipWarning. - DTARR: migrate to 12 fields with feed_type column; generic names primary (sefd_p1/sefd_p2/d_p1/d_p2), legacy names as title aliases. - DTPOL_CIRC: add title aliases (p1p1vis/... ↔ rrvis/...); legacy names remain primary. - DTPOL_LIN, DTPOL_MIXED: new dtypes for linear and mixed feed-space observations; DTPOL_MIXED carries per-row polbasis field (populated in Phase 3a). - DTCAL_CIRC, DTCAL_LIN: new dtypes with D-term fields (Phase 4 needs data[site]['d_p1']); legacy DTCAL kept as alias of DTCAL_CIRC. - feed_poldict, feed_dtype_for_polrep helpers. - upgrade_tarr / upgrade_dtpol_circ / upgrade_dtcal_circ helpers used by the class-level schema-upgrade plumbing (next commit). * Wire mixed-pol schema upgrade into Array/Obsdata/Caltable - Array.__init__ / __setstate__: silently upgrade legacy DTARR recarrays (no feed_type column) via upgrade_tarr; fills feed_type='rl'. - Obsdata.__init__ / __setstate__: zero-copy view-cast legacy DTPOL_CIRC to the title-aliased dtype; upgrade tarr. - Caltable.__init__ / __setstate__: upgrade per-site DTCAL recarrays (legacy time/rscale/lscale → adds dr/dl); upgrade tarr. - add_site, add_satellite_tle, add_satellite_elements (array.py): fill feed_type='rl' on the new DTARR row. Full feed_type kwarg lands in Phase 2 (TODO noted on each). - load_array_txt, load_obs_txt, load_uvfits, load_obs_oifits, vex.py: same minimal feed_type='rl' fix on positional DTARR constructions. - load_array_txt and load_obs_txt: accept new column count (14 / 15 respectively) carrying an explicit feed_type token; legacy formats unchanged. Versioned-header parsing lands in Phase 2. - obsdata.py: fix `for f in <dtype-list>: f = f[0]` iteration in switch_polrep (two sites) and data_conj — `f[0]` is a (title, name) tuple after title-alias migration. Switch to iterating dtype.names. * Add Phase 1 mixedpol schema tests 30 new tests in tests/test_mixedpol.py covering: - Warning class identity (subclass of UserWarning). - DTARR / DTPOL_CIRC / DTPOL_LIN / DTPOL_MIXED / DTCAL_CIRC / DTCAL_LIN alias equivalence, wrong-feed KeyError/ValueError, primary-name conventions. - feed_dtype_for_polrep + feed_poldict (circ/lin/mixed pairs, '??' raises). - upgrade_tarr / upgrade_dtpol_circ / upgrade_dtcal_circ correctness + idempotence + zero-copy view-cast verification. - End-to-end: Array, Obsdata, Caltable constructed from legacy-dtype recarrays produce upgraded recarrays with feed_type='rl' populated and the original numerical values preserved. - Pickle round-trip from legacy state through __setstate__ produces upgraded objects. Tests are namespaced test_phase1_* and grouped under a Phase 1 header so subsequent phases of obsdata_mixedpol_plan_v2.md append cleanly. * ruff lint array.py * ruff lint const_def.py * ruff lint vex.py * ruff lint test_mixedpol.py * ruff lint const_def.py format issues * ruff lint array.py format issues * updated exceptions for Array.remove_site * ruff lint vex.py string formatting * ruff lint vex.py string formatting
CI: trigger on dev-backend-mixpol
…eed_type column) (#260) * Phase 2 (1/N): Array query methods for mixed-pol feed dispatch Adds four read-only query methods to Array, building on the Phase 1 DTARR schema (feed_type column + sefd_p1/p2, d_p1/p2 generic names): - is_homogeneous_feeds() -> bool - feed_types() -> set[str] - sefd_for_feed(site, feed) -> float - dterm_for_feed(site, feed) -> complex The two lookup methods dispatch on the station's feed_type ('rl', 'xy', mixed bases, ...) and raise on absent feeds rather than silently returning a wrong-feed SEFD/D-term. 13 new test_phase2_* tests in tests/test_mixedpol.py. Phase 1 tests unchanged. Implements obsdata_mixedpol_plan_v2.md Phase 2, Commit 1. * Phase 2 (2/N): add_site / add_satellite_* feed_type and per-feed kwargs Extends Array.add_site, Array.add_satellite_tle, and Array.add_satellite_elements with mixed-pol kwargs: - feed_type='rl' (lowercase 2-char, validated against VALID_FEED_TYPES) - sefd_p1 / sefd_p2: per-feed SEFDs (alternative to legacy `sefd`) - d_p1 / d_p2: per-feed D-terms Legacy paths preserved bit-identically: bare add_site('A', coords) still yields a row with sefd_p1=sefd_p2=10000, d_p1=d_p2=0+0j, feed_type='rl'. Legacy `sefd`, `dr`, `dl` kwargs continue to work for feed_type='rl'. Mixing legacy and generic kwargs for the same field is rejected with ValueError: - sefd + (sefd_p1 or sefd_p2) - dr + d_p1 - dl + d_p2 - dr/dl with feed_type != 'rl' - sefd_p1 without sefd_p2 (or vice versa) - unrecognised feed_type strings Resolution helpers live as module-level functions (_resolve_sefd_pair, _resolve_dterm_pair, _validate_feed_type) so they can be reused by later commits (Obsdata polrep validation, etc.). 15 new test_phase2_* tests; 43 total in test_mixedpol.py (Phase 1: 30 + Phase 2 so far: 13 + 15 = 28). No regressions in the broader Array/Obsdata/imager test surface. TODOs at array.py:227-229, :274, :302 removed. Implements obsdata_mixedpol_plan_v2.md Phase 2, Commit 2. * Phase 2 (3/N): TarrView wrapper raising legacy R/L access on non-RL arrays Array.tarr now returns a TarrView wrapping the underlying ndarray. Whole-column access of 'sefdr'/'sefdl'/'dr'/'dl' raises KeyError on any array with non-RL stations, pointing callers at sefd_for_feed / dterm_for_feed. Homogeneous-RL arrays unchanged. Row-form (tarr[i]['sefdr']) is unguarded — known limitation, numpy owns the void. Storage stays as plain ndarray; setter unwraps views. __array_interface__ / __array_struct__ are hidden from __getattr__ so np.asarray() routes through our __array__ and preserves the structured dtype. 19 tests added; full suite 1072 passed, no regressions. * Phase 2 (4/N): Array.obsdata(polrep=...) extension to 'lin' and 'mixed' Widens valid polreps to {'stokes', 'circ', 'lin', 'mixed'} with explicit feed-type validation: - 'circ' requires every station feed_type == 'rl' - 'lin' requires every station feed_type == 'xy' - 'mixed' requires at least two distinct feed types 'lin' and 'mixed' validate cleanly but then raise NotImplementedError pointing to Phase 5 — make_uvpoints is still circular-only. This gives callers a clear error instead of a downstream pipeline break. 9 tests added. * Phase 2 (5/N): save_array_txt v2 emit + feed_type column save_array_txt now writes a leading '# ehtim array format v2' header and a 14th feed_type column. Numeric layout, column order, and printf precision are unchanged from the legacy 13-column format; the v2 header is a forward-compat marker so future schema changes can bump to v3+ cleanly. load_array_txt continues to dispatch on column count (5/13/14) so all legacy files load bit-identically. 13 tests added, including a parametrized regression that loads every arrays/*.txt file and asserts numeric columns match a fresh np.loadtxt read. * Phase 2 (6/N): save_obs_txt embedded-tarr feed_type column save_obs_txt now appends feed_type (14th column) to each station row in the obs.txt embedded tarr. load_obs_txt's Phase 1 len==15 branch (14-col + leading '#' marker) handles this without further change. 1 round-trip test added.
* Collapse sX + reg_X regularizer pairs and uniformise scm/scmgrad masking (#258) * Uniformise scm/scmgrad masking pattern * Collapse Stokes-I sX + reg_X pairs into unified reg_X functions * Collapse polarimetric sX + reg_X pairs into unified reg_X functions * Collapse multifrequency sX + reg_X pairs into unified reg_X functions * Drop now-unused sX functions and exports * Sort imports / split semicolons in test_regularizers * Refresh stale section headers in regularizer files * Fix reg_tv2log embed to use clipfloor=epsilon_tv * Restore math-derivation tags and carried TODOs in regularizers * Add cm gradient FD test under a partial mask * Add explicit norm=1 placeholder to reg_l1w; document rgauss normalization * Keep dev-backend-mixpol in CI triggers (mirror of #261) (#262) * Add initial test suite for image.py (#263) * Add initial test suite for image.py 159 tests covering construction, polrep getters/setters, switch_polrep, summary quantities, geometric transforms, blur/mask/grad, source-adders, multifrequency, sampling/observe, compare/align, fit_gauss, save/load, and module-level helpers. Reuses session fixtures from conftest.py. Two latent bugs discovered while writing the suite are captured as xfail tests so a future fix flips them to passing: - Image.evpa() circ branch missing 0.5 factor (image.py:979) - Image.grad(gradtype='x') aliases 'abs' due to missing `elif` (image.py:1786-1791) * fixed import sort ruff fail in test_image.py * Fix two Image bugs surfaced by test_image.py xfails - Image.evpa() circ branch: add missing 0.5 factor so it matches stokes - Image.grad(): make second `if` an `elif` so gradtype='x' is not overwritten by the `else` branch - Drop the two xfail markers in tests/test_image.py (now 161 passed) * Add initial test suite for array.py (#264) * Add initial test suite for array.py 43 tests covering construction, listbls, make_subarray, add_site, remove_site, add_satellite_elements, obsdata, save_txt/load_txt round-trip, and deepcopy/pickle interop. Reuses the eht_array session fixture from conftest.py. Designed to be forward-compatible with PR #260 (Phase 2 mixed-pol Array migration): tests interact with arr.tarr only via behaviour the upcoming TarrView wrapper preserves, use only legacy 'rl' feed semantics, and avoid asserting the full DTARR field tuple. * fixed lint import order in test_array.py * Add tests for obs_simulate (#265) * Add tests for obs_simulate * removed unused copy import in test_obs_simulate.py * removed unneeded np.random.seed(0) calls in test_obs_simulate.py * Fix sig_ll using sefdr instead of sefdl in add_noise. add_noise's SEFD recomputation used (sefdr1, sefdl2) for sig_ll instead of (sefdl1, sefdl2). Invisible when sefdr == sefdl; breaks for asymmetric arrays. Add asymmetric-array tests in stokes and circ polreps in test_obs_simulate.py * Add SEFD-pairing tests for jones noise paths. Pin the four per-correlation SEFD pairings in add_jones_and_noise and apply_jones_inverse using the asymmetric-array fixture, so a future wrong-SEFD-column bug in those paths is caught the same way. * Fix tlist/bllist collapsing to 2-D when scans share a shape (#266) np.array(datalist, dtype=object) silently stacks a list of equal-shape recarrays into a 2-D ndarray, dropping the structured dtype. Downstream field-name indexing (e.g. bispectra() -> tdata[0]['time']) then raises IndexError. Pre-allocate the 1-D object array and fill element-by-element to force the intended shape. Closes #186. * Fix mixpol regressions surfaced by new dev-backend tests - obs_simulate.make_jones: DTCAL gained dr/dl in Phase 1, so the caltable-write row needs all 5 fields, not 3. - Array.remove_site: self.tkey[site] raises KeyError, not IndexError; catch both so the informative message surfaces on a missing site. - test_array.test_init_tarr_dtype_has_legacy_columns: check dtype.fields (includes title aliases) rather than dtype.names (primary only). --------- Co-authored-by: Rohan Dahale <rohan.dahale@utoronto.ca>
* Add ehtim/observing/pol_conventions.py (Phase 3 step 1)
Single home for basis transforms (circ/lin/stokes), per-component sigma
propagation (with TODO: full covariance), and Jones-matrix scaffolding
(J = G @ (I + D)). Convention: IAU/HBS, R = (X - iY)/sqrt(2).
MixedPolConventionWarning fires once per session on first transform.
No callers yet; obsdata.py migration is step 3.
* Add docs/polarization_conventions.md (Phase 3 step 2)
12-section reference doc: convention statement (IAU/HBS), basis
transforms, Stokes derivations for circ and lin, visibility-sigma
propagation (with the per-component / no-cross-covariance limitation
spelled out), Jones-matrix machinery, and the polrep-specific slot
table. Each section cites the implementing line range in
pol_conventions.py.
* Migrate Obsdata circ<->stokes to pol_conventions (Phase 3 step 3)
The inline if/elif field-name dispatch at obsdata.py:282-334 is replaced
with direct calls into pol_conventions.{circ_to_stokes,stokes_to_circ}
and the matching *_sigma helpers. Bit-identical numerical behavior;
allow_singlepol and NaN-mask handling stay in place.
Also: rename stdlib import in ehtim/__init__.py to _stdlib_warnings.
Bare `import warnings` was clobbering the ehtim.warnings submodule
attribute, which broke `import ehtim.warnings as ehw` once the new
pol_conventions import chain pulled the submodule in before
__init__.py's import line ran (latent bug, exposed by step 3).
* Widen Image to 'lin' polrep (Phase 3 step 4)
- Accept polrep='lin' with pol_prim in {XX, YY}; _imdict gets XX/YY/XY/YX
keys, mirroring the circ pattern.
- Add cross-polrep getters: xxvec/yyvec/xyvec/yxvec compose through the
stokes pair on circ/stokes sources; ivec/qvec/uvec/vvec also gain a
'lin' branch. The stokes-derived pvec/mvec/chivec widen 'stokes' ->
'stokes or lin' (formulas unchanged).
- add_pol_image and get_polvec learn the XX/YY/XY/YX pol keys.
- Simplify switch_polrep: since the cross-polrep accessors already
return empty arrays when a transform can't close, the per-source
branching collapses to one line per output polrep. circ <-> lin
composes through stokes per plan Decision 7.
- Flip the two test_image.py tests that asserted 'lin' was rejected.
test_init_accepts_lin_polrep added.
Existing test suite (162 image tests + ~1200 others) passes unchanged.
* Factor pol_conventions into pair-based primitives (Phase 3 step 4a)
The 4-in/4-out functions (circ_to_stokes etc.) become thin wrappers
around 8 pair-based primitives:
- circ_to_stokes_parallel/cross (RR,LL <-> I,V; RL,LR <-> Q,U)
- stokes_to_circ_parallel/cross (inverses)
- lin_to_stokes_diag/offdiag (XX,YY <-> I,Q; XY,YX <-> U,V)
- stokes_to_lin_diag/offdiag (inverses)
This gives image/movie property getters a single-Stokes-component
entry point without paying for the other three or constructing dummy
zeros. Image refactor lands in step 4b; Movie uses these from day one
in step 5.
Argument names also drop the 'vis' suffix (rrvis -> rr etc.) — these
are polarization-basis transforms, not visibility-specific. Section
heading "Visibility transforms" -> "Polarization transforms" to match.
Obsdata callers are positional so unaffected.
* Refactor Image polrep math to pol_conventions pair helpers (Phase 3 step 4b)
All 12 polrep-conversion code paths in Image's vec getters
(ivec/qvec/uvec/vvec, rrvec/llvec/rlvec/lrvec, xxvec/yyvec/xyvec/yxvec)
now call the pair primitives from pol_conventions instead of inlining
the basis-transform math. np.real() projections on cross-hand and
off-diagonal outputs stay at the call site (they're a real-domain
coercion, not a basis-transform concern).
Numerical behavior unchanged; 444 image/obsdata/mixedpol tests pass.
docs/polarization_conventions.md line refs updated for the
pol_conventions reshape from step 4a.
* Widen Movie to 'lin' polrep + harmonize frame accessors (Phase 3 step 5)
Largest behavior change in Phase 3. Previously every Movie frame
accessor (iframes, qframes, rrframes, rlvec, etc.) raised when polrep
didn't match. Now they compose through the canonical pair for the
source polrep via pol_conventions, matching Image's pattern.
- Constructor accepts polrep='lin' with pol_prim in {XX, YY}.
- Adds xxframes/yyframes/xyframes/yxframes accessors.
- Existing 8 accessors (iframes/qframes/uframes/vframes/
rrframes/llframes/rlframes/lrframes) get cross-polrep getters.
Setters keep their polrep guard (you can only write to storage in
the storage basis) and continue rebuilding _fundict interpolators.
- Cross-polrep reads do not populate _fundict; callers needing time
interpolation in a different basis should switch_polrep first.
- rlvec / lrvec renamed to rlframes / lrframes (consistent with the
rest of Movie). The old names are kept as shims that raise
AttributeError with a migration message; no in-tree callers used
them on Movie objects (only Image.rlvec/lrvec, which stay).
- switch_polrep widened to accept and produce 'lin'; the per-source
cross-polrep math collapses to single-line dict assembly because
the accessors do the work. circ <-> lin composes through stokes.
- add_pol_movie learns the XX/YY/XY/YX pol keys with the matching
_fundict interp1d construction.
- Constructor now explicitly raises on unknown polrep (was silently
falling through).
1382 existing tests pass unchanged.
* Widen Model to 'lin' polrep (Phase 3 step 6)
- switch_polrep accepts polrep_out='lin' with pol_prim_out='XX'.
- observe_same_nonoise populates xxvis/yyvis/xyvis/yxvis on a 'lin' Obsdata.
- 9 sample_uv* dispatch sites (sample_1model_uv, sample_1model_graduv_uv,
sample_1model_grad_leakage_uv_re/im, sample_1model_grad_uv, plus the
no-Jones recursive fallbacks and get_const_polfac) gain XX/YY/XY/YX
clauses with a pointer comment to pol_conventions.py.
Each Jones-applied site uses the inline formulas:
XX = 0.5*(RRp + LLp + LRp + RLp)
YY = 0.5*(RRp + LLp - LRp - RLp)
XY = 0.5j*(LRp - RLp - RRp + LLp)
YX = 0.5j*(LRp - RLp + RRp - LLp)
Each non-Jones / recursive site delegates to the I/Q/U/V calls per the
stokes_to_lin formulas. Behavior validated via cross-check against
pol_conventions.stokes_to_lin and pol_conventions.circ_to_lin.
1382 existing tests pass unchanged.
* Extend const_def poldicts with XX/YY/XY/YX entries (Phase 3 step 8)
vis_poldict, amp_poldict, sig_poldict gain entries for the linear-basis
pol keys, mirroring the existing RR/LL/RL/LR entries. Downstream callers
that do pol-name -> field-name lookups (e.g. Obsdata.dirtyimage at
obsdata.py:1548) now work on 'lin' polrep observations.
Also reword the migration TODO above feed_dtype_for_polrep: the
pol_conventions.py module exists now (created in Phase 3 step 1); the
helpers themselves still need to move when Obsdata.switch_polrep is
wired up to the new module.
* Add tests/test_pol_conventions.py (Phase 3 step 7a)
25 tests covering:
- pair-based primitives (parallel/cross for circ, diag/offdiag for lin)
and their 4-in/4-out wrappers
- round-trips in all four bases
- cross-convention consistency (stokes -> circ vs stokes -> lin
produces the same physics via lin_to_circ)
- IAU/HBS sign convention (pure-V Stokes -> XY=-i, YX=+i)
- sigma propagation (matches the existing inline obsdata.py formulas
bit-for-bit)
- Jones-matrix scaffolding (identity, D-term off-diagonals,
apply_inverse_jones round-trip with small D-terms)
- MixedPolConventionWarning fires exactly once per session
* Add lin-polrep tests to tests/test_image.py (Phase 3 step 7b)
13 new tests in Section 5b covering:
- Image construction with polrep='lin' + pol_prim XX/YY
- switch_polrep('lin') round-trips through stokes
- explicit stokes_to_lin formulae (XX=I+Q, YY=I-Q, XY=U-iV, YX=U+iV)
- circ -> lin via two-step composition (plan Decision 7)
- cross-convention check: Stokes recovered identically via circ and lin
- xxvec/yyvec/xyvec/yxvec computed-property semantics on stokes images
- xxvec setter raises on non-lin polrep
- add_pol_image learns XX/YY/XY/YX pol keys
* Add tests/test_movie.py (Phase 3 step 7c)
24 tests for Movie's Phase 3 additions (Movie had no dedicated test
file before this commit). Covers:
- polrep='lin' construction with pol_prim XX/YY
- harmonized cross-polrep frame accessors: iframes/qframes/rrframes/etc.
all compose on-demand for any source polrep (was: raise on mismatch)
- xxframes/yyframes/xyframes/yxframes accessors and computed semantics
- cross-polrep read does not populate _fundict (interp1d stays on
storage basis only)
- setter polrep guards intact (storage basis is the only writable one)
- switch_polrep round-trips through stokes/circ/lin
- circ <-> lin via stokes composition (Decision 7)
- rlvec/lrvec deprecation shims raise AttributeError with migration
message; rlframes/lrframes are the new names
- add_pol_movie learns XX/YY/XY/YX pol keys
* Add tests/test_model.py (Phase 3 step 7d)
11 passing + 1 xfail covering:
- Model.switch_polrep('lin') accepts the new polrep_out (validation
widening). Note: Model.switch_polrep is a no-op by design; only the
validation surface is verified here.
- get_const_polfac dispatches XX/YY/XY/YX via the stokes_to_lin
pair formulas.
- sample_1model_uv produces lin visibilities consistent with both
pol_conventions.stokes_to_lin (per Stokes component) and
pol_conventions.circ_to_lin (full cross-basis route).
- observe_same_nonoise on a lin Obsdata is xfail (strict): the path
goes through model.py:1470 np.sum(generator) which was removed in
numpy 2.0. Pre-existing bug, out of scope for Phase 3; the xfail
flips to a pass once the np.sum is fixed.
* Document Model pre-Phase-3 issues in test_model.py header
The polrep slot on a Model is always 'stokes' (init hardcodes it,
switch_polrep is a no-op) and sample_uv/observe_same_nonoise hit a
numpy 2.0 incompat in sample_model_uv. The header spells these out
so a reader doesn't wonder why the assertions are narrow.
* linted ehtim/__init__.py
* lined test_movie and test_model
* reverted import order in __init__.py to avoid circular imports breaking
* Address PR #270 review + fix §2/§4/§5 convention inconsistency
- pol_conventions: switch §2 basis to R = (X+iY)/sqrt2 (engineering convention,
matching historical §4); flip V sign in lin_to_stokes_offdiag and
stokes_to_lin_offdiag accordingly
- docs/polarization_conventions.md: rewrite §1 time convention, §2 basis, §5
derivation; add §5a TMS Eq. 4.28 comparison
- image.py: route inline cross-Stokes through circ_to_stokes_cross (Rohan #3)
- pol_conventions: add "reference only" notes on BASIS_LIN_TO_CIRC and
invert_jones (Rohan #2)
- tests: flip expected XY/YX signs; add regression test for §2/§4/§5
self-consistency
* Route model.py pol transformations through pol_conventions
Replace inline §4/§5 formulas in the 5 Jones-leakage dispatch blocks with
circ_to_stokes / circ_to_lin calls, and the 4 stokes-recursion blocks with
the stokes_to_circ_* / stokes_to_lin_* pair primitives. Catches a missed
XY/YX V-sign bug in get_const_polfac (escaped the earlier sweep because
its inline form didn't match the grep pattern).
* const_def: add POLDICT_MIXED, polrep lookup, LIN/generic FIELDS entries
Adds POLDICT_MIXED (generic p1p1vis/... slot names) and a
polrep_to_poldict lookup so Obsdata.__init__ can dispatch by dict.
Extends FIELDS and FIELDS_AMPS/SIGS/PHASE/SIGPHASE/SNRS with
xx/yy/xy/yx and p1p1/p2p2/p1p2/p2p1 entries so unpack() can resolve
LIN and generic-slot field requests.
* obsdata: dispatch polrep via lookup; enforce dtype/polrep agreement
Replaces the stokes/circ-only Obsdata.__init__ dispatch with a lookup
over ehc.polrep_to_poldict, so 'lin' and 'mixed' are accepted as
valid polreps. Cross-checks datatable.dtype against the declared
polrep instead of just whitelisting it. Rejects polrep='circ' when
tarr feed types are not all 'rl', and 'lin' when not all 'xy'.
Updates two test_obsdata.py error-message assertions to match the
new error strings (polrep='lin' is no longer a "bad polrep" case).
* feed_type: centralise VALID_FEED_TYPES; populate polbasis losslessly
Moves the VALID_FEED_TYPES frozenset from array.py to const_def.py so
both Array and Obsdata can validate against the same source.
Obsdata.__init__ now rejects tarr feed_type='??' or any value outside
VALID_FEED_TYPES, for every polrep. For polrep='mixed' it also
requires at least two distinct feed types and that every (t1,t2)
station is present in tarr, then populates data['polbasis'] as the
lossless 4-char concatenation of t1 and t2 feed_types (e.g. 'rlrl'
for both circular, 'rlxy' for circ/lin). polbasis dtype changes
from U2 to U4 in DTPOL_MIXED to fit the lossless encoding.
* obsdata: raise on dirtyimage for polrep='mixed'
Image has no 'mixed' polrep and per-baseline feed-basis interpretation
varies across rows, so a single dirty image is not well-defined.
Raise informatively pointing the caller at switch_polrep('stokes').
LIN flows through unchanged via the existing vis_poldict reverse map
and Image's existing XX/YY/XY/YX support.
* obsdata: raise on avg_coherent / avg_incoherent for LIN/MIXED
The pandas-routed averaging code in statistics/dataframes.py only
understands CIRC/STOKES dtypes; let the underlying code handle the
existing polreps and surface a NotImplementedError early for LIN
and MIXED so callers get a clear message instead of obscure pandas
errors later.
* Fix reorder_baselines: distinguish conjugate pairs from data loss (#267)
* Fix reorder_baselines: distinguish conjugate pairs from data loss
The default reorder_baselines path deduped by set((t1, t2)) per time
bucket and silently kept only the first row, with no signal to the
caller. This collapsed true (A,B)/(B,A) conjugate pairs (intended)
along with genuine duplicates and multi-channel rows flattened into a
single-frequency Obsdata (silent data loss; the symptom reported in
#140).
The opt-in reorder_baselines_trial_speedups variant (only reached when
the caller explicitly passed trial_speedups=True; off by default in
Obsdata.__init__, reorder_baselines, and load_uvfits) had a separate
dead-code bug: dat_unique = dat[deletemask] was built but never
assigned back, so when invoked it skipped dedup entirely while
printing a WARNING that it had removed duplicates.
Unify both paths into one vectorized implementation that:
- swaps reversed-baseline rows into canonical order;
- identifies a true conjugate pair as exactly two rows with opposite
original orderings and matching (u, v) after the swap, and drops
one silently;
- warns loudly via UserWarning on any other collision (true duplicate,
>2 rows, or mismatched (u, v) -- the multi-channel symptom).
Keep trial_speedups on the signature as a no-op for API compatibility;
it still controls separate optimizations in load_uvfits.
Closes #140.
* Add load_obs_uvfits trial_speedups parity test
load_obs_uvfits accepts trial_speedups to enable a vectorized site lookup
and an alternate datatable assembly path. The two paths were not covered
by any equivalence test. Add one against data/sample.uvfits asserting
identical data and tarr.
* Refine warning for dropped duplicate rows
Updated warning message to clarify the reason for dropped duplicates.
* obsdata: reorder_baselines LIN branch
Adds the linear-feed conjugate/swap analogue of the existing circ
branch in the vectorized reorder path: xxvis/yyvis conjugate,
xyvis<->yxvis swap-and-conjugate, xysigma<->yxsigma swap. Widens
the trailing raise to include the polrep value for clarity.
* obsdata: reorder_baselines MIXED branch
Adds the generic-slot conjugate/swap analogue for polrep='mixed', and
swaps the two halves of polbasis on reordered rows so the t1+t2 feed
concatenation stays consistent with the new (t1, t2) ordering. Same
algebra as the circ/lin branches, just expressed in p1p1/p2p2/p1p2/
p2p1 slot names.
* tests/mixedpol: phase 4a obsdata coverage (22 tests)
Covers LIN/MIXED Obsdata.__init__ (synthetic construction, attribute
wiring, polbasis population, tarr/polrep and dtype agreement
rejection, ?? and unknown feed_type rejection, mixed-station presence
check), reorder_baselines LIN and MIXED swap with polbasis half-flip,
and the NotImplementedError guards on avg_coherent / avg_incoherent
(LIN/MIXED) and dirtyimage (MIXED). LIN dirtyimage coverage is
intentionally deferred until unpack_dat gains the LIN branch.
* obsdata: switch_polrep accepts 'lin' out; rejects mixed source
Widens the polrep_out whitelist to ('stokes','circ','lin') and adds
an early raise for self.polrep='mixed' (no path out of the mixed
representation without Jones-level processing, not yet implemented).
The lin output branch itself is added in a follow-up. Updates the
test_switch_polrep_invalid_raises sentinel from 'lin' to 'bogus'
since 'lin' is no longer a "bad polrep" case.
* obsdata: switch_polrep lin<->stokes; warn on singlepol fallback
Adds lin->stokes and stokes->lin branches via pol_conventions, forks
the dispatch on (self.polrep, polrep_out), and extends singlepol_hand
to accept 'X'/'Y' for lin target (validation lives inside the
allow_singlepol block, so the kwarg is only checked when used).
Emits a UserWarning whenever allow_singlepol substitutes any rows,
noting that cross-hand visibilities are not filled. circ<->lin
paths remain a NotImplementedError pending the two-step composition.
* obsdata: switch_polrep circ<->lin via two-step composition
Adds the circ<->lin paths as a single user-facing call that routes
through stokes. The intermediate Obsdata is built with the original
tarr; to make that work, the chunk-3 strict tarr/polrep agreement
check is relaxed to a UserWarning -- polrep declares the data-layout
basis, tarr.feed_type describes the physical array, and the two are
no longer required to match. The mismatch still surfaces (warning,
not silent), and Obsdata constructed directly from data on a
non-matching tarr will trigger it. Two phase-4a tests updated from
'asserts raise' to 'asserts warn'.
* tests/mixedpol: phase 4b switch_polrep coverage (17 tests)
Covers lin<->stokes round-trip, cross-convention check (circ and lin
constructions of the same Stokes match when both convert back),
circ<->lin two-step composition equality with the explicit chain, and
the circ -> lin -> circ round-trip. Also exercises the mixed source
raise, invalid polrep_out raise, allow_singlepol fill on lin->stokes,
the singlepol warning's fire/no-fire conditions, and singlepol_hand
validation (X/Y vs R/L, case-insensitive, skipped when disabled).
A tarr-preservation test asserts that switch_polrep does not mutate
tarr.feed_type.
* removed references to plan phases in test_mixedpol.py
* removed trailing whitespace in test_image.py
* Generalize data_conj to lin and mixed polreps
Drive the conjugate-baseline swap off the generic poldict slots so circ,
lin, and mixed share one path; flip the polbasis halves on conjugate rows
for mixed. circ/stokes output is unchanged.
* Support lin and mixed polreps in closure quantities
Consolidate make_bispectrum/make_closure_amplitude onto one per-correlation
helper that synthesizes across bases via pol_conventions, adding lin and
stokes->lin support. On mixed-feed observations, form closures only across
baselines sharing one feed basis and skip the rest with a
MixedPolClosureSkipWarning (with a minimal-set caveat); Stokes and
generic-slot closures on mixed raise. The diagonal-closure covariance now
routes through the same helper, fixing a latent crash on cross-synthesized
vtypes (e.g. cphase_diag(vtype='vis') on a circ obs).
Regression-checked against v1.3 on data/sample.uvfits and the HOPS M87 file:
closure values and triangle assignments are bit-identical for stokes and
circ polreps across all vtypes and the diagonal closures; only
synthesized-vtype bispectrum sigmas differ at ~1e-13 (a sqrt/square
round-trip, i.e. machine precision).
* Add general coherency<->Stokes transform for mixed feeds
Add feed_matrix and coherency_to_stokes / stokes_to_coherency (+ sigma) to
pol_conventions. Under ideal feeds a baseline measures the full coherency
V = F1 B F2^dagger, so Stokes is recoverable from any feed pairing --
including heterogeneous (circular x linear) and hybrid, non-orthogonal feeds
-- via B = F1^-1 V (F2^dagger)^-1. Feed matrices are built from
BASIS_LIN_TO_CIRC (engineering convention R=(X+iY)/sqrt2) and the operator is
cached per feed pairing. Cross-validated to ~1e-13 against circ_to_stokes /
lin_to_stokes on homogeneous pairings. Foundation for mixed-pol unpack and
later mixed-pol imaging; not yet wired into unpack.
* Support lin polrep in unpack via pol_conventions consolidation
Route unpack_dat's direct-correlation blocks (vis/qvis/uvis/vvis/
rrvis/llvis/rlvis/lrvis + new xxvis/yyvis/xyvis/yxvis) through
obs_helpers.vis_component, so all basis transforms come from
pol_conventions; add generic-slot fields (p1p1vis ...) via the DTPOL
title alias, and extend the suffix lists. evis/bvis source q,u from
vis_component too; pvis/m keep the native circ path to preserve circ's
sigma. This adds lin support and makes unpack on a lin obs match
switch_polrep('stokes') exactly. unpack on mixed raises NotImplementedError
for now (next commit).
Verified bit-identical to v1.3 for stokes and circ unpack across all legacy
fields (including every sigma) on data/sample.uvfits and the HOPS M87 file.
* Support mixed polrep in unpack (row-aligned NaN-fill)
Add correlation_slot to pol_conventions, and an "Unpack Helpers" section to
obs_helpers (alongside vis_component, which moves there). unpack_vis_mixed
dispatches a mixed-feed vis-family field per row via polbasis: generic slots
read directly; physical/cross names (rrvis, xxvis, ...) return the matching
slot where the baseline provides it and NaN elsewhere (warning with per-field
NaN counts via the new MixedPolUnpackNaNWarning); Stokes-derived fields are
recovered on every row, including heterogeneous baselines, via
coherency_to_stokes. unpack_dat delegates the mixed case and reads generic
slots through obs_helpers. stokes/circ/lin unpack is unchanged.
* Extract unpack_vis_standard for symmetry with unpack_vis_mixed
Move the stokes/circ/lin vis-family field dispatch out of Obsdata.unpack_dat
into obs_helpers.unpack_vis_standard. unpack_dat now delegates the vis-family
case to unpack_vis_standard (stokes/circ/lin) or unpack_vis_mixed (mixed) and
applies the shared suffix stage; all polarization-field logic now lives in
obs_helpers. Pure relocation -- verified bit-identical to v1.3 for stokes and
circ unpack on data/sample.uvfits and the HOPS M87 file.
* Detect mixed-pol uvfits in load_uvfits and raise
Peek at the AIPS AN POLTYA and POLTYB columns; if any feed type is not R/L,
raise NotImplementedError instead of silently loading the data as circular
(this also catches hybrid stations, e.g. POLTYA=R with POLTYB=X). Full
POLTYA/POLTYB -> feed_type parsing is deferred (Phase 7). All-R/L files load
unchanged.
* Test Stokes-I closure imaging is polrep-transparent (circ and lin)
Parametrize the end-to-end polrep-transparency test over circ and lin obs:
imaging Stokes I from a circ- or lin-polrep observation reproduces the
stokes-obs reconstruction (exact for the unpolarized Gaussian, since I
synthesis from rr/ll or xx/yy is exact).
* reorganized obs_helpers.py
* refactored unpack_vis_mixed and unpack_vis_standard into single unpack_vis
* Fix mixed-pol unpack_vis regressions and require numpy>=2.0
Restore mixed-pol pvis/m/rrllvis (dropped in the unpack_vis merge) by
routing them through vis_component, and make unpack_mixed_stokes/
unpack_mixed_correlation return scalars for single-row input so
bispectra/c_phases work under numpy 2.x (numpy 1.x hid this by
squeezing size-1 arrays).
Bump deps to numpy>=2.0/scipy>=1.13/astropy>=6.1/matplotlib>=3.9 and
requires-python>=3.10; add mixed pvis/m/rrllvis tests, drop a stale
xfail fixed by merged #260, and tidy whitespace/docstrings.
* Replace lru_cache with cache decorator
* Make polrep-transparent e2e test robust to optimizer drift
Assert the exact invariant (vis/amp data products equal to 1e-12 between
the stokes obs and its switch_polrep) plus nxcorr > 0.99 on the images,
instead of a 1e-10 exact-imvec check. L-BFGS-B amplifies ~1e-12 input
deltas into ~1e-3 pixel drift depending on BLAS/scipy reduction order
(cf. #232); the old assertion only passed by luck of the local stack.
* Fix mixed-feed diagonal closures producing NaN sigmas / IndexError
c_phases_diag and c_log_amplitudes_diag assembled the per-baseline
covariance over all baselines at a timestamp, so NaN-filled cross-feed
baselines on a mixed obs poisoned the surviving triangle/quad via
0*NaN. Restrict viss_here to the baselines actually referenced by the
surviving (polbasis-filtered) closures before building sigma_mat. Also
return [] when no closure survives, instead of indexing a fieldless
empty array (the IndexError in c_log_amplitudes_diag).
Strengthen the mixed diag test to assert finite sigmas and add
c_log_amplitudes_diag mixed tests (empty-set + surviving-quad).
* Test closure-amplitude family on lin / mixed obs
Add c_amplitudes/camp_quad lin-matches-stokes equality tests and mixed
quad-skip tests (cross-feed quad skipped, homogeneous quad kept),
mirroring the existing bispectra/c_phases coverage.
* Regression-test moved unpack fields stokes vs circ
The inline stokes/circ unpack for evis/bvis/m/rrllvis (and rr/ll/rl/lr)
was deleted and routed through unpack_vis; assert the stokes obs and its
circ switch still agree to 1e-12 on all of them.
* Add independent ground-truth test for mixed unpack
Build a mixed obs from known Stokes via stokes_to_coherency per row and
assert unpack recovers I/Q/U/V on the heterogeneous rlxy rows.
* Drop pyproject packaging bump from Phase 4c data PR
The mixpol fix does not require numpy>=2.0: the single-row unpack path
returns scalars explicitly and behaves identically on numpy 1.24 and 2.4.
The numpy-2 / scipy / python-version policy is already decided in #271 on
dev-backend (requires-python >=3.11,<3.13) and should reach this branch
via the dev-backend sync, not as divergent hand-rolled floors here.
* Reject degenerate same-feed codes in feed_matrix
A same-feed code like 'rr' builds a rank-1 singular matrix that would
fail deep in linalg; reject it at the boundary so the "always invertible"
docstring holds. Add a negative test.
* Guard correlation_slot against corrupt feed codes
Reject feeds that aren't two distinct letters so a corrupt polbasis
fails loudly instead of mis-resolving via str.find. Add a test.
* Drop redundant polrep revalidation in data_conj
The constructor already enforces polrep against the central
polrep_to_poldict definition, so the hardcoded-tuple check was dead code.
Resolve 11 conflicts: keep mixpol's feed_type/TarrView schema and lin-polrep widening; take dev-backend's new functions, NumPy2 import cleanup, KeyError fixes, and parametrized/deprecation tests; union the add/add test files. Alias pol_cal's `import warnings` to avoid clobbering the ehtim.warnings submodule via `from ...pol_cal import *`. Known: dev-backend's #276 calibration tests expose a pre-existing mixpol bug (DTCAL widened to 5 fields but write-sites still emit 3-tuples); fixed separately on dev-backend-mixpol.
mixpol widened DTCAL to 5 fields (gain scales + d_p1/d_p2) but left the gain-table write-sites emitting 3-tuples, which fail against the widened dtype. dev-backend's #276 calibration suite exposes this. Append zero D-terms (no leakage) at the 9 gain write-sites in caltable, self_cal, network_cal, polgains_cal, and the unity-caltable test helper. Also ruff-clean polgains_cal.py, which the fix pulls into CI's changed-file lint scope for the first time.
A zero-initialised np.zeros(dtype=DTARR) has the feed_type field present
but blank (''), so upgrade_tarr's field-presence check treated it as
already-upgraded and left '', which Obsdata validation then rejected.
Fill blank feed_types with the legacy 'rl' default; the '??' must-declare
sentinel still raises. Fixes dev-backend's test_averaging.
modeler_func(fit_gains=True) appended 3-tuple gain rows, failing the widened 5-field DTCAL cast. Pad with zero D-terms, matching the other write-sites. Add a minimal fit_gains test so the path isn't CI-blind.
…end-into-mixpol-287
Merge dev-backend (v1.4.0) into dev-backend-mixpol
Add package build and check steps to CI workflow
* Add physical_grad_slots helper Maps the Stokes DOF mask to the physical gradout slots the chisq/reg kernels must fill, centralizing the mcv/vcv cross-coupling that mirrors transform_gradients' Jacobian sparsity. Not yet wired in. + unit tests. * Wire physical_grad_slots into chisq and reg gradient dicts Feed the cross-coupling-aware mask to the pol gradient kernels in both compute_chisqgrad_dict and compute_reggrad_dict. Behavior-identical for now (kernels still carry the or-patches). Guard physical_grad_slots against sub-4-wide single-pol masks (Stokes-I carries 'mcv' inertly). + regression test. * Revert vvis kernels to diagonal pol_solve gating The mcv/vcv cross-coupling now lives in physical_grad_slots, so drop the 'or pol_solve[3]' patches (#296) in chisqgrad_vvis / chisqgrad_vvis_nfft; each physical slot keys on its own bit again. Note in each pol chisqgrad docstring that pol_solve flags required physical gradients, not DOFs. * Fix reggrad_ptv first-row/col boundary masking + epsilon_tv Zero the back-neighbor (m2/m3) terms on the first row/column in reggrad_ptv slots 0/1/3 (the back-neighbor is the zero pad), matching reggrad_vtv/reggrad_tv. Pre-fix the whole first row+col of those slots was wrong (corner ~4x off vs FD). Add epsilon_tv to reg_ptv/reggrad_ptv denominators (default 0, byte-identical) for #295 parity. Note pol_solve = physical-gradient slots in the 8 pol reggrad docstrings. Add full-grid boundary FD regression tests for ptv, vtv, and Stokes-I tv. * Note pol_solve semantics in polchisqgrad docstring polchisqgrad is a legacy shim (parity tests only); document that its pol_solve is a physical-gradient mask, not a raw DOF mask. * Drive pol regularizer FD with all four physical slots _pol_solve_for now returns [1,1,1,1] so the previously-blind cross- coupling slots are FD-checked: reggrad_ptv psi (3), reggrad_vflux/l1v/ l2v/vtv rho (1), and slot 0 for every pol reg. Proves the reg-grad slots are individually correct against finite differences. * Add pol chisq FD + cross-ttype tests in test_chisquared.py New pol coverage in its final-home file: TestPolChisqGradFD checks chisqgrad_p/m/vvis against finite differences of the chisq value in all four physical slots (pol_solve=[1,1,1,1]) for direct+nfft, asserting vvis slot 2 (EVPA) is identically zero. TestPolChisq{,Grad}Consistency check direct-vs-nfft agreement. Closes the m / p-slot-3 blind spots. * Add parametrized pol objective-FD sampling the polarization DOF block TestObjectiveGradPolarimetricFD checks objgrad vs FD for IP/IV/IQUV x {direct,nfft}, with each case bundling its pol data terms + a pol reg, so both the chisq and reg gradient paths through physical_grad_slots are exercised. Samples the pol DOF block (past the Stokes-I block), where the mcv/vcv cross-coupling lives -- the existing global-sampling FD tests missed it (the dropped IP slot-3 term is ~4% off FD at V=0.02*I, ~430% at V=0.2*I). Comments out the now-subsumed test_fd_matches_analytic_polarimetric (backend) and test_iv_gradient_matches_finite_difference (e2e). * Use an asymmetric image for chisq/regularizer/gradient FD fixtures Add make_asym_image (broad offset/elongated/rotated double-Gaussian) and switch the Stokes-I FD fixtures (chisq_setup, reg_setup, mfreg_setup, grad_setup) to it. Breaking the reflection/rotation/x<->y symmetry of the centered Gaussian surfaces boundary/axis-ordering bugs a symmetric image hides. Blobs kept broad (grid-filling, no dead pixels) so the |.|-kink TV gradients stay FD-well-conditioned at epsilon_tv=0; all tolerances unchanged. * Use asymmetric + spatially-varying pol in pol FD test fixtures chisq_setup_pol and a new asym_pol_setup build on make_asym_image and use add_random_pol (ccorr>0) so EVPA, vfrac, rho, and psi all vary spatially instead of a constant pol fraction. polreg_setup switches its Stokes I to the asymmetric image (keeping the per-pixel pol jitter that keeps TV denominators non-degenerate). TestObjectiveGradPolarimetricFD now uses the structured-pol obs. Widen the pol chisq FD check to a median+max split (median 1e-5, max 1e-3): the structured-pol imcur has sharper local curvature, so 2nd-order FD truncation pushes a few small-gradient pixels to ~2.6e-4 -- well below any real pol-gradient bug (%-level), which the tight median still catches. * Comment cleanup + epsilon_tv consistency in pol_imager_utils Manual review pass: per-slot dR/dX labels, docstrings on the reg kernels, a module-level CONVENTIONS block (imarr = [I, rho, phi=2chi, psi]), and removal of stale TODOs. Two behavior touches, both byte-identical at the defaults: - reg_vtv / reggrad_vtv now honor epsilon_tv (kwargs, default 0) like the ptv pair, instead of the value ignoring it while the grad pinned it to 0. - reggrad_ptv masks the chi-slot back-neighbor terms (c2/c3) too, for uniformity (they already self-zero at the pad). Plus an mcv_r exception-message fix and ruff-clean whitespace. * Comment cleanup in imager_utils (no behavior change) Manual review pass: docstrings on the Stokes-I reg kernels, per-block comments, 'fourier/transform matrices' labels on the diag Amatrices unpacking, and removal of dead commented-out systematic-noise code in the bispectrum data functions (the intent is now documented in apply_systematic_noise_snrcut). Purely cosmetic; ruff-clean. * fixed lint errors in test_regularizers and test_chisquared
The dev->dev-backend merge took dev-backend's bare `import warnings`, which shadows the ehtim.warnings submodule pulled in from dev and broke `import ehtim.warnings as ehw` (test_mixedpol failures). Re-apply dev's _stdlib_warnings rename.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## dev-backend #308 +/- ##
===============================================
+ Coverage 43.78% 47.54% +3.75%
===============================================
Files 53 55 +2
Lines 26333 26977 +644
Branches 4477 4595 +118
===============================================
+ Hits 11530 12825 +1295
+ Misses 13538 12663 -875
- Partials 1265 1489 +224 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
merge fix for pol gradients (#306) from dev into dev-backend