Skip to content

feat: auto ID explains candidates whose peak went to a detected sum peak - #61

Merged
Paullo9 merged 12 commits into
mainfrom
issue-9-possible-sum-question
Nov 1, 2026
Merged

Paullo9 merged 12 commits into
mainfrom
issue-9-possible-sum-question

Conversation

@Paullo9

@Paullo9 Paullo9 commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Requested by Paul · project thread

Part of #9

Stacked on issue-8-pooled-sum-peaks (#59). This PR's base is that branch; retarget to main once #59 merges. It is stacked because ElementProposer.swift is already in two open PRs (#59 and the issue-9-autoid-leftovers branch). This is a third PR on that file, so expect a merge after those land. The touched regions differ: the loser loop, the candidate tiers and the notes.

What changes

Auto ID (proposeOnRefinedAxis, nothing listed) already gave the Al+Al sum peak (2.973 keV) its own column. The element candidates that lost that peak to it stayed in the possible tier, because only proposed candidates got the sum-peak question. On the Al-rich demo pools these were Ar K-alpha (15 to 27 L_D), Ag L-alpha (6 to 11 L_D) and Pd L-alpha.

Now a candidate that clearly lost its peak to a sum column is explained:

  • ElementCandidate.explainedBySumPeak holds the sum label, and ProposalResult.explainedBySumPeaks is the new tier.
  • "Clearly" means three things:
    • its peak gain is outside the ambiguity margin (30) of the sum's;
    • its total gain is no larger than the sum's;
    • the sum column stays in the final fit at or above its own L_D.
  • Explained candidates leave possible. They are not proposed, not a sum-peak question and not withheld.
  • A note names them, for example "Explained as pile-up: Ag, Ar, Pd lost the peak they were tested on to the Al+Al sum column".
  • ProposalRules: an explained candidate no longer counts as a K peak in the beside-K rule (the peak is pile-up).

What it cannot do: one spectrum cannot separate a real line from the sum when the two are within the detector resolution.

  • A planted Ar K-alpha of 1000 counts (about half the sum, 16 eV away) is still explained. Its peak gain is 105591, against 105800 for the sum.
  • It keeps its sum-peak conflict and the two-beam-current remedy, so the question stays visible.
  • At 4000 Ar counts it is a sum-peak question in the proposed tier.

Additive public API: one new init parameter (default nil), one new property and one new computed tier. Apps that list possible should also show explainedBySumPeaks; otherwise those candidates drop from the list. The app change is not in this PR.

Test evidence

The truth is planted on synthetic data, not a measurement: the public AlMgSi demo pools (mac4DSTEM demo-edx, sim_edx.py v2.0, edx-demo-pools.json from #59). No Ar, Ag or Pd is planted.

  • Red first, before this change, on the Poisson counts of L5, M_A and M_matrix: Ar, Ag and Pd were in possible, with reason "The peak at 2.97 keV went to Al+Al sum".

  • After:

    pool proposed possible explained by Al+Al
    L5 Al, Si none Ag, Ar, Pd
    M_A Al Si, Ta Ag, Ar, Pd
    M_matrix Al, Si none Ag, Ar, Pd

    On L7 (pure Mg5Si6, no Al) there is no Al+Al column and nothing is explained by it.

  • New test file: DemoEDXAutoIDTests (the three pools, a real Ar planted at 1000 and 4000 counts, and L7).

  • Golden baselines: one line per dumped candidate (the new field) and three notes lines in eds-proposer-sim. No number changed. These are in their own commit.

    • In that scenario nine unplanted candidates are now explained by the Al+Mg, Al+Si and Si+Si sums: Rh, Pd, Cl, Ag, Ar, In, Sn, Cd and K.
  • Fast gate: swift test -c release --parallel --num-workers 1 -j 1 --skip SpectroKitParityTests is green: 287 SpectroEDX tests, 2 known issues (unchanged).

  • Parity: SpectroKitParityTests ran once, because the golden baselines moved. Green, 83 tests.

  • One numerics review before the commit. It found no blocking bug. Fixed from it:

    • only a clear loss explains (the ambiguity margin and the total-gain guard);
    • the beside-K rule ignores explained candidates;
    • a planted real Ar test was added.

What remains for #9

🤖 Generated with Claude Code

Paullo9 and others added 5 commits October 31, 2026 14:42
…aks as explained

With the sum/escape model on, the unlisted-line check proposed on the bare spectrum and named the modelled pile-up and escape peaks as elements. SumEscapeResult.componentCounts (the fitted c S(A) + f E(A)) is now carried by the quantification and handed to ElementProposer.propose(explained:), which adds it to every fit as a fixed offset. nil (the default, and the model off) runs the proposer as before.

Planted pools (tools/edx-sim proposer_sim, model on): 100 counts/px no longer names Ar, Ag, Pd; 300 counts/px no longer names Rh, Ar, In, Sn, Pd, Ag, Cd, Ru, Bi, K; a planted Fe escape with the free escape columns off no longer names La, Ce, Cr, Cs. A planted K K-alpha between two sum peaks is still named first. Model off: unchanged.

Done: sum/escape peaks are explained counts for the proposer and the check. Next: docs rows for parity.md and open-items.md item 8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pair is significant

The pooled quantification fitted one Gaussian per line and no sum (pile-up) peaks, exSpy's model. On the public
synthetic AlMgSi demo (planted truth: the simulation recipe of sim_edx.py v2.0, not a measurement) that fit read
noise-free Si K-alpha 15-18 % low on the Al-rich map and fixture pools and 57 % low on the highest-rate pool, held
a planted Cu K-alpha at 0 at high rate, and had a reduced chi2 of 2.1-31 on the Poisson counts. The Al+Al sum peak
(2.97 keV) and its neighbours pull the continuum segment above the Al K edge, and Si K-alpha with it.

FitPolicy.sumPeaks (SumPeakRule, default .whenSignificant) now tests the strongest line's self pair alone with
EDSSumEscapeFit; when its pile-up coefficient is 3 sigma above 0 (sigma scaled by sqrt(max(chi2_red, 1)) of that
fit) the sum peaks of the four strongest lines are fitted and kept, otherwise the result is the old fit bit for
bit. The self pair is the test because the cross pairs absorbed an unlisted Cu K-alpha (Ni K-alpha + O K-alpha) on a
synthetic spectrum with no pile-up. The footer names the decision with its number, and names the alpha lines within
one FWHM of the self pair (Ar K-alpha, Ag L-alpha under Al+Al): a real line there is read as pile-up.
.off (FitPolicy.exspyEquivalent) is the old fit; a caller passing sumEscape keeps full control. The unlisted-line
and fit-range refits repeat the reported decision; the proposer axis refinement runs without sum peaks.

Noise-free Si K-alpha against planted: -0.0 to -4.0 % on five pools, -7.9 % at the highest rate; Cu K-alpha 92-97 %
of planted at high rate (0-42 % before); reduced chi2 on the Poisson counts 0.70-1.08. The absorption-truth Si pin
moves from -0.203 to -0.067 (+- 0.051), and the old pin stays on the .off fit. The Al K-alpha tail stays a detector
calibration: measured once with TailEstimate on a high-count spectrum and typed, it brings Mg K-alpha from -22 to
-32 % to within 5 % (matrix map -15 %). New fixture edx-demo-pools.json from tools/edx-sim/demo_pools.py. The
golden scenarios pin the explicit off model; their numbers are unchanged, only the two new nil fields are added.
Results stay validation none. Reviewed (numerics, two independent reviews): sigma scaling, the coincident-line
caveat, the axis refinement cost and the wording fixed.

BREAKING CHANGE: PooledQuantifier.run fits sum peaks by default when the data support them, so nets, at% and wt%
change on spectra with pile-up. Migration: pass a policy with fit.sumPeaks = .off (or FitPolicy.exspyEquivalent),
or sumEscape: SumEscapeModel(), to keep the old numbers. A stored QuantificationPolicy without the new field
decodes as .off and its footer names the non-default policy.

Done: sum peaks on by default in the pooled fit, with planted-truth tests on the demo pools. Next: the parity and
open-items rows once the open PRs on those files land, a check of the default on real data with published values,
and the sum-peak question for the auto ID possible candidates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The identification loop already gave the Al+Al sum peak (2.973 keV) its own column, but the element candidates that lost that peak to it (Ar K-alpha, Ag L-alpha, Pd L-alpha) stayed in the possible tier. Only proposed candidates got the sum-peak question.

A candidate that clearly lost its peak to a sum column is now explained (ElementCandidate.explainedBySumPeak, ProposalResult.explainedBySumPeaks). Clearly means outside the ambiguity margin at the peak, no more total gain than the sum, and the sum column at or above its own L_D in the final fit. Explained candidates leave the possible list and the beside-K rule, and a note names them. A real line within the detector resolution of the sum cannot be told apart in one spectrum. It keeps its sum-peak conflict and the two-beam-current remedy.

Red-first on the public AlMgSi demo pools (synthetic, planted truth from sim_edx.py v2.0): before, Ar, Ag and Pd were possible on L5, M_A and M_matrix. Now they are explained by Al+Al, and L7 (no Al) has no Al+Al explanation.

Done: possible candidates get the sum-peak answer. Next: the golden baselines gain the new field (next commit).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reason: ElementCandidate gained explainedBySumPeak, so every dumped candidate gains one line. Only lines are added: no number in eds-realistic-l-sim, eds-unlisted-sim or eds-proposer-sim changed.

In eds-proposer-sim (beta'' region, planted sum peaks) nine unplanted candidates are now explained by the Al+Mg, Al+Si and Si+Si sums. Three notes lines name them.

Done: baselines match. Next: PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Paullo9 Paullo9 self-assigned this Oct 31, 2026
@Paullo9
Paullo9 changed the base branch from issue-8-pooled-sum-peaks to main October 31, 2026 17:37
Main gained #59 (squashed, the base this branch was stacked on) and the
continuum dead-layer term. Conflicts:
- PooledQuantification.swift, QuantificationPolicy.swift: this branch does not
  change them; main's side kept (it is the branch side plus continuumDeadLayer).
- the four golden baselines: main's files taken, then re-recorded with
  SPECTRO_GOLDEN_DIR. Against main the recording adds only the
  explainedBySumPeak = nil lines of this branch (35 in eds-realistic-l-sim,
  60 in eds-unlisted-sim); demo-edx-tiny-dm4, eds-weakline-sim and
  eds-proposer-sim match exactly. No number moves.

After the merge the diff against main is this branch's two commits only.

Done: branch up to date with main, builds, gate green.
Next: review and merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paullo9 and others added 2 commits November 1, 2026 10:02
…ns) into issue-9-possible-sum-question

ElementProposer.swift conflicted on the notes block: keep both, the explained-as-pile-up notes and then the overlap-check note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Paullo9
Paullo9 enabled auto-merge (squash) November 1, 2026 12:42
Paullo9 and others added 4 commits November 1, 2026 14:35
…e-9-possible-sum-question

The only conflicts were spectro goldens. They take the main side here; the next commit re-records
them from the merged code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…default, no number moves

Recorded from the merged code with SPECTRO_GOLDEN_DIR. Against main the only changes are this
branch's own: the explainedBySumPeak field on proposal candidates and the three pile-up notes in
the eds-proposer-sim region2 proposal. No numeric line moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Auto ID note for elements explained by a sum peak now reads
"Explained as pile-up: ..." with no validation tag; the rest of the
sentence is unchanged. DemoEDXAutoIDTests asserts the note carries no
"(validation" text (red before, green after).

Done: the pile-up note has no validation tag. Next: re-record the goldens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recorded from the code with SPECTRO_GOLDEN_DIR into a temporary folder and
copied in. The only change is the three pile-up notes in the
eds-proposer-sim region2 proposal; no numeric line and no csv hash moves.

Done: goldens match the code. Next: CI on the branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Paullo9
Paullo9 merged commit 500e5bd into main Nov 1, 2026
7 checks passed
@Paullo9
Paullo9 deleted the issue-9-possible-sum-question branch November 1, 2026 14:09
Paullo9 added a commit that referenced this pull request Nov 1, 2026
Brings in the squash-merged EPQ line-weight table (#67), the sum-peak
explanation for auto ID (#61) and the docs move (#76).

One conflict, a doc comment on FitPolicy.init(from:): kept this branch
wording, which names every key the decoder defaults (sumPeakSignificance
included); main wording listed fewer.

Goldens re-recorded with SPECTRO_GOLDEN_DIR after the merge: all six
scenario files are byte-identical to the merged fixtures, so no number
moved and no fixture changes in this merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant