Skip to content

Fix inactive JAX diagonal cells poisoning path gradients - #8

Open
geekbeast wants to merge 1 commit into
mainfrom
fix/mask-inactive-diagonal-cells
Open

geekbeast wants to merge 1 commit into
mainfrom
fix/mask-inactive-diagonal-cells

Conversation

@geekbeast

Copy link
Copy Markdown
Owner

Summary

Prevent inactive lanes in the fixed-width JAX diagonal sweep from overflowing and poisoning native JAX gradients. The final kernel value can be finite while unused internal states produce NaNs during backpropagation; outer gradient clipping cannot fix that.

  • Mask boundary vectors and path operands before static-kernel/power-series arithmetic; use safe gather indices and where for boundary selection.
  • Apply the same rule to chunked and unchunked forward sweeps.
  • Use row-consistent rectangular geometry so the active-lane mask is correct for unequal path lengths.
  • Remove mutation of self.exponents inside the jitted single-pair entry point, which leaked a tracer on reuse in the new differentiation tests.
  • Add synthetic FP32 regressions and an independent row-major reference; wire them into CPU CI.

No custom-VJP implementation is imported from another repository. No dataset samples, learned weights, preprocessing changes, gradient clipping changes, or dtype changes are included. Real active-tile overflow remains visible.

Validation

  • Before fix: a dataset-free 64-knot linear example at order 9 returns a finite kernel value but 46 non-finite query-gradient entries and 2 non-finite bank-gradient entries.
  • CPU: pytest tests/test_core_jax.py tests/test_diagonal_masking.py -q — 46 passed.
  • GPU: pytest tests/test_diagonal_masking.py -q — 25 passed; strict FP32, highest matmul precision, NVIDIA_TF32_OVERRIDE=0.
  • Tests cover linear/RBF kernels, both rectangular orientations, one-increment paths, chunking, poisoned inactive inputs, independent gradient agreement and active overflow preservation.

See docs/diagonal-masking.md for the numerical rationale and stress-test tolerance.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant