Set-valued interventions, interference estimands, and continuous states with function-valued bounds - #36
Closed
raphaelrrcoelho wants to merge 4 commits into
Closed
Conversation
Five additive capabilities, each a gap the library already implied but could
not express. Nothing existing changes behaviour; Agent and fit_scm keep their
signatures.
1. Set-valued interventions (causalrl.intervention, agents/interventional).
do() takes a Mapping and pomis() returns sets, but Agent.act returned an arm
index -- so a multi-variable intervention could be identified and never
executed. InterventionSpace states which variables are manipulable in a
given context and to which values; assignments() turns an intervention set
into arms; InterventionalAgent returns an intervention; ScalarAgentAdapter
lifts existing arm-indexed agents into the contract unchanged.
2. AdmissibleInterventions -- POMIS recomputed per context, memoised on the
manipulable set. Restricting that set is a latent projection (r40 Thm. 4),
NOT a filter: the test pins a witness graph where the unconstrained POMIS is
[{C,D}] but the correct answer under manipulable={D} is [{}, {D}], so a
filtering implementation returns nothing and loses optimality.
3. Interference (causalrl.interference) -- direct / spillover / total effects
under an exposure mapping (Aronow & Samii 2017), the case every other module
rules out by assumption. Positivity failures raise NotIdentifiableError
naming the empty cell instead of extrapolating. No standard errors: under
interference the rows are dependent and the variance must come from the
randomisation design, not the outcome column.
4. PinnedMechanism -- deploy a known structural equation while fit_scm learns
the rest. Additive noise keeps a pinned node invertible. Mixed models carry
the new provenance="mixed", gated for L3 exactly like "fitted"; all-pinned
is "specified". Pinned nodes are still scored, and that score tests the
asserted equation rather than measuring a fit.
5. Deadline -- a monotonic per-decision budget for agents in a live loop.
Cooperative and advisory; nothing interrupts a running computation.
968 tests pass (80 new), coverage 97%, ruff/pyright/generality-lint clean and
mkdocs --strict builds.
Closes the seam left by the set-valued action work. Actions became assignments; states were still `int`. Both sides now carry a real type. Tier 1 -- causalrl.state. A StateEncoder maps an observation to a feature vector, and everything downstream works in feature space. The tabular case is CONTAINED, not discarded: an indicator basis spans every function on a finite state set, so OneHotEncoder plus a least-squares learner reproduces the tabular backup to ~1e-8. That property is asserted directly against a hand-rolled tabular recursion, because a generalisation that does not contain its own special case is not trustworthy. Ships OneHotEncoder, IdentityEncoder, RBFEncoder (the same basis FunctionApproxBackdoorAgent already uses for a continuous confounder -- a continuous state and a continuous confounder want the same machinery, which is why the tabular state interface was the odd one out) and FeatureTransition. Tier 2 -- FittedQIteration. DOVI's backward induction with the (H,S,A) table and (S,A,S) transition tensor replaced by one fitted regressor per (step, action). The regressor is the duck-typed estimate.nuisance.Regressor protocol the DML estimators already use, so the control layer went continuous without a new dependency. What it gives up, stated rather than inherited: DOVI caps optimism with a Manski bound per (state, action) cell. Feature space has no cells, so the cap falls back to a global one -- with per-step reward bounded by reward_max, the return from step h cannot exceed reward_max * (H - h + 1). Valid with no function-class assumption, and much weaker. certificate() returns Kind.EMPIRICAL with an explicit downgraded_from="bounded" hedge; is_certified is False where DOVI's can be True; and the inherited int-typed observe_transition hook raises rather than silently discarding a transition it cannot represent. Tests cover both directions: one-hot features match the tabular Q and policy exactly, and an RBF encoder recovers a threshold policy that a single-bucket encoder cannot represent at all (100% vs chance). 1002 tests pass (34 new), ruff/pyright/generality-lint clean, mkdocs --strict builds.
Tier 3: the part I argued against on scope grounds, built to the shape that
does not compromise the library's identity. It turns out neither half needs a
generative model.
FunctionalManskiBounds (causalrl.bounds.functional). causal_q_bounds bounds
E[R|do(a),s] for a discrete s from that cell's propensity and mean. Feature
space has no cells, so both become fitted functions:
lower(x,a) = mu(x,a)*e(x,a) + r_min*(1-e(x,a))
upper(x,a) = mu(x,a)*e(x,a) + r_max*(1-e(x,a))
Same logic -- the logged fraction contributes its observed mean, the unlogged
fraction is bounded only by the reward range -- and with indicator features it
reproduces causal_q_bounds to ~1e-4, asserted against the tabular function.
Nuisances are cross-fitted. This is not decoration: an in-sample plug-in makes
the interval optimistically TIGHT, which is the anti-conservative direction and
the dangerous one for a bound. An OverlapDiagnostic separates the two regimes
that look alike from the outside -- propensities near zero (wide, honest,
uninformative) from propensities a flexible model drove near one (tight, and
only as good as that model).
BoundedFittedQIteration (causalrl.agents.bounded_fitted). Recovers the causal
bound FittedQIteration had to give up:
U_h(x,a) = upper_reward(x,a) + E[ max_a' U_{h+1}(X',a') | X=x, A=a ]
L_h(x,a) = lower_reward(x,a) + E[ max_a' L_{h+1}(X',a') | X=x, A=a ]
The recursion preserves the envelope because max and conditional expectation
are both monotone. The transition tensor I said had no continuous analogue
turns out not to be needed: the successor distribution enters only through an
expectation, and an expectation is a regression. No generative model, and no
drift toward model-based deep RL.
Verified on a confounded-action / unconfounded-transition MDP where the true
interventional Q* is computable: the envelope contains it at every cell and
every step, at ~38% of the vacuous width. A separate test guards the fixture
itself -- the naive confounded mean has 0.245 bias and picks the wrong action
at state 0 -- because a bound containing the truth is only interesting when a
point estimate would have missed.
Gated as DOVI is. Multi-step propagation raises UnverifiedAssumptionError
unless transition_assumption='unconfounded': the continuation expectation runs
over the LOGGED successor distribution, so a confounder driving the dynamics
makes the interval a bound on nothing. allow_heuristic=True runs it and the
certificate drops to EMPIRICAL with downgraded_from='bounded'. Even a certified
run keeps a hedge -- correct specification of mu and e is an assumption the
tabular bound never needed, and every certificate records it with the overlap
diagnostic attached.
1037 tests pass (35 new), new modules 97-100% covered, ruff/pyright/
generality-lint clean, mkdocs --strict builds.
Acting on the end-to-end review. Four items, all additive; no exported name
changed meaning and no test was weakened to make something pass.
1. Documentation completeness. 127 of 253 exported names had no API-reference
entry -- among them Agent, CausalMBRLAgent and CausalThompsonSampling, which
is the README's own headline example. mkdocs --strict never caught this
because mkdocstrings validates the references that exist, never the ones
that are absent. All 127 are now written, grouped by tier, and
test_every_export_appears_in_the_api_reference keeps it closed -- resolving
through the lazy export map so a renamed export (canonical ->
canonical_intervention) is checked against the attribute it points at.
2. Two defects from the continuous-state work, both mine.
- bounded_fitted.py redefined TransitionAssumption as bare `str` while
dovi.py already had Literal["unknown", "unconfounded"]. The sibling
definition silently discarded the narrowing; it now imports DOVI's.
- The two fitted agents carried hand-written copies of the same
transition unpacking and validation, with paraphrased error messages --
worse than duplication that reads identically, since the two could drift
and no reader could tell whether a wording difference was meaningful. Now
one unpack_transitions/TransitionBatch in state.py.
Fixing the first surfaced a third: FunctionalManskiBounds.fit annotated
`actions` as a float array when it has always required and coerced ints.
3. Agent interface split. update() was an empty body in ten subclasses --
CertifiedPolicyAgent, all five *BackdoorAgents, CounterfactualOptimalPolicy,
NaiveOffline, and both fitted agents. These are batch learners whose policy
comes from fit/ingest_offline; a single reward carries nothing they can act
on, so the ABC was asserting something untrue of a third of its implementers.
BatchAgent supplies the no-op once and says why. Additive: it subclasses
Agent, so isinstance checks and online harnesses are untouched.
4. API tiering. 253 names in a flat alphabetical __all__ said nothing about
where to start. API_TIERS partitions them by intent -- core (14),
identification, modelling, decision, inference, integration -- and is tested
to be a true partition, so a new export cannot be added without being placed.
Verified: 1040 tests pass (3 new), coverage 97%, ruff/pyright/generality-lint
clean, mkdocs --strict builds, and the torch-free core still imports in a fresh
venv without torch.
Owner
Author
|
Superseded — closing rather than merging. Every symbol this branch contributes is now on
Verified: no top-level Two things here are deliberately not on
The branch is left in place, so the interference work stays recoverable if it ever earns a home behind an agent. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes a set of gaps the library already implied but could not express. Everything is additive: no exported name changed meaning,
Agentandfit_scmkeep their signatures, and the 888 pre-existing tests pass unmodified.The through-line is that several layers already spoke a richer language than their neighbours.
do()takes aMappingandpomis()returns sets, but agents returned an arm index. The estimation core is continuous (cross-fitted DML, ANM/neural mechanisms) while the control layer indexed states byint. Each commit closes one of those seams.166c0bbbf87d02794322e4d6f88cNew public surface:
InterventionSpace,InterventionalAgent,ScalarAgentAdapter,AdmissibleInterventions,ExposureMapping+direct_effect/spillover_effect/total_effect,PinnedMechanism,Deadline,StateEncoderfamily,FittedQIteration,FunctionalManskiBounds,BoundedFittedQIteration,BatchAgent,API_TIERS.30 files changed, +4365 / −46.
Causal / software contract
Set-valued interventions.
pomis()returns intervention sets anddo()takes assignments, butAgent.actreturned anint— so a multi-variable intervention could be identified and never executed.AdmissibleInterventionsrecomputes POMIS per context, and restricting the manipulable set is a latent projection (Lee & Bareinboim, AAAI 2019, Thm. 4), not a filter: on the witness graphC→B, C→D, C→YwithB↔D, the unconstrained POMIS is[{C,D}]but the correct answer undermanipulable={D}is[{}, {D}]. A filtering implementation returns nothing and loses optimality. Pinned intest_admissible_interventions.py.Interference. Direct/spillover/total effects under an exposure mapping (Aronow & Samii, Ann. Appl. Stat. 2017), assuming
Y_i(A) = Y_i(A_i, f(i,A)). That mapping is an untestable assumption and is documented as such. Positivity failures raiseNotIdentifiableErrornaming the empty cell rather than extrapolating. No standard errors are reported: under interference the rows are dependent, so the variance must come from the randomisation design, not the outcome column.Pinned mechanisms. A known equation deploys at one node while
fit_scmlearns the rest. Additive noise keeps the node invertible, so pinning can restore an identification a fitted node would lose. Mixed models carry the newprovenance="mixed", gated for L3 exactly like"fitted". Pinned nodes are still scored — there the holdout score tests the asserted equation rather than measuring a fit.Continuous states. The tabular case is contained, not replaced: an indicator basis spans every function on a finite state set, so
OneHotEncoder+ least squares reproduces the tabular backup to ~1e-8, asserted against a hand-rolled tabular recursion.Function-valued bounds — and what they cost.
causal_q_boundsuses a cell's propensity and mean; in feature space both become fitted functions, reproducing the tabular bound to ~1e-4 under indicator features. Nuisances are cross-fitted, because an in-sample plug-in makes the interval optimistically tight — the anti-conservative direction, which is the dangerous one for a bound.The envelope recursion
U_h(x,a) = upper_reward(x,a) + E[max_a' U_{h+1}(X',a') | x,a]preserves the bound becausemaxand conditional expectation are monotone. No transition tensor or generative model appears — the successor distribution enters only through an expectation, which is a regression. Verified on a confounded-action / unconfounded-transition MDP: the envelope contains the true interventionalQ*at every cell and step, at ~38% of the vacuous width, on a fixture where the naive confounded mean carries 0.245 bias and picks the wrong action outright.Guarantees are declared, never inherited. The ladder now reads:
DOVIBOUNDED per-cell →BoundedFittedQIterationBOUNDED up-to-specification →FittedQIterationEMPIRICAL under a global cap. Multi-step propagation raisesUnverifiedAssumptionErrorunlesstransition_assumption="unconfounded", since the continuation expectation runs over the logged successor distribution;allow_heuristic=Truedowngrades the certificate toEMPIRICALwithdowngraded_from="bounded". Even a certified run keeps a hedge, because correct specification of the outcome and propensity models is an assumption the tabular bound never needed.Open, and not claimed: a sharp bound uniform over a function class. What is here is a valid pointwise bound conditional on the nuisance models.
Validation
uv run pytestpassesuv run ruff check .anduv run ruff format --check .cleanuv run pyright srcclean1040 passed, 3 skipped, coverage 97% (CI gate 90%); new modules 97–100%.
mkdocs build --strictbuilds and the custom generality lint is clean.Three checks worth calling out beyond the boxes:
test_the_confounding_this_fixture_carries_is_materialguards the fixture itself — a bound containing the truth only matters if a point estimate would have missed.API_TIERStier and must appear indocs/api.md. The second closed a pre-existing gap of 127 undocumented exports (includingAgentandCausalThompsonSampling) thatmkdocs --strictcould never catch, since mkdocstrings validates references that exist rather than ones that are absent.Generated by Claude Code