RL recentering: act() as a policy, the conformal gate, the CGFA critic, and a no-regret learner

[raphaelrrcoelho]

Four conversions from the library-wide RL-recentering audit. Each had to pass the audit's own test — can something under agents/ or envs/ actually reach it? — demonstrated by a test that exercises the path, not by an import that merely exists.

act() returns a policy, not a constant

The audit's worst finding was inside the RL half: five of six back-door planners' act() returned a constant and discarded the observation — return int(self._best_action), identical at five sites. The parameter existed only to satisfy the ABC.

Meanwhile interop/econml.py:34 already did tau = effect(X); pi = (tau > 0) for a third-party CATE model. The library performed CATE→policy for EconML and not for itself.

All five now read their own fitted interventional outcome model at the observation: tabular stratum lookup for BackdoorAdjustedAgent / DiscoveryBackdoorAgent / TransportBackdoorAgent, qhat(a, z) at the observed confounder for FunctionApproxBackdoorAgent, and the per-action T-learner at the observed covariate row for GFormulaBackdoorAgent — where act is asserted row-by-row to equal sign(cate), the per-unit decision the T-learner already estimated and threw away.

Two tests say something the library could not say before:

• on ContinuousConfoundedBandit the best constant action is arm 0 (E[Y|do(1)] ≈ 0.38 < 0.5), yet act({"Z": 0.85}) plays arm 1 inside the reward bump;
• on TransportableConfoundedBandit the transported marginal says "never arm 1" in every phase-diagram cell, yet act({"Z": 1, "W": 0}) plays it.

An observation carrying none of the conditioning variables returns the marginal argmax (correct, backwards compatible); a partial one raises rather than silently answering a different query.

conformal_action_value — the conformal layer becomes a safety gate

conformal/ was orphaned statistics: nothing under agents/ or envs/ could reach it. But conformal_quantile(weights=…) already accepted likelihood ratios dP_test/dP_cal, and for off-policy that ratio is π_target/π_behavior.

The path now runs CertifiedPolicyAgent(alpha=0.1).ingest_offline() → certify_policy(alpha=…) → conformal_action_value(...) → the existing weighted path, reported in the new DecisionCertificate.conformal_lcb. Proven live rather than merely importable: on a log where action 1 pays 1.5 with probability 0.95 and −5.0 otherwise, its mean is higher (1.175 vs 0.5) so the ungated agent ships it — and the calibrated downside gate picks up the tail the mean hides, so the gated agent abstains to action 0.

Stated plainly in the docs: this bounds a quantile, not V(π). HCOPE-style safe policy improvement bounds the mean, which split conformal cannot.

Honesty fix: certify_conformal_interval's query parameter is removed; it always emits query="see". It previously accepted query="counterfactual" — a string label with no counterfactual mathematics behind it.

The CGFA-PPO K-head critic

factored_advantage.py was named for CGFA-PPO (Cunha, Mian, French & Liu 2026, arXiv:2605.06066) but the entire computation was V_i − baseline — a subtraction. The example itself conceded "in a full CGFA-PPO, you'd have K value heads; here we use the same value."

Implemented from the paper, no code ported: FactoredCritic with one value head per SCM parent of the return on a trunk shared with the scalar critic, each regressed on its own per-factor return (Eq. 8–9); learnable mixture logits; a state-conditional residual gate (Eq. 11); and the intervention-calibration loss (Eq. 12). Pure-NumPy factor_rewards / factor_gae / blend_advantages stay torch-free; the critic sits behind the existing [torch] extra.

Tests pin head differentiation — each head fits its own target >20× better than its neighbour's, and swapping target columns inflates the residual >10×. Both ratios collapse to 1 under a shared value head, which is the defect being fixed.

Two holes found in the paper, both documented as deviations rather than silently patched:

1. L_cal's advantages are read from the rollout buffer — detached — so Eq. 12 would be a pure diagnostic and the paper's own ablation of it a no-op. Recomputed under current parameters instead.
2. The gate and mixture logits have no gradient path under Algorithm 1 as written: line 12 computes A_used outside the epoch loop and Table 6 sets c_e = 0, zeroing the only other term — so the two parameters the paper calls learnable never move. Both readings ship, with opposing tests.

run_no_regret — the multi-agent vocabulary gets a learner

magames/ was 874 lines of RL vocabulary with no learner: cce_regret's own docstring says "it is what a no-regret population drives to 0", and the package contained no such population. The evidence that this was a missing function rather than a mislabelled package: experiments/eqcf/common.py hand-rolls a Hedge population purely to feed cce_regret.

Adds RegretMatching (Hart & Mas-Colell 2000) and MultiplicativeWeights (Freund & Schapire 1997) as real Agent subclasses, and run_no_regret(...) producing the empirical joint that cce_regret / certify_cce_do already accept. Convergence is pinned against a payoff-blind null available as a first-class knob: blind regret 0.75 versus 0.000 / 0.029 for the learners; zeroing both update bodies fails 13 of 23 tests.

PopulationAgentView → LinearGaussianPopulationEnv (it was a linear-Gaussian DGP, not an agent).

Honest boundary: the new edge runs magames/ → agents/, so the halves touch through real code — but nothing under agents/ imports magames, so on the audit's forward definition it remains unreached. Re-exporting it from causalrl.agents to fake an edge would need an import cycle and would be the disguised conversion this programme exists to eliminate.

Registry drift the enforcement tests caught

Sixteen new exports had no API tier; four had no api.md entry (the docs had the functions, not the classes the lazy map resolves to). And PopulationAgentView / agent_causal_env_view were still listed in API_TIERS and api.md after the rename — the test working in the direction it was not written for: not a missing export, a stale one.

Verification

ruff check clean · ruff format --check 360 files · pyright src 0 errors, 0 warnings, 0 informations · pytest --cov-fail-under=90 exit 0, coverage 97.33%.

Not included

T2 (real-data examples calling the RL front door), T3 (causalrl.ope), T4 (deleting the statistics tooling), T5 (parameter renames) and T8 (docs framing) remain. T8 is deliberately last: reframing the README before the code is true would be the same deviation, better disguised.