feat: volatility targets, CZAR loss, and testnet topic examples

[jefferythewind]

Summary

• czar_loss.py — CZAR directional loss with gradient/hessian for custom LightGBM objectives; exported via __init__.py
• workflow.py — adds target_type="log_return" | "volatility" parameter and compute_volatility_target_polars() for realized volatility targets (std of 1-min log returns × √target_bars, matching the reputer's ground truth)
• notebooks/testnet/ — per-topic example scripts for all active testnet/mainnet topics: price (38, 41, 42), log-return (57, 61–63, 71, 83, 84), and volatility (79–82, 85)
• tests/test_volatility_target.py — 8 tests covering correctness, null rows, scaling, and ddof=1 behavior
• skills/allora-model-builder/SKILL.md — new skill for building price, log-return, and volatility workers
• README — new top-level "Topic reference" section with target-type definitions and topic tables; file map updated

Notes

This is a rebased and cleaned-up version of PR #31. The original branch was branched before several main-branch additions (wallet linking, export hosting, engineered features). This PR rebases cleanly on top of all of that with conflicts resolved.

Key fixes vs. the original PR:

• Stale script paths in AGENTS.md and SKILL.md corrected (model_e_calibrated.py → model_grid_retrain.py, notebooks/topic_* → notebooks/testnet/topic_*)
• SKILL.md deploy section updated to use WorkerManager instead of raw AlloraWorker
• README topic tables restructured: playground / volatility / mainnet-equivalents separated cleanly; topic reference promoted to its own top-level section
• No changes to existing notebooks/ root-level scripts or deploy tooling

Test plan

• pytest tests/test_volatility_target.py -v — all 8 tests pass
• from allora_forge_builder_kit import czar_loss, make_czar_objective — imports cleanly
• AlloraMLWorkflow(..., target_type="volatility") — accepted without error
• AlloraMLWorkflow(..., target_type="log_return") — default behavior unchanged
• Spot-check one testnet example script runs to backfill/feature stage

🤖 Generated with Claude Code

---

Summary by cubic

Adds realized-volatility targets, a CZAR LightGBM objective, and hardened testnet examples with secure env and network selection. Replaces per-bar std and pre-override interval checks with √horizon-scaled std of 1‑minute log returns and post‑override validation; switches hardcoded testnet and embedded API keys to ALLORA_NETWORK/ALLORA_API_KEY; replaces OOS bias correction with leakage‑free per‑checkpoint train‑residual correction.

• Workflow/tests: add target_type ("log_return" | "volatility"); compute volatility as std of 1‑minute log returns × √target_bars (ddof=1); validate the effective interval after any data_manager override; non‑1m interval now errors; tests/test_volatility_target.py covers scaling, wrong‑interval, nulls.
• CZAR: new czar_loss with grad/hess and make_czar_objective exported via allora_forge_builder_kit.__init__ for LightGBM custom objectives.
• Security/runtime: example artifacts no longer capture API keys; predict closures rebuild from ALLORA_API_KEY; worker_runtime and notebooks/deploy_worker_raw.py export the resolved key before artifact execution; smoke‑test failures exit non‑zero; artifacts save next to scripts; run_all_examples.sh continues and summarizes results.
• Network/tooling: WorkerManager, WorkerMonitor, AlloraTopicDiscovery, and worker_runtime read network from ALLORA_NETWORK (fallback "testnet"); notebooks/deploy_worker.py passes it through; README unified for v10 on both networks.
• Examples: consolidate testnet topic scripts; add Topic 85 (ETH 4h vol); apply Jensen bias correction for log‑space vol models and use per‑checkpoint train‑residual bias correction for selection/evaluation (no leakage); proper train/val/test split with validation ending at test_split − TARGET_BARS; top‑K deploy deduplicated by model config; defensive snapshot fallbacks and current_price validation added; vol feature engineering vectorized.

Migrate

• Volatility: use AlloraMLWorkflow(..., interval="1m", target_type="volatility", target_bars=<minutes>); models must output a non‑negative σ over the horizon.
• CZAR: use LGBMRegressor(objective=make_czar_objective()) from allora_forge_builder_kit.
• Env/Network: set ALLORA_API_KEY before running artifacts; set ALLORA_NETWORK=mainnet to switch networks (set env before constructing managers/monitors).

^{Written for commit 993fca2. Summary will update on new commits.}

[cubic-dev-ai]

All reported issues were addressed across 31 files

Architecture diagram

sequenceDiagram
    participant User as Researcher/Dev
    participant API as AlloraMLWorkflow API
    participant DM as DataManager
    participant DF as Polars DataFrame
    participant LGB as LightGBM
    participant CZAR as CZAR Loss Module
    participant WM as WorkerManager
    participant ALL as Allora Network

    Note over User,ALL: Core Training Flow (Volatility & Log-Return)

    User->>API: Initialize with target_type & config
    API->>API: Validate target_type (log_return/volatility)
    API->>DM: backfill(days, interval)
    DM-->>API: Historical OHLCV data
    API->>DM: get_full_feature_target_dataframe()
    DM->>DF: Build base features + engineered features

    alt target_type = "volatility"
        DF->>DF: compute_volatility_target_polars()
        Note over DF: Shift log returns, forward rolling std<br/>scale by sqrt(target_bars) - matches reputer
        DF-->>API: DataFrame with volatility target (non-negative)
    else target_type = "log_return"
        DF->>DF: compute_target_polars()
        Note over DF: log(future_close / current_close)
        DF-->>API: DataFrame with log-return target
    end

    API-->>User: Training-ready DataFrame

    Note over User,LGB: Model Training with CZAR Support

    User->>CZAR: make_czar_objective(std, alpha)
    CZAR->>CZAR: z-score by local volatility
    CZAR-->>User: Custom LightGBM objective (grad + hess)

    User->>LGB: LGBMRegressor(objective=czar_obj)
    LGB->>DF: Fit (features, target)
    DF-->>LGB: Trained model

    Note over User,WM: Deployment & Live Inference

    User->>WM: deploy_worker(topic_id, artifact_path)
    WM-->>User: Worker deployed

    User->>ALL: Submit predictions (live loop)
    ALL->>User: Topic ground truth (for volatility: std of<br/>1-min log returns over horizon)

    Note over User,ALL: New topic resources
    Note over User: Topic example scripts for price (38/41/42),<br/>log-return (57/61-63/71/83/84),<br/>volatility (79-82/85) topics

Loading

<sub>Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 23 files (changes from recent commits).

<sub>Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 23 files (changes from recent commits).

<sub>Tip: instead of fixing issues one by one fix them all with cubic
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[jefferythewind]

also slipped in the main net upgrade into that last commit.

[cubic-dev-ai]

All reported issues were addressed across 7 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[jefferythewind]

This is thorough and comprehensive to upgrade the builder kit for volatility target with examples. Suggest approval.

[clementupshot]

Verdict — needs attention

• 🔴 Critical: 0
• 🔴 High: 3
• 🟡 Medium: 6
  ▸ Minor (non-blocking): 16

Review summary

This PR adds a CZAR custom LightGBM objective (czar_loss.py), a target_type="volatility" path in the workflow, ALLORA_NETWORK env plumbing for mainnet readiness, and ~30 per-topic training/deployment scripts. The review focused on the numerical correctness of the new loss (checked against the canonical allora-standard-loss-functions reference and against finite differences), the volatility target semantics, and credential handling in the deployable artifacts.

Verdict — needs attention

• 🔴 Critical: 0
• 🔴 High: 3
• 🟡 Medium: 6
  ▸ Minor (non-blocking): 22

What looked solid: the volatility target itself is correct — the forward-looking window alignment (row t sees returns t+1..t+T), the √T scaling, and the tail-null behavior were verified empirically against a hand-computed reference, and tests/test_volatility_target.py pins it. The runtime-reconstruction pattern used for the pickled predict() in the vol grid-retrain scripts and topic 61 is the right fix for the credential-leak class; it just hasn't been rolled out to all the scripts. The target_type validation ordering (after the data-manager override) is correct.

The headline issue is in the new CZAR module: the shipped loss matches the canonical reference to ~7e-15, but the gradient's region-3 branch is missing the linear term of L3, and the hessian uses the wrong multiplier for H2/h3 — both proven numerically below. Separately, 14 of the new scripts still embed the Atlas API key in the deployable pickles, and the deployed predict() chain breaks under the documented file-only API-key flow.

Next actions

• Fix czar_gradient region 3: add the missing linear term s * d2p1 * derivative(d_true) (czar_loss.py:117)
• Fix czar_hessian H2/h3 multipliers: (1+x²) → d2p1 (czar_loss.py:140-141)
• Add finite-difference grad/hess tests for czar_loss — the region-3 bug shipped exactly where coverage is absent
• Roll the runtime-reconstruction predict() pattern out to the remaining scripts that still pickle the workflow (list in the inline comment)
• Export the resolved key in worker_runtime.main (os.environ["ALLORA_API_KEY"] = api_key after _load_api_key) so file-only auth works for deployed artifacts
• Either add the documented bias correction to the vol predictors (AGENTS.md:121) or fix the docs
• Fail the vol grid scripts (non-zero exit / skip the dump) when the save-time smoke test fails

Minor findings — 16 · non-blocking (style, tests, nits)

6 runtime-semantics · 4 correctness · 3 architecture · 1 test-coverage · 1 security · 1 deep-analysis

🟡 No unit tests for the new CZAR loss/gradient/hessian or make_czar_objective — a numerically-verified gradient bug shipped exactly where coverage is absent
allora_forge_builder_kit/czar_loss.py · test-coverage

🔵 Pickled predict() reconstructs the Booster and a fresh AlloraMLWorkflow (network clients included) on every inference call
notebooks/testnet/topic_79_btc_vol/model_grid_retrain.py · runtime-semantics

🔵 ALLORA_NETWORK resolution (env fallback to 'testnet') is duplicated across five call sites instead of one shared resolver — and the duplication is what let the case-normalization inconsistency (see the worker_runtime.py:244 finding) slip in
allora_forge_builder_kit/worker_manager.py · architecture

🔵 czar_gradient G1 uses np.sign(z_true) (0 at z_true==0) while czar_loss L1 uses s (=1) — gradient off by exactly 1 (z-space) for exactly-zero targets
allora_forge_builder_kit/czar_loss.py · correctness

🔵 make_czar_objective's docstring claims compatibility with LightGBM's native fobj parameter but implements the sklearn (y_true, y_pred) order — correct for LGBMRegressor (as all notebooks use), silently wrong for lgbm.train
allora_forge_builder_kit/czar_loss.py · runtime-semantics

🔵 compute_volatility_target_polars uses rolling_std(min_samples=...) (polars≥1.21 spelling) while pyproject declares unpinned 'polars' — older installed environments fail with TypeError
allora_forge_builder_kit/workflow.py · runtime-semantics

🔵 CZAR price predictors can return a non-finite price when the live-price fallbacks all fail
notebooks/testnet/topic_41_eth_8h_price/model_czar.py · correctness

🔵 ALLORA_NETWORK=mainnet deploys a real-funds worker with no confirmation prompt and no network echo in deploy output
notebooks/deploy_worker.py · security

🔵 topic_63 example passes an explicit api_key_file that bypasses get_api_key's canonical fallback chain (confirms the still-open prior finding)
notebooks/testnet/topic_63_eth_24h_logreturn/example.py · deep-analysis

🔵 czar_loss.py exports generic names (derivative, antiderivative, double_derivative) at module scope — namespace pollution for a public package module
allora_forge_builder_kit/czar_loss.py · architecture

🔵 target_type is stringly-typed ('log_return'/'volatility') where the codebase has no enum precedent — boundary validation exists, but a StrEnum would make the dispatch self-documenting
allora_forge_builder_kit/workflow.py · architecture

🔵 make_czar_objective captures the std array by reference — a caller that mutates std after construction silently changes the objective
allora_forge_builder_kit/czar_loss.py · runtime-semantics

🔵 ALLORA_NETWORK value normalization is inconsistent across the new env plumbing: lenient .lower() comparison at three sites vs strict case-sensitive argparse choices at the worker subprocess — ALLORA_NETWORK=MAINNET gives mainnet discovery/monitoring but a worker that dies at argparse after the DB row is marked running
allora_forge_builder_kit/worker_runtime.py · runtime-semantics

🔵 czar_loss default alpha=1 diverges 100x from the canonical reference default alpha=0.01 — the only default not carried over; public-API callers omitting alpha silently get the maximum-curvature regime
allora_forge_builder_kit/czar_loss.py · correctness

🔵 All six log-return example.py predict() docstrings claim price output ('Predicted BTC price in USD' — even on SOL/ETH clones) while the code correctly returns the raw log return — doc rot on the repo's self-declared critical correctness rule
notebooks/testnet/topic_57_sol_8h_logreturn/example.py · correctness

🔵 run_all_examples.sh and dashboard.sh hardcode the repo-root .venv interpreter, bypassing the activated environment AGENTS.md mandates — dashboard.sh falls through to PATH python without set -e and can run against the wrong environment silently
run_all_examples.sh · runtime-semantics

[jefferythewind]

All comments addressed and handled. All czar loss adjustments were ignored since the implementation is based on Research's exact methods. I left more comments about that in the code.

[cubic-dev-ai]

All reported issues were addressed across 22 files (changes from recent commits).

<sub>Tip: instead of fixing issues one by one fix them all with cubic
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 6 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 13 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[clementupshot]

Re-review (verify mode) on commits 54e6881..993fca2 — 24 files changed across 9 fix commits.

All 9 previously flagged important findings (3 high + 6 medium) are resolved:

High findings — resolved

1. czar_gradient region 3 missing linear term (czar_loss.py:117)
The canonical reference (https://research.allora.network/t/czar-loss-function-for-returns-prediction-topics/155, post #1) explicitly documents this as an intentional pseudo-gradient for numerical stability. The reference shows both the actual gradient (G3 = np.minimum(h3, h1) * (z_pred - z_true) + s * d2p1 * derivative(d_true)) and the pseudo form (G3 = np.minimum(h3, h1) * (z_pred - z_true)) — the shipped code matches the reference exactly. The added comment now cites the reference. Not a bug.

2. czar_hessian H2/h3 multipliers (czar_loss.py:140-141)
Same — the reference documents (1 + x²) as the pseudo-hessian form (vs d2p1 in the actual hessian), chosen to bound H2/h3 and prevent overshooting for large |x|. Shipped code matches the reference. The added comment cites the reference. Not a bug.

3. API key capture in pickled predict closures
Fixed. All 14 affected scripts now use the runtime-reconstruction pattern: _make_predict captures only plain data (model string, feature lists, tickers, interval constants); predict() reconstructs AlloraMLWorkflow from os.environ["ALLORA_API_KEY"] at call time. No _wf=wf closure capture remains. worker_runtime.py and deploy_worker_raw.py both export the resolved key to os.environ after _load_api_key().

Medium findings — resolved

4. Jensen bias correction in log-space vol predictors — Fixed. bias_correction = exp(0.5 * var(train_residuals)) computed per-checkpoint, captured in each predict closure, applied in grid scoring and scatter evaluation.

5. Smoke test failure exit code — Fixed. n_smoke_failures counter incremented on each failed smoke test; sys.exit(n_smoke_failures) at script end.

6. Val/test split for model selection — Fixed. Three-way 70/10/20 split; grid search scores on df_val; df_test held out for final scatter-plot evaluation only. Val boundary ends at test_split - TARGET_BARS to prevent target leakage.

7. Top-K deploy dedup by model_num — Fixed. drop_duplicates(subset=["model_num"]).head(TOP_K_DEPLOY) applied to all vol grid scripts (was only in topic_79).

8. Defensive snapshot fallback — Fixed. try/except around float(snap["close"].iloc[-1]), "close" in snap.columns guard, np.isfinite(snap_price) and snap_price > 0 validation before assignment.

9. Per-checkpoint train residuals for grid bias correction — Fixed. Final commit (993fca2) switched from OOS val residuals (which leaked targets into scoring) to per-checkpoint train residuals: model.predict(df_train, num_iteration=n_est) gives a leakage-free correction that varies correctly with n_est.

Verification

• tests/test_volatility_target.py — 9/9 passed
• CZAR loss/gradient/hessian — all produce finite values across 100 random samples
• Finite-difference gradient check confirms the region-3 divergence from the analytical derivative — this is the documented pseudo-gradient, not a bug
• No new regressions detected in the scoped diff

Minor findings — non-blocking

These were minor-tier in the prior review and remain non-blocking. Some have been addressed; the rest are nits/suggestions.

• No unit tests for CZAR loss/gradient/hessian — still absent. The finite-difference check above confirms the pseudo-gradient diverges from the analytical derivative by design, but a test pinning the loss value and confirming the pseudo-forms match the reference would prevent future drift.
• alpha=1 default on czar_loss — the reference uses alpha=0.01 for the loss and alpha=1 for gradient/hessian (post #5 confirms alpha=1 is for training stability). The PR uses alpha=1 for both. Since the loss is only used for evaluation (LightGBM uses grad/hess), this is a cosmetic divergence, but aligning the loss default with the reference would be cleaner.
• ALLORA_NETWORK resolution duplication — partially addressed (moved to None default with env read in __init__), but the case-normalization inconsistency across sites remains.
• make_czar_objective docstring — claims fobj compatibility but implements sklearn (y_true, y_pred) order. Correct for LGBMRegressor (as all notebooks use); silently wrong for lgbm.train. Minor since the kit's examples all use the sklearn API.
• target_type stringly-typed — boundary validation exists; a StrEnum would be self-documenting but there's no enum precedent in the codebase.
• Generic function names at module scope — derivative, antiderivative, double_derivative are exported at module scope. Could prefix with _ but they match the canonical reference verbatim.