docs: advanced-AutoML sample + full doc/PDF coverage of the new features#16
Merged
Merged
Conversation
The latest features (explainability, calibration, stacking ensembles, PR-AUC selection, CV strategies, persisted audit trail) shipped with reference docs but were missing from the samples, the landing pages, and the PDF. This closes those gaps so every feature is demonstrated, discoverable, and in the guide. - samples/advanced_automl.py (NEW): one runnable, test-covered sample on the real breast_cancer data β StratifiedKFold CV, PR-AUC selection, a calibrated stacking ensemble, global explainability, and a persisted JSONL audit trail of every gate decision. Auto-detects an LLM key: deterministic StaticFeatureProposer offline, real AgentFeatureProposer when a key is present (so the same sample runs in CI and against a live model). - tests/samples/test_advanced_automl.py (NEW): offline smoke test + an @integration test that runs the sample against a real LLM. All three real-LLM integration tests (this + the showcase) pass with a live key. - README.md, docs/README.md: add Explainability to the doc tables; note calibration/ensembling/PR-AUC/CV on the AutoML rows. - docs/index.md: enrich the AutoML card; add an "Explainable & trustworthy" pillar. - docs/samples.md: five samples now, with the advanced sample documented. - docs/brief/firefly-datascience-complete-guide.pdf: regenerated (62pp) β Ch.8 gains calibration/ensembling/PR-AUC/CV + explainability; Ch.14 the hands-on advanced selection; Ch.15 the persisted audit trail; glossary terms added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The latest features β explainability, calibration, stacking ensembles, PR-AUC selection, CV strategies, persisted audit trail β shipped with reference docs but were missing from the samples, the landing pages, and the PDF. This PR closes those gaps so every feature is demonstrated, discoverable, and in the guide.
New runnable sample (test-covered, real data)
samples/advanced_automl.pyβ one cohesive sample on the realbreast_cancerdataset that exercises all the new capabilities:StratifiedKFoldcross-validation strategymetric="average_precision")calibrate=True, ensemble=True) β Brier scoreresult.explain(...))It auto-detects an LLM key: deterministic
StaticFeatureProposeroffline, realAgentFeatureProposerwhen a key is present β so the same sample runs in CI and against a live model.Tested for real (incl. a live LLM)
tests/samples/test_advanced_automl.py: offline smoke test (CI) + an@integrationtest against a real LLM.Docs + PDF
README.md,docs/README.md: Explainability added to the doc tables; AutoML rows note calibration/ensembling/PR-AUC/CV.docs/index.md: enriched AutoML card + a new "Explainable & trustworthy" pillar.docs/samples.md: five samples now, with the advanced sample documented (offline + real-LLM run lines).docs/brief/firefly-datascience-complete-guide.pdf: regenerated (62pp) β Ch.8 gains calibration/ensembling/PR-AUC/CV + explainability; Ch.14 the hands-on advanced selection; Ch.15 the persisted audit trail; glossary terms added. Rendered pages were visually verified.Gates
ruff format --check+ruff checkclean; nosrcchanges this PR (docs/sample/test/PDF only), so existing suites are unaffected; the new offline sample test passes.