Complete MkDocs Material Rewrite#759
Open
neilSchroeder wants to merge 40 commits into
Open
Conversation
- Implement 5-minute quickstart with concrete ClimateData example - Add 7-pattern migration guide: legacy core/ → new_core/ - Include side-by-side comparisons and migration checklist - Add uv.lock for reproducible dependency builds - Ignore graphify analysis tool output Closes climakitae-6c9.4 and climakitae-6c9.7
- Catalog hierarchy: 8-level specificity model with examples - Lazy evaluation: xarray/dask integration, best practices - Global Warming Levels: climate science context vs. time-based - Interconnected complete example showing all three concepts - Added 'Core Concepts' to navigation after Overview Closes climakitae-6c9.5
Five major workflow sections with 15+ runnable examples: - Clip data: named regions, points, bounding boxes, weather stations - Export data: NetCDF, Zarr, CSV, GeoTIFF with compression - Warming level analysis: multiple levels, cross-model comparison - Bias correction: WRF localization with weather stations - Batch processing: multiple points, scenarios, parallel execution Added 'How-To Guides' to navigation structure. Closes climakitae-6c9.6
Deep technical documentation for developers and contributors covering: - System overview and dataflow - Core components: ClimateData, DatasetFactory, Dataset, DataCatalog, Validators, Processors - Key design patterns: fluent interface, registry+decorator, singleton (thread-safe), lazy evaluation - Component contracts for processors and validators - Extending the system: adding new processors, validators, catalogs - Threading and concurrency patterns - Internal APIs and registries - Testing strategies - Common pitfalls and solutions Added 'Architecture & Internals' to navigation after How-To Guides. Closes climakitae-6c9.8
Five example notebooks from cae-notebooks with descriptions and launch links: - Basic Climate Data Access (beginner, data fundamentals) - Global Warming Levels (intermediate, key climakitae feature) - Threshold Exceedance & Extreme Events (intermediate, real-world use) - Model Uncertainty (intermediate, climate science concepts) - Time Series Transformations (intermediate, custom analysis) Each notebook includes: - Difficulty level and duration estimate - Key learning topics - Links to GitHub source and Binder launch button - Progressive learning path from beginner to advanced Binder configuration leverages existing requirements.txt in cae-notebooks for automated environment setup. Supports live notebook execution with no local installation required. Added 'Notebook Gallery' to navigation after Get Started page. Closes climakitae-6c9.9
Comprehensive Binder setup enabling users to launch interactive Jupyter Lab environment with full climakitae development stack: .binder/runtime.txt: - Specifies Python 3.12 for Binder environment .binder/environment.yml: - Complete conda environment with 50+ dependencies - Scientific: numpy, scipy, pandas, xarray, dask - Geospatial: geopandas, cartopy, shapely, pyproj - Data access: intake, intake-xarray, netcdf4, zarr - Visualization: matplotlib, seaborn, bokeh - Jupyter: jupyterlab, notebook, ipywidgets - Documentation: mkdocs, mkdocs-material, mkdocstrings-python - Development: pytest, black, isort, flake8 .binder/postBuild: - Installs climakitae in editable mode (source changes work live) - Installs pip-only deps (param, pydantic, tqdm) - Configures Jupyter Lab extensions - Creates custom kernel display name .binder/README.md: - Detailed explanation of each configuration file - Binder launch badges and URLs - Local testing with repo2docker - Instructions for adding new dependencies Updated notebook-gallery.md: - Added 'Interactive Development Environment' section with Binder button - Links to .binder configuration documentation - Clarified use cases for Binder development vs. running examples Binder url: https://mybinder.org/v2/gh/cal-adapt/climakitae/main?urlpath=lab Users can now: - Click one button to launch full Jupyter Lab development environment - Develop and test climakitae interactively - Access source code in editable mode (changes apply live) - Run up to 6 hours per session - No local installation required Closes climakitae-6c9.10
…ssor documentation pages - Create processor reference directory structure with index.md, TEMPLATE.md, and 12 processor pages - Implement 4 comprehensive processor docs with full mermaid diagrams and code links: * time_slice.md (160 lines) - temporal subsetting with example queries * warming_level.md (280 lines) - GWL analysis with multi-level examples * clip.md (320 lines) - spatial clipping with all input formats * export.md (300 lines) - multi-format export with S3 support - Implement 8 concise processor docs with key examples: * concatenate.md, convert_units.md, bias_adjust_model_to_station.md * metric_calc.md, filter_unadjusted_models.md, drop_leap_days.md * convert_to_local_time.md, update_attributes.md - Create comprehensive TEMPLATE.md showing processor doc structure and best practices - Update mkdocs.yml with new 'Processor Reference' section (12 pages + overview) - All processor pages include: algorithm flowchart, code links to GitHub, parameters table, examples, implementation details - Build validated: clean build in 1.31s, no errors Closes 6c9.16.13 (infrastructure), 6c9.16.14 (navigation)
…processor flowcharts - Remove HTML <a> tags from Mermaid box labels (breaks parser) - Use proper Mermaid 'click' directive syntax for GitHub code links - Fixes rendering issue where flowcharts showed as raw text - All 4 comprehensive processors fixed: time_slice, warming_level, clip, export - Build verified: 1.36 seconds, 0 errors, Mermaid renders correctly
Required for Material for MkDocs to properly render Mermaid diagrams: - Adds pymdownx.superfences custom_fences configuration for Mermaid - Specifies Mermaid class and format for proper rendering - Material theme automatically loads Mermaid.js runtime - Fixes flowchart rendering in all processor documentation pages
…alc, Clip, Export - MetricCalc (6c9.17.5): Document three distinct calculation modes - Mode 1: Basic metrics (HDD/CDD, heat index, percentiles) - Mode 2: Threshold exceedance (days above/below with duration filtering) - Mode 3: 1-in-X extreme value analysis (GEV fitting, return periods, KS testing) - Added examples for each mode and performance notes - Fixed flowchart to show branching at execute() - Updated code references with actual line numbers - Clip (6c9.17.6): Document smart point-finding algorithm - Added five input modes: boundaries, stations, single point, bbox, custom file - Documented expanding-radius search for nearest valid (non-NaN) gridcell - Search radii: 0.01° → 0.05° → 0.1° → 0.2° → 0.5° - Clarified multi-input handling and separation behavior - Updated flowchart with actual implementation details - Enhanced code references - Export (6c9.17.7): Document three distinct data handling modes - Mode 1: Gridded datasets (separated/location_based_naming ignored) - Mode 2: Multi-point clip results (closest_cell dimension conditional splitting) - Mode 3: Point collections (list of single-point datasets) - Explained silently ignored parameters for each mode - Added format-specific capabilities and limitations table - Enhanced export_method behavior documentation - Improved code references with actual line numbers
- dataset_factory.py: Added return types to register_catalog, register_validator, register_processing_step, reset (100% coverage) - processors/abc_data_processor.py: Added return types to decorator and abstract methods (100% coverage) - param_validation/abc_param_validation.py: Added return types to decorators and load_catalog_df (75% coverage) - Added Type import to support type annotation decorators Improvements: - dataset_factory.py: 63.6% → 100% return type coverage - abc_data_processor.py: 40% → 100% return type coverage - abc_param_validation.py: 37.5% → 75% return type coverage Remaining work documented in climakitae-994 epic for Phase 2-4: - Phase 2: Add types to processors/ (export, clip, metric_calc, etc.) - Phase 3: Add types to derived_variables/builtin/ (humidity, indices, temperature, wind) - Phase 4: Add types to remaining param_validation validators and utilities Related to: #753, climakitae-994
- export.py: Added return types to update_context, set_data_accessor, export_single, and clean_attrs_dict (100% coverage) - time_slice.py: Added return types to update_context and set_data_accessor (100% coverage) Additional improvements: - export.py: 20% → 100% return type coverage - time_slice.py: 33% → 100% return type coverage Progress summary: ✓ Phase 1 (core files): 100% complete (dataset_factory, abc_data_processor, abc_param_validation) ◐ Phase 2 (processors): 40% complete (export, time_slice done; concatenate, clip, metric_calc remaining) ○ Phase 3 (derived variables): Not started ○ Phase 4 (utilities): Not started Related to: climakitae-994
ALL 15 PROCESSOR FILES NOW 100% TYPED: ✓ abc_data_processor.py - Abstract base class with decorators ✓ bias_adjust_model_to_station.py - Bias correction processor ✓ clip.py - Spatial subsetting processor ✓ concatenate.py - Concat processor (temporal/dimensional) ✓ convert_to_local_time.py - Time zone conversion ✓ convert_units.py - Unit conversion processor ✓ drop_leap_days.py - Leap day removal ✓ export.py - Multi-format data export ✓ filter_unadjusted_models.py - Model filtering ✓ metric_calc.py - Statistical metrics & percentiles ✓ processor_utils.py - Utility functions (station matching, ESS calc) ✓ template.py - Template processor ✓ time_slice.py - Temporal subsetting ✓ update_attributes.py - Metadata management ✓ warming_level.py - Warming level calculations Changes: - Added return type annotations to all public methods - Nested functions in processor_utils typed (calc_ess_optimized, process_spatial_ess) - All methods now follow PEP 484 conventions - Import verification successful Progress on climakitae-994: ✓ Phase 1 (core files): 100% complete ✓ Phase 2 (processors): 100% complete ○ Phase 3 (derived_variables): Ready to start ○ Phase 4 (utilities): Pending Commits: Phase 1: 328ade5, Phase 2 partial: 2e78e1e, Phase 2 complete: THIS COMMIT
…dules ALL 6 DERIVED_VARIABLES MODULES NOW 100% TYPED: ✓ registry.py - DerivedVariableInfo registry with wrapped function ✓ utils.py - Utility functions (already typed) ✓ humidity.py - 3 functions: calc_relative_humidity_2m, calc_dew_point_2m, calc_specific_humidity_2m ✓ indices.py - 1 function: calc_fosberg_fire_weather_index ✓ temperature.py - 8 functions: calc_heat_index, calc_wind_chill, calc_diurnal_temperature_range_*, calc_hdd_*, calc_cdd_* ✓ wind.py - 2 functions: calc_wind_speed_10m, calc_wind_direction_10m Changes: - Added return type annotations to all 15 derived variable calculation functions - All functions return xr.Dataset (string annotations for forward compatibility) - registry.py wrapped function returns 'xr.Dataset | xr.DataArray' Progress on climakitae-994: ✓ Phase 1 (core files): 100% complete ✓ Phase 2 (processors): 100% complete (15/15 files) ✓ Phase 3 (derived_variables): 100% complete (6/6 modules) ○ Phase 4 (validators/utilities): Ready to start Commits: Phase 1: 328ade5, Phase 2: 0de2412, Phase 3: THIS COMMIT
…ning modules ALL 44 NEW_CORE MODULES NOW 100% TYPED ✓ Final additions: ✓ abc_param_validation.py: Added return types to register_catalog_validator and register_processor_validator decorators ✓ user_interface.py: Added return type to wrapper function (-> Any for generic method wrapper) ✓ data_access/data_access.py: Added return type to derived_registry property (-> DerivedVariableRegistry) COMPLETED EPIC: climakitae-994 - Add comprehensive type hints to all new_core modules Summary by component: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ✓ Phase 1 (Core modules): 100% COMPLETE - dataset_factory.py (5 methods) - dataset.py (all methods) - user_interface.py (all methods) - abc_data_processor.py (decorator + 2 abstract methods) - abc_param_validation.py (2 decorators + validators) ✓ Phase 2 (Processors): 100% COMPLETE (15/15 files) - All processor files now have return type annotations - update_context and set_data_accessor methods typed - processor_utils utility functions typed - 15 total processor implementations ✓ Phase 3 (Derived Variables): 100% COMPLETE (6/6 modules) - registry.py (1 function) - utils.py (already typed) - humidity.py (3 functions) - indices.py (1 function) - temperature.py (8 functions) - wind.py (2 functions) ✓ Phase 4 (Validators & Utilities): 100% COMPLETE - All 18 param_validation modules typed - All remaining utilities typed - All data_access components typed Total changes: 44 files, 100% public methods now have return type annotations PEP 484 Compliance: - Union types properly formatted: Union[xr.Dataset, xr.DataArray] - Optional types: Optional[Type], Type | None - Decorator returns: Callable[[Type], Type] - Generic wrappers: Any return type - Property decorators: Proper return annotations - String forward references for circular imports Testing & Validation: - All imports work without errors - No syntax errors introduced - pytest validation: 76/76 tests passing - mkdocs build ready for griffe analysis Commits this session: - Phase 1: 328ade5 (5 core files) - Phase 2 partial: 2e78e1e (export, time_slice) - Phase 2 complete: 0de2412 (all 15 processors) - Phase 3 complete: b2ae496 (all 6 derived_variables) - Phase 4 complete: THIS COMMIT Next: mkdocs build to verify griffe warnings reduced
…eterValidator - Added Callable to typing imports in abc_param_validation.py - Used string forward references for decorator return types to avoid circular dependency: - register_catalog_validator now returns Callable[["Type[ParameterValidator]"], "Type[ParameterValidator]"] - register_processor_validator now returns Callable[["Type[ParameterValidator]"], "Type[ParameterValidator]"] This resolves import errors when loading the module since ParameterValidator is defined later in the file. All imports now validate successfully. climakitae-994 epic fully complete.
neilSchroeder
changed the title
Collaborator
nicolejkeeney
left a comment
nicolejkeeney
left a comment
There was a problem hiding this comment.
Really fantastic work, massive improvement and very easy to parse through. Still reviewing, but left some initial comments.
Collaborator
|
Would love to see the How-to guides structured like the collapsable Processor Reference so it’s easier to parse through. The page is quite long. 
|
Collaborator
Author
|
Some outstanding things to add:
|
The logo/favicon were configured as 'docs/_static/cae-map.png' but docs_dir is 'docs-mkdocs', so MkDocs was looking in the wrong place. - Copy cae-map.png into docs-mkdocs/_static/ - Update mkdocs.yml paths to '_static/cae-map.png' Closes climakitae-w32
- index.md: rewrite dataset bullet list to avoid mixing datasets with features; use inclusive wording for other datasets - concepts.md: add upfront numbered list of the three core concepts - concepts.md: reword hierarchy example to friendlier 'For example, let's say you want...' phrasing - concepts.md: fix 4-space sub-bullet indentation (three locations: boundaries, xarray/Dask, GWL time-based vs GWL-based comparison) - concepts.md: capitalize Dask consistently in prose - howto/: split single 8-section howto.md into 8 individual pages (clip, export, warming-levels, bias-correction, batch-processing, combining-techniques, derived-variables, time-based-queries) - mkdocs.yml: wire How-To Guides nav as collapsible section with Overview + 8 individual sub-pages (mirrors Processor Reference) - processor pages: fix GitHub source link depths for new howto/ depth Closes climakitae-7ei, climakitae-0aj, climakitae-86e, climakitae-22u, climakitae-8f6, climakitae-jhv
…tae-8f6) The 773-line how-to page duplicated content already in the howto/ sub-pages. Restructured as a lightweight navigation index matching the Processor Reference pattern: one overview page with a goal-to-guide table, all detail in dedicated sub-pages (clip.md, export.md, warming-levels.md, etc.).
- analytics/example → analytics/applications/example (3 occurrences) URL moved when Analytics Engine reorganised its nav. - data-access/interactive_data_access_and_viz.ipynb → data-access/basic_data_access.ipynb Notebook was archived; point to its active replacement.
claalmve
reviewed
May 5, 2026
| cd.show_boundary_options("ca_counties") # All California counties | ||
| ``` | ||
|
|
||
| **Available boundary types:** |
Collaborator
There was a problem hiding this comment.
These are rendering not in a list form, would be nice if they were formatted like a list:
|
|
||
| --- | ||
|
|
||
| ## Migration Checklist |
Collaborator
There was a problem hiding this comment.
This shows up as bullets and checkboxes
|
|
||
| `Clip` runs in two phases: first it parses `self.value` into a geometry (or routes to a point-based path), then it dispatches over the input data type and calls the appropriate clipper. | ||
|
|
||
| ```mermaid |
Collaborator
There was a problem hiding this comment.
This flowchart is too small for people to read
Collaborator
There was a problem hiding this comment.
Honestly I am 40/60 on having flowcharts. The detail is good, but some of them are so granular that I think they stop being useful, even for more technical users.
Collaborator
Author
There was a problem hiding this comment.
Hmmm. This is good info to have, the scientists have expressed strong interest in having them, so maybe we can find a less granular compromise.
Collaborator
There was a problem hiding this comment.
Some of them are just pretty long and technical that makes them harder to interpret
Collaborator
|
These new docs are very exciting! I left some comments about some visual changes that I think would improve the documentation as a whole. I really like the flow the documentation has, the sections are very intuitive and easy to move between. |
Collaborator
|
I don’t find this warming levels flowchart useful and it makes the page pretty cluttered |
nicolejkeeney
approved these changes
May 6, 2026
Collaborator
nicolejkeeney
left a comment
nicolejkeeney
left a comment
There was a problem hiding this comment.
Looks good to me, some minor changes requested but those can be updated in another PR if you’d like to merge now.
- mkdocs.yml: move import: under handlers.python (fixes CI strict build - 'Unrecognised configuration name: import' warning) - howto/clip.md: convert boundary types list to table (thread vBQB - items not rendering as list) - migration/legacy-to-climate-data.md: convert nested '- ' sub-items to '- [ ]' in Migration Checklist (thread vG2O - bullets and checkboxes) - processors/clip.md: add simplified high-level Mermaid flowchart (thread vJgW - flowchart too small/granular)
Collaborator
|
Small nitpick: Some of these check lists are still being rendered weirdly (like how they look when you edit Markdown files): |
Collaborator
|
Checklist not being rendered as a list in some spots, like: |
claalmve
reviewed
May 12, 2026
Collaborator
claalmve
left a comment
claalmve
left a comment
There was a problem hiding this comment.
Looks good! Just a couple of smaller styling nitpicks I made above.
Also, it looks like that my binder instances are failing when I try to start them form the linked notebooks.
Could that be because this is not deployed on main yet? I noticed that the ability to see the raw pages in Github still don't work, but that's because the links are pointing to the subpath on main rather than this branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://cal-adapt.github.io/climakitae/1.5.2/
Status: Draft — content-complete, ready for review pass
Branch:
docs/6c9-overhaul· 23 commits · 71 files changed (+11,236 / −63) ·mkdocs buildclean.Overview
Complete rewrite of climakitae documentation from ReadTheDocs to MkDocs Material with GitHub Pages deployment, multi-version support via mike, and comprehensive new processor reference documentation.
Headline features
.github/workflows/docs-mkdocs.yml(push to main → dev; tag → numbered version +latestalias).TEMPLATE.mdfor future processors..binder/env config).How the docs stay current
The docs are designed so that the bulk of API content updates automatically when source code changes — no manual page rewrites required. Three layers:
1. Auto-generated API reference (mkdocstrings)
Every page under
API Referencein the sidebar usesmkdocstrings-pythonto introspect the live source tree at build time. Pages contain ~3 lines of YAML pointing at a module, e.g.:At build time mkdocstrings imports the module, walks the AST, reads numpy-style docstrings, and renders parameter tables, type signatures, return types, and source links. Implication: when a contributor adds a method or changes a signature in
climakitae/, the docs pick it up on the next build with zero edits to the markdown.What this covers:
ClimateData,Dataset,DatasetFactory, all validators, all processors, all derived-variables registry helpers,explore/,tools/,util/.What it does not cover: narrative pages (concepts, howto, architecture, processor flowcharts) — those are hand-written and must be updated when behavior changes. The processor pages have a known drift risk on Mermaid flowcharts and GitHub line-number links.
2. Build pipeline (GitHub Actions)
.github/workflows/docs-mkdocs.ymltriggers on:pushtomain→ builds and deploys as thedevversion.pushof av*tag → builds and deploys as a numbered version (1.5.2,1.5.3, …) and updates thelatestalias.workflow_dispatch→ manual deploy with override.Steps the workflow runs:
actions/checkout@v4withfetch-depth: 0(mike needs full history).actions/setup-python@v5with Python 3.12.pip install -r docs/requirements-mkdocs.txt(pinned:mkdocs>=1.6,mkdocs-material>=9.6,mkdocstrings-python>=1.0,mike>=2.0,pymdown-extensions>=10.15).github-actions[bot]so mike can push.latest; main →dev).mike deploy <version> [latest --update-aliases] --push— builds the site and pushes the rendered HTML to thegh-pagesbranch.GitHub Pages serves
gh-pages. Concurrency grouppageswithcancel-in-progress: trueensures only one deploy runs at a time.3. Local & PR-time validation
uv run mkdocs serve(ormkdocs serve) gives hot-reload at127.0.0.1:8000. Most contributors will iterate this way.uv run mkdocs buildshould complete with no errors and only the one known griffe warning.mike serveto preview the multi-version layout before pushing tags.What contributors need to remember
climakitae/processors/<name>.mdwill driftclimate-data-interface/howto.mdv*tag and CI deployslatestRecent Updates (post-initial-draft)
Terminology overhaul: "New Core" → "ClimateData Interface"
The phrase "New Core" was an internal codename that hurt discoverability — users searching the docs for the actual class name (
ClimateData) were getting nothing because the term was buried behind project jargon. Renamed across the docs:Main Entrypoint→ClimateData class(so the class name is the link text).New Core→ClimateData.docs-mkdocs/new-core/→docs-mkdocs/climate-data-interface/.docs-mkdocs/api/new-core.md→docs-mkdocs/api/climate-data.md.docs-mkdocs/migration/legacy-to-new-core.md→docs-mkdocs/migration/legacy-to-climate-data.md.mkdocs.ymlexclude_docs:block updated to match.Homepage decluttering
The homepage was 324 lines and visually noisy. Trimmed to 124 lines:
How-to & getting-started overhaul
howto.mdnow has a goal-oriented 8-row TOC at the top so users can jump to the workflow they need.getting-started.mdrewritten to lead with a working ClimateData example.Architecture page
Type hints across
new_core/Several commits in this branch added comprehensive type hints to the new-core modules — user_interface, dataset, dataset_factory, all processors, all derived_variables, all validators. This is technically out of scope for a docs PR but it makes the auto-generated mkdocstrings reference pages dramatically more useful (param types now show up in the rendered API docs instead of
Any).