diff --git a/docs/source/user_guide/hooks_and_presets.md b/docs/source/user_guide/hooks_and_presets.md index 6993be9..afed74d 100644 --- a/docs/source/user_guide/hooks_and_presets.md +++ b/docs/source/user_guide/hooks_and_presets.md @@ -1,4 +1,4 @@ -# šŸŖ Hooks and Presets +# šŸŖ Hooks Fetchez is designed to be highly extendable. Using **hooks** and **presets**, you can build automated pipelines that process fetched or local data on the fly. @@ -94,16 +94,20 @@ global_hooks: file: charts_audit_full.json ``` -### Extending Hooks and Presets (Plugins and Extensions) -Fetchez is generic. If you are building a custom tool and want to create your own processing hooks and presets, you can register your own hooks and presets either in your project or in the `.fetchez` configuration directory and they will be discoverable with the `fetchez.registry.HookRegistry` and `fetchez.registry.PresetRegistry` +## Extending Hooks and Presets (Plugins and Extensions) +Fetchez is generic. If you are building a custom tool and want to create your own processing hooks and presets, you can register your own hooks and presets either in your project or in the `.fetchez` configuration directory and they will be discoverable with the `fetchez.registry` -In your project, make a directory called 'hooks' and/or 'hooks/presets'; add any python hooks and YAML presets to the appropriate directory and register them with Fetchez in your `pyproject.toml`: +In your project, make a directory called 'hooks' and/or 'hooks/presets'; add any python source hooks and YAML presets to the appropriate directory and register them with Fetchez in your `pyproject.toml`: + +**Hooks** ```toml [project.entry-points."fetchez.hooks"] my_project_hooks = "my_project.hooks" ``` +**Presets** + ```toml [project.entry-points."fetchez.hooks.presets"] my_project_presetes = "my_project.hooks.presets" diff --git a/docs/source/user_guide/index.md b/docs/source/user_guide/index.md index 5f2af41..46b487b 100644 --- a/docs/source/user_guide/index.md +++ b/docs/source/user_guide/index.md @@ -9,9 +9,8 @@ installation cli_usage modules_and_bundles hooks_and_presets -plugins_and_extensions +streams recipes -modifiers -schemas +plugins_and_extensions data_persistence ``` diff --git a/docs/source/user_guide/modifiers.md b/docs/source/user_guide/modifiers.md deleted file mode 100644 index 5a9c908..0000000 --- a/docs/source/user_guide/modifiers.md +++ /dev/null @@ -1,47 +0,0 @@ -# šŸ› ļø Domain Modifiers - -Fetchez includes a **Modifier Engine** in its `ModifierRegistry` that can automatically mutate your YAML recipes as they are loaded, allowing for complete runtime control of the pipeline. - -## Using a Modifier - -Add a `modifiers` argument to the top of your YAML recipe: - -```yaml ---- -project: - name: "My_Project" - -modifiers: - - name: exclude_module - args: - modules: margrav/charts -region: [-120.0, -119.75, 33.0, 33.25] -modules: - - bundle: my-bathymetry-bundle -``` - -*What happens under the hood?* - -By specifying the modifier: `exclude_module`, the engine intercepts your recipe and removes the named `margrav` and `charts` modules from the module Bundle `my-bathymetry-bundle`. Modifiers take an input recipe config and do something to or with it and return the possibly mutated bundle, right before sending to the core Fetchez engine for processing. - -## Extending Modifiers (Plugins and Extensions) -Fetchez is generic. If you are building a custom tool (like a specialized DEM engine), you can register your own recipe modifiers in Python. Make a directory called 'my_project/recipes/modifiers' and put all your modifier python files within it: - -```python -from fetchez.recipes.modifiers import BaseModifier - -class WeatherModifier(BaseModifier): - name = "wrf_weather" - - @classmethod - def apply(cls, config): - config["region"] = [-180, 180, -90, 90] # Force global fetch - return config -``` - -Then register your project with fetchez in your `pyproject.toml`: - -```toml -[project.entry-points."fetchez.recipes.modifiers"] -my_project_modifiers = "my_project.recipes.modifiers" -``` diff --git a/docs/source/user_guide/modules_and_bundles.md b/docs/source/user_guide/modules_and_bundles.md index 6696e71..4d5c257 100644 --- a/docs/source/user_guide/modules_and_bundles.md +++ b/docs/source/user_guide/modules_and_bundles.md @@ -1,4 +1,4 @@ -# šŸŒŽ Modules and Bundles +# šŸŒŽ Modules Fetchez comes builtin with [70+ different modules](https://fetchez.readthedocs.io/en/latest/modules/index.html) to access geospatial data from various remote apis and local file-systems. @@ -69,12 +69,21 @@ project: args: {weight: 1.0} ``` -### Extending Bunldes (Plugins and Extensions) -Fetchez is generic. If you are building a custom tool and want to bundle your own modules, you can register your own bundles either in your project or in the `.fetchez` configuration directory and they will be discoverable with the `fetchez.registry.BundleRegistry` +## Extending Bunldes (Plugins and Extensions) +Fetchez is generic. If you are building a custom tool and want to create a custom module or bundle your own custom set of modules, you can register them either in your project or in the `~.fetchez` configuration directory and they will be discoverable with the `fetchez.registry` -To create an extension where your bundles can be installed and used by `fetchez`, make a directory called 'bundles' in your project; add any YAML module bundles to that directory and register them with `fetchez` in your `pyproject.toml`: +To create an extension where your bundles can be installed and used by `fetchez`, make a directory called 'modules' in your project; add any python source files defining the module there or make a directory called `modules/bundles` add any YAML bundle definitions there and register them with `fetchez` in your `pyproject.toml`: + +**Modules** + +```toml +[project.entry-points."fetchez.modules"] +my_project_modules = "my_project.modules" +``` + +**Bundles** ```toml [project.entry-points."fetchez.modules.bundles"] -my_project_bundles = "my_project.hooks.bundles" +my_project_bundles = "my_project.modules.bundles" ``` diff --git a/docs/source/user_guide/plugins_and_extensions.md b/docs/source/user_guide/plugins_and_extensions.md index 8f690ca..4ad5e14 100644 --- a/docs/source/user_guide/plugins_and_extensions.md +++ b/docs/source/user_guide/plugins_and_extensions.md @@ -1,6 +1,6 @@ # šŸ„ Plugins & Extensions -Fetchez is designed to be highly extendable. Fetchez can extended by adding Modules, Bundles, Hooks, Presets, Recipes, Modifiers and Schemas. Using the fetchez registry system, its simple to create custom personal plugins or widely distributed Fetchez extensions. +Fetchez is designed to be highly extendable. Fetchez can extended by adding Modules, Bundles, Hooks, Presets, Readers, Profiles, Recipes, Modifiers and Schemas. Using the fetchez registry system, its simple to create custom personal plugins or widely distributed Fetchez extensions. There are two ways to extend `fetchez`: **Local Plugins** (for quick, personal scripts and plugins) and **Full Extensions** (for distributable Python packages). diff --git a/docs/source/user_guide/recipes.md b/docs/source/user_guide/recipes.md index 9bcad41..7caa014 100644 --- a/docs/source/user_guide/recipes.md +++ b/docs/source/user_guide/recipes.md @@ -140,22 +140,85 @@ global_hooks: ``` ### Modifiers -Modifiers dynamically mutate your recipe configuration before execution begins. This is useful for temporarily excluding large modules, or injecting arguments during a specific run. Modifiers can be invoked via the CLI: +Fetchez includes a **Modifier Engine** in its `ModifierRegistry` that can automatically mutate your YAML recipes as they are loaded, allowing for complete runtime control of the pipeline. + +#### Using a Modifier + +Add a `modifiers` argument to the top of your YAML recipe: + +```yaml +--- +project: + name: "My_Project" + +modifiers: + - name: exclude_module + args: + modules: margrav/charts +region: [-120.0, -119.75, 33.0, 33.25] +modules: + - bundle: my-bathymetry-bundle +``` + +*What happens under the hood?* + +By specifying the modifier: `exclude_module`, the engine intercepts your recipe and removes the named `margrav` and `charts` modules from the module Bundle `my-bathymetry-bundle`. Modifiers take an input recipe config and do something to or with it and return the possibly mutated bundle, right before sending to the core Fetchez engine for processing. + + +**Use the modifier in the cli** ```bash -# Injecting a new cache_dir argument into the stream_reproject module at runtime -fetchez recipes run my_recipe.yaml --modifier inject_args:match=stream_reproject,cache_dir=socal_data +fetchez recipes run my_project.yaml --modifier exclude_module:modules=margrav/charts ``` ### Schemas -Schemas act as quality control for your pipelines. They enforce strict validation rules to ensure a recipe is formatted correctly and contains all required fields before `fetchez` attempts to run it. If a recipe fails schema validation, the engine will warn you before any downloads begin, preventing partial or failed executions. +Fetchez includes a **Schema Engine** in its `SchemaRegistry` that automatically scans your YAML recipes to enforce rules or otherwise validate the recipe structure or purpose. + +#### Using a Schema + +Add a `schemas` argument to the top of your YAML recipe, in this example we'll use a theoretical `schema` that would make sure the `region` parameter is a strict 1/4 degree tile: + +```yaml +project: + name: "My_Strict_Project" + +schemas: + - name: "quarter-degree-tile" +region: [-120.0, -119.75, 33.0, 33.25] # Your exact delivery tile +``` + +*What happens under the hood?* + +By specifying schema: `quarter-degree-tile`, the engine intercepts your recipe and checks your region to make sure it snaps directly to a quarter degree tile in WGS84. It will return the validity of the recipe based on that schema along with any errors it found. + +**Use the schema in the CLI** + +```bash +fetchez recipes run -R -120/-119.75/33/33.25 --schema quarter-degree-tile my_strict_project.yaml +``` ## Extending Recipes (Plugins and Extensions) -Fetchez is generic. If you are building a custom tool (like a specialized DEM engine), you can register your own recipes either in your project or in the .fetchez configuration directory and they will be discoverable with the `fetchez.registry.RecipeRegistry` +Fetchez is generic. If you are building a custom tool (like a specialized DEM engine), you can register your own recipes, modifiers and schems either in your project or in the `~/.fetchez` configuration directory and they will be discoverable with the `fetchez.registry` + +In your project, make a directory called 'recipes'; add any YAML recipes to that directory, add any python source files in 'recipes/modifiers' or 'recipe/schemas' and register them with `fetchez` in your `pyproject.toml`: -In your project, make a directory called 'recipes'; add any YAML recipes to that directory and register them with `fetchez` in your `pyproject.toml`: +**Recipes** ```toml [project.entry-points."fetchez.recipes"] my_project_recipes = "my_project.recipes" ``` + +**Modifiers** + +```toml +[project.entry-points."fetchez.recipes.modifiers"] +my_project_modifiers = "my_project.recipes.modifiers" +``` + +**Schemas** + +```toml +[project.entry-points."fetchez.recipes.schemas"] +my_project_schemas = "my_project.recipes.schemas" +``` diff --git a/docs/source/user_guide/schemas.md b/docs/source/user_guide/schemas.md deleted file mode 100644 index ed93df4..0000000 --- a/docs/source/user_guide/schemas.md +++ /dev/null @@ -1,47 +0,0 @@ -# šŸ›ļø Domain Schemas - -Fetchez includes a **Schema Engine** in its `SchemaRegistry` that automatically scans your YAML recipes to enforce rules or otherwise validate the recipe structure or purpose. - -## Using a Schema - -Add a `schemas` argument to the top of your YAML recipe, in this example we'll use a theoretical `schema` that would make sure the `region` parameter is a strict 1/4 degree tile: - -```yaml -project: - name: "My_Strict_Project" - -schemas: - - name: "quarter-degree-tile" -region: [-120.0, -119.75, 33.0, 33.25] # Your exact delivery tile -``` - -*What happens under the hood?* - -By specifying schema: `quarter-degree-tile`, the engine intercepts your recipe and checks your region to make sure it snaps directly to a quarter degree tile in WGS84. It will return the validity of the recipe based on that schema along with any errors it found. - -## Extending Schemas (Plugins and Extensions) -Fetchez is generic. If you are building a custom tool (like a specialized DEM engine), you can register your own schemas in Python. Make a directory called 'my_project/recipes/schemas' and put all your schema python files within it: - -```python -from fetchez.schema import BaseSchema - -class WeatherSchema(BaseSchema): - name = "wrf_global_weather" - - @classmethod - def validate(cls, config): - from fetchez.spatial import Region - local_region = Region(*config.get("region")) - global_region = Region(-180, 180, -90, 90]) - if local_region != global_region: - return False, ["local_region is not global"] - return True, [] - -``` - -Then register your project with fetchez in your `pyproject.toml`: - -```toml -[project.entry-points."fetchez.recipes.schemas"] -my_project_schemas = "my_project.recipes.schemas" -``` diff --git a/docs/source/user_guide/streams.md b/docs/source/user_guide/streams.md new file mode 100644 index 0000000..1accec9 --- /dev/null +++ b/docs/source/user_guide/streams.md @@ -0,0 +1,26 @@ +# šŸ—ƒļø Streams + +## Data Streams + +## Format Readers + +## Reader Profiles + +### Extending Streams (Plugins and Extensions) +Fetchez is generic. If you are building a custom tool and want to create your own format readers or profiles, you can register them either in your project or in the `~/.fetchez` configuration directory and they will be discoverable with the `fetchez.registry.ReaderRegistry` and `fetchez.registry.ProfileRegistry`. +xs +In your project, make a directory called 'streams/readers' and/or 'streams/profiles'; add any python readers and YAML profiles to the appropriate directory and register them with Fetchez in your `pyproject.toml`: + +**Readers** + +```toml +[project.entry-points."fetchez.streams.readers"] +my_project_readers = "my_project.streams.readers" +``` + +**Profiles** + +```toml + [project.entry-points."fetchez.streams.profiles"] +my_project_presetes = "my_project.streams.profiles" +``` diff --git a/src/fetchez/cli/recipes.py b/src/fetchez/cli/recipes.py index 7208795..5d683f6 100644 --- a/src/fetchez/cli/recipes.py +++ b/src/fetchez/cli/recipes.py @@ -382,7 +382,7 @@ def run_recipe( click.secho(f"āœØ Successfully executed {name} recipe!", fg="green", bold=True) except Exception as e: - click.secho(f"Failed to execute {name} recipe!: {str(e)}", fg="red") + click.secho(f"Failed to execute {name} recipe!: {str(e)}", fg="red", bold=True) recipes_group.add_command(schemas_group, name="schemas") diff --git a/src/fetchez/cli/schemas.py b/src/fetchez/cli/schemas.py index becb198..698bba7 100644 --- a/src/fetchez/cli/schemas.py +++ b/src/fetchez/cli/schemas.py @@ -86,7 +86,7 @@ def schemas_info(name): click.secho(f"Error: Schema '{name}' not found.", fg="red") sys.exit(1) - click.secho(f"\nšŸŖ SCHEMA: {name}", fg="cyan", bold=True) + click.secho(f"\nšŸ›ļø SCHEMA: {name}", fg="cyan", bold=True) click.echo("=" * 60) click.echo(f" Description : {meta.get('desc', 'N/A')}") click.echo(f" Category : {meta.get('category', 'N/A')}\n") diff --git a/src/fetchez/hooks/transfer_log.py b/src/fetchez/hooks/transfer_log.py index 8e04ff6..36f997c 100644 --- a/src/fetchez/hooks/transfer_log.py +++ b/src/fetchez/hooks/transfer_log.py @@ -19,10 +19,10 @@ class TransferLog(FetchHook): - """Write a clear summary of failed and successful downloads.""" + """Write a summary of failed and successful downloads.""" name = "transfer-log" - meta_desc = "Generates a clear report of download successes and failures." + meta_desc = "Generates a report of download successes and failures." meta_stage = "collection" # post meta_category = "metadata" meta_aliases = ["transfer_log"]