Add streams docs

docs/source/user_guide/hooks_and_presets.md

@@ -1,4 +1,4 @@
-# 🪝 Hooks and Presets
+# 🪝 Hooks
 
 Fetchez is designed to be highly extendable. Using **hooks** and **presets**, you can build automated pipelines that process fetched or local data on the fly.
 
@@ -94,16 +94,20 @@ global_hooks:
           file: charts_audit_full.json
 ```
 
-### Extending Hooks and Presets (Plugins and Extensions)
-Fetchez is generic. If you are building a custom tool and want to create your own processing hooks and presets, you can register your own hooks and presets either in your project or in the `.fetchez` configuration directory and they will be discoverable with the `fetchez.registry.HookRegistry` and `fetchez.registry.PresetRegistry`
+## Extending Hooks and Presets (Plugins and Extensions)
+Fetchez is generic. If you are building a custom tool and want to create your own processing hooks and presets, you can register your own hooks and presets either in your project or in the `.fetchez` configuration directory and they will be discoverable with the `fetchez.registry`
 
-In your project, make a directory called 'hooks' and/or 'hooks/presets'; add any python hooks and YAML presets to the appropriate directory and register them with Fetchez in your `pyproject.toml`:
+In your project, make a directory called 'hooks' and/or 'hooks/presets'; add any python source hooks and YAML presets to the appropriate directory and register them with Fetchez in your `pyproject.toml`:
+
+**Hooks**
 
 ```toml
 [project.entry-points."fetchez.hooks"]
 my_project_hooks = "my_project.hooks"
 ```
 
+**Presets**
+
 ```toml
 [project.entry-points."fetchez.hooks.presets"]
 my_project_presetes = "my_project.hooks.presets"

docs/source/user_guide/index.md

@@ -9,9 +9,8 @@ installation
 cli_usage
 modules_and_bundles
 hooks_and_presets
-plugins_and_extensions
+streams
 recipes
-modifiers
-schemas
+plugins_and_extensions
 data_persistence
 ```

docs/source/user_guide/modules_and_bundles.md

@@ -1,4 +1,4 @@
-# 🌎 Modules and Bundles
+# 🌎 Modules
 
 Fetchez comes builtin with [70+ different modules](https://fetchez.readthedocs.io/en/latest/modules/index.html) to access geospatial data from various remote apis and local file-systems.
 
@@ -69,12 +69,21 @@ project:
     args: {weight: 1.0}
 ```
 
-### Extending Bunldes (Plugins and Extensions)
-Fetchez is generic. If you are building a custom tool and want to bundle your own modules, you can register your own bundles either in your project or in the `.fetchez` configuration directory and they will be discoverable with the `fetchez.registry.BundleRegistry`
+## Extending Bunldes (Plugins and Extensions)
+Fetchez is generic. If you are building a custom tool and want to create a custom module or bundle your own custom set of modules, you can register them  either in your project or in the `~.fetchez` configuration directory and they will be discoverable with the `fetchez.registry`
 
-To create an extension where your bundles can be installed and used by `fetchez`, make a directory called 'bundles' in your project; add any YAML module bundles to that directory and register them with `fetchez` in your `pyproject.toml`:
+To create an extension where your bundles can be installed and used by `fetchez`, make a directory called 'modules' in your project; add any python source files defining the module there or make a directory called `modules/bundles` add any YAML bundle definitions there and register them with `fetchez` in your `pyproject.toml`:
+
+**Modules**
+
+```toml
+[project.entry-points."fetchez.modules"]
+my_project_modules = "my_project.modules"
+```
+
+**Bundles**
 
 ```toml
 [project.entry-points."fetchez.modules.bundles"]
-my_project_bundles = "my_project.hooks.bundles"
+my_project_bundles = "my_project.modules.bundles"
 ```

docs/source/user_guide/plugins_and_extensions.md

@@ -1,6 +1,6 @@
 # 🐄 Plugins & Extensions
 
-Fetchez is designed to be highly extendable. Fetchez can extended by adding Modules, Bundles, Hooks, Presets, Recipes, Modifiers and Schemas. Using the fetchez registry system, its simple to create custom personal plugins or widely distributed Fetchez extensions.
+Fetchez is designed to be highly extendable. Fetchez can extended by adding Modules, Bundles, Hooks, Presets, Readers, Profiles, Recipes, Modifiers and Schemas. Using the fetchez registry system, its simple to create custom personal plugins or widely distributed Fetchez extensions.
 
 There are two ways to extend `fetchez`: **Local Plugins** (for quick, personal scripts and plugins) and **Full Extensions** (for distributable Python packages).
 

docs/source/user_guide/recipes.md

@@ -140,22 +140,85 @@ global_hooks:
 ```
 
 ### Modifiers
-Modifiers dynamically mutate your recipe configuration before execution begins. This is useful for temporarily excluding large modules, or injecting arguments during a specific run. Modifiers can be invoked via the CLI:
+Fetchez includes a **Modifier Engine** in its `ModifierRegistry` that can automatically mutate your YAML recipes as they are loaded, allowing for complete runtime control of the pipeline.
+
+#### Using a Modifier
+
+Add a `modifiers` argument to the top of your YAML recipe:
+
+```yaml
+---
+project:
+  name: "My_Project"
+
+modifiers:
+  - name: exclude_module
+    args:
+      modules: margrav/charts
+region: [-120.0, -119.75, 33.0, 33.25]
+modules:
+  - bundle: my-bathymetry-bundle
+```
+
+*What happens under the hood?*
+
+By specifying the modifier: `exclude_module`, the engine intercepts your recipe and removes the named `margrav` and `charts` modules from the module Bundle `my-bathymetry-bundle`. Modifiers take an input recipe config and do something to or with it and return the possibly mutated bundle, right before sending to the core Fetchez engine for processing.
+
+
+**Use the modifier in the cli**
 
 ```bash
-# Injecting a new cache_dir argument into the stream_reproject module at runtime
-fetchez recipes run my_recipe.yaml --modifier inject_args:match=stream_reproject,cache_dir=socal_data
+fetchez recipes run my_project.yaml --modifier exclude_module:modules=margrav/charts
 ```
 
 ### Schemas
-Schemas act as quality control for your pipelines. They enforce strict validation rules to ensure a recipe is formatted correctly and contains all required fields before `fetchez` attempts to run it. If a recipe fails schema validation, the engine will warn you before any downloads begin, preventing partial or failed executions.
+Fetchez includes a **Schema Engine** in its `SchemaRegistry` that automatically scans your YAML recipes to enforce rules or otherwise validate the recipe structure or purpose.
+
+#### Using a Schema
+
+Add a `schemas` argument to the top of your YAML recipe, in this example we'll use a theoretical `schema` that would make sure the `region` parameter is a strict 1/4 degree tile:
+
+```yaml
+project:
+  name: "My_Strict_Project"
+
+schemas:
+  - name: "quarter-degree-tile"
+region: [-120.0, -119.75, 33.0, 33.25] # Your exact delivery tile
+```
+
+*What happens under the hood?*
+
+By specifying schema: `quarter-degree-tile`, the engine intercepts your recipe and checks your region to make sure it snaps directly to a quarter degree tile in WGS84. It will return the validity of the recipe based on that schema along with any errors it found.
+
+**Use the schema in the CLI**
+
+```bash
+fetchez recipes run -R -120/-119.75/33/33.25 --schema quarter-degree-tile my_strict_project.yaml
+```
 
 ## Extending Recipes (Plugins and Extensions)
-Fetchez is generic. If you are building a custom tool (like a specialized DEM engine), you can register your own recipes either in your project or in the .fetchez configuration directory and they will be discoverable with the `fetchez.registry.RecipeRegistry`
+Fetchez is generic. If you are building a custom tool (like a specialized DEM engine), you can register your own recipes, modifiers and schems either in your project or in the `~/.fetchez` configuration directory and they will be discoverable with the `fetchez.registry`
+
+In your project, make a directory called 'recipes'; add any YAML recipes to that directory, add any python source files in 'recipes/modifiers' or 'recipe/schemas' and register them with `fetchez` in your `pyproject.toml`:
 
-In your project, make a directory called 'recipes'; add any YAML recipes to that directory and register them with `fetchez` in your `pyproject.toml`:
+**Recipes**
 
 ```toml
 [project.entry-points."fetchez.recipes"]
 my_project_recipes = "my_project.recipes"
 ```
+
+**Modifiers**
+
+```toml
+[project.entry-points."fetchez.recipes.modifiers"]
+my_project_modifiers = "my_project.recipes.modifiers"
+```
+
+**Schemas**
+
+```toml
+[project.entry-points."fetchez.recipes.schemas"]
+my_project_schemas = "my_project.recipes.schemas"
+```

docs/source/user_guide/streams.md

@@ -0,0 +1,26 @@
+# 🗃️ Streams
+
+## Data Streams
+
+## Format Readers
+
+## Reader Profiles
+
+### Extending Streams (Plugins and Extensions)
+Fetchez is generic. If you are building a custom tool and want to create your own format readers or profiles, you can register them either in your project or in the `~/.fetchez` configuration directory and they will be discoverable with the `fetchez.registry.ReaderRegistry` and `fetchez.registry.ProfileRegistry`.
+xs
+In your project, make a directory called 'streams/readers' and/or 'streams/profiles'; add any python readers and YAML profiles to the appropriate directory and register them with Fetchez in your `pyproject.toml`:
+
+**Readers**
+
+```toml
+[project.entry-points."fetchez.streams.readers"]
+my_project_readers = "my_project.streams.readers"
+```
+
+**Profiles**
+
+```toml
+	[project.entry-points."fetchez.streams.profiles"]
+my_project_presetes = "my_project.streams.profiles"
+```

src/fetchez/cli/recipes.py

@@ -382,7 +382,7 @@ def run_recipe(
         click.secho(f"✨ Successfully executed {name} recipe!", fg="green", bold=True)
 
     except Exception as e:
-        click.secho(f"Failed to execute {name} recipe!: {str(e)}", fg="red")
+        click.secho(f"Failed to execute {name} recipe!: {str(e)}", fg="red", bold=True)
 
 
 recipes_group.add_command(schemas_group, name="schemas")

src/fetchez/cli/schemas.py

@@ -86,7 +86,7 @@ def schemas_info(name):
         click.secho(f"Error: Schema '{name}' not found.", fg="red")
         sys.exit(1)
 
-    click.secho(f"\n🪝 SCHEMA: {name}", fg="cyan", bold=True)
+    click.secho(f"\n🏛️ SCHEMA: {name}", fg="cyan", bold=True)
     click.echo("=" * 60)
     click.echo(f"  Description : {meta.get('desc', 'N/A')}")
     click.echo(f"  Category    : {meta.get('category', 'N/A')}\n")

src/fetchez/hooks/transfer_log.py

@@ -19,10 +19,10 @@
 
 
 class TransferLog(FetchHook):
-    """Write a clear summary of failed and successful downloads."""
+    """Write a summary of failed and successful downloads."""
 
     name = "transfer-log"
-    meta_desc = "Generates a clear report of download successes and failures."
+    meta_desc = "Generates a report of download successes and failures."
     meta_stage = "collection"  # post
     meta_category = "metadata"
     meta_aliases = ["transfer_log"]