Initial Project Design & Architecture

# Iolaus — Initial Project Design & Architecture

## Overview

**Iolaus** is a lightweight Python framework for research data analysis projects. It wires together [Dynaconf](https://www.dynaconf.com/), [Typer](https://typer.tiangolo.com/), and a custom run-logging system into a single decorator-based API that feels like FastAPI/Typer but adds automatic configuration management and reproducible run artifacts on every invocation.

**Core value proposition:** every command run produces a timestamped output directory containing a merged config snapshot and a log file, with zero boilerplate in the user's code.

---

## Motivating Example

```python
# myproject/cli.py
from pathlib import Path
from dynaconf import Dynaconf
import typer
from iolaus import command

app = typer.Typer()
settings = Dynaconf(settings_files=["settings.toml"], envvar_prefix="MYAPP")
cmd = command(app, settings)

@cmd
def analyze(
    input: Path,
    verbose: bool = False,
    settings=None,   # injected by Iolaus if declared
    run_dir: Path = None,  # injected by Iolaus if declared
):
    """Run the analysis pipeline."""
    ...

if __name__ == "__main__":
    app()
```

```bash
# Base invocation
python cli.py analyze data.csv

# Merge an extra config file on top of the base settings
python cli.py analyze data.csv --config prod.yaml

# Override individual keys (Dynaconf __ separator for nesting)
python cli.py analyze data.csv --config prod.yaml --set model__lr=0.01 --set db__host=remote
```

Every run produces:

```
outputs/
└── analyze/
    └── 2025-03-29/
        └── 14-32-05/
            ├── run.log
            └── config.json   # full merged config snapshot
```

---

## Technical Design

### Decorator architecture

The central challenge is that Typer derives CLI parameters from function signatures at decoration time. Iolaus injects extra parameters into the signature *before* Typer sees the function, using `inspect.Parameter` + `__signature__` replacement.

```python
# iolaus/decorators.py
import inspect
import functools
import logging
from datetime import datetime
from pathlib import Path
from typing import Annotated, Optional
import typer
from dynaconf import Dynaconf


_EXTRA_PARAMS = [
    inspect.Parameter(
        "extra_config",
        inspect.Parameter.KEYWORD_ONLY,
        default=None,
        annotation=Annotated[
            Optional[Path],
            typer.Option("--config", "-c", help="Extra config file to merge on top of base settings."),
        ],
    ),
    inspect.Parameter(
        "override",
        inspect.Parameter.KEYWORD_ONLY,
        default=[],
        annotation=Annotated[
            list[str],
            typer.Option(
                "--set", "-s",
                help="Override a config value. Use __ for nesting: --set model__lr=0.01",
            ),
        ],
    ),
]


def command(app: typer.Typer, base_settings: Dynaconf, output_dir: Path = Path("outputs")):
    """
    Decorator factory. Bind once, reuse as a decorator on every command.

    Usage:
        cmd = command(app, settings)

        @cmd
        def my_command(...): ...
    """
    def decorator(func):
        sig = inspect.signature(func)
        existing = list(sig.parameters.values())
        existing_names = {p.name for p in existing}
        extra = [p for p in _EXTRA_PARAMS if p.name not in existing_names]
        new_sig = sig.replace(parameters=existing + extra)

        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            extra_config: Optional[Path] = kwargs.pop("extra_config", None)
            overrides: list[str] = kwargs.pop("override", [])

            merged_settings = _build_settings(base_settings, extra_config, overrides)
            run_dir = _setup_logging(func.__name__, output_dir)
            _save_config_snapshot(merged_settings, run_dir)

            if "settings" in sig.parameters:
                kwargs["settings"] = merged_settings
            if "run_dir" in sig.parameters:
                kwargs["run_dir"] = run_dir

            return func(*args, **kwargs)

        wrapper.__signature__ = new_sig
        app.command()(wrapper)
        return func  # return unwrapped original so it is unit-testable without Typer

    return decorator


def _build_settings(base: Dynaconf, extra_config: Optional[Path], overrides: list[str]) -> Dynaconf:
    files = list(base.options.get("settings_file", []))
    if extra_config:
        files.append(str(extra_config))

    merged = Dynaconf(
        settings_files=files,
        envvar_prefix=base.options.get("envvar_prefix", "APP"),
    )

    # __ is Dynaconf's native nested-key separator (DYNACONF_DOTTED_LOOKUP not required)
    for item in overrides:
        key, _, value = item.partition("=")
        merged.set(key.replace("__", "."), value)

    return merged


def _setup_logging(command_name: str, base_dir: Path) -> Path:
    run_dir = base_dir / command_name / datetime.now().strftime("%Y-%m-%d/%H-%M-%S")
    run_dir.mkdir(parents=True, exist_ok=True)

    logging.basicConfig(
        level=logging.INFO,
        format="[%(asctime)s][%(name)s][%(levelname)s] %(message)s",
        handlers=[
            logging.FileHandler(run_dir / "run.log"),
            logging.StreamHandler(),
        ],
        force=True,
    )
    return run_dir


def _save_config_snapshot(settings: Dynaconf, run_dir: Path):
    import json
    snapshot = settings.as_dict()
    (run_dir / "config.json").write_text(json.dumps(snapshot, indent=2, default=str))
```

### Key design decisions

| Decision | Choice | Rationale |
|---|---|---|
| Config override separator | `__` (double underscore) | Dynaconf native; no ambiguity with dotted key names |
| `settings` / `run_dir` injection | Opt-in via signature declaration | Keeps simple commands simple; no forced parameters |
| Decorator return value | Unwrapped `func` | Allows direct import and unit testing without Typer |
| `basicConfig(force=True)` | Yes | Ensures each command invocation gets its own log file handlers |
| Config snapshot format | JSON | Human-readable, diff-friendly, no extra dependency |

---

## Repository Structure

```
iolaus/
├── .github/
│   └── workflows/
│       ├── ci.yml          # run tests on every PR
│       └── publish.yml     # bump version + publish to PyPI on merge to main
├── docs/
│   ├── index.md
│   ├── getting-started.md
│   ├── configuration.md
│   ├── cli.md
│   └── api-reference.md
├── src/
│   └── iolaus/
│       ├── __init__.py
│       ├── decorators.py
│       ├── settings.py     # Dynaconf helpers / factory
│       ├── logging.py      # _setup_logging, _save_config_snapshot
│       └── py.typed        # PEP 561 marker
├── tests/
│   ├── conftest.py
│   ├── test_decorators.py
│   ├── test_settings.py
│   └── test_logging.py
├── .python-version         # managed by uv
├── CHANGELOG.md
├── LICENSE
├── README.md
├── mkdocs.yml
└── pyproject.toml
```

`src/` layout is used to prevent accidental imports of the local package during testing (standard best practice with `uv` + `pytest`).

---

## Dependency Management — `uv`

```toml
# pyproject.toml
[project]
name = "iolaus"
version = "0.1.0"
description = "Decorator-based CLI + config + logging framework for research projects."
readme = "README.md"
requires-python = ">=3.11"
license = { text = "MIT" }
dependencies = [
    "dynaconf>=3.2",
    "typer>=0.12",
]

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "pytest-cov",
    "ruff",
    "mypy",
    "pre-commit",
]
docs = [
    "mkdocs-material>=9.0",
    "mkdocstrings[python]",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/iolaus"]

[tool.ruff.lint]
select = ["E", "F", "I", "UP"]

[tool.mypy]
strict = true
```

```bash
# Bootstrap the project
uv init iolaus
uv add dynaconf typer
uv add --dev pytest pytest-cov ruff mypy pre-commit
uv add --group docs mkdocs-material mkdocstrings

# Run tests
uv run pytest

# Run linter
uv run ruff check src/
```

---

## Testing — `pytest`

```python
# tests/conftest.py
import pytest
from pathlib import Path
from dynaconf import Dynaconf
import typer

@pytest.fixture
def base_settings(tmp_path):
    cfg = tmp_path / "settings.toml"
    cfg.write_text('[default]\nmodel__lr = 0.001\ndb__host = "localhost"\n')
    return Dynaconf(settings_files=[str(cfg)], envvar_prefix="TEST")

@pytest.fixture
def app():
    return typer.Typer()
```

```python
# tests/test_decorators.py
from pathlib import Path
from typer.testing import CliRunner
from iolaus.decorators import command

runner = CliRunner()

def test_command_runs_without_extra_config(app, base_settings, tmp_path):
    cmd = command(app, base_settings, output_dir=tmp_path)
    captured = {}

    @cmd
    def train(epochs: int = 10, settings=None, run_dir: Path = None):
        captured["settings"] = settings
        captured["run_dir"] = run_dir

    result = runner.invoke(app, ["train", "--epochs", "5"])
    assert result.exit_code == 0
    assert captured["settings"] is not None
    assert captured["run_dir"].exists()


def test_set_override_applies(app, base_settings, tmp_path):
    cmd = command(app, base_settings, output_dir=tmp_path)
    captured = {}

    @cmd
    def train(settings=None):
        captured["settings"] = settings

    runner.invoke(app, ["train", "--set", "model__lr=0.1"])
    assert float(captured["settings"].model.lr) == pytest.approx(0.1)


def test_extra_config_merges(app, base_settings, tmp_path):
    extra = tmp_path / "extra.toml"
    extra.write_text('[default]\ndb__host = "remote"\n')
    cmd = command(app, base_settings, output_dir=tmp_path)
    captured = {}

    @cmd
    def run(settings=None):
        captured["settings"] = settings

    runner.invoke(app, ["run", "--config", str(extra)])
    assert captured["settings"].db.host == "remote"


def test_original_function_is_callable_directly(app, base_settings, tmp_path):
    """The unwrapped function should be importable and callable without Typer."""
    cmd = command(app, base_settings, output_dir=tmp_path)

    @cmd
    def process(x: int = 1):
        return x * 2

    # Direct call — no Typer involved
    assert process(x=3) == 6


def test_run_artifacts_created(app, base_settings, tmp_path):
    cmd = command(app, base_settings, output_dir=tmp_path)
    captured = {}

    @cmd
    def analyze(run_dir: Path = None):
        captured["run_dir"] = run_dir

    runner.invoke(app, ["analyze"])
    run_dir = captured["run_dir"]
    assert (run_dir / "run.log").exists()
    assert (run_dir / "config.json").exists()
```

---

## GitHub Actions

### CI — run on every push and PR

```yaml
# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: ["main"]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.11", "3.12", "3.13"]
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: uv sync --all-extras
      - run: uv run ruff check src/ tests/
      - run: uv run mypy src/
      - run: uv run pytest --cov=iolaus --cov-report=xml
      - uses: codecov/codecov-action@v4
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
```

### Publish — triggered by merge to main

Version bumping uses `bump-my-version`, which updates `pyproject.toml`, commits, and tags. The tag push then triggers PyPI publishing via Trusted Publishing (no stored API token needed).

```yaml
# .github/workflows/publish.yml
name: Publish

on:
  push:
    branches: ["main"]

permissions:
  contents: write
  id-token: write  # required for PyPI Trusted Publishing

jobs:
  bump-and-publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          token: ${{ secrets.GITHUB_TOKEN }}

      - uses: astral-sh/setup-uv@v5

      - name: Configure git
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"

      - name: Bump patch version
        run: |
          uv add --dev bump-my-version
          uv run bump-my-version bump patch --commit --tag

      - name: Push commit and tag
        run: git push --follow-tags

      - name: Build
        run: uv build

      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
```

> **Setup required:** In the PyPI project settings, add a Trusted Publisher for this repository (publisher: GitHub Actions, workflow: `publish.yml`). No `PYPI_TOKEN` secret needed.

Add `bump-my-version` config to `pyproject.toml`:

```toml
[tool.bumpversion]
current_version = "0.1.0"
commit = true
tag = true

[[tool.bumpversion.files]]
filename = "pyproject.toml"
search = 'version = "{current_version}"'
replace = 'version = "{new_version}"'
```

---

## Documentation — MkDocs + Material

```yaml
# mkdocs.yml
site_name: Iolaus
site_description: Decorator-based CLI, config, and logging framework for research projects.
repo_url: https://github.com/your-org/iolaus
repo_name: iolaus

theme:
  name: material
  palette:
    - scheme: default
      primary: indigo
      toggle:
        icon: material/brightness-7
        name: Dark mode
    - scheme: slate
      primary: indigo
      toggle:
        icon: material/brightness-4
        name: Light mode
  features:
    - navigation.tabs
    - navigation.sections
    - content.code.copy

plugins:
  - search
  - mkdocstrings:
      handlers:
        python:
          paths: [src]
          options:
            docstring_style: google
            show_source: true

nav:
  - Home: index.md
  - Getting Started: getting-started.md
  - Configuration: configuration.md
  - CLI: cli.md
  - API Reference: api-reference.md

markdown_extensions:
  - pymdownx.highlight
  - pymdownx.superfences
  - admonition
  - pymdownx.details
```

Serve locally:

```bash
uv run mkdocs serve
```

Deploy to GitHub Pages (add to `ci.yml` or a separate workflow):

```yaml
- name: Deploy docs
  if: github.ref == 'refs/heads/main'
  run: uv run mkdocs gh-deploy --force
```

---

## Pre-commit Hooks

```yaml
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.4.0
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-toml
      - id: check-yaml
```

```bash
uv run pre-commit install
```

---

## Open Questions / Future Work

- [ ] Should `output_dir` be configurable via Dynaconf itself (e.g. `iolaus.output_dir` key) rather than only at decorator-factory time?
- [ ] Support `--multirun` style sweeps (comma-separated values like Hydra): `--set model__lr=0.001,0.01,0.1`
- [ ] Add a `@cmd.group` helper for nested Typer subcommands
- [ ] Async command support (`async def` functions with Typer)
- [ ] Optional structured logging (JSON log lines) as an alternative to plain text
- [ ] Plugin system for custom config loaders (e.g. load from S3, Vault)
- [ ] `iolaus replay <run_dir>` CLI command to re-run from a saved config snapshot


Navigation Menu

Initial Project Design & Architecture #1

Iolaus — Initial Project Design & Architecture

Overview

Motivating Example

Technical Design

Decorator architecture

Key design decisions

Repository Structure

Dependency Management — `uv`

Testing — `pytest`

GitHub Actions

CI — run on every push and PR

Publish — triggered by merge to main

Documentation — MkDocs + Material

Pre-commit Hooks

Open Questions / Future Work

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Decision	Choice	Rationale
Config override separator	`__` (double underscore)	Dynaconf native; no ambiguity with dotted key names
`settings` / `run_dir` injection	Opt-in via signature declaration	Keeps simple commands simple; no forced parameters
Decorator return value	Unwrapped `func`	Allows direct import and unit testing without Typer
`basicConfig(force=True)`	Yes	Ensures each command invocation gets its own log file handlers
Config snapshot format	JSON	Human-readable, diff-friendly, no extra dependency

Initial Project Design & Architecture #1

Description

Iolaus — Initial Project Design & Architecture

Overview

Motivating Example

Technical Design

Decorator architecture

Key design decisions

Repository Structure

Dependency Management — uv

Testing — pytest

GitHub Actions

CI — run on every push and PR

Publish — triggered by merge to main

Documentation — MkDocs + Material

Pre-commit Hooks

Open Questions / Future Work

Activity

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Dependency Management — `uv`

Testing — `pytest`