Restructure the harness: layers, one loop, a session tree, hooks, compaction

CHANGELOG.md

@@ -1,5 +1,44 @@
 # Changelog
 
+### Unreleased
+
+Structural refactor, taking its shape from the pi agent harness. Every
+previous import path still works, and resolves to the same object rather than
+a copy.
+
+- **Layers.** The package was one flat namespace where the loop constructed a
+  `SessionCache` and called `format_tool_output`. It is now `llm` -> `core` ->
+  `data` -> `app`, each importing only the layers below it, enforced
+  statically by `tests/test_layers.py`. The loop takes a `RunEnvironment`
+  instead of a cache, so `data_harness.core` runs an agent with no pandas
+  anywhere.
+- **One loop.** Four near-copies (sync/async x result/stream) collapsed into
+  one generator driven by two drivers. They had drifted: the streaming copy
+  discarded token usage on provider errors, and `AsyncAgent` was missing
+  `from_dataframe`, subagents, MCP, the replay cache, and the approval gate.
+  The sync driver runs inline, so Ctrl-C lands promptly and a handler holding
+  a `sqlite3` connection still works.
+- **Session tree.** A session is an append-only tree of typed entries and the
+  conversation is *derived* from it, never stored. Brings resume across
+  processes, forking a conversation without losing the original, reversible
+  compaction, and linear rather than quadratic writes.
+- **Hooks.** `BeforeTurn`, `BeforeToolCall`, `AfterToolCall`, `AfterTurn`,
+  returning `Reminder`, `Block`, `Replace`, or `Stop`. The interpreter
+  approval gate is now built from this mechanism rather than hardcoded inside
+  the dispatcher. `AfterTurn` is where a spend cap belongs.
+- **Compaction.** `max_turns` was a wall, not a strategy. Cuts land on turn
+  boundaries so a tool call is never separated from its result, and nothing is
+  deleted: the summary is an entry, so stepping the leaf back undoes it.
+- **Typed errors.** Every failure carries a stable `code` under a common
+  `DataHarnessError`, so a caller can tell a rate-limited provider from a bug
+  in the model's code from a sandbox timeout. Existing base classes are kept,
+  so code catching `RuntimeError`/`ValueError` still works.
+- `RunResult` gains `stopped_by`; `Agent` gains `last_result` and `hooks`;
+  `resolve_async_adapter` mirrors `resolve_adapter`.
+
+913 tests, up from 547.
+
+
 ### 0.13.0
 - **MCP bridge** (`[mcp]` extra): data-harness is now an MCP client. `Agent.add_mcp_server(name, command, args=...)` connects any stdio [MCP](https://modelcontextprotocol.io) server and exposes its tools as a connector — hidden until `load_connectors`, with large results routed through the `SessionCache`. Swap the command for a Postgres/SQLite/filesystem server; no per-source code.
 - `MCPClient` / `MCPServer` / `mcp_tool_specs` exported for wiring MCP into a `Harness` directly; `Agent.close()` shuts servers down. `examples/mcp_demo.py` shows it live against `mcp-server-time`.

data_harness/__init__.py

@@ -1,23 +1,79 @@
-from data_harness.agent import Agent, AgentSession, AsyncAgent, AsyncAgentSession
-from data_harness.artifacts import ChartArtifact
-from data_harness.exceptions import (
+"""data-harness: a constrained agent harness for data analysis.
+
+The package is layered, bottom up:
+
+- `data_harness.llm` — provider adapters and the wire types they speak
+- `data_harness.core` — the ReAct loop, `RunResult`, run logging
+- `data_harness.data` — the session cache, interpreter, SQL, connectors
+- `data_harness.app` — `Agent`, `ask`, the CLI
+
+Each layer may import the ones below it and no others; `tests/test_layers.py`
+enforces that. The names below are the stable public surface and are unaffected
+by where a class physically lives.
+"""
+
+from data_harness._legacy_paths import install as _install_legacy_paths
+
+_install_legacy_paths()
+
+from data_harness.app.agent import (  # noqa: E402
+    Agent,
+    AgentSession,
+    AsyncAgent,
+    AsyncAgentSession,
+)
+from data_harness.app.quickstart import (  # noqa: E402
+    Chat,
+    SmartFrame,
+    ask,
+    resolve_adapter,
+    resolve_async_adapter,
+)
+from data_harness.core.artifacts import ChartArtifact  # noqa: E402
+from data_harness.core.compaction import (  # noqa: E402
+    CompactionSettings,
+    estimate_tokens,
+    make_compactor,
+)
+from data_harness.core.exceptions import (  # noqa: E402
+    ConfigurationError,
+    DataHarnessError,
+    ExecutionError,
     MaxTurnsExceeded,
+    ProviderError,
     SubagentRecursionError,
     ToolNotFoundError,
 )
-from data_harness.exec_cache import ExecutionCache
-from data_harness.io import load_dataframe
-from data_harness.loop import AsyncHarness
-from data_harness.mcp import MCPClient, MCPServer, mcp_tool_specs
-from data_harness.providers.base import (
+from data_harness.core.hooks import (  # noqa: E402
+    AfterToolCall,
+    AfterTurn,
+    BeforeToolCall,
+    BeforeTurn,
+    Block,
+    HookError,
+    HookRegistry,
+    Reminder,
+    Replace,
+    Stop,
+)
+from data_harness.core.result import CacheStorageInfo, RunResult, Usage  # noqa: E402
+from data_harness.core.session import (  # noqa: E402
+    JsonlSessionStore,
+    MemorySessionStore,
+    Session,
+    SessionStoreError,
+)
+from data_harness.data.exec_cache import ExecutionCache  # noqa: E402
+from data_harness.data.harness import AsyncHarness  # noqa: E402
+from data_harness.data.io import load_dataframe  # noqa: E402
+from data_harness.data.mcp import MCPClient, MCPServer, mcp_tool_specs  # noqa: E402
+from data_harness.llm.providers.base import (  # noqa: E402
     AsyncProviderAdapter,
     NormalizedResponse,
     ProviderAdapter,
     StopReason,
 )
-from data_harness.quickstart import Chat, SmartFrame, ask, resolve_adapter
-from data_harness.result import CacheStorageInfo, RunResult, Usage
-from data_harness.streaming import (
+from data_harness.llm.streaming import (  # noqa: E402
     ContentBlockDeltaEvent,
     ContentBlockStartEvent,
     ContentBlockStopEvent,
@@ -30,7 +86,7 @@
     TextDelta,
     ToolResultEvent,
 )
-from data_harness.types import (
+from data_harness.llm.types import (  # noqa: E402
     ContentBlock,
     Message,
     TextBlock,
@@ -41,6 +97,25 @@
 )
 
 __all__ = [
+    "Stop",
+    "SessionStoreError",
+    "Session",
+    "Replace",
+    "Reminder",
+    "ProviderError",
+    "MemorySessionStore",
+    "JsonlSessionStore",
+    "HookRegistry",
+    "HookError",
+    "ExecutionError",
+    "DataHarnessError",
+    "ConfigurationError",
+    "CompactionSettings",
+    "Block",
+    "BeforeTurn",
+    "BeforeToolCall",
+    "AfterTurn",
+    "AfterToolCall",
     "Agent",
     "AgentSession",
     "AsyncAgent",
@@ -81,7 +156,10 @@
     "ToolUseBlock",
     "Usage",
     "ask",
+    "estimate_tokens",
+    "make_compactor",
     "load_dataframe",
     "mcp_tool_specs",
     "resolve_adapter",
+    "resolve_async_adapter",
 ]

data_harness/_legacy_paths.py

@@ -0,0 +1,112 @@
+"""Keep the pre-layering module paths importable.
+
+Modules moved into `llm`, `core`, `data`, and `app` when the package grew
+layers. `data_harness.loop` and friends were documented and in use, so they
+stay resolvable.
+
+These are aliases, not copies: `sys.modules["data_harness.loop"]` *is*
+`data_harness.core.loop`, so ``isinstance`` checks and identity comparisons
+across the two paths agree. Re-exporting names into a shim module would give
+the same names but a second module object, and a class imported through one
+path would not be the class imported through the other.
+
+New code should use the layered path. The alias exists so a rename does not
+become an upgrade barrier.
+"""
+
+from __future__ import annotations
+
+import importlib
+import importlib.abc
+import importlib.machinery
+import importlib.util
+import sys
+
+#: legacy dotted path -> the module that now implements it
+LEGACY_PATHS: dict[str, str] = {
+    "data_harness.types": "data_harness.llm.types",
+    "data_harness.streaming": "data_harness.llm.streaming",
+    "data_harness.testing": "data_harness.llm.testing",
+    "data_harness.providers": "data_harness.llm.providers",
+    "data_harness.providers.base": "data_harness.llm.providers.base",
+    "data_harness.providers.openai": "data_harness.llm.providers.openai",
+    "data_harness.providers.anthropic": "data_harness.llm.providers.anthropic",
+    "data_harness.loop": "data_harness.data.harness",
+    "data_harness.result": "data_harness.core.result",
+    "data_harness.artifacts": "data_harness.core.artifacts",
+    "data_harness.exceptions": "data_harness.core.exceptions",
+    "data_harness.observe": "data_harness.core.observe",
+    "data_harness.logger": "data_harness.core.logger",
+    "data_harness.schema": "data_harness.core.schema",
+    "data_harness.serialize": "data_harness.core.serialize",
+    "data_harness.cache": "data_harness.data.cache",
+    "data_harness.format": "data_harness.data.format",
+    "data_harness.io": "data_harness.data.io",
+    "data_harness.exec_cache": "data_harness.data.exec_cache",
+    "data_harness.mcp": "data_harness.data.mcp",
+    "data_harness.tools": "data_harness.data.tools",
+    "data_harness.tools.interpreter": "data_harness.data.tools.interpreter",
+    "data_harness.tools.sandbox": "data_harness.data.tools.sandbox",
+    "data_harness.tools.sql": "data_harness.data.tools.sql",
+    "data_harness.tools.variables": "data_harness.data.tools.variables",
+    "data_harness.tools.connectors": "data_harness.data.tools.connectors",
+    "data_harness.tools.planner": "data_harness.data.tools.planner",
+    "data_harness.tools.subagent": "data_harness.data.tools.subagent",
+    "data_harness.agent": "data_harness.app.agent",
+    "data_harness.quickstart": "data_harness.app.quickstart",
+    "data_harness.cli": "data_harness.app.cli",
+    "data_harness.notebook": "data_harness.app.notebook",
+    "data_harness.pandas": "data_harness.app.pandas",
+}
+
+
+class _AliasLoader(importlib.abc.Loader):
+    """Bind a legacy name to an already-imported module.
+
+    `create_module` hands back the real module and `exec_module` does nothing,
+    because the source has already run under its own name. Returning the
+    target's own spec instead would make the import machinery build a second
+    module object and execute the file again: the two names would then hold
+    separate classes, and `isinstance` across them would quietly be false.
+    """
+
+    def __init__(self, target: str) -> None:
+        self._target = target
+
+    def create_module(self, spec: importlib.machinery.ModuleSpec):
+        return importlib.import_module(self._target)
+
+    def exec_module(self, module: object) -> None:
+        return None
+
+
+class _LegacyPathFinder(importlib.abc.MetaPathFinder):
+    """Resolve a legacy path to the module that replaced it, on first import.
+
+    A meta-path finder rather than eager aliasing: importing every module at
+    package import time would drag the whole library, and every optional
+    dependency sitting behind a lazy import, into memory just to register
+    names nobody may use.
+    """
+
+    def find_spec(
+        self, fullname: str, path: object = None, target: object = None
+    ) -> importlib.machinery.ModuleSpec | None:
+        real = LEGACY_PATHS.get(fullname)
+        if real is None:
+            return None
+        return importlib.util.spec_from_loader(fullname, _AliasLoader(real))
+
+
+def install() -> None:
+    """Make the legacy paths importable. Idempotent.
+
+    Inserted at the front of `sys.meta_path`, not appended. An aliased package
+    such as `data_harness.providers` resolves to a real package whose
+    ``__path__`` still contains `base.py`, so the default `PathFinder` can
+    resolve `data_harness.providers.base` by itself and would load a second
+    copy of it. Going first means the alias wins for the names it owns; every
+    other name falls through on a dict miss.
+    """
+    if not any(isinstance(f, _LegacyPathFinder) for f in sys.meta_path):
+        sys.meta_path.insert(0, _LegacyPathFinder())

data_harness/app/__init__.py

@@ -0,0 +1,8 @@
+"""Facades: the API most users touch.
+
+`Agent`/`AsyncAgent`, the zero-config `ask`/`Chat`/`SmartFrame` entry points,
+the CLI, and notebook helpers. Composition only; no behaviour that belongs to
+a lower layer.
+
+May import everything below it.
+"""