SnapGraph3D

A constraint-guided 3D scene assembly agent and reliability layer for industrial simulation authoring.

SnapGraph3D makes agent-driven 3D assembly safer and more reproducible. A typed Python scene graph owns authoritative state; deterministic geometry code mates connector frames, applies hierarchy-aware transforms, reads state back, validates explicit constraints, and either repairs a bounded error or rolls back. The same state drives an interactive NiceGUI demo, versioned Unity export, and MCP tools.

What It Demonstrates

• Reliable tool-agent execution through typed actions, postcondition checks, bounded repair, abstention, and rollback.
• Exact 3D transform handling with local/world frames, quaternions, connector mating, and compound-object hierarchy.
• Functional validation beyond visual plausibility, including conveyor continuity, route traversal, support, gaps, and alignment.
• One authoritative scene represented consistently in Python, exported JSON, browser projections, and Unity GameObjects.
• Deterministic comparison of an open-loop baseline against the full closed-loop workflow.

Why This Matters

LLM and tool-using agents can produce plausible sequences of 3D edits while still missing the precision needed by simulation systems. Small height errors leave objects floating or intersecting. Rotation mistakes break connector orientation. Repeated relative moves accumulate drift. Parent/child confusion can deform compound objects or move only part of an assembly.

SnapGraph3D separates high-level intent from exact execution. The agent requests a typed operation, while deterministic transform and constraint code decides whether the resulting state is geometrically and functionally valid. Invalid bounded changes can be repaired locally; unsafe or conflicting changes are rolled back.

Demo Scene

The canonical industrial scene contains:

• a floor plane;
• three connected, thin conveyor belts with four supporting legs each;
• a moving box that traverses the validated conveyor route;
• an inspection station represented by one thin physical platform;
• a sensor-arch proxy and control cabinet mounted as children of the inspection platform.

The inspection station is a compound object: translating or rotating its parent moves the platform, sensor arch, and control box together without moving the separate conveyor passing through the station. Stable object IDs and child-local transforms are preserved across the browser and Unity representations.

Architecture

Typed scene graph
        -> connector-frame solver
        -> guarded scene action
        -> authoritative state readback
        -> constraint and route validation
        -> accept | bounded repair | rollback
        -> NiceGUI views | JSON/audit outputs | Unity bridge

The main layers are:

• Schema and state: Pydantic DTOs for scene objects, transforms, connectors, constraints, and connections.
• Geometry: NumPy/SciPy rigid-transform math and exact connector-frame mating.
• Execution: hierarchy-aware service operations and rollback-safe scripted agent loops.
• Reliability: geometry, hierarchy, connection, and route validators with local repair policies.
• Integration: NiceGUI, MCP tools, deterministic benchmark outputs, and a C# Unity bridge.

More detail is available in architecture.md and coordinate_conventions.md.

Quick Start

Python 3.11 or newer is required. Create and activate an isolated environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e ".[dev,app,mcp]"

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1.

Run the quality checks:

python -m pytest -q
python -m ruff check src tests scripts

Start the interactive web application:

snapgraph3d-app

The local demo is served at http://127.0.0.1:8080 by default. Set SNAPGRAPH3D_PORT to select another port.

Run the deterministic benchmark:

snapgraph3d-benchmark

Export the authoritative canonical scene for Unity:

python scripts/export_unity_scene.py --output outputs/unity/scene_state.json

Run the optional MCP server over standard I/O:

snapgraph3d-mcp --transport stdio

Interactive Workflow

The NiceGUI application exposes top and vertical projections derived directly from SceneState. It supports object selection, bounded translation and yaw controls, connector validation and repair, failure injection, reset/rollback workflows, and moving-part route playback.

The SVG views are regenerated after operations. They do not maintain a separate UI-only geometry model, so what the user sees reflects the same typed state used by validation and export.

Benchmark Results

The latest local summary in outputs/metrics/benchmark_summary.json contains 420 deterministic trials: 210 open-loop baseline trials and 210 closed-loop SnapGraph3D trials.

System	Trials	Scene + route success	Route traversal success	Mean connector position error	Recovery rate
Open-loop baseline	210	4.3%	4.3%	0.3282 m	0.0%
SnapGraph3D closed loop	210	100.0%	100.0%	0.0000476 m	100.0%

These are prototype results on controlled translation and yaw perturbations generated by the repository's deterministic benchmark, not claims about arbitrary industrial scenes. Reproduce them with snapgraph3d-benchmark.

Validation And Recovery

After each meaningful state change, SnapGraph3D evaluates:

• finite, normalized, and internally consistent local/world transforms;
• connector position, height, orientation, width, and type compatibility;
• hierarchy integrity and preservation of child-local transforms;
• support, penetration, forbidden overlap, and required clearance;
• directed conveyor continuity and moving-part route traversal.

The recovery policy escalates by risk:

small bounded residual -> snap to the valid connector transform
local constraint error -> repair the affected neighborhood
large/conflicting error -> restore the previous snapshot and abstain or replan

Machine-readable audit traces record actions, residuals, repairs, rollbacks, and final outcomes.

Injected connector error	Deterministic repair

Unity + MCP Integration

The Python core remains the source of truth for SceneState, transform solving, validation, repair, rollback, and route decisions.

The Unity package under unity/SnapGraph3DBridge consumes the versioned snapgraph3d.unity.v1 JSON contract. It updates GameObjects by stable object ID, preserves the DTO parent-child hierarchy and local quaternion transforms, and attaches base-aligned generated box meshes without introducing hidden visual children.

The MCP server exposes typed tools for scene inspection, object movement and yaw rotation, connector mating and validation, repair, reset, route placement, and closed-loop demo execution. See unity_mcp_integration.md for the contract and tool list.

Current Unity synchronization is export and reload: Python writes a JSON snapshot and the bridge loads it. Automatic live refresh, file watching, and Unity route playback are roadmap items.

Unity setup and the reproducible Editor capture workflow are documented in the visual asset guide.

Repository Layout

src/snapgraph3d/             Typed scene, transforms, solver, service, agent, validation
tests/                       Unit and integration tests
scripts/                     App, benchmark, MCP, and Unity export entry points
unity/SnapGraph3DBridge/     C# Unity package and JSON importer
docs/                        Architecture, conventions, integration, release notes
docs/assets/                 Public README screenshots and GIFs
outputs/unity/               Versioned example Unity scene exports
outputs/screenshots/         Captured web-demo evidence
outputs/videos/              Captured repair and route demonstrations
configs/                     Tolerance and scene configuration

Known Limitations

• sensor_arch is currently a simplified box/mesh proxy rather than a detailed hollow industrial arch.
• The Unity bridge is a visualization and integration layer; it does not independently validate or repair scene state.
• Unity updates currently require JSON export/reload rather than continuous synchronization.
• Conveyor movement is kinematic route playback, not a rigid-body or manufacturing-process simulation.
• This is a focused research prototype, not a production simulator or safety-certified planning system.

Roadmap

• Richer Unity geometry, materials, labels, and connector gizmos.
• Unity-side route animation with play, pause, reset, and progress controls.
• Live Unity refresh through a file watcher or streaming transport.
• Broader collision, clearance, support, and reachability constraints.
• A natural-language planner front end that compiles intent into the existing typed action API.

Project Highlights

Python · Pydantic · NumPy · SciPy · NiceGUI · Unity · C# · MCP · scene graphs · 3D transforms · quaternions · connector constraints · spatial reasoning · validation and repair · rollback-safe agents · deterministic benchmarking · pytest · Ruff

Reproducibility

The prototype uses typed schemas, deterministic seeds, versioned scene exports, explicit tolerances, machine-readable traces, and automated tests. Generated benchmark metrics are intentionally ignored by Git; run the benchmark locally to regenerate them.

License

Licensed under the Apache License 2.0.

Author

Shuang Gao (Rebecca)