RADI: LLMs as World Models for Robotic Action Decomposition and Imagination

A neuro-symbolic framework for robot task planning that decouples action proposal from commitment. RADI uses an LLM for symbolic planning and a GNN-based World Model for internal verification—eliminating the need for external simulators at inference time.

Architecture Overview

Task τ → [LLM Planner] → action proposal aₜ
                                    ↓
                          [GNN Critic] → feasible?
                           ↓ no              ↓ yes
                    write to Memory    [GNN Simulator] → predicted G̃ₜ₊₁
                    + replan                    ↓
                                      [SymCheck] → subgoal met?
                                       ↓ no           ↓ yes
                                    discard +        COMMIT
                                    replan           + execute

Key Components

Module	Description	Location
Scene Graph Builder	Tri-layer state representation (State/Event/Memory graphs)	src/core/, src/memory/
LLM Planner	Two-stage prompt: subgoal decomposition → action grounding	src/planning/
GNN World Model	2-layer GAT with Critic + Simulator + SymCheck	src/world_model/
Evolutionary Memory	Selective write policy with minimal witness extraction	src/memory/, src/correction/
Task Generator	Adversarial benchmark generation with hardness metric H(τ)	src/task/
Perception	ConceptGraphs integration for open-vocabulary scene understanding	src/perception/

Installation

Additional dependencies:

• PyTorch Geometric: follow official install guide for your CUDA version.
• AI2-THOR / ProcTHOR: pip install ai2thor (requires a display or Xvfb for headless rendering).

Project Structure

RADI/
├── src/
│   ├── core/                  # Graph schema, relations, data structures
│   │   └── graph_schema.py
│   ├── world_model/           # GNN World Model (Layer 5)
│   │   ├── gnn_model.py       # NeuroSymbolicWorldModel (2-layer GAT)
│   │   ├── verifier.py        # ActionVerifier (Critic → Simulator → SymCheck)
│   │   └── rule_dynamics.py   # Rule-based dynamics engine (fallback)
│   ├── planning/              # LLM Planner (Layer 4)
│   │   ├── prompt_builder.py  # Two-stage prompt construction
│   │   ├── decomposer.py      # Subgoal decomposition
│   │   └── llm_interface.py   # LLM API wrapper
│   ├── perception/            # Perception Pipeline (Layer 1-2)
│   │   ├── oracle_interface.py
│   │   ├── vlm_interface.py
│   │   └── spatial_lifter.py
│   ├── memory/                # Evolutionary Memory (Layer 5 - Module IV)
│   │   └── graph_manager.py
│   ├── env/                   # ProcTHOR Environment Wrapper
│   │   ├── procthor_wrapper.py
│   │   ├── action_adapter.py
│   │   └── symbolic_executor.py
│   ├── task/                  # Adversarial Task Generator (Layer 3)
│   │   └── task_generator.py
│   ├── correction/            # Error parsing & RAG
│   │   ├── error_parser.py
│   │   └── rag_engine.py
│   └── utils/
│       ├── semantic_matcher.py
│       └── visualizer.py
├── scripts/
│   ├── data_collection.py     # Collect transition tuples from ProcTHOR
│   └── train_gnn.py           # Train the GNN World Model
├── configs/
│   ├── env_config.yaml
│   ├── model_config.yaml
│   └── rag_config.yaml
├── main.py                    # Main pipeline entry point
└── requirements.txt