Skip to content

CLAUDE.md

Latest commit

 

History

History
131 lines (101 loc) · 6.58 KB

CLAUDE.md

File metadata and controls

131 lines (101 loc) · 6.58 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

This is AgentSDK — a hierarchical, decoupled Agentic RL training/inference framework for Ascend NPU, enhanced with an automatic skill optimization system (agent-evolution). The repo has two main layers:

Layer 1: AgentSDK Core (agentic_rl/)

  • base/ — Logging, utilities, weight loaders
  • configs/ — Configuration management (agentic_rl_config, ray_env_config)
  • data_manager/ — Dataset loading and transformation
  • memory/ — Token-counted conversation memory (simple, summary)
  • runner/ — Agent execution engine wrapper + vLLM inference adapter
  • trainer/ — GRPO training via MindSpeed-RL and VERL adapters

Layer 2: Agent Evolution (added from agent-evolution repo)

  • rllm_train/ — Self-contained agent RL training pipeline (HuggingFace backend, runs on Mac/CPU)
  • traj_opt/ — Trajectory capture, segmentation, analysis, and skill-bank patch generation
  • skill-bank/ — Skill management system (base + patch + compile architecture)
  • docs/ — Design documents for training, optimization, and skill-bank architecture

Running Training

AgentSDK Core (Ascend NPU / MindSpeed-RL / VERL)

# Via agentic_rl CLI (see docs/zh/quick_start.md)
agentic_rl train --config configs/GPUexample.yaml

rllm_train (HuggingFace backend)

# Default config (Qwen2.5-0.5B, 64 problems, 2 epochs)
python -m rllm_train.train

# Natural language config (supports Chinese and English)
python -m rllm_train.train "用 qwen-0.5b 训练数学 agent，64 个问题，2 个 epoch"
python -m rllm_train.train "quick test with 16 problems"

# From config file (generated by rllm-config skill)
python -m rllm_train.run_training rllm_train/output/runs/<run_id>/config.json

Training outputs go to rllm_train/output/runs/<run_id>/.

rllm_train Architecture

The training pipeline: train.py → GRPOTrainer → rollout_func → HFAgentExecutionEngine → agent/env loop.

Key modules:

  • train.py — Entry point, builds dataset/model/tokenizer, wires into GRPOTrainer
  • config.py — TrainingConfig dataclass, parse_natural_language() for free-text config
  • rollout.py — make_rllm_rollout_func(), bridge between TRL and rllm-style agent execution
  • hf_engine.py — HFAgentExecutionEngine, manages async parallel trajectories with token masks
  • base.py — Core abstractions: BaseAgent, BaseEnv, Step, Action, Trajectory
  • tool_agent.py — ToolAgent(BaseAgent), manages conversation history and tool calls
  • math_env.py — MathCalcEnv(BaseEnv), calculator environment with binary reward
  • parsers.py — Chat template parsers, convert_messages_to_tokens_and_masks() for GRPO masking
  • logger.py / perf_stats.py — Training logging and performance tracking
  • trajectory_writer.py — Per-step JSONL trajectory files

traj_opt Architecture

The optimization pipeline: hooks/ capture → adapter/ convert → store/ persist → segmenter/ split → analyzer/ extract → optimizer/ patch.

Key modules:

  • hooks/post_tool.py — PostToolUse hook, reads stdin JSON, appends to events.jsonl
  • hooks/on_stop.py — Stop/SubagentStop hook
  • adapter/hooks_adapter.py — Converts Claude Code Hooks JSON → TrajectoryEvent
  • adapter/schema.py — Data models: TrajectoryEvent, Trajectory
  • store/ — JSONL persistence (writer, reader, index)
  • segmenter/ — Skill and free segmenters with registry chain
  • analyzer/ — Base analyzer + report writer
  • optimizer/ — Patch generator + compiler bridge

Dual CLI Architecture

Training and optimization run in two independent Claude Code processes:

  • CLI-1: Executes rllm-xx skills, produces training results
  • CLI-2: Executes traj-xx skills, analyzes trajectories, generates patches
  • Coordination via traj_opt/output/rounds/round_{n}/status.json

Skill Bank

Skills managed via skill-bank/ using base + patch + compile architecture. Do not edit .claude/skills/*/SKILL.md directly.

python skill-bank/compile.py rllm-config              # compile single skill
python skill-bank/compile.py --group rllm              # compile all rllm skills
python skill-bank/compile.py --all                     # compile everything
python skill-bank/compile.py --diff rllm-config        # preview changes
python skill-bank/compile.py --status                  # patch status summary
python skill-bank/compile.py --validate                # validate package registry and path invariants
python skill-bank/compile.py --list-packages           # show stable/experimental/vertical/task/lineage packages
python skill-bank/compile.py --smoke-test              # smoke-test stable/current vertical packages
python skill-bank/compile.py --package vertical --domain <domain> --name <vertical-id> --review  # review promotion classification
python skill-bank/compile.py --package task-package --name <task-id> --run-id <run-id>  # freeze a completed run
python skill-bank/compile.py --package lineage-archive --domain <domain> --name <lineage-id> --round-range 1-3  # archive traj evolution rounds

Skill Package Layer

skill-bank remains the source/build system. The package layer under skill-bank/packages/ is an overlay for release and archive workflows:

  • stable/ — generic base skill package, like a foundation model for skills
  • experimental/ — domain work package copied from stable/vertical and evolved by traj
  • vertical/ — validated domain skill package for future same-domain tasks
  • task-packages/ — reproducible package for one concrete trained agent/task
  • lineage-archive/ — archived traj evolution history for audit/recovery

New sessions should read skill-bank/registry.json first when selecting a skill package. For new training tasks, prefer matching vertical.current, otherwise stable.current. Do not use task-packages/ or lineage-archive/ unless the user asks to reproduce or trace historical work.

During package-layer work, do not move these compatibility paths:

skill-bank/compile.py
skill-bank/bank.yaml
skill-bank/rllm/
skill-bank/traj/
skill-bank/compiled/
.claude/skills/
traj_opt/output/
rllm_train/output/

Key Design Decisions

  • Response mask system (1=model tokens, 0=env tokens) is central to GRPO training — only model-generated tokens get gradients
  • Rollout function handles asyncio event loop edge cases for TRL compatibility
  • traj_opt Python code provides infrastructure only; analysis strategies live in SKILL.md files
  • AgentSDK core targets Ascend NPU; rllm_train targets HuggingFace (CPU/MPS/GPU)