AdaptLoop Architecture

Event-driven continuous evaluation & adapter fine-tuning platform.

System diagram

flowchart TB
  subgraph sources [Signal sources]
    GW[LLMOps Postgres traces]
    SYN[Synthetic EPC producer]
  end

  subgraph streaming [Streaming plane]
    RP[Redpanda llm.signals.v1]
    BX[Bytewax aggregator]
  end

  subgraph state [State]
    Redis[(Redis rollups and lag)]
    PG[(Postgres training_examples)]
    MinIO[(MinIO Parquet and adapters)]
  end

  subgraph trainPlane [Training plane]
    Curate[Dataset builder]
    LoRA[PEFT LoRA job]
    Eval[Holdout eval]
  end

  subgraph promotePlane [Promotion]
    Gate[Lag and eval gates]
    MLF[MLflow registry]
    Canary[Canary watcher]
    Route[LLMOps adapter_routes]
  end

  GW --> RP
  SYN --> RP
  RP --> BX
  BX --> Redis
  BX --> PG
  PG --> Curate
  Curate --> MinIO
  MinIO --> LoRA
  LoRA --> Eval
  Eval --> MinIO
  Eval --> Gate
  Gate --> MLF
  Gate --> Route
  Canary -->|breach| Gate
  Redis --> Gate
  Redis --> Canary
  Route -->|base_model plus adapter_id| GW

Portfolio loop: EPCdash (domain) → AEVAR (knowledge) → LLMOps (gateway) → AdaptLoop (adaptation).

Spine (Phase 1 — implemented)

LLMOps Postgres (metadata) ──► gateway_trace_bridge ──┐
                                                       ├──► Redpanda llm.signals.v1
Synthetic EPC Q&A producer ──► (gated prompt_text) ───┘              │
                                                                     ▼
                                                          Bytewax signal_aggregator
                                                            ├── Redis rollups / dedup
                                                            └── Postgres training_examples

Components

Piece	Role
Event contract	`LLMSignalEvent` on `llm.signals.v1` — see EVENT_CONTRACT.md
Shared normalization	`adaptloop/normalization` — train/serve skew defense
Gateway bridge	Polls LLMOps `requests` + `token_usage`; metadata-only signals
Synthetic producer	EPC terminology Q&A with gated text + occasional `needs_review` poison samples
Bytewax aggregator	Validate → Redis dedup → Postgres append → live aggregates
Promotion API	Lag/eval gates, MLflow stages, gateway route sync, canary rollback

Local ports

Service	Host port
Redpanda Kafka	`19092`
Redpanda Console	`8085`
Postgres	`5433`
Redis	`6380`
MinIO API / Console	`9000` / `9001` (profile `training`)

Failure modes (spine defenses)

Training/serving skew — shared normalize_prompt / hash_prompt / chat template + golden tests.
Feedback poisoning — Pydantic validation; needs_review flag on low-confidence synthetic samples.
Consumer lag / stale promotion — Redis watermarks + processed counters; lag-aware gates in Phase 2.

Curation (Phase 2 — implemented)

Postgres training_examples
  ├── needs_review=true  ──► review_queue.parquet
  ├── no prompt_text     ──► skipped (gateway metadata)
  └── eligible text rows ──► dedup(prompt_hash)
                              ├── train.parquet   (~80%)
                              └── holdout.parquet (~20%, never trains)
                                    │
                                    ▼
                              MinIO adaptloop-datasets/

docker compose --profile training up -d
adaptloop-curate --dataset-id epc_qa_demo --local-dir .artifacts/datasets/latest
./scripts/smoke_curation.sh

Training (Phase 2 — implemented)

train.parquet ──► PEFT LoRA (Qwen2.5-3B or smoke tiny model)
holdout.parquet ─► eval harness (exact match + token F1)
                      │
                      ▼
                 MinIO adaptloop-checkpoints/adapters/...

pip install -e '.[training]'
adaptloop-train \
  --train-uri .artifacts/datasets/latest/train.parquet \
  --holdout-uri .artifacts/datasets/latest/holdout.parquet \
  --smoke
# Worker:
#   arq adaptloop.workers.worker_app.WorkerSettings
#   adaptloop-train ... --enqueue
./scripts/smoke_train.sh

Holdout never enters training. Smoke mode uses hf-internal-testing/tiny-random-LlamaForCausalLM for CPU demos.

Promotion (Phase 2 — implemented)

run_report.json + Redis lag (produced - processed)
        │
        ▼
 lag-aware gate ──► block if lag high or holdout metrics low
        │
        ▼
 MLflow model registry stages: Staging / Production / Archived

docker compose --profile training up -d mlflow
adaptloop-promote smoke_lora --decision approve --target-stage Staging
adaptloop-promote smoke_lora --decision approve --target-stage Production --min-token-f1 0.99  # often blocked for smoke
adaptloop-api   # http://localhost:8090/docs
./scripts/smoke_promotion.sh

MLflow UI: http://localhost:5001

Gateway adapter routing (implemented)

AdaptLoop promotions can upsert LLMOps adapter_routes:

adaptloop-promote → MLflow Staging/Production
                 → PUT /v1/admin/adapter-routes
                      model_alias=epc-qa
                      base_model=gpt-4o-mini
                      adapter_id=s3://.../adapter
LLMOps GatewayService remaps ChatRequest.model → base_model
  and sets X-Adapter-Id / X-Model-Alias / X-Base-Model / X-Adapter-Stage
  Staging.canary_percent sticky-hashes a % of alias traffic to the Staging adapter

# In LLMOps repo:
alembic upgrade head   # includes 0005_adapter_canary_percent
# In AdaptLoop:
export LLMOPS_GATEWAY_URL=http://localhost:8000
export LLMOPS_ADMIN_API_KEY=llmops_dev_default_key
adaptloop-promote smoke_lora --decision approve --target-stage Staging
./scripts/smoke_gateway_route.sh

Canary + auto-rollback (implemented)

After Staging promote, adaptloop-canary watches Redis error_rate / latency_p95. On breach it issues ROLLBACK (MLflow Archived + gateway route disable).

adaptloop-canary smoke_lora --inject-fault   # demo rollback
make canary
make demo                                    # full closed loop

Remaining follow-ons

Custom inference provider that applies LoRA weights server-side (today adapter_id is routed/echoed)
Wire judge API provider key into CI secretly for stronger offline+online judge parity

FilesExpand file tree

ARCHITECTURE.md

Latest commit

History