Inference Lab

Local-first AI inference platform for Ollama-powered applications.

Inference Lab is Milestone 0 (M0) of a larger local-first AI engineering roadmap. It provides the reusable inference layer that powers future projects such as PDF RAG, conversational memory, evaluation pipelines, and autonomous agents.

It ships as a reusable CLI plus a thin HTTP API so other local projects can share the same inference layer instead of connecting to Ollama directly.

Why It Exists

Local AI applications often duplicate the same infrastructure: model requests, streaming, persistence, benchmarking, and telemetry. Inference Lab centralizes that work behind a shared API and CLI so future projects can focus on product logic instead of inference plumbing.

What It Ships

• CLI for completions, benchmarking, reports, and exports
• REST API for inference, health, and metrics
• Docker Compose deployment
• SQLite persistence
• Type-safe Ollama client
• Modular workspace packages

Model Selection

Inference Lab does not download, load, unload, or otherwise manage model lifecycle. Ollama remains responsible for that.

When a request includes model:

1. Inference Lab validates that the model is installed locally.
2. The request is forwarded to that model.
3. If the model is missing, the API returns an error that lists the installed models.

When model is omitted:

1. Inference Lab queries GET /api/ps for a live snapshot of running models.
2. If one or more models are running, the first running model is used automatically.
3. If no models are running and DEFAULT_MODEL is configured, that model is used.
4. If no models are running and DEFAULT_MODEL is not configured, the API returns HTTP 503.

DEFAULT_MODEL is optional. It is useful when you want a stable fallback without forcing a specific model to stay loaded in memory.

Architecture

flowchart LR
  CLI["apps/cli"] --> Client["packages/inference-client"]
  CLI --> Benchmark["packages/benchmark"]
  API["apps/api"] --> Client
  API --> Metrics["packages/metrics"]
  API --> Storage["packages/storage"]
  Benchmark --> Client
  Benchmark --> Metrics
  Benchmark --> Storage
  Storage --> SQLite["SQLite"]
  Client --> Ollama["Ollama"]

Loading

Inference Lab (M0) is the foundational inference layer for every subsequent milestone in this roadmap, including PDF RAG, conversational memory, evaluation pipelines, and autonomous agents. Those applications consume a stable API instead of integrating directly with model runtimes.

The CLI keeps the benchmark workflow intact. The API adds a thin route/controller/service layer on top of the shared client and storage packages. Nothing in the repository talks to Ollama directly except the shared inference client.

The repository is organized as a pnpm workspace with independently reusable packages.

Repository Structure

apps/
  api/              # HTTP API entrypoint
  cli/              # Terminal entrypoint
benchmarks/         # Sample benchmark artifacts and notes
docs/               # Architecture, API, schema, and migration docs
packages/
  benchmark/        # Benchmark orchestration
  inference-client/ # Ollama client and typed request/response models
  metrics/          # Benchmark summaries and formatting helpers
  storage/          # SQLite persistence for benchmarks and inference history
scripts/            # Maintenance helpers

Prerequisites

Install these before you start:

• Node.js 22 or newer
• pnpm v11 or newer
• Ollama 0.32+ or another compatible local Ollama build

Installation

Install the workspace dependencies:

pnpm install

Make sure at least one model is installed locally in Ollama. Inference Lab will never pull a model for you.

ollama pull llama3.1:8b

Then start Ollama:

ollama serve

If you want a fallback model when no model is running, configure DEFAULT_MODEL in your environment:

$env:DEFAULT_MODEL = 'llama3.1:8b'

CLI Quick Start

Single benchmark run:

pnpm benchmark --prompt "Explain KV Cache"

Explicit model:

pnpm benchmark --model llama3.1:8b --prompt "Hello"

Benchmark run:

pnpm benchmark --prompt "Explain KV Cache"

Streaming benchmark:

pnpm benchmark --stream --prompt "Explain RAG"

The CLI remains backward compatible with the previous release.

HTTP API Quick Start

Milestone 1 exposes the shared inference service at http://localhost:4000/v1/generate. The API uses the same local Ollama backend as the CLI, so once Ollama is running and a model is available, you can reach the service from any HTTP client on your machine.

Start the API:

pnpm api

Health check:

curl http://localhost:4000/v1/health

Generate a response without specifying a model:

curl http://localhost:4000/v1/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Explain KV Cache in one paragraph.",
    "stream": false,
    "temperature": 0.7,
    "max_tokens": 512
  }'

Generate a response with an explicit model:

curl http://localhost:4000/v1/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Explain KV Cache in one paragraph.",
    "model": "llama3.1:8b",
    "stream": false,
    "temperature": 0.7,
    "max_tokens": 512
  }'

PowerShell example without model:

$body = @{
  prompt = 'Explain KV Cache in one paragraph.'
  stream = $false
  temperature = 0.7
  max_tokens = 512
} | ConvertTo-Json

Invoke-RestMethod `
  -Uri 'http://localhost:4000/v1/generate' `
  -Method Post `
  -ContentType 'application/json' `
  -Body $body

PowerShell example with model:

$body = @{
  prompt = 'Explain KV Cache in one paragraph.'
  model = 'llama3.1:8b'
  stream = $false
  temperature = 0.7
  max_tokens = 512
} | ConvertTo-Json

Invoke-RestMethod `
  -Uri 'http://localhost:4000/v1/generate' `
  -Method Post `
  -ContentType 'application/json' `
  -Body $body

Stream a response:

curl -N http://localhost:4000/v1/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Explain KV Cache in one paragraph.",
    "model": "llama3.1:8b",
    "stream": true,
    "temperature": 0.7,
    "max_tokens": 512
  }'

Inspect aggregated metrics:

curl http://localhost:4000/v1/metrics

For complete API examples, see docs/http-api.md.

Expected response structure:

{
  "created_at": "2026-07-28T06:28:02.6505566Z",
  "completion_tokens": 64,
  "id": "db9c0406-2ab9-4aa6-aafe-cdebedb97a0b",
  "latency_ms": 2891,
  "model": "llama3.1:8b",
  "prompt_tokens": 19,
  "response": "Generated text from Ollama...",
  "session_id": "dd75b404-ae83-4f03-97cb-0aab5199aa32",
  "tokens_per_second": 22.1,
  "total_tokens": 83
}

Database Schema

inference-lab.sqlite stores both shared benchmark/timing history and the newer API request history.

• benchmark_runs stores benchmark and timing history from both CLI benchmark runs and API-driven requests.
• sessions records inference metadata for each session.
• runs records each persisted request.
• system_metrics stores optional snapshots for future trend analysis.

See docs/database-schema.md for the full schema.

Version History

• v1.0.0 introduced the CLI foundation, Ollama client, benchmarks, and SQLite persistence.
• v1.1.0 adds a REST API, request-level persistence, Docker Compose support, model digests, Git commit tracking, Ollama versioning, and runtime metadata capture.
• v1.1.1 introduces model-agnostic model selection, automatic running-model discovery, explicit model validation, and DEFAULT_MODEL fallback.

See CHANGELOG.md for the canonical release history.

Migration Notes

v1.1.1 does not require manual SQL work on a fresh install. Existing SQLite databases are extended automatically on startup.

Read docs/migration-notes.md before upgrading an existing deployment.

Development Workflow

pnpm format
pnpm lint
pnpm typecheck
pnpm test

If you touch public behavior, update the relevant documentation in the same change.

Troubleshooting

• If pnpm api fails immediately, verify that Node.js 22+ and pnpm are installed and that pnpm install completed successfully.
• If the health check does not return healthy, confirm that ollama serve is running and that at least one model is installed locally.
• If the API returns No model is currently running and DEFAULT_MODEL is not configured., either start a model in Ollama or set DEFAULT_MODEL to an installed model.
• If the API returns Model <name> is not installed., install that model with Ollama or request a model that is already installed.
• If the CLI or API cannot reach Ollama, confirm ollama serve is running and OLLAMA_BASE_URL is correct.
• If the API fails to start on Linux containers, ensure host.docker.internal is available or set OLLAMA_BASE_URL manually.
• If SQLite files do not appear, verify the current working directory or INFERENCE_LAB_DATABASE_FILE.

See docs/troubleshooting.md for more details.

License

Released under the MIT License.