Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/skills/develop
1 change: 0 additions & 1 deletion .agents/skills/ndif-develop

This file was deleted.

1 change: 0 additions & 1 deletion .agents/skills/ndif-operate

This file was deleted.

1 change: 0 additions & 1 deletion .agents/skills/ndif-selfhost

This file was deleted.

1 change: 0 additions & 1 deletion .agents/skills/ndif-troubleshoot

This file was deleted.

1 change: 1 addition & 0 deletions .agents/skills/operate
1 change: 1 addition & 0 deletions .agents/skills/selfhost
1 change: 1 addition & 0 deletions .agents/skills/troubleshoot
1 change: 1 addition & 0 deletions .codex/skills/develop
1 change: 0 additions & 1 deletion .codex/skills/ndif-develop

This file was deleted.

1 change: 0 additions & 1 deletion .codex/skills/ndif-operate

This file was deleted.

1 change: 0 additions & 1 deletion .codex/skills/ndif-selfhost

This file was deleted.

1 change: 0 additions & 1 deletion .codex/skills/ndif-troubleshoot

This file was deleted.

1 change: 1 addition & 0 deletions .codex/skills/operate
1 change: 1 addition & 0 deletions .codex/skills/selfhost
1 change: 1 addition & 0 deletions .codex/skills/troubleshoot
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,10 +113,10 @@ you want.)

| Skill | Use when... |
| --- | --- |
| `ndif-selfhost` | Standing a server up: the published `ndif/ndif` image, the compose dev stack, or a from-source `ndif start`. Prerequisites, tags, ports, volumes, configuration, and the first remote trace. |
| `ndif-operate` | Running models on it: deploy, evict, pin, scale, `models.yaml`, sizing and padding, HOT/WARM/COLD, the dashboard, telemetry, turning on auth. |
| `ndif-troubleshoot` | It won't start, requests hang, a deploy OOMs, a result won't download, versions disagree β€” symptom to cause to fix, and where the logs actually are. |
| `ndif-develop` | Changing the server itself: the request lifecycle, the process map, trusted vs untrusted execution, the model-actor hooks, the test suite, release mechanics. |
| `selfhost` | Standing a server up: the published `ndif/ndif` image, the compose dev stack, or a from-source `ndif start`. Prerequisites, tags, ports, volumes, configuration, and the first remote trace. |
| `operate` | Running models on it: deploy, evict, pin, scale, `models.yaml`, sizing and padding, HOT/WARM/COLD, the dashboard, telemetry, turning on auth. |
| `troubleshoot` | It won't start, requests hang, a deploy OOMs, a result won't download, versions disagree β€” symptom to cause to fix, and where the logs actually are. |
| `develop` | Changing the server itself: the request lifecycle, the process map, trusted vs untrusted execution, the model-actor hooks, the test suite, release mechanics. |

These skills document a server, so β€” unlike the nnsight ones β€” their code blocks
are reference material rather than executed examples.
Expand Down Expand Up @@ -212,10 +212,10 @@ skills/
β”‚ └── ndif/
β”‚ β”œβ”€β”€ .claude-plugin/plugin.json
β”‚ └── skills/
β”‚ β”œβ”€β”€ ndif-selfhost/
β”‚ β”œβ”€β”€ ndif-operate/
β”‚ β”œβ”€β”€ ndif-troubleshoot/
β”‚ └── ndif-develop/
β”‚ β”œβ”€β”€ selfhost/
β”‚ β”œβ”€β”€ operate/
β”‚ β”œβ”€β”€ troubleshoot/
β”‚ └── develop/
β”œβ”€β”€ tests/ # executes every code block
└── Makefile
```
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: ndif-develop
description: Change NDIF's own server code. Use when working inside the ndif repo β€” "how does an NDIF request actually run", "where do I add an endpoint / a model actor / a provider / a CLI command", "what is the dispatcher", "trusted vs untrusted execution", "the sandbox runner", "how do I test NDIF", "how do I force the untrusted path", "how is nnsight pinned", "how do I cut an NDIF release" β€” or when reading a traceback from NDIF's internals rather than from a user's block. Covers the repo layout, the process map, the request lifecycle, the five model-actor hooks, the live-server test suite (CI runs none of it), and the tag-driven publish workflows. For running a server rather than changing one, use `ndif-selfhost`, `ndif-operate` and `ndif-troubleshoot`.
name: develop
description: Change NDIF's own server code. Use when working inside the ndif repo β€” "how does an NDIF request actually run", "where do I add an endpoint / a model actor / a provider / a CLI command", "what is the dispatcher", "trusted vs untrusted execution", "the sandbox runner", "how do I test NDIF", "how do I force the untrusted path", "how is nnsight pinned", "how do I cut an NDIF release" β€” or when reading a traceback from NDIF's internals rather than from a user's block. Covers the repo layout, the process map, the request lifecycle, the five model-actor hooks, the live-server test suite (CI runs none of it), and the tag-driven publish workflows. For running a server rather than changing one, use `selfhost`, `operate` and `troubleshoot`.
---

# Developing NDIF
Expand Down Expand Up @@ -329,6 +329,6 @@ that is the most common drift in the repo.

## Related skills

- `ndif-selfhost` β€” getting a dev stack up.
- `ndif-troubleshoot` β€” reading the logs a change produces.
- `ndif-operate` β€” the operator surface your change has to keep working.
- `selfhost` β€” getting a dev stack up.
- `troubleshoot` β€” reading the logs a change produces.
- `operate` β€” the operator surface your change has to keep working.
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: ndif-operate
description: Run models on an NDIF you host β€” deploy, evict, pin, scale, restart, inspect the queue, kill a request, export a models.yaml, and size a model so it actually fits. Use for "deploy a model on NDIF", "ndif deploy/evict/scale/status/queue/kill", "pin a model", "models.yaml", "NDIF_DEPLOYMENTS", "how much GPU will this model take", "padding factor", "HOT/WARM/COLD", "NDIF dashboard", "schedule a model", "add a GPU node", "turn on NDIF auth", "API keys", "NDIF in production", or NDIF Grafana/Loki/InfluxDB telemetry. Assumes a server is already up β€” standing one up is `ndif-selfhost`.
name: operate
description: Run models on an NDIF you host β€” deploy, evict, pin, scale, restart, inspect the queue, kill a request, export a models.yaml, and size a model so it actually fits. Use for "deploy a model on NDIF", "ndif deploy/evict/scale/status/queue/kill", "pin a model", "models.yaml", "NDIF_DEPLOYMENTS", "how much GPU will this model take", "padding factor", "HOT/WARM/COLD", "NDIF dashboard", "schedule a model", "add a GPU node", "turn on NDIF auth", "API keys", "NDIF in production", or NDIF Grafana/Loki/InfluxDB telemetry. Assumes a server is already up β€” standing one up is `selfhost`.
---

# Operating an NDIF
Expand Down Expand Up @@ -317,7 +317,7 @@ hardened jail. Terminate TLS yourself and never expose 8001 directly.

## Related skills

- `ndif-selfhost` β€” getting a server up in the first place.
- `ndif-troubleshoot` β€” when a deploy fails or a request hangs.
- `ndif-develop` β€” changing placement, the queue, or the actor.
- `selfhost` β€” getting a server up in the first place.
- `troubleshoot` β€” when a deploy fails or a request hangs.
- `develop` β€” changing placement, the queue, or the actor.
- nnsight plugin `remote` β€” the client side of what these deployments serve.
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ itself defaults to `trusted=True`. Follow that flag:
An explicit `trusted` in the envelope is honored either way, which is the one
escape hatch: a client can send `trusted: false` to force the sandbox path with
no Postgres at all. That is how you exercise the untrusted path in dev β€” see the
`ndif-develop` skill.
`develop` skill.

## Turning it on

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
name: ndif-selfhost
name: selfhost
description: Stand up your own NDIF server β€” the backend that nnsight's remote=True talks to β€” on your own GPUs. Use for "run my own NDIF", "self-host NDIF", "docker run ndif/ndif", "local NDIF server", "point nnsight at my server", "ndif start", "install NDIF from source", "NDIF docker compose", or any question about NDIF prerequisites, ports, volumes, tags, or configuration. Covers all three routes (the published ndif/ndif image, the compose dev stack from a checkout, and a from-source install with the ndif CLI), how to prove it works, and the one default that matters β€” an NDIF with no Postgres runs every caller's Python inside the model process. NOT for using the public ndif.us service; that is the nnsight plugin's `remote` skill.
---

Expand Down Expand Up @@ -28,7 +28,7 @@ that. A trusted request:
That is the right default for an NDIF you run for yourself, and the wrong one the
moment a second person can reach port 8001. Before anyone else can:
`docs/runbooks/enable-auth.md`, then `docs/operating/production.md`. The
`ndif-operate` skill covers what turning it on changes.
`operate` skill covers what turning it on changes.

## Pick a route

Expand Down Expand Up @@ -80,7 +80,7 @@ change:
other people hold.** `ndif status` will say a 80 GB card is 80 GB free while a
colleague's job holds 25 GB of it, and the placer will put a model there.
Subtract other tenants' `nvidia-smi` usage yourself and force the placement
with `ndif deploy --gpus N` (or `--size-bytes`); see the `ndif-operate` skill's
with `ndif deploy --gpus N` (or `--size-bytes`); see the `operate` skill's
sizing section for the arithmetic.
- **The container runs as root**, so anything it writes into a bind mount β€” the
Ray temp dir, new files in the HF cache β€” is root-owned afterwards. Clean up
Expand Down Expand Up @@ -340,7 +340,7 @@ and exhaustively in `docs/reference/env-vars.md` and `docs/reference/ports.md`.

## Related skills

- `ndif-operate` β€” deploying and sizing models, the dashboard, auth, production.
- `ndif-troubleshoot` β€” when it does not come up, or a request hangs.
- `ndif-develop` β€” changing the server's code.
- `operate` β€” deploying and sizing models, the dashboard, auth, production.
- `troubleshoot` β€” when it does not come up, or a request hangs.
- `develop` β€” changing the server's code.
- nnsight plugin `remote` β€” writing the client-side code that talks to it.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
name: ndif-troubleshoot
name: troubleshoot
description: Diagnose a self-hosted NDIF that is broken β€” container won't start, no GPU, 503 "compute backend is reconnecting", requests that hang in QUEUED or DEPLOYING, CANT_ACCOMMODATE, CUDA out of memory on deploy or inside a block, "COMPLETED but the client can't download", "The model architecture on this server doesn't match", "Your request payload could not be read", "Starting Ray client server failed", code changes with no effect, a blank dashboard, empty Grafana panels. Use when someone pastes an NDIF server log, an nnsight RemoteError from their own deployment, or asks where NDIF's logs are, how to read `ndif queue`, or how to kill or reconstruct a request. For the public ndif.us service use the nnsight plugin's `remote` skill instead.
---

Expand Down Expand Up @@ -187,7 +187,7 @@ to serve a half-CPU model. Read it as "the estimate was too low".
RAM for the WARM cache. For GPU pressure: evict explicitly, deploy the incoming
model `--pinned` (which waives the age check), lower
`NDIF_MINIMUM_DEPLOYMENT_TIME_SECONDS`, or tune the padding knobs. The whole
procedure, including what the ledger cannot see, is in the `ndif-operate` skill
procedure, including what the ledger cannot see, is in the `operate` skill
and `docs/runbooks/model-oom-on-deploy.md`.

## COMPLETED but no result
Expand Down Expand Up @@ -314,7 +314,7 @@ And the label that hides model-actor logs: they are **`service="model"`**, not

## Related skills

- `ndif-selfhost` β€” prerequisites, tags, ports, and the configuration model.
- `ndif-operate` β€” deploys, sizing, eviction, auth.
- `ndif-develop` β€” when the fix is a code change.
- `selfhost` β€” prerequisites, tags, ports, and the configuration model.
- `operate` β€” deploys, sizing, eviction, auth.
- `develop` β€” when the fix is a code change.
- nnsight plugin `debugging` β€” when the failure is in the user's own trace.
Loading
Loading