borrowed-gpu

LLM server toolkit for opencode, pi, and Hermes Agent, running on borrowed (free) GPUs.

Run an LLM on free GPUs (Google Colab / Kaggle), expose it through an authenticated Cloudflare quick tunnel, and let opencode, pi, or Hermes Agent on your local machine connect automatically — no manual config edits every session.

Architecture

Colab/Kaggle GPU                          Local machine
┌──────────────────────────┐
│ Ollama (model in VRAM)   │              ┌────────────────────┐
│   ▲                      │   publish    │ sync-opencode.py / │
│ Auth proxy (Bearer key)  │──URL+key──▶  │ sync-pi.py /       │
│   ▲                      │  ntfy.sh     │ sync-hermes.py     │
│ cloudflared quick tunnel │◀─────────────│   │ updates        │
│ Keep-alive watchdog      │  HTTPS+key   │ client config      │
└──────────────────────────┘              │   ▲                │
                                          │ opencode/pi/hermes │
                                          └────────────────────┘

Every time the notebook opens a tunnel it publishes a small JSON payload (base_url, api_key, model, num_ctx, source) to your secret ntfy.sh topic. The sync scripts on your local machine poll that topic and rewrite the client config. The topic name doubles as the shared secret.

Components

File	Purpose
colab_llm_server.ipynb	Main server on Colab (T4, 14B model by default, model cache on Google Drive)
kaggle_llm_server.ipynb	Fallback when the Colab quota runs out (2x T4 32GB, 30B model by default)
local/sync-opencode.py	Updates the borrowed-gpu provider in ~/.config/opencode/opencode.json
local/sync-pi.py	Updates the borrowed-gpu provider in ~/.pi/agent/models.json
local/sync-hermes.py	Updates the model: block in ~/.hermes/config.yaml
local/com.user.sync-*.plist	Optional launchd agents: auto-sync every 60 seconds (macOS)
local/systemd/sync-llm@.{service,timer}	Optional systemd user units: auto-sync every 60 seconds (Rocky Linux 9 / any systemd distro)
opencode.json.example	opencode config template for manual setup
pi-models.json.example	pi config template for manual setup
hermes-config.yaml.example	Hermes Agent config template for manual setup
tests/mock_ntfy.ndjson	Mock ntfy payload for offline testing of the sync scripts

Requirements

• A Google account (Colab + Drive). For the Kaggle fallback: a phone-verified Kaggle account.
• A local machine (desktop or laptop) running macOS or Linux — this project targets macOS and Rocky Linux 9; Windows is not supported.
• Python 3.8+ (the sync scripts use only the standard library).
• At least one client installed: opencode, pi, or Hermes Agent.

One-time setup

1. Create a secret ntfy topic

The topic is your only secret: anyone who knows its name can read your server URL + API key. Make it long and random:

echo "borrowed-gpu-$(openssl rand -hex 8)"
# example output: borrowed-gpu-3fa9c21b74d0e6a8

You don't need an ntfy account — publishing and polling public topics is free.

2. Save the topic on your local machine

mkdir -p ~/.config/borrowed-gpu
echo "borrowed-gpu-3fa9c21b74d0e6a8" > ~/.config/borrowed-gpu/topic

The sync scripts read this file automatically (you can also use the NTFY_TOPIC env var or the --topic flag).

3. Save the topic in Colab

1. Open colab_llm_server.ipynb in Google Colab (upload it or open from GitHub/Drive).
2. Click the key icon ("Secrets") in the left sidebar.
3. Add a secret named NTFY_TOPIC with your topic as the value.
4. Toggle Notebook access on for this notebook.

4. (Optional) Save the topic in Kaggle

1. Open kaggle_llm_server.ipynb on Kaggle (Create > New Notebook > File > Import Notebook).
2. Add-ons > Secrets > Add secret: name NTFY_TOPIC, same value.
3. Make sure the secret is attached to the notebook (checkbox).

Daily usage

Start the server

1. Open colab_llm_server.ipynb in Colab.
2. Runtime > Change runtime type > T4 GPU > Save.
3. Runtime > Run all.
4. Watch the cells complete top to bottom:
   • Cell 2 detects the GPU/RAM/disk and prints which coding models fit this hardware (with the matching NUM_CTX). If your configured MODEL isn't recommended for the detected GPU, it tells you what to change in cell 1.
   • Cell 3 asks for Google Drive permission the first time (this is where the model cache lives).
   • Cell 5 pulls the model. First run: ~5-15 minutes. Later runs: ~1-2 minutes (loaded from the Drive cache).
   • Cell 7 prints the public URL, the session API key, and a test response from the model. If you see the test response, the whole chain works.
5. Leave cell 8 (the watchdog) running while you code. It restarts anything that dies and re-publishes the URL if the tunnel changes.

Connect your client

Run the sync script for the client(s) you use — that's it:

python3 local/sync-opencode.py   # opencode -> ~/.config/opencode/opencode.json
python3 local/sync-pi.py         # pi       -> ~/.pi/agent/models.json
python3 local/sync-hermes.py     # Hermes   -> ~/.hermes/config.yaml

Each script fetches the latest payload from your ntfy topic, rewrites the config idempotently, and prints what changed. Re-run it whenever the tunnel URL changes (or install the launchd agents below and forget about it).

Stop the server

Stop the watchdog cell (square stop button), then Runtime > Disconnect and delete runtime to free your GPU quota.

Client notes

opencode

The sync script maintains a borrowed-gpu provider. Select the model with /models inside opencode, or set it as default in opencode.json ("model": "borrowed-gpu/<model-id>" — the sync script does this automatically unless you've pointed the default at another provider). Manual template: opencode.json.example.

pi

The sync script maintains a borrowed-gpu provider in ~/.pi/agent/models.json with the compat flags Ollama needs (supportsDeveloperRole: false, supportsReasoningEffort: false). pi reloads the file every time you open /model, so no restart is needed. Manual template: pi-models.json.example.

Hermes Agent

Important: Hermes Agent requires a context window of at least 64,000 tokens and rejects smaller endpoints at startup. Before using Hermes:

1. In the notebook's cell 1, set NUM_CTX to 65536.
2. On a 16GB T4 a 14B model won't fit with a 64k KV cache — switch MODEL to a smaller one, e.g. qwen2.5-coder:7b-instruct-q4_K_M. On Kaggle (2x T4, 32GB) the default 30B model works with 64k context.

The sync script rewrites the five managed keys of the model: block in ~/.hermes/config.yaml (default, provider: custom, base_url, api_key, context_length) and leaves the rest of your config untouched. It warns if the published context is below 64k. Manual template: hermes-config.yaml.example.

Auto-sync (optional)

Instead of re-running the sync scripts by hand, schedule them every 60 seconds.

macOS (launchd):

# pick the one(s) you need
cp local/com.user.sync-opencode.plist ~/Library/LaunchAgents/
cp local/com.user.sync-pi.plist ~/Library/LaunchAgents/
cp local/com.user.sync-hermes.plist ~/Library/LaunchAgents/

launchctl load ~/Library/LaunchAgents/com.user.sync-opencode.plist
launchctl load ~/Library/LaunchAgents/com.user.sync-pi.plist
launchctl load ~/Library/LaunchAgents/com.user.sync-hermes.plist

Adjust the script path inside the plist if the project is not at ~/Projects/borrowed-gpu. Logs go to /tmp/sync-<client>.log.

Rocky Linux 9 (systemd user timers):

The units are templates — the part after @ picks the client (opencode, pi, or hermes):

mkdir -p ~/.config/systemd/user
cp local/systemd/sync-llm@.service local/systemd/sync-llm@.timer ~/.config/systemd/user/
systemctl --user daemon-reload

# enable the one(s) you need
systemctl --user enable --now sync-llm@opencode.timer
systemctl --user enable --now sync-llm@pi.timer
systemctl --user enable --now sync-llm@hermes.timer

Adjust the project path in the service's ExecStart if it is not at ~/Projects/borrowed-gpu. Check logs with journalctl --user -u sync-llm@opencode.service. To stop: systemctl --user disable --now sync-llm@opencode.timer.

When the Colab GPU quota runs out

Open kaggle_llm_server.ipynb on Kaggle (accelerator GPU T4 x2, Internet on), Run all. The sync scripts need no changes — same ntfy payload, same provider. Bonus: 32GB of VRAM fits qwen3-coder:30b, which is far more capable for agentic coding.

Customizing the notebooks

Everything user-specific lives in cell 1 (Configuration) of each notebook. If you fork this project, these are the knobs:

colab_llm_server.ipynb

Variable	Default	What to change
MODEL	qwen2.5-coder:14b-instruct-q4_K_M	Any model tag from the Ollama library. See the VRAM table below.
NUM_CTX	32768	Context window. Use 65536 for Hermes Agent (mandatory) or long agentic sessions; lower it if the model doesn't fit.
USE_DRIVE_CACHE	True	Set False if you don't want the model stored on your Google Drive (it will re-download every session).
NTFY_TOPIC	empty	Normally left empty (read from the Colab Secret). Fill it in only for quick experiments — anyone who can see the notebook sees the topic.
OLLAMA_PORT / PROXY_PORT	11434 / 8000	Only change on port conflicts (rare on Colab).

Other things people commonly tweak:

• Hardware check (cell 2): detects GPU/VRAM/RAM/disk via nvidia-smi and recommends coding models per VRAM tier. If you fork this for different hardware, the TIERS table in that cell is the single place to tune the recommendations.
• Drive cache location (cell 3): /content/drive/MyDrive/ollama_models. Change the path if you keep your Drive organized differently. The cache grows to the size of every model you've pulled — prune it from Drive if space is tight.
• Ollama tuning (cell 5, OLLAMA_ENV): OLLAMA_KV_CACHE_TYPE="q8_0" halves KV-cache memory with negligible quality loss; set to "f16" if you suspect cache-quantization artifacts. OLLAMA_KEEP_ALIVE="-1" keeps the model in VRAM permanently. OLLAMA_NUM_PARALLEL="1" — raise only if you have VRAM to spare and multiple clients.
• Watchdog cadence (cell 8): checks every 30 seconds and prints a heartbeat every 10 minutes; both are plain constants in the loop.

kaggle_llm_server.ipynb

Same knobs, plus:

Variable	Default	Notes
MODEL	qwen3-coder:30b	2x T4 = 32GB fits 30B-class MoE models.
WORK_DIR	/kaggle/working	Logs and the proxy script live here. No Drive on Kaggle, so there is no model cache — every session re-pulls.
OLLAMA_SCHED_SPREAD	"1" (cell 4)	Forces the model to spread across both GPUs. Leave it on.

Which model fits which GPU?

You don't have to guess: cell 2 of each notebook detects the actual hardware and prints these recommendations for you. The table below is the same logic for reference:

GPU (VRAM)	Recommended MODEL	Max practical NUM_CTX
Colab T4 (16GB)	qwen2.5-coder:14b-instruct-q4_K_M (~9GB)	32768
Colab T4 (16GB), Hermes	qwen2.5-coder:7b-instruct-q4_K_M (~4.7GB)	65536
Kaggle 2x T4 (32GB)	qwen3-coder:30b (~19GB)	65536
Colab Pro L4 (24GB)	qwen2.5-coder:14b / 30B q4	65536
Colab Pro A100 (40GB)	qwen3-coder:30b and up	131072

Rule of thumb: model weights + KV cache must fit in VRAM with ~1GB headroom. Halving NUM_CTX roughly halves KV-cache memory; q8_0 KV cache halves it again versus f16.

Things you should NOT need to change

• The auth proxy, tunnel, publish, and watchdog cells are plumbing — they work as-is for any model/client combination.
• The payload format (base_url, api_key, model, num_ctx, source) is what the three sync scripts expect. If you add fields, add them in both notebooks and in the scripts.

How the Colab limitations are mitigated

Limitation	Mitigation
Session dies / disconnects	Watchdog auto-restarts Ollama/proxy/tunnel; the notebook is idempotent, a single "Run all" fully recovers
Slow model pull every session	OLLAMA_MODELS points at Google Drive (persistent cache)
GPU quota exhausted	Twin notebook on Kaggle (30 GPU hours/week, 2x T4)
Tunnel URL changes	Automatic publish to ntfy + sync scripts rewrite the client config
Public endpoint without auth	Auth proxy requires Authorization: Bearer <key> with a random per-session key
16GB VRAM	14B q4_K_M model + flash attention + q8_0 KV cache → a 32k context still fits
Model unloads (latency)	OLLAMA_KEEP_ALIVE=-1, the model stays in VRAM
Context overflow in clients	Context limits propagated from the notebook (num_ctx) into every client config

Limitations that CANNOT be mitigated

• Google Colab ToS. Running a long-lived non-interactive server is a grey area under Colab's terms of service. Use it session-based only while you're coding, not 24/7. The risk of account restrictions remains and is your responsibility.
• Model quality ceiling. A 14B (or even 30B) model is still no replacement for Claude/GPT in heavy agentic coding — it sometimes fails at tool calling or produces incorrect diffs. This is a physical limit of free VRAM, not of the setup.
• Session duration. Free Colab still disconnects sessions after a few hours. The watchdog makes recovery fast, but it cannot prevent the disconnect itself.

Offline testing

All sync scripts can be tested without a running server:

python3 local/sync-opencode.py --input tests/mock_ntfy.ndjson --config /tmp/opencode.json --dry-run
python3 local/sync-pi.py --input tests/mock_ntfy.ndjson --config /tmp/models.json --dry-run
python3 local/sync-hermes.py --input tests/mock_ntfy.ndjson --config /tmp/config.yaml --dry-run

License

MIT — use and modify as you like.