LLM Manager

A lightweight, cross-platform (Windows & Linux) Rust utility for managing local LLM inference servers (specifically llama.cpp).

🚀 Overview

llm-manager provides a clean CLI interface to start, stop, and monitor local LLM inference processes. It uses a Launcher-Wrapper architecture, delegating the complex execution logic to lightweight wrapper scripts (.bat on Windows, .sh on Linux).

The same binary (llamactl) runs on a Windows desktop or a headless Linux server — only the per-machine config and the wrapper scripts differ.

🛠 Prerequisites

• Windows or Linux — all OS-specific process handling is selected at compile time (cmd/taskkill/tasklist on Windows, sh/pkill/kill on Linux).
• Rust Toolchain (latest stable).
• llama.cpp binaries: You must have llama-server (llama-server.exe on Windows) and its dependencies available on your system.

📦 Installation & Setup

Option 1: Build from Source (Recommended for Developers)

1. Clone the repository:

   git clone <repository-url>
   cd llm-manager

2. Build the project:

   cargo build --release

Option 2: Download Pre-built Release

1. Go to the GitHub Releases page.
2. Download the archive for your platform:
   • llamactl-windows-x86_64.zip (Windows)
   • llamactl-linux-x86_64.tar.gz (Linux)
3. Extract the contents to your desired directory and put the binary on your PATH.

Release archives are built automatically by GitHub Actions when a v* tag is pushed (see Continuous Integration).

---

Post-Installation Steps (Required for both options)

Regardless of how you install, you must complete the following:

1. Prepare your configuration: Create a file named llm-config.toml (see llm-config.toml.example).
2. Create your model scripts: Create a folder (e.g., scripts/) and add wrapper scripts that contain the specific llama-server commands for your models — .bat files on Windows, .sh files on Linux.

Linux note: make your wrapper scripts executable (chmod +x scripts/*.sh). The server listens on http://127.0.0.1:8080 on both platforms.

Recommended Directory Structure

llm-manager/
├── Cargo.toml
├── llm-config.toml          <-- Your model registry
├── src/
├── scripts/                 <-- Your model wrappers
│   ├── run-model-a.bat      <-- Windows: calls llama-server.exe with specific flags
│   └── run-model-a.sh       <-- Linux:   calls llama-server with specific flags
└── target/

⚙️ Configuration

Models are defined in a configuration file. You can specify the location of this file using the --config flag or by setting the LLM_CONFIG_PATH environment variable.

Configuration Hierarchy

The application resolves the configuration path in this order:

1. CLI Flag: --config <path>
2. Environment Variable: LLM_CONFIG_PATH
3. Default: llm-config.toml in the current directory.

Example entry:

# Windows
[[models]]
name = "my-model"
aliases = ["my-alias"]
script = 'C:\llama.cpp\scripts\my-model.bat'

# Linux
[[models]]
name = "my-model"
aliases = ["my-alias"]
script = "/opt/llama.cpp/scripts/my-model.sh"

The script field is the only OS-specific part of the config — point it at a .bat (Windows) or .sh (Linux) wrapper that launches llama-server with your flags.

⌨️ Commands

llamactl (or cargo run --) supports the following commands:

Command	Description
<name>	Shorthand: Starts the model matching the name or alias (replacing any running model).
start <name>	Starts a model by name or alias. If another model is already running, it is stopped first.
stop	Stops all running llama-server processes.
status	Shows the status of the currently running model (if any).
list	Lists all configured models and their aliases.

♻️ Replace-on-Start

By default, starting a model automatically stops whatever model is already running (freeing the port and VRAM) before launching the new one. Detection prefers the server's /health endpoint — which responds even when the model is asleep under --sleep-idle-seconds — and falls back to the tracked PID.

Pass --no-replace to start to refuse instead: if a model is already running, the command reports it and exits without touching the running server.

llamactl start my-model --no-replace

🛠 Options

Option	Description
--config <path>	Specify a custom path to the configuration file. (global)
--no-replace	start only: refuse to start if a model is already running, instead of replacing it.

Examples

Start a model (shorthand):

llamactl my-model

List all models:

llamactl list

Start a model with a custom config:

llamactl --config ./custom-config.toml start my-model

🧠 Architecture & Design Decisions

The "Launcher-Wrapper" Pattern

To keep the Rust codebase maintainable and avoid "flag bloat," we do not pass complex arguments directly through Rust. Instead:

• Rust manages the lifecycle (Start/Stop/Monitor).
• Wrapper Scripts manage the configuration (Quantization, Layers, Context).

Cross-Platform Design

All OS-specific behaviour is isolated in src/platform.rs behind #[cfg(windows)] / #[cfg(unix)], so the rest of the code is platform-agnostic. Three operations differ per OS:

Operation	Windows	Linux
Launch script	cmd /C <script> (detached, no window)	sh -c <script> (own process group)
Stop server	taskkill /F /IM llama-server.exe /T	pkill -f llama-server
PID liveness	tasklist /FI "PID eq <pid>"	kill -0 <pid>

Each machine keeps its own llm-config.toml pointing at .bat or .sh wrappers. A later refactor could move the llama-server flags into the TOML itself so a single config serves both OSes.

PID vs. Sledgehammer Termination

A critical design choice was made regarding how processes are stopped:

• The Problem: When Rust spawns a wrapper script, the captured PID belongs to the shell runner (cmd.exe / sh), not the actual llama-server process. Killing the shell PID often leaves the inference engine running as an "orphan" process, hogging GPU memory.
• The Solution (Sledgehammer): The stop command targets the server by process name (taskkill … llama-server.exe / pkill -f llama-server), killing the actual workload and ensuring a clean release of system resources.

🔁 Continuous Integration

This repo ships two GitHub Actions workflows (.github/workflows/):

• ci.yml — on every push/PR, runs cargo fmt --check, cargo clippy -D warnings, build, and tests on both ubuntu-latest and windows-latest.
• release.yml — on pushing a v* tag, builds native binaries for Windows and Linux and attaches them to a GitHub Release:

  git tag v0.1.0
  git push origin v0.1.0

🛠 Development

Building

cargo build --release

Cross-checking the other OS

cargo check the Linux target from Windows (or vice versa) to catch #[cfg] mistakes early. Note that the TLS dependency needs a C toolchain for the target, so a full cross-build requires a cross-compiler — CI builds each OS on its own native runner to avoid this.