tokenkit

Agentic-first token counting and text chunking service for LLM context management. Count tokens, chunk text to fit context windows, estimate API costs, and manage context limits.

Quick Start

# Build
make build

# Run (no auth for development)
./tokenkit -no-auth

# Count tokens
curl -X POST http://localhost:7198/count -d '{"text":"Hello, world!"}'

# Chunk text
curl -X POST http://localhost:7198/chunk -d '{"text":"long text...","max_tokens":512}'

# List models
curl http://localhost:7198/models

# Estimate cost
curl -X POST http://localhost:7198/estimate -d '{"model":"gpt-4o","input_tokens":1000,"output_tokens":500}'

API

Method	Endpoint	Description
POST	/count	Count tokens in text
POST	/chunk	Split text into chunks within token limit
POST	/truncate	Truncate text to token count
POST	/split	Split text into overlapping chunks for RAG
GET	/models	List popular LLM models with pricing
POST	/estimate	Estimate API cost for tokens and model
GET	/help	Self-documenting API manual
POST	/mcp	MCP JSON-RPC 2.0 endpoint

Tokenizers

Tokenizer	Description	Chars/Token
openai	Approximates OpenAI cl100k_base/p50k_base	~4.0
claude	Approximates Anthropic tokenizer	~3.5
simple	Word-based counting	~5.0
char	Pure character ratio	~4.0

Configuration

Flag	Env	Default	Description
-addr	TOKENKIT_ADDR	:7198	Listen address
-no-auth	TOKENKIT_NO_AUTH	false	Disable auth (dev mode)

Build

CGO_ENABLED=0 go build -trimpath -o tokenkit ./cmd/tokenkit

Single static binary, zero external dependencies, no database needed.

License

MIT