Tiktokenizer

Online token calculator for diffusion language models, providing tokenization and token count for prompts.

Adapted from tiktokenizer.vercel.app.

Quick Start

# Install dependencies
yarn install

# Dev mode (port 22345)
yarn dev

# Build
yarn build

# Production
yarn start

Acknowledgments

• T3 Stack
• shadcn/ui
• openai/tiktoken
• tiktokenizer