MetaInfer

Let AI generate high-performance, compact, purpose-built LLM inference frameworks

Describe what you want — MetaInfer builds your optimized inference engine

(An LLM-powered AI Infra optimization toolkit)

---

中文文档

---

What is this? (Our vision)

• Let AI generate simple, compact inference frameworks that match or surpass SOTA performance. (Killer app)

  • Today's SOTA inference frameworks are buckling under their own weight.

    • The traditional approach — multi-layer abstractions to support every model, every hardware target, every parallelism strategy, every quantization scheme, every kernel optimization — no longer keeps pace with the pace of LLM innovation.
    • The vast majority of code in an inference framework is unnecessary once your deployment constraints are fixed, yet that compatibility logic drags down performance.

  • Tell MetaInfer what you need, and it writes a high-performance, purpose-built inference framework for your exact setup.

    • Which model? What hardware? What parallelism? What quantization? Optimize for throughput or latency?

• AI-assisted kernel optimization

  • Given a target kernel, the LLM drives an automated [optimize → benchmark] loop.

• Model porting

  • New model unsupported on older GPUs? Point MetaInfer at the upstream framework code and let it backport to a lower-version target.

• Embedded / edge inference

  • MetaInfer targets C/C++ and Rust as first-class output languages — laying the groundwork for edge computing and embodied AI.

Built-in task types

Type	Description
gen-infer-framework	Build a model-specific inference server with an OpenAI-compatible HTTP API. An immutable oracle boots serve.sh, sends fixed prompts, and dispatches an LLM judge to verdict correctness.
calc-theoretical-value	Compute theoretical FLOPs and memory-traffic for a single forward pass of an LLM. Fully read-only deterministic pipeline: model inspection → memory modeling → compute graph → visualization.
example	Canonical skeleton for new task types. Copy, rename, uncomment register(), implement your pipeline — no shared code touched.

Quick start

Step 1: Install ccb (MetaInfer currently uses the open-source Claude Code CLI; other coding agents are not yet supported — contributions welcome)

Open-source ccb repository: https://github.com/claude-code-best/claude-code

npm i -g claude-code-best

Step 2: Install MetaInfer

git clone https://github.com/MetaInfer/MetaInfer.git
cd MetaInfer
pip install -r requirements.txt
./serve.py

Open http://127.0.0.1:8765, click + New Task, pick a task type, fill in your requirements, and the LLM gets to work.

# Other ways to start
./serve.py --host 0.0.0.0 --port 9000
METAINFER_PORT=9000 ./serve.py
python -m metainfer.server.app

License

MIT

Academic Research

The initial ideas and experimental data of MetaInfer are publicly available at https://arxiv.org/abs/2607.12875. The related code is on the arxiv-paper branch.

Citation:

@misc{miao2026metainferknowledgellminference,
      title={MetaInfer: A Knowledge Only LLM Inference Engine Generator SKILL Toolbox}, 
      author={Zhenwen Miao and Honglin Wang and Mingheng Mi},
      year={2026},
      eprint={2607.12875},
      archivePrefix={arXiv},
      primaryClass={cs.MA},
      url={https://arxiv.org/abs/2607.12875}, 
}

Contributing

See CONTRIBUTING.md for architecture details, design principles, and how to add new task types.