zinfer

An ML/AI inference and power cost estimation library for low-power embedded devices and phones, written in Rust.

Focused on optimizing for joules per inference, not tokens per second.

See BACKLOG.md for the current state of the project.

The thesis

Generally speaking energy usage estimated derives from time in a particular power state. Most of the design of this library follows from that.

FLOPs don't predict it accurately. Neither does model size: pack weights tighter and you can spend more unpacking them than you saved fetching them. There's no fixed ratio between two chips either, since a core's extensions help some kernels and do nothing for others.

"Memory is expensive, arithmetic is cheap" is a rule that can get misinterpeted. True with DRAM, or flash behind a cache. Much weaker once the weights already sit in on-chip memory. Which term wins is a property of the part, so it belongs in a model rather than a rule of thumb.

zinfer doesn't measure hardware. An operator declares what it does: operations, bytes moved, time awake. A device description you write prices it. Out comes a ranking and a breakdown of where the largest power usage went, not a certified figure.

If cost comes from what an operator declares rather than a table of special cases, adding a new kind of model stays cheap. That matters, because large savings probably aren't in dense quantized networks. Hyperdimensional computing, Tsetlin machines, spiking models, tensor networks all have plausible arguments, with almost no tooling to compare them honestly.

Direction

Where this is heading, not a specification yet.

• Describe a model as a typed graph in Rust.
• Estimate per-stage energy against a named target, on the host, before flashing.
• Fail the build when it exceeds the declared budget, naming the stage and the term that blew it.
• Compile to a bundle that runs on a microcontroller, a phone or a desktop through one executor abstraction.

Early numbers from pricing a few classifier styles against two Cortex-M parts are on Hugging Face in data/benchmarks/.

Hardware

The boards are not used with a measurement rig. They run compiled bundles and check the executor and device abstraction; no energy data is collected from them. The set spans Armv6-M through AArch64, flat SRAM through DRAM, and datasheets of deliberately uneven quality, because that is what a user's device will be like. docs/devices.md covers which board are tested with zinfer.

Shape of the workspace

graph TD
    op["«the extension contract»<br/>zinfer-op"]

    ir["«graph»<br/>zinfer-ir"]
    paradigms["«paradigms»<br/>zinfer-nn · zinfer-hdc<br/>zinfer-tsetlin · zinfer-snn<br/>zinfer-tn"]

    energy["«cost model»<br/>zinfer-energy"]

    plan["zinfer-plan"]
    sim["zinfer-sim"]

    codegen["zinfer-codegen"]
    fmt["zinfer-fmt"]
    macro["zinfer-macro"]

    core["«executor and device abstraction»<br/>zinfer-core"]

    facade["zinfer"]
    tauri["zinfer-tauri"]
    cli["zinfer-cli"]

    op --> ir
    op --> paradigms

    ir --> energy
    ir --> plan
    energy --> plan

    plan --> codegen
    plan --> fmt

    codegen --> core
    fmt --> core

    core --> facade
    core --> tauri
    core --> cli

Loading

Only zinfer-op, zinfer-core, zinfer-fmt and the paradigm crates are meant to reach a microcontroller. Everything else is a host-side tool.

Crates

Crate	Role
zinfer	Facade
zinfer-op	Operator and cost declaration contract
zinfer-core	Executor and device abstraction
zinfer-energy	Device description schema and action pricing
zinfer-ir	Graph intermediate representation
zinfer-plan	Energy-aware planner
zinfer-macro	Graph definition macros
zinfer-fmt	Tensor and event container format
zinfer-cli	Command line interface
zinfer-sim	Host reference implementations
zinfer-codegen	Static Rust code generation
zinfer-tauri	Tauri v2 plugin
zinfer-target	Hardware target descriptions
zinfer-nn	Dense quantized baseline
zinfer-hdc	Hyperdimensional computing
zinfer-tsetlin	Tsetlin machines
zinfer-snn	Spiking and temporal models
zinfer-tn	Tensor networks

License

Apache-2.0