Green Compress

Shrink transformer weights. Keep the quality. Run on low-RAM machines.

Green Compress (greencompress) is a Rust toolkit for post-training weight compression and layer inference. Quantize transformer weights to Q4/Q8, apply green-format repair (low-rank, sparse, outliers), benchmark quality vs RAM vs speed, and run matmul with AVX2 SIMD — optional CUDA for the heavy GEMM.

What it does today: tensor extraction from GGUF, per-layer compression and benchmarking, and individual layer inference.

Chat path today: Phase 1 export-gguf → llama.cpp. pack-model emits dense-complete .green packages; Green Engine native generate works for engineering/smoke (dense, tiny MoE, 1B) but production chat still prefers GGUF.

---

Three reasons to use Green Compress

1. ~45% less RAM — Real model layers at ~99.9% quality vs FP32 (see benchmarks below).
2. CPU-first — AVX2 SIMD with portable x86-64 fallback; optional CUDA GEMM.
3. Fits the Green stack — Standalone CLI or via ge install / ge compress in Green Engine.

---

Installation

Prebuilt binaries (Linux, macOS, Windows): GitHub Releases

From source:

git clone https://github.com/VeyrForge/GreenCompress.git && cd GreenCompress
make
bin/greencompress help

Requires Rust stable. Linux is recommended for the POSIX shared-memory infer-server; other platforms can use pipe transport.

make                # portable x86-64 (AVX2 at runtime if present)
make native         # CPU-tuned for this machine
make rust-gpu       # CUDA matmul (--features gpu)
make rust-test

---

30-second example

make
bash benchmarks/run.sh
bin/greencompress compare-benchmark --dir out/benchmark/synthetic

Green Engine workflows (one-liners)

Goal	Command
Smaller GGUF for llama.cpp	greencompress export-gguf --gguf source.gguf --out smaller-q4.gguf --verify then ge chat serve --model smaller-q4.gguf
Native .green package (Llama-3.2-1B)	python scripts/rebuild_llama32_1b_green.py then ge chat serve --model ~/.green/models/Llama-3.2-1B.green
Layer RAM / codec research	greencompress benchmark --type green_optimal --in weights.mx --activations acts.mx --out-dir out/layer

---

See it work

No bundled demo video yet — synthetic benchmark output:

method                 quality%    ram_mib   vs_fp32
fp32_reference         100.00      1.000     —
green_optimal           99.71      0.390     ~2.6× less RAM
green_spqr_svd          99.92      0.417     ~2.4× less RAM

Real-model example (Llama-3.2-1B ffn_down): green_optimal at 99.56% quality, 20.75 MiB vs 64 MiB FP32.

---

Supported platforms

Platform	Notes
Linux	Full support; SHM infer-server
macOS	Build + infer via pipe transport
Windows	Build + infer via pipe transport

---

How it works

Method	Best for
green_optimal	Default — 99.50% quality floor, smart repair skip
green_adaptive	Skip repair when Q8 already passes the quality gate
green_smart	AWQ Q8 + imatrix sparse repair
green_spqr_svd	Higher-quality escalation
green_q7	Sub-8-bit codec (−12% RAM vs Q8)
fp32	Uncompressed reference

Policy file: config/tensor_policy.json

Model export roadmap

Phase	Command	Output	Status
1	greencompress export-gguf --gguf MODEL.gguf --out MODEL-green-q4.gguf [--method green_optimal] [--verify]	Runnable compressed GGUF for llama.cpp fallback (metadata, tokenizer, norms, embeddings, output weights preserved; 2D weights re-quantized to Q4_0 baseline)	Available (chat path today)
2 pack	greencompress pack-model --gguf MODEL.gguf --out MODEL.green [--requant q4_0|none] [--verify]	Dense-complete .green (production: Q4_0 requant; Q4_K passthrough experimental), or MoE raw-f32 greenpack	Available for dense/tiny + 1B + tiny MoE
2+ runtime	Green Engine native generate from .green	Token generation	Available (experimental) — dense/tiny-MoE/1B engineering path; production chat still prefers GGUF

Research pipeline (per-tensor benchmarks, no single-file export): python3 scripts/compress_model.py --gguf MODEL.gguf --out WORK --methods green_optimal

Phase 1 export-gguf (F16 / BF16 / F32 -> Q4 GGUF)

Supported input: a full-precision or already-quantized GGUF. 2D float tensors (F32, F16, BF16) and dequantizable ggml types are re-quantized to Q4_0; 1D tensors, norms, and non-2D payloads are passed through. Metadata and tokenizer fields are preserved. This is a llama.cpp fallback (not native .green token generation).

Exact CLI shape (--help is not implemented for subcommands; use greencompress help):

greencompress export-gguf --gguf MODEL-f16.gguf --out MODEL-green-q4.gguf --method green_optimal --verify

Or via Green Engine:

ge compress export-gguf --gguf MODEL-f16.gguf --out MODEL-green-q4.gguf --method green_optimal --verify

Optional env: GREENCOMPRESS_ROOT (checkout with scripts/), GREEN_PYTHON (Windows default probes python3 / python / py).

Tiny dense .green for forward / loader smoke

# from green-compress checkout
python scripts/make_test_gguf.py --out scripts/fixtures/mini-test.gguf --seed 0
greencompress pack-model --gguf scripts/fixtures/mini-test.gguf --out scripts/fixtures/tiny.green --verify
# or without installed binary:
# python scripts/pack_model.py --gguf scripts/fixtures/mini-test.gguf --out scripts/fixtures/tiny.green --verify

Package contents: manifest.json (format=green-model, source_model + tensor_files + files.{metadata,dense,tokenizer}), metadata.gguf, dense.gguf, checksums.json, config.json, tokenizer.json. No expert rows in tensors[].

Production Llama-3.2-1B .green (Q4_0 requant — known-good)

# Source: Instruct Q4_K_M GGUF under ~/.green/models (or --gguf PATH)
python scripts/rebuild_llama32_1b_green.py --verify
# Check an existing package without rebuilding:
python scripts/rebuild_llama32_1b_green.py --check-only

Production default is --requant q4_0 with pack-model engine_pack layout (reshape dequant flat → (out, in) before Q4_0 so GGUFWriter stores [in, out] for Green Engine GEMV). Tied embeddings set config.tied_embeddings=true without cloning token_embd → output.weight.

Completion-style prompts (e.g. The sky is) produce readable English. Instruct chat (ge run Auto/Force wrap) is improved vs Q4_K passthrough but can still be soft on 1B greedy — prefer llama.cpp for demos until Engine chat logits catch up.

Experimental — do not use for demos: --requant none (Q4_K/Q6_K passthrough) opens and loads quickly, but native decode is currently garbage until the engine Q4_K GEMV contract matches llama.cpp. Prefer q4_0 until that is fixed.

Tiny MoE .green (M4 expert greenpack)

python scripts/make_test_moe_gguf.py --out scripts/fixtures/mini-moe-test.gguf --seed 1
python scripts/pack_model.py --gguf scripts/fixtures/mini-moe-test.gguf --out scripts/fixtures/tiny-moe.green --verify

Adds experts-000.greenpack (GRNP + raw f32) and expert rows in tensors[] (expert_tensors_pending empty).

Llama-8B / full MoE gaps (honest)

• Float sources / production 1B packs use Q4_0 re-quant (--requant q4_0). Q4_K passthrough (--requant none) is experimental (opens; decode broken until GEMV contract fix).
• Expert .greenpack is raw-f32 today (quantized/compressed experts still TBD for large MoE). See quantized-expert-greenpack-plan.md.
• Native Green Engine MoE token generate is wired for tiny-moe.green (ge run / native_moe_generate_smoke). Larger MoE still blocked by quantized expert greenpack.
• HF tokenizer.json / tokenizer_config.json when source is safetensors (pass --tokenizer / --config or convert to GGUF first).
• Dense 8B Q4_0 packs layout-verify OK; native ge run gravity smoke on Llama-3.1-8B is currently engine-side gibberish (1B/3B OK) — not a pack transpose regression.

---

Benchmarks

	FP32 baseline	green_optimal
~1.1B model down_proj RAM	~0.94 GiB	~0.52 GiB (~45% less)
Quality vs FP32	100%	~99.9%

Reproduce: bash benchmarks/run.sh · Full tables in synthetic runs under out/benchmark/.

---

Documentation

• CHANGELOG.md — version history
• bin/greencompress help — CLI reference
• config/tensor_policy.json — per-tensor policy
• quantized-expert-greenpack-plan.md — MoE expert pack roadmap

---

Limitations

• Native Green Engine token generation from .green is now available for engineering and smoke, but production chat still prefers export-gguf + llama.cpp today.
• Phase 1 export-gguf applies Q4_0 re-quantization on 2D weights (llama.cpp fallback; Python gguf lacks Q4_K quantize).
• pack-model production packs use --requant q4_0. Q4_K passthrough (--requant none) is experimental (opens; decode garbage until engine GEMV fix). MoE packages emit raw-f32 expert .greenpack shards.
• FP4 (--requant nvfp4|mxfp4) is an experimental codec/pack scaffold only — not a production speed or memory path.
• If FP32 already fits in RAM and is fastest, stay on FP32.
• GGUF compression requires Python 3 (python3 or Windows python/py), numpy, and gguf.
• GPU inference needs CUDA toolkit for make rust-gpu.
• Linux SHM infer-server is not available on all platforms (pipe fallback exists).

---

Contributing

Issues, benchmark results, and suggested improvements are welcome on VeyrForge/GreenCompress.

Pull requests improving Green Compress are welcome. You may also keep private/internal forks for your own deployment under the BUSL-1.1 Additional Use Grant. Do not offer Green Compress (or a substantially similar substitute) to third parties as a hosted or competing product. See License and permitted use.

---

Public release history

See CHANGELOG.md and GitHub Releases.

---

License and permitted use

Green Compress is source-available under the Business Source License 1.1 (BUSL-1.1). It is not OSI open source until the Change License applies.

You may:

• Use and run Green Compress (including in production) for personal use or internal business purposes (including employees and contractors acting on your behalf)
• Copy, modify, and create derivative works for those same purposes — without having to contribute changes back
• Use Green Compress to develop, test, maintain, review, or operate software for yourself or your customers (without offering Green Compress itself as a product or service)
• Study the published source

You may not (until the Change License applies):

• Offer Green Compress or a modified version to third parties as a hosted, managed, embedded, or distributed product or service whose primary purpose is to provide functionality substantially similar to Green Compress as a substitute
• Sell a renamed fork or embed Green Compress as the main feature of another paid product without a commercial license

Change Date: 2029-07-31 — on that date, or the fourth anniversary of the first public BSL distribution of this version (whichever is earlier), this version becomes available under Apache License 2.0.

Tutorials and blog posts may include short illustrative snippets from the published source for explanation, provided they do not redistribute the software as a competing product or imply an OSI open-source grant before the Change Date.

For commercial redistribution, OEM licensing, or other usage not covered above, contact licensing@veyrforge.com.

This section is a plain-language summary. The binding terms are in LICENSE.