Implement Bits Per Byte Metric Computation#61

Open

ilkerkesen wants to merge 12 commits into

ilkerkesen:bits-per-byte

Commits on Feb 26, 2026

Commits on Mar 2, 2026

handle uneqaul non-PAD counts across batches correctly
ilkerkesen
committed

Commits on Mar 3, 2026

make evaluation correctly working for text evaluation metrics
ilkerkesen
committed

Commits on Mar 5, 2026

add support for utf-16 and utf-32 encodings also as well
ilkerkesen
committed

Commits on Mar 6, 2026

Commits on Mar 11, 2026

remove huggingface/evaluate dependency for accuracy: it complicates things on no-internet-access clusters.
ilkerkesen
committed

Commits on Mar 21, 2026