Skip to content

Refine level 9 matches with dynamic Huffman prices - #37

Closed
rafambn wants to merge 2 commits into
dev-1.1.0from
cost-parser-ratio
Closed

rafambn wants to merge 2 commits into
dev-1.1.0from
cost-parser-ratio

Conversation

@rafambn

@rafambn rafambn commented Sep 9, 2026 •

Copy link
Copy Markdown
Owner

Level 9 currently prices every token with fixed Huffman codes even when the encoder chooses dynamic trees. Try one refinement using prices from the first parse, rebuild the frequency estimate, and restore the original path unless the estimate improves.

Actual level 9 output across seven fixtures is 1.263% smaller, including 2.475% on Maltese and 0.899% on text, with no fixture size regression. This adds parsing work and allocations; final development JMH measured Maltese at 1.65 s versus the 1.21 s baseline. Raw samples, sizes and trusted saved Kompress comparisons are in performance/2026-09-09-dynamic/RESULTS.md.

The estimate covers a parser window rather than each emitted block, so arbitrary inputs can still regress. Replace the Huffman builder's 25001 sentinel to support window frequencies, and reject packed two-byte no-match results in the window-crossing fallback. Integration testing exposed that otherwise dynamic prices could select length two and emit a length-three token.

Linux JVM tests, 100% instruction/branch coverage, and all 70 corpus/level round trips through JDK and KFlate passed. Levels 0–8 are unchanged. The combined candidate also passes all 70 cases. Independent PR targeting dev-1.1.0; it overlaps #35 and requires resolving the parser changes if both are adopted.

@rafambn rafambn closed this Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant