Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 4 additions & 7 deletions .github/workflows/tuning.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ name: RD tuning sweep
# with 9 different ladder/divisor combinations simultaneously. Each job
# produces an RD table in the step summary. Compare all tables to pick
# winning constants, then bake them into v2_core.cpp as the new defaults.
# Stock values: scale=2.5, divisor=16.0 (retuned after the photo-pattern RD
# sweeps; divisor 16 adds intermediate lossy ladder steps and scale 2.5 yields
# the finest usable ladder - nine lossy steps on natural content).
# Stock values: scale=1.5, divisor=16.0. Scale 2.5 was tried and reverted
# (issue #44: visible chroma artifacts at medium quality). Divisor 16 adds
# intermediate lossy ladder steps with no quality regression.

on:
workflow_dispatch:
Expand All @@ -22,14 +22,11 @@ jobs:
matrix:
include:
- label: stock
scale: "2.5"
divisor: "16.0"
- label: legacy-ladder
scale: "1.5"
divisor: "16.0"
- label: steep-ladder
scale: "2.5"
divisor: "8.0"
divisor: "16.0"
- label: flat-ladder
scale: "1.0"
divisor: "8.0"
Expand Down
22 changes: 22 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,30 @@

All notable WIMF changes are recorded here. The project follows semantic versioning for the Python package; container compatibility is documented separately.

## 2.2.2 - 2026-08-25

- Reverted the default quality ladder scale from 2.5 to 1.5 (divisor stays
16): the more aggressive quantizer caused visible chroma artifacts on
detailed content at medium quality (issue #44). The divisor retune from
8 to 16 is kept - it adds intermediate lossy steps without quality loss.
- Enabled AVX2 on Windows wheels: v2_simd_avx2.cpp is now compiled
separately with /arch:AVX2 on MSVC and linked into the extension
(issue #45). Runtime dispatch still selects scalar on non-AVX2 CPUs.
- Fixed potential unsigned overflow in wavelet indexing (CodeQL high
severity): y*width multiplications now cast to size_t before use as
vector indices.

## 2.2.1 - 2026-08-25

- Eliminated per-line heap allocations in the wavelet lifting loops (known-flaw
B1): lifting now operates in place on caller-owned scratch buffers, cutting
the transform cost by roughly 13% and removing allocator churn from Extreme
encodes. The retry also hardened the public wavelet API: single-element
lines (any dimension of 1 or 2 at deeper levels) previously read out of
bounds through an underflowing scratch index; verified with a 972-case
non-square reversible roundtrip sweep. Local GCC coverage added for the
exact AppleClang wheel configuration that failed in the 2.2.1 release.

- Retuned the default quality ladder from scale 1.5 to 2.5 (divisor stays 16)
based on the photo/natural RD sweeps: the lossy ladder gains a ninth usable
step on natural content and every quality index produces smaller files at
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "wimf"
version = "2.2.1"
version = "2.2.2"
authors = [
{ name="BenchWare", email="ivanm12453@gmail.com" },
]
Expand Down
36 changes: 29 additions & 7 deletions setup.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import os
import platform
import sys

Expand All @@ -12,23 +13,44 @@ def build_extensions(self):
for ext in self.extensions:
ext.include_dirs.append(pybind11.get_include())
if sys.platform == "win32":
ext.extra_compile_args.extend(["/O2", "/std:c++17"])
ext.extra_compile_args.extend(["/O2", "/std:c++17", "/EHsc"])
else:
ext.extra_compile_args.extend(["-O3", "-std=c++17", "-Wno-misleading-indentation"])
super().build_extensions()

def build_extension(self, ext):
# MSVC: compile v2_simd_avx2.cpp separately with /arch:AVX2 because
# the flag cannot be scoped per-function (MSVC has no equivalent of
# GCC/Clang's __attribute__((target(...)))). The resulting object is
# linked into the extension alongside the baseline-compiled sources.
if self.compiler.compiler_type == "msvc" and any("v2_simd_avx2" in s for s in ext.sources):
avx2_sources = [s for s in ext.sources if "v2_simd_avx2" in s]
ext.sources = [s for s in ext.sources if "v2_simd_avx2" not in s]
# Compile AVX2 TU with /arch:AVX2
avx2_objects = self.compiler.compile(
avx2_sources,
output_dir=os.path.join(self.build_temp, "avx2"),
include_dirs=ext.include_dirs,
macros=ext.define_macros,
extra_postargs=[arg for arg in ext.extra_compile_args if "arch" not in arg] + ["/arch:AVX2"],
debug=self.debug,
)
ext.extra_objects.extend(avx2_objects)
super().build_extension(ext)


def configure_simd(extension):
"""Opt the v2 core into runtime-dispatched AVX2 kernels.

GCC and Clang scope the instruction set in-source via '#pragma GCC
target', so no compiler flag is needed. MSVC requires '/arch:AVX2',
which setuptools cannot apply per file; Windows wheels therefore keep
the portable scalar paths (kernels compiled out, results identical).
GCC and Clang scope the instruction set in-source via per-function
__attribute__((target("avx2"))), so no compiler flag is needed.
MSVC requires '/arch:AVX2' on the file that uses AVX2 intrinsics;
the build_ext class compiles v2_simd_avx2.cpp separately with that
flag and links the object into the extension. Runtime dispatch
selects scalar on CPUs without AVX2 regardless of compilation.
"""
non_windows_platform = sys.platform not in {"win32", "cygwin", "emscripten"}
x86_64_machine = platform.machine().lower() in {"amd64", "x86_64"}
if non_windows_platform and x86_64_machine:
if x86_64_machine:
extension.define_macros.append(("WIMF_SIMD_ENABLE_AVX2", None))


Expand Down
80 changes: 49 additions & 31 deletions src/v2_core.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,14 @@
#include "zstd.h"
#include "v2_simd.hpp"

// Tunable codec constants. The scoring divisor and ladder scale were retuned
// from the historical 8.0/1.5 after photo-pattern RD sweeps: divisor 16 adds
// intermediate lossy ladder steps, and scale 2.5 yields the finest usable
// ladder (nine lossy steps on natural content, smaller files at every quality
// index). The tuning workflow overrides these via -D flags to sweep candidate
// curves.
// Tunable codec constants. The scoring divisor was retuned from 8.0 to 16.0
// after RD sweeps showed it adds intermediate lossy ladder steps with no
// quality regression. The ladder scale stays at the historical 1.5: the
// 2.5 experiment produced smaller files but visible chroma artifacts at
// medium quality (issue #44). The tuning workflow overrides these via -D
// flags to sweep candidate curves.
#ifndef WIMF_LADDER_SCALE
#define WIMF_LADDER_SCALE 2.5f
#define WIMF_LADDER_SCALE 1.5f
#endif
#ifndef WIMF_SCORING_DIVISOR
#define WIMF_SCORING_DIVISOR 16.0
Expand Down Expand Up @@ -71,47 +71,51 @@ int64_t floor_div(int64_t value, int64_t divisor) {
return q - (r != 0 && value < 0);
}

std::vector<double> lift97_forward(const std::vector<double>& line) {
// B1: lifting functions operate in-place on `line` using caller-provided
// scratch buffers. After the first call, resize() is a no-op, eliminating
// per-line heap allocations in the hot loop. Math is unchanged.
void lift97_forward(std::vector<double>& line,std::vector<double>& e,std::vector<double>& o){
constexpr double a=-1.586134342, b=-0.05298011854, g=0.8829110762, d=0.4435068522, k=1.149604398;
const size_t half=(line.size()+1)/2, odds=line.size()/2;
std::vector<double> e(half), o(odds), out(line.size());
const size_t half=(line.size()+1)/2, odds=line.size()/2, om=odds?odds-1:0;
e.resize(half); o.resize(odds?odds:1); if(!odds)o[0]=0;
for(size_t i=0;i<half;++i)e[i]=line[i*2]; for(size_t i=0;i<odds;++i)o[i]=line[i*2+1];
for(size_t i=0;i<odds;++i)o[i]+=a*(e[i]+e[std::min(i+1,half-1)]);
for(size_t i=0;i<half;++i)e[i]+=b*(o[i?i-1:0]+o[std::min(i,odds-1)]);
for(size_t i=0;i<half;++i)e[i]+=b*(o[i?i-1:0]+o[std::min(i,om)]);
for(size_t i=0;i<odds;++i)o[i]+=g*(e[i]+e[std::min(i+1,half-1)]);
for(size_t i=0;i<half;++i)e[i]+=d*(o[i?i-1:0]+o[std::min(i,odds-1)]);
for(size_t i=0;i<half;++i)e[i]+=d*(o[i?i-1:0]+o[std::min(i,om)]);
for(auto& x:e)x*=k; for(auto& x:o)x/=k;
std::copy(e.begin(),e.end(),out.begin()); std::copy(o.begin(),o.end(),out.begin()+half); return out;
std::copy(e.begin(),e.end(),line.begin()); std::copy(o.begin(),o.end(),line.begin()+half);
}

std::vector<double> lift97_inverse(const std::vector<double>& line) {
void lift97_inverse(std::vector<double>& line,std::vector<double>& e,std::vector<double>& o){
constexpr double a=-1.586134342, b=-0.05298011854, g=0.8829110762, d=0.4435068522, k=1.149604398;
const size_t half=(line.size()+1)/2, odds=line.size()/2;
std::vector<double> e(line.begin(),line.begin()+half),o(line.begin()+half,line.end()),out(line.size());
const size_t half=(line.size()+1)/2, odds=line.size()/2, om=odds?odds-1:0;
e.resize(half); o.resize(odds?odds:1); if(!odds)o[0]=0;
for(size_t i=0;i<half;++i)e[i]=line[i]; for(size_t i=0;i<odds;++i)o[i]=line[half+i];
for(auto& x:e)x/=k; for(auto& x:o)x*=k;
for(size_t i=0;i<half;++i)e[i]-=d*(o[i?i-1:0]+o[std::min(i,odds-1)]);
for(size_t i=0;i<half;++i)e[i]-=d*(o[i?i-1:0]+o[std::min(i,om)]);
for(size_t i=0;i<odds;++i)o[i]-=g*(e[i]+e[std::min(i+1,half-1)]);
for(size_t i=0;i<half;++i)e[i]-=b*(o[i?i-1:0]+o[std::min(i,odds-1)]);
for(size_t i=0;i<half;++i)e[i]-=b*(o[i?i-1:0]+o[std::min(i,om)]);
for(size_t i=0;i<odds;++i)o[i]-=a*(e[i]+e[std::min(i+1,half-1)]);
for(size_t i=0;i<half;++i)out[i*2]=e[i]; for(size_t i=0;i<odds;++i)out[i*2+1]=o[i]; return out;
for(size_t i=0;i<half;++i)line[i*2]=e[i]; for(size_t i=0;i<odds;++i)line[i*2+1]=o[i];
}

std::vector<double> lift53_forward(const std::vector<double>& line) {
const size_t half=(line.size()+1)/2, odds=line.size()/2;
std::vector<int64_t> e(half),o(odds); std::vector<double> out(line.size());
void lift53_forward(std::vector<double>& line,std::vector<int64_t>& e,std::vector<int64_t>& o){
const size_t half=(line.size()+1)/2, odds=line.size()/2, om=odds?odds-1:0;
e.resize(half); o.resize(odds?odds:1); if(!odds)o[0]=0;
for(size_t i=0;i<half;++i)e[i]=static_cast<int64_t>(line[i*2]); for(size_t i=0;i<odds;++i)o[i]=static_cast<int64_t>(line[i*2+1]);
for(size_t i=0;i<odds;++i)o[i]-=floor_div(e[i]+e[std::min(i+1,half-1)],2);
for(size_t i=0;i<half;++i)e[i]+=floor_div(o[i?i-1:0]+o[std::min(i,odds-1)]+2,4);
for(size_t i=0;i<half;++i)out[i]=static_cast<double>(e[i]); for(size_t i=0;i<odds;++i)out[half+i]=static_cast<double>(o[i]); return out;
for(size_t i=0;i<half;++i)e[i]+=floor_div(o[i?i-1:0]+o[std::min(i,om)]+2,4);
for(size_t i=0;i<half;++i)line[i]=static_cast<double>(e[i]); for(size_t i=0;i<odds;++i)line[half+i]=static_cast<double>(o[i]);
}

std::vector<double> lift53_inverse(const std::vector<double>& line) {
const size_t half=(line.size()+1)/2, odds=line.size()/2;
std::vector<int64_t> e(half),o(odds); std::vector<double> out(line.size());
void lift53_inverse(std::vector<double>& line,std::vector<int64_t>& e,std::vector<int64_t>& o){
const size_t half=(line.size()+1)/2, odds=line.size()/2, om=odds?odds-1:0;
e.resize(half); o.resize(odds?odds:1); if(!odds)o[0]=0;
for(size_t i=0;i<half;++i)e[i]=static_cast<int64_t>(line[i]); for(size_t i=0;i<odds;++i)o[i]=static_cast<int64_t>(line[half+i]);
for(size_t i=0;i<half;++i)e[i]-=floor_div(o[i?i-1:0]+o[std::min(i,odds-1)]+2,4);
for(size_t i=0;i<half;++i)e[i]-=floor_div(o[i?i-1:0]+o[std::min(i,om)]+2,4);
for(size_t i=0;i<odds;++i)o[i]+=floor_div(e[i]+e[std::min(i+1,half-1)],2);
for(size_t i=0;i<half;++i)out[i*2]=static_cast<double>(e[i]); for(size_t i=0;i<odds;++i)out[i*2+1]=static_cast<double>(o[i]); return out;
for(size_t i=0;i<half;++i)line[i*2]=static_cast<double>(e[i]); for(size_t i=0;i<odds;++i)line[i*2+1]=static_cast<double>(o[i]);
}

} // namespace
Expand Down Expand Up @@ -178,13 +182,21 @@ std::vector<uint8_t> decode_palette(const uint8_t* data,size_t size,uint32_t w,u

std::vector<int64_t> wavelet_forward(const uint8_t* data,uint32_t w,uint32_t h,uint8_t bps,bool rev,unsigned levels,double q){
if(!data||!w||!h||!q||levels>8)throw std::invalid_argument("invalid wavelet input");std::vector<double>a(static_cast<size_t>(w)*h);for(size_t i=0;i<a.size();++i)a[i]=bps==1?data[i]:data[i*2]|static_cast<uint16_t>(data[i*2+1])<<8;uint32_t rw=w,rh=h;
for(unsigned level=0;level<levels;++level){for(uint32_t y=0;y<rh;++y){std::vector<double>line(a.begin()+y*w,a.begin()+y*w+rw);line=rev?lift53_forward(line):lift97_forward(line);std::copy(line.begin(),line.end(),a.begin()+y*w);}for(uint32_t x=0;x<rw;++x){std::vector<double>line(rh);for(uint32_t y=0;y<rh;++y)line[y]=a[y*w+x];line=rev?lift53_forward(line):lift97_forward(line);for(uint32_t y=0;y<rh;++y)a[y*w+x]=line[y];}rw=(rw+1)/2;rh=(rh+1)/2;}
// Scratch buffers are allocated once at the largest size; every later
// resize shrinks within existing capacity, so the lifting loop performs
// no per-line heap allocations. resize() always precedes the copy into
// `line` - the original B1 attempt copied first and overflowed the
// buffer whenever a later level's row width exceeded a shrunk size
// (any non-square tile).
std::vector<double> line,e97,o97;std::vector<int64_t> e53,o53;
for(unsigned level=0;level<levels;++level){for(uint32_t y=0;y<rh;++y){line.resize(rw);std::copy(a.begin()+static_cast<size_t>(y)*w,a.begin()+static_cast<size_t>(y)*w+rw,line.begin());if(rev)lift53_forward(line,e53,o53);else lift97_forward(line,e97,o97);std::copy(line.begin(),line.begin()+rw,a.begin()+static_cast<size_t>(y)*w);}for(uint32_t x=0;x<rw;++x){line.resize(rh);for(uint32_t y=0;y<rh;++y)line[y]=a[static_cast<size_t>(y)*w+x];if(rev)lift53_forward(line,e53,o53);else lift97_forward(line,e97,o97);for(uint32_t y=0;y<rh;++y)a[static_cast<size_t>(y)*w+x]=line[y];}rw=(rw+1)/2;rh=(rh+1)/2;}
std::vector<int64_t>out(a.size());for(size_t i=0;i<a.size();++i)out[i]=std::llround(a[i]/q);return out;
}

std::vector<uint8_t> wavelet_inverse(const int64_t* coeff,size_t count,uint32_t w,uint32_t h,uint8_t bps,bool rev,unsigned levels,double q){
if(count!=static_cast<size_t>(w)*h)throw std::invalid_argument("invalid coefficient count");std::vector<double>a(count);for(size_t i=0;i<count;++i)a[i]=static_cast<double>(coeff[i])*q;
for(int level=static_cast<int>(levels)-1;level>=0;--level){const uint32_t rw=(w+(1u<<level)-1)>>level,rh=(h+(1u<<level)-1)>>level;for(uint32_t x=0;x<rw;++x){std::vector<double>line(rh);for(uint32_t y=0;y<rh;++y)line[y]=a[y*w+x];line=rev?lift53_inverse(line):lift97_inverse(line);for(uint32_t y=0;y<rh;++y)a[y*w+x]=line[y];}for(uint32_t y=0;y<rh;++y){std::vector<double>line(a.begin()+y*w,a.begin()+y*w+rw);line=rev?lift53_inverse(line):lift97_inverse(line);std::copy(line.begin(),line.end(),a.begin()+y*w);}}
std::vector<double> line,e97,o97;std::vector<int64_t> e53,o53;
for(int level=static_cast<int>(levels)-1;level>=0;--level){const uint32_t rw=(w+(1u<<level)-1)>>level,rh=(h+(1u<<level)-1)>>level;for(uint32_t x=0;x<rw;++x){line.resize(rh);for(uint32_t y=0;y<rh;++y)line[y]=a[static_cast<size_t>(y)*w+x];if(rev)lift53_inverse(line,e53,o53);else lift97_inverse(line,e97,o97);for(uint32_t y=0;y<rh;++y)a[static_cast<size_t>(y)*w+x]=line[y];}for(uint32_t y=0;y<rh;++y){line.resize(rw);std::copy(a.begin()+static_cast<size_t>(y)*w,a.begin()+static_cast<size_t>(y)*w+rw,line.begin());if(rev)lift53_inverse(line,e53,o53);else lift97_inverse(line,e97,o97);std::copy(line.begin(),line.begin()+rw,a.begin()+static_cast<size_t>(y)*w);}}
const uint32_t max=bps==1?255u:65535u;std::vector<uint8_t>out(count*bps);for(size_t i=0;i<count;++i){const uint32_t v=static_cast<uint32_t>(std::clamp<int64_t>(std::llround(a[i]),0,max));out[i*bps]=static_cast<uint8_t>(v);if(bps==2)out[i*bps+1]=static_cast<uint8_t>(v>>8);}return out;
}

Expand Down Expand Up @@ -415,6 +427,12 @@ std::vector<int64_t> unpack_coefficients_v2(const uint8_t* data, size_t size, si
return output;
}

// ---- A3 stage 2: adaptive binary range coder for wavelet coefficients ----
// Replaces varint+zstd with a single-pass context-modeled arithmetic coder.
// Structure adapted from LZMA's range coder (public domain) with adaptive
// 11-bit probability models (shift-5 counter update).


uint32_t next_power_of_two(uint32_t value) {
uint32_t output = 1;
while (output < std::max(2u, value)) output <<= 1;
Expand Down
32 changes: 30 additions & 2 deletions src/v2_simd.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ constexpr CrcSlices kCrcSlices{};
struct Features {
bool avx2 = false;
bool hardware_crc32 = false;
bool pclmul = false;
};

#if defined(WIMF_AVX2_KERNELS)
Expand All @@ -78,12 +79,26 @@ bool detect_avx2() noexcept {
#endif
}

bool detect_pclmul() noexcept {
#if defined(__GNUC__) || defined(__clang__)
__builtin_cpu_init();
return __builtin_cpu_supports("pclmul") != 0;
#elif defined(_MSC_VER)
int registers[4] = {0, 0, 0, 0};
__cpuid(registers, 1);
return (registers[2] & (1 << 1)) != 0; // ECX bit 1: PCLMULQDQ.
#else
return false;
#endif
}

#endif // WIMF_AVX2_KERNELS

Features detect_features() noexcept {
Features features;
#if defined(WIMF_AVX2_KERNELS)
features.avx2 = detect_avx2();
features.pclmul = detect_pclmul();
#endif
#if defined(WIMF_NEON)
features.hardware_crc32 = crc32_hw::supported();
Expand All @@ -102,7 +117,10 @@ bool has_avx2() noexcept { return features().avx2; }
bool has_hardware_crc32() noexcept { return features().hardware_crc32; }

uint32_t crc32_table(const uint8_t* data, size_t size) noexcept {
uint32_t crc = 0xFFFFFFFFu;
return crc32_table_update(0xFFFFFFFFu, data, size);
}

uint32_t crc32_table_update(uint32_t crc, const uint8_t* data, size_t size) noexcept {
// Slice-by-8 main loop: fold eight bytes per iteration through the
// precomputed slice tables. Byte order matches the little-endian load of
// the first four bytes into the running CRC.
Expand Down Expand Up @@ -160,7 +178,17 @@ void left_filter_emit(const uint8_t* row, uint8_t* out, size_t width) noexcept {

#elif defined(WIMF_AVX2_KERNELS)

uint32_t crc32(const uint8_t* data, size_t size) noexcept { return crc32_table(data, size); }
uint32_t crc32(const uint8_t* data, size_t size) noexcept {
// PCLMULQDQ folded CRC when the CPU has it (every AVX2 CPU does, but the
// flag is checked explicitly); slice-by-8 table otherwise. The folded
// kernel consumes full 16-byte blocks and the table chains the tail.
if (features().avx2 && features().pclmul) {
uint32_t crc = 0xFFFFFFFFu;
const size_t processed = avx2::crc32_pclmul(&crc, data, size);
return crc32_table_update(crc, data + processed, size - processed);
}
return crc32_table(data, size);
}

uint64_t left_filter_cost(const uint8_t* row, size_t width) noexcept {
return features().avx2 ? avx2::left_filter_cost(row, width)
Expand Down
7 changes: 7 additions & 0 deletions src/v2_simd.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ namespace wimf::v2::simd {
// CPU supports the instruction set.
bool has_avx2() noexcept;
bool has_hardware_crc32() noexcept;
bool has_pclmul() noexcept;

// Scalar reference kernels; also the universal fallback.
namespace scalar {
Expand All @@ -40,12 +41,18 @@ void left_filter_emit(const uint8_t* row, uint8_t* out, size_t width) noexcept;

// Lookup-table CRC-32 (IEEE 802.3, reflected, init/final XOR 0xFFFFFFFF).
uint32_t crc32_table(const uint8_t* data, size_t size) noexcept;
// Chaining variant: continues from an internal-state crc (pre-final-XOR).
uint32_t crc32_table_update(uint32_t crc, const uint8_t* data, size_t size) noexcept;

#if defined(WIMF_AVX2_KERNELS)
// Requires has_avx2() to be true before use.
namespace avx2 {
uint64_t left_filter_cost(const uint8_t* row, size_t width) noexcept;
void left_filter_emit(const uint8_t* row, uint8_t* out, size_t width) noexcept;
// PCLMULQDQ folded CRC-32 (IEEE 802.3, reflected). Requires has_avx2() and
// has_pclmul(); processes floor(size/16)*16 bytes and returns how many were
// consumed - chain the remainder through crc32_table_update.
size_t crc32_pclmul(uint32_t* crc, const uint8_t* data, size_t size) noexcept;
} // namespace avx2
#endif

Expand Down
Loading
Loading