Fix catastrophic hash collisions in FreqTable::make_hash and vecHasher

[gvegayon]

Summary

DEFM::init() appeared to hang for models with many response columns (reported with a 16-column, Markov order-1 panel model built through the defm R package). Profiling showed 99.9% of the time inside a single unordered_map emplace called from FreqTable::add.

Root cause: std::hash of arithmetic types is typically the identity (a bit-cast for doubles), so hashing statistics that are small integers (0.0, 1.0, ...) yields values whose low ~50 bits are all zero. The boost-style combine in FreqTable::make_hash/vecHasher only propagates entropy toward the low bits ~2 bits per element, so for arrays with up to ~20 columns the low 16+ bits of the final hash are identical across all keys. libc++'s unordered_map masks the hash with a power-of-two bucket count — exactly those bits — so every key lands in one bucket (verified: all 65,536 distinct stat vectors of a 16-column support produced a single bucket with a 65,536-long chain). Each insert scans the whole chain, making Support::calc O(4^ncol) instead of O(2^ncol).

Fix: apply a splitmix64 finalizer to each element hash before combining, in both FreqTable::make_hash (freqtable.hpp) and vecHasher (typedefs.hpp — it backs keys2support in Model::add_array, which had the same degeneration). The top-level barry.hpp amalgam is regenerated.

Measurements

Benchmark	Before	After
Insert all 2^16 stat vectors of one 16-col support into FreqTable	2.37 s	0.015 s
defm::DEFM::init(), 16 cols, 1 unique Markov state	14.4 s	0.05 s
defm::DEFM::init(), 16 cols, 20 unique Markov states	>5 min (timed out)	~1.4 s

Collision hardening (third commit)

FreqTable::index keyed rows by the 64-bit hash alone, so two distinct stat vectors hashing to the same value would silently merge into one support row (birthday-bound probability, ~1e-10 for a 65,536-row support, but silent if it ever fired). add now verifies the stored row against the incoming vector on a hash hit — the stats are already in data, so no extra storage — and sends true collisions to a per-hash overflow chain that stays empty otherwise. size() returns the true row count n (with chains, index.size() would undercount); get_index() has no consumers, and the h_precomp path in Support is currently unreachable (hashes_initialized is never set to true), so no caller is affected.

Complexity is unchanged — the verify compare is O(k), which make_hash already costs, and misses (the common case) pay nothing new. Measured overhead ~5% on the worst-case benchmark above. A new unit test (tests/19-freqtable-collisions.cpp) forces collisions through the precomputed-hash argument and checks rows stay separate and clear() resets the chains.

Also in this PR

The second commit regenerates the stale top-level defm.hpp amalgam: running barry-hpp.R showed it was missing the multi-group formula parser changes (#19) already merged in include/barry/models/defm/. Purely mechanical output of the regeneration script — happy to drop it from this PR if you prefer it separate.

Testing

• make test-portable: all 122 assertions in 21 test cases pass
• Downstream check with the R stack: patched headers in barryr, rebuilt defm; its full tinytest suite (25 results) passes, and a synthetic 16-outcome panel model (mirroring the report) initializes and fits instead of hanging

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.04%. Comparing base (57b12fe) to head (fdaaf1a).

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #24      +/-   ##
==========================================
+ Coverage   88.88%   89.04%   +0.16%     
==========================================
  Files          46       46              
  Lines        3166     3204      +38     
==========================================
+ Hits         2814     2853      +39     
+ Misses        352      351       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gvegayon]

Thanks @copilot — addressed all five points in fdaaf1a:

1. 32-bit portability (both mix64 sites). The finalizer now operates on an explicit std::uint64_t and casts back to size_t only at the end, so the avalanche is well-defined and identical regardless of whether size_t is 32- or 64-bit. On 64-bit targets the produced hashes are byte-identical to the previous commit (verified: benchmark and all existing assertions unchanged). <cstdint> is already pulled in via barry.hpp.

2. Empty-vector UB in make_hash and vecHasher::operator(). Both now return a fixed value for empty input instead of reading element [0]. Added an assertion exercising make_hash on an empty vector.

3. lenght → length typo fixed in the FreqTable length_error message.

make test-portable now passes 123 assertions in 21 test cases, and defm rebuilt against these headers still passes its full tinytest suite (25 results).