Legacy 64-bit dictionary layout is misdetected on big-endian platforms (v1.4.x regression) #1471

The current opencc v1.4.1 release fails to build from source when compiling for Debian on big-endian platforms. You may visit https://buildd.debian.org/status/package.php?p=opencc&suite=experimental for details.

The following analysis are written by Claude Code for your reference. I don't guarantee its correctness.

What happens

The BinaryDict and DartsDict deserializers accept two on-disk layouts: the
current fixed-width one, where every integer field is a uint32_t, and the
legacy one written by old 64-bit builds, where the fields were native 8-byte
size_t / id_type. Both readers tell the two apart by looking at the first
8 bytes of the integer section and checking whether the high 32 bits of that
field are zero — true for the legacy uint64_t field, false for a valid
fixed-width file.

The problem is how that check is written. In src/BinaryDict.cpp:

const bool highBytesZero =
    data[4] == 0 && data[5] == 0 && data[6] == 0 && data[7] == 0;

and the same idea twice in src/DartsDict.cpp (in NewFromFile and in
NewFromBuffer):

bool is64bit =
    (probe[4] == 0 && probe[5] == 0 && probe[6] == 0 && probe[7] == 0);

Bytes [4..7] are the high half of a native uint64_t only on little-endian
machines. The legacy files were written in native byte order, so on a
big-endian host the high half sits at bytes [0..3] instead, and the check
always says "not legacy". A legacy file is then read as fixed-width, the
first uint32_t picked up is the zero high half, and so numItems (or
dartsSize) comes out as 0. The dictionary loads as empty instead of either
loading correctly or throwing InvalidFormat.

Where it came from

Three commits are involved, all of them introducing or copying the same
byte-offset idiom:

• bd177d1 (Fix id_type to uint32_t in vendored darts.h; add legacy 64-bit OCD reader with full validation #1373, "Fix id_type to uint32_t in vendored darts.h; add legacy
  64-bit OCD reader with full validation", 2026-06-30) added the probe in
  DartsDict::NewFromFile, together with all three of the failing DartsDict
  tests. This is where the regression starts.
• 18c09df (Keep Darts support always enabled #1374, "Keep Darts support always enabled", 2026-06-30) added
  DartsDict::NewFromBuffer and carried the same check over to it.
• 58c1836 (Serialize BinaryDict with fixed-width fields for word-size-independent OCD output #1414, "Serialize BinaryDict with fixed-width fields for
  word-size-independent OCD output", 2026-07-08) added DetectFieldWidth to
  BinaryDict with the same idiom, added AcceptsLegacy64BitLayout, and
  extended RejectsHugeValueTotalLength to cover the legacy layout too.

Before #1373 there was no dual-layout detection at all — the files were
written and read with native fields throughout, so there is nothing in
DartsDict.cpp or DartsDictTest.cpp at bd177d1~1 that mentions the legacy
path.

How it showed up

I ran into this packaging 1.4.1 for Debian, which builds on a fairly wide set
of architectures. The split is exactly along byte order:

• failing: s390x, powerpc, ppc64, sparc64, hppa (all big-endian)
• passing: amd64, arm64, armhf, i386, loong64, ppc64el (all little-endian)

Five tests fail, all of them the ones that exercise the legacy layout:

BinaryDictTest.AcceptsLegacy64BitLayout
BinaryDictTest.RejectsHugeValueTotalLength
DartsDictTest.LoadsLegacy64bitFormat
DartsDictTest.Legacy64PrefixSearchBeyond64Matches
DartsDictTest.Legacy64SerializeToFileThrows

From the s390x build log:

[ RUN      ] BinaryDictTest.AcceptsLegacy64BitLayout
./src/BinaryDictTest.cpp:198: Failure
Expected equality of these values:
  deserialized->GetLexicon()->Length()
    Which is: 0
  1

[ RUN      ] BinaryDictTest.RejectsHugeValueTotalLength
./src/BinaryDictTest.cpp:151: Failure
Expected: SerializableDict::NewFromFile<BinaryDict>(path) throws an exception
of type InvalidFormat.
  Actual: it throws nothing.

The DartsDict ones fail the same way, just reported as "Unknown C++ exception
thrown in the test body" since the empty lexicon trips an out-of-range access
later on.

The full logs are at
https://buildd.debian.org/status/package.php?p=opencc&suite=experimental if
they are useful.

Suggested fix

Load the 8 bytes as a native uint64_t and shift, instead of naming byte
offsets. That is correct on both byte orders, and on little-endian it compiles
to the same thing as the current code:

uint64_t probeWord;
memcpy(&probeWord, probe, sizeof(probeWord));
bool is64bit = (probeWord >> 32) == 0;

Applied to all three sites. As far as I can tell nothing else needs changing —
the rest of the legacy paths (ReadField with an 8-byte width, LegacyUnit64,
ValidateLegacy64) already do native loads of natively written data, so they
are fine on both byte orders.

One thing worth checking on your side: on big-endian the new test degenerates
to "the first field is zero", i.e. numItems == 0 for BinaryDict. That only
holds for an empty dictionary, which is exactly 12 bytes and is already
excluded by the existing size != 3 * sizeof(uint32_t) guard, so I believe it
is safe. But it is a different failure mode from the little-endian side (where
the same test means keyTotalLength == 0), and you know the format history
better than I do.

I verified that this does not regress little-endian by running BinaryDictTest
and DartsDictTest on amd64 — 22/22 pass before and after. I have not been
able to run the patched code on a big-endian machine yet; I will follow up once
I have done that on a Debian porterbox.

Note on portability

This only fixes reading a legacy file on the same kind of host that wrote it.
Legacy .ocd files stay non-portable across byte orders by construction, since
the format is native-endian in both word size and byte order. The default
.ocd2 (marisa) format is not affected by any of this.

Version

OpenCC 1.4.1 as well as git HEAD (6b1538b), Debian Testing/Sid.