Issue: _decode_langid SUBLANG table is structurally incorrect for multilingual binaries

[malx-labs]

Issue: _decode_langid SUBLANG table is structurally incorrect for multilingual binaries

Status: Open, deferred from requirement 4 Severity: Moderate (incorrect output for multilingual edge cases; correct for the common English case) Component: iocx.parsers.pe_parser._decode_langid

Summary

The current _decode_langid implementation models SUBLANG values as language-independent (a flat SUBLANG table indexed by sublang value). This is structurally incorrect per the Windows LCID specification. A SUBLANG value's meaning depends on its associated primary language.

Concrete examples of the bug

Given the current SUBLANG table containing only {0x02: "GB"}:

LANGID	Primary lang	Sublang	Expected language_name	Current language_name
0x0809	English (0x09)	0x02	"en-GB"	"en-GB" ✓
0x0807	German (0x07)	0x02	"de-CH"	"de-GB" ✗
0x080A	Spanish (0x0A)	0x02	"es-MX"	"es-GB" ✗
0x080C	French (0x0C)	0x02	"fr-BE"	"fr-GB" ✗

The bug exists because the same sublang value (0x02) is mapped to:

• SUBLANG_ENGLISH_UK (UK) when primary is English
• SUBLANG_GERMAN_SWISS (Switzerland) when primary is German
• SUBLANG_SPANISH_MEXICAN (Mexico) when primary is Spanish
• SUBLANG_FRENCH_BELGIAN (Belgium) when primary is French

A flat SUBLANG table cannot represent these distinctions.

Why this was deferred from requirement 4

Requirement 4 specified "extract language and codepage" as structured metadata. The current implementation extracts both correctly — language is the raw LCID, language_name is a decoded string. The SUBLANG correctness issue affects the quality of the decoded string, not the structural correctness of the metadata extraction.

The deferral was a deliberate choice to:

1. Land requirement 4 on its bounded scope without expanding into LCID-table rework.
2. Track the SUBLANG fix as a separately reviewable change with its own tests and migration notes.
3. Avoid blocking the requirement 4 closure on a non-trivial table refresh that affects snapshot output across multiple fixtures.

Proposed fix

Replace the (primary, sublang) decomposition with a flat LCID → BCP-47 mapping for the common cases. This is the pattern used in iocx.projections.version_info._LCID_NAMES:

_LCID_NAMES = {

    0x0409: "en-US",

    0x0809: "en-GB",

    0x0407: "de-DE",

    0x0807: "de-CH",

    0x040A: "es-ES",

    0x080A: "es-MX",

    0x040C: "fr-FR",

    0x080C: "fr-BE",

    # ... extended to cover common locales

}

def _decode_langid(langid):

    if not isinstance(langid, int):

        return None

    name = _LCID_NAMES.get(langid)

    if name:

        return name

    # Fallback: primary-language only for unmapped LCIDs

    primary = langid & 0x3FF

    return PRIMARY_LANG.get(primary)

This approach:

• Avoids the cross-language sublang ambiguity entirely.
• Is easier to maintain — each entry is independently correct.
• Matches the pattern already established in project_version_info.
• Returns None for unmapped LCIDs cleanly.

Migration impact

Snapshot refresh required for any fixture whose resources use sublang values where the current implementation produces incorrect output. Likely affects:

• Any fixture with German Swiss, Spanish Mexican, French Belgian, or other multilingual sublang resources.
• Likely zero or minimal impact on existing fixtures, since most fixtures use auto-generated manifests with LANGID 0x0409 (en-US) or 0x0000 (neutral).

A blanket scan of existing fixtures for non-English LCIDs would identify the affected set before the fix is applied.

Acceptance criteria

1. _decode_langid(0x0807) returns "de-CH" (not "de-GB").
2. _decode_langid(0x080A) returns "es-MX".
3. _decode_langid(0x080C) returns "fr-BE".
4. _decode_langid(0x0409) returns "en-US" (regression guard for the common case).
5. _decode_langid(0x0999) (unmapped primary, unmapped sublang) returns None.
6. New unit tests pin each of the above behaviours.
7. Snapshot diffs reviewed and fixtures updated as needed.

Related context

• The flat-mapping pattern is the same approach used by Python's babel.locale.Locale and by Microsoft's own LCIDToLocaleName API. The (primary, sublang) decomposition is a Win32-era abstraction that predates BCP-47 by a decade.
• The current DEFAULT_REGION table is a partial workaround that handles a few common cases via primary-language defaults. It would be superseded by the flat mapping.
• The issue is structurally identical to the language-display issue we discussed during the version-info validator work; the same flat-mapping table can serve both modules (worth extracting to a shared location if both modules need it).

Out of scope for this issue

• Wholesale Win32 LCID → BCP-47 mapping coverage. The fix focuses on the locales most likely to appear in real binaries (the common Western European and US/UK/Latin American sublangs). Comprehensive coverage of all ~700 Windows LCIDs is a separate concern with its own maintenance burden.
• LCID → IETF language tag refinement (handling neutral sublangs, deprecated subtags, etc.). The fix produces BCP-47-shaped strings but doesn't claim full BCP-47 conformance.