diff --git a/CHANGELOG.md b/CHANGELOG.md index d9b0a86..f3afee5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,81 @@ +# **v0.7.6.1 — Exception Directory Validator, and a Silent Output Defect** +**Released: 2026‑08‑27** + +## Added +- **Exception directory validator (§2.15)** — deep semantic validation of the + PE exception (`.pdata`) directory: the x64 `RUNTIME_FUNCTION` table and its + `.xdata` `UNWIND_INFO` records, plus the ARM/ARM64 8-byte record walk. + Detects the loader-visible sortedness invariant, function-range validity, + adjacent-range overlap, RVA bounds, unwind version/flag anomalies, and + chained-unwind target faults. +- **`pe_exception` parser** — pure `struct`-level decoder over + `pe.get_data`-acquired bytes, independent of pefile's + `DIRECTORY_ENTRY_EXCEPTION` interpretation. Machine-gated to AMD64 / + ARM64 / ARM64EC / ARMNT; unsupported machines are reported once and the + walk is skipped. UNWIND_INFO V3 (APX preview) is recognised but not + deeply parsed. +- **14 exception reason codes** — `EXCEPTION_DIRECTORY_INVALID_HEADER`, + `_OUT_OF_BOUNDS`, `_UNALIGNED`, `_SIZE_NOT_MULTIPLE`, + `EXCEPTION_TABLE_TRUNCATED`, `EXCEPTION_UNSUPPORTED_MACHINE`, + `EXCEPTION_ENTRY_INVALID`, `EXCEPTION_FUNCTION_RANGE_INVALID`, + `_RVA_OUT_OF_BOUNDS`, `EXCEPTION_ENTRIES_NOT_SORTED`, + `EXCEPTION_FUNCTION_OVERLAP`, `EXCEPTION_UNWIND_INFO_UNALIGNED`, + `_INVALID`, `EXCEPTION_UNWIND_CHAIN_INVALID`. +- `ExceptionStruct` / `ExceptionFunctionEntry` / `ExceptionUnwindInfo` in + the internal schema. +- Reason-code contract regression suite + (`tests/unit/analysis/test_reason_codes.py`) pinning the emission + contract at both the source and output ends. +- Sub-reason taxonomies documented for nine previously undocumented codes. + +## Fixed +- **Validator `details` could overwrite the parent reason code.** The + emission layer merged `details` over its own `reason` field, so any + validator using a top-level `reason` key clobbered it. Eleven documented + reason codes had never appeared in output; consumers saw bare sub-reason + strings instead. Validators now use `sub_reason`, the parent code is + written last, and a legacy `reason` payload is re-keyed defensively. + **Output-visible.** +- **`SizeOfImage` was read from the wrong layer**, returning `None` in + production and silently disabling `EXPORT_DIRECTORY_OUT_OF_BOUNDS`, + `DELAY_IMPORT_DIRECTORY_OUT_OF_BOUNDS`, and the SizeOfImage fallback in + `DEBUG_ENTRY_RVA_INVALID` / `RELOCATION_ENTRY_RVA_INVALID`. Now sourced + from `metadata["optional_header"]` and threaded explicitly into + `_directory_invariants`. **Output-visible.** +- **`DATA_DIRECTORY_NOT_MAPPED_TO_SECTION` was suppressed for any file + carrying an overlay.** The `raw_offset is None` guard used a bare + `continue`, skipping the section-mapping checks entirely; it is now + scoped to the overlay check alone. **Output-visible.** +- **`RESOURCE_DIRECTORY_OUT_OF_BOUNDS` was declared and documented but + never emitted** — a directory outside `.rsrc` caused a silent `return`. + Now reported, with `depth` distinguishing the root case from a + subdirectory whose extent overflows the section end. **Output-visible.** +- `version_info` no longer raises on an analysis dict missing `sections`. +- Removed a dead `zero_length_sections` set in `rva_graph`. + +## Changed +- `validate_exports` and `validate_delay_imports` are now + `@depends_on("internal", "metadata")`; `validate_debug` and + `validate_relocations` are now + `@depends_on("internal", "metadata", "analysis")`. +- `rva_in_any_section` / `region_in_any_section` take an explicit optional + `size_of_image` argument. Existing two- and three-argument call sites keep + working. +- Tests: 1620 → 2136. Coverage held at 100%. + +## Documentation +- `reason-codes.md`: `reason` → `sub_reason` throughout; nine sub-reason + taxonomies added; six descriptions corrected against implementation + (`RESOURCE_ENTRY_OUT_OF_BOUNDS` is subdirectory-only; + `RESOURCE_DATA_OVERLAPS_OTHER_DATA` does not compare blobs to each other; + entropy uniformity and size thresholds stated precisely). +- `structural-validation-deterministic-heuristics.md`: §2.13 and §2.14 no + longer claim placement checks they do not perform, and no longer cite two + reason codes that do not exist. Placement ownership and the `sub_reason` + output contract are now stated explicitly. + +--- + # **v0.7.6 — Structural validator expansion: debug, relocations directories** **Released: 2026‑08‑10** diff --git a/README-pypi.md b/README-pypi.md index 3669890..46e266a 100644 --- a/README-pypi.md +++ b/README-pypi.md @@ -40,6 +40,13 @@ If you need predictable, automatable IOC extraction — IOCX is built for you. --- +### v0.7.6.1 — Exception Directory Validator + +- Adds deep semantic validation of the PE exception (`.pdata`) directory; 14 new reason codes; 15 validators total. +- Fixes a defect that had been suppressing structural findings across the engine. +- **Output-visible:** findings previously suppressed or mislabelled will now appear. +- Tests: 1620 → 2136. Coverage: 100%. + ## Version highlights (v0.7.6) - Added new PE structural validators for relocations and debug directories @@ -47,13 +54,6 @@ If you need predictable, automatable IOC extraction — IOCX is built for you. - Never crashes on malformed input - byte-level parsing with structured error tombstones - 1620 tests at 100% coverage - deterministic output, snapshot-stable -## Version highlights (v0.7.5) - -- Added detection for malformed exports, delay-load tables, resources, VS_VERSIONINFO, and Optional Header fields via 24 structural reason codes -- Surfaces security metadata — DLL characteristics flags, subsystem/machine decoding, per-resource Shannon entropy -- Never crashes on malformed input — byte-level parsing with structured error tombstones -- 1370 tests at 100% coverage — deterministic output, snapshot-stable, cross-verified against `dumpbin` - --- ## **Performance** diff --git a/README.md b/README.md index a66016e..ed6108f 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@

- + @@ -202,6 +202,16 @@ Fast path — no PE parsing.

Show Version History
+### v0.7.6.1 — Exception Directory Validator + +- Adds deep semantic validation of the PE exception (`.pdata`) directory; 14 new reason codes; 15 validators total. +- Fixes a defect that had been suppressing structural findings across the engine. +- Four further checks found to be dead in production: two directory placement, a section-mapping, and a resource-directory bounds check. +- **Output-visible:** findings previously suppressed or mislabelled will now appear. +- Tests: 1620 → 2136. Coverage: 100%. + +--- + ### **v0.7.6 — Structural Validator Expansion: Debug and relocations directories** - Two new PE structural validators - relocations and debug - WIN_CERTIFICATE and tls validators now source structural truth from dedicated struct parsers, independent of pefile diff --git a/docs/specs/reason-codes.md b/docs/specs/reason-codes.md index b082488..fb25391 100644 --- a/docs/specs/reason-codes.md +++ b/docs/specs/reason-codes.md @@ -1,5 +1,18 @@ # **PE Structural Reason Codes** +> **Truncation detail keys are not uniform.** `EXPORT_TABLE_TRUNCATED`, +> `DELAY_IMPORT_TABLE_TRUNCATED` and `EXCEPTION_TABLE_TRUNCATED` name the +> affected sub-table in a **`table`** key. `DEBUG_TABLE_TRUNCATED`, +> `RELOCATION_TABLE_TRUNCATED` and `TLS_DIRECTORY_TRUNCATED` use **`region`** +> instead. Both are stable; consumers handling truncation generically must +> read either. + +> **On the `sub_reason` field.** The key is deliberately not named `reason`: +> the heuristics layer emits the parent code under `reason`, and a details key +> of the same name would overwrite it. Sub-reasons are priority-resolved where +> noted — a single entry carrying several tags reports exactly one, the first +> in the documented order. + ## **SECTION ANOMALIES** | Reason Code | What Triggers It | Example Malformed Pattern | Scope | @@ -20,6 +33,18 @@ | **SECTION_DISCARDABLE_CODE** | Section is executable AND discardable | `.text` with `MEM_EXECUTE | MEM_DISCARDABLE` | Per‑section | | **SECTION_FLAGS_INCONSISTENT** | Contradictory flags: code/write/exec without read | `.text` with `EXECUTE` but missing `READ` | Per‑section | +## SECTION SUB‑REASONS + +### SECTION_FLAGS_INCONSISTENT + +Emitted once per violated combination, so one section may raise several: + +| Sub‑reason | Meaning | +|------------|---------| +| code_without_read | `CNT_CODE` set but `MEM_READ` absent | +| write_without_read | `MEM_WRITE` set but `MEM_READ` absent | +| exec_without_read | `MEM_EXECUTE` set but `MEM_READ` absent | + --- ## **ENTRYPOINT ANOMALIES** @@ -35,6 +60,17 @@ | **ENTRYPOINT_IN_NON_CODE_SECTION** | EP inside `.rsrc`, `.reloc`, or non‑code section | EP inside `.rsrc` | Per‑file | | **ENTRYPOINT_IN_DISCARDABLE_SECTION** | EP inside discardable section | EP inside `.upx0` with discardable flag | Per‑file | +## ENTRYPOINT SUB‑REASONS + +### ENTRYPOINT_IN_TRUNCATED_REGION + +Mutually exclusive — the zero-length case takes precedence: + +| Sub‑reason | Meaning | +|------------|---------| +| zero_length_section | The mapped section has `VirtualSize == 0` | +| beyond_virtual_size | EP lies at or past `VirtualAddress + VirtualSize` | + --- ## **OPTIONAL HEADER ANOMALIES** @@ -50,6 +86,36 @@ | **OPTIONAL_HEADER_INVALID_NUMBER_OF_RVA_AND_SIZES** | `NumDirs` < actual directories OR > 16 | `NumDirs = 1`, actual = 3 | Per‑file | | **OPTIONAL_HEADER_SIZE_OF_IMAGE_MISALIGNED** | `SizeOfImage % SectionAlignment != 0` | `SizeOfImage = 512`, `SectionAlignment = 4096` | Per‑file | +## OPTIONAL HEADER SUB‑REASONS + +### OPTIONAL_HEADER_INVALID_SECTION_ALIGNMENT + +| Sub‑reason | Meaning | +|------------|---------| +| not_power_of_two | `SectionAlignment` is not a power of two | +| *(none)* | `SectionAlignment < FileAlignment` — this branch carries no sub‑reason and is identified by the presence of a `file_alignment` key in details | + +### OPTIONAL_HEADER_INVALID_FILE_ALIGNMENT + +Both checks are independent, so a single value may raise both: + +| Sub‑reason | Meaning | +|------------|---------| +| not_power_of_two | `FileAlignment` is not a power of two | +| out_of_range | `FileAlignment` outside the recommended 512–65536 range | + +### Codes distinguished by detail keys rather than sub‑reasons + +Two optional‑header codes have multiple emission sites with no `sub_reason`. +They are separable by a discriminating key: + +| Reason Code | Branch | Discriminating key | +|-------------|--------|--------------------| +| OPTIONAL_HEADER_INVALID_SIZE_OF_HEADERS | misaligned to FileAlignment | `file_alignment` | +| OPTIONAL_HEADER_INVALID_SIZE_OF_HEADERS | below required minimum | `required_minimum` | +| OPTIONAL_HEADER_INVALID_NUMBER_OF_RVA_AND_SIZES | count outside 0–16 | *(absent)* | +| OPTIONAL_HEADER_INVALID_NUMBER_OF_RVA_AND_SIZES | count below actual directories | `actual_directories` | + --- ## **RVA / DIRECTORY ANOMALIES** @@ -64,10 +130,12 @@ | **DATA_DIRECTORY_OUT_OF_RANGE** | Directory extends beyond `SizeOfImage` | RVA = 0x5000, Size = 0x2000, SizeOfImage = 0x4000 | Per‑directory *(primary error, mapping suppressed)* | | **DATA_DIRECTORY_IN_OVERLAY** | Directory maps to a raw offset ≥ overlay start | RVA maps to raw offset 0x6000, overlay starts at 0x5800 | Per‑directory | | **DATA_DIRECTORY_RAW_MISMATCH** | Directory RVA maps into a section’s virtual range but the computed raw offset lies outside that section’s raw data | RVA=0x2500 maps to .text, but raw offset=0xC00 is outside .text raw range | Per‑directory | -| **DATA_DIRECTORY_NOT_MAPPED_TO_SECTION** | Directory is in range but does not fall inside any section | RVA = 0x9000, Size = 0x200, no section covers it | Per‑directory *(suppressed for empty, zero‑RVA, out‑of‑range, zero‑length‑section)* | +| **DATA_DIRECTORY_NOT_MAPPED_TO_SECTION** | Directory is in range but does not fall inside any section | RVA = 0x9000, Size = 0x200, no section covers it | Per‑directory *(suppressed for empty, zero‑RVA, zero‑size, out‑of‑range and zero‑length‑section directories)* | | **DATA_DIRECTORY_SPANS_MULTIPLE_SECTIONS** | Directory range overlaps more than one section | RVA = 0x1800, Size = 0x1000 spans .text → .rdata | Per‑directory | | **DATA_DIRECTORY_OVERLAP** | Two directories’ RVA ranges overlap | Import and IAT overlap | Global | -| **IMPORT_RVA_INVALID** | Import RVA does not map to a valid import table structure (import validator) | Import RVA = 0x9000 | Per‑directory | +| **IMPORT_RVA_INVALID** *coming soon* | Import RVA does not map to a valid import table structure (import validator) | Import RVA = 0x9000 | Per‑directory | + +> Prior to the `raw_offset` guard fix, `DATA_DIRECTORY_NOT_MAPPED_TO_SECTION` was additionally suppressed for any directory in a file carrying an overlay, because the raw-mapping guard skipped the section-mapping checks entirely. Files analysed before that fix may under-report it. --- @@ -75,8 +143,8 @@ | Reason Code | What Triggers It | Example Pattern | Scope | |------------|------------------|-----------------|--------| -| **TLS_CALLBACK_OUTSIDE_RANGE** | Callback RVA not within the TLS directory’s `start, end)` range | Callback = `0x5000`, TLS range = `0x4000–0x4100` | Per‑file | -| **[TLS_MULTIPLE_DIRECTORIES** | More than one TLS directory is present in the PE | Two `tls_directory` entries in `extended` | Per‑file | +| **TLS_CALLBACK_OUTSIDE_RANGE** | Callback RVA not within the TLS directory’s `(start, end)` range | Callback = `0x5000`, TLS range = `0x4000–0x4100` | Per‑file | +| **TLS_MULTIPLE_DIRECTORIES** | More than one TLS directory is present in the PE | Two `tls_directory` entries in `extended` | Per‑file | | **TLS_INVALID_RANGE** | TLS directory has `start >= end` (structurally impossible) | Start = `0x6000`, End = `0x6000` | Per‑file | | **TLS_ZERO_LENGTH_DIRECTORY** | TLS directory exists but `start == end` (zero‑length region) | Start = `0x7000`, End = `0x7000` | Per‑file | | **TLS_CALLBACKS_MISSING** | TLS directory is non‑empty but callback pointer is `0` | Start = `0x4000`, End = `0x4100`, Callbacks = `0` | Per‑file | @@ -85,6 +153,35 @@ | **TLS_CALLBACK_IN_HEADERS** | Callback RVA falls inside the PE headers (`< SizeOfHeaders`) | Callback = `0x200`, SizeOfHeaders = `0x600` | Per‑file | | **TLS_CALLBACK_IN_OVERLAY** | Callback RVA maps to a raw offset beyond the last section (overlay) | Raw offset = `0x1F000`, overlay starts at `0x1E000` | Per‑file | | **TLS_CALLBACK_ARRAY_NOT_TERMINATED** *(optional future rule)* | Callback array exists but is not 0‑terminated | Callback list ends with non‑zero RVA | Per‑file | +| **TLS_DIRECTORY_TRUNCATED** *(v0.7.6)* | `pe_tls` struct decoder failed: the IMAGE_TLS_DIRECTORY header could not be read, or the callback array was truncated or looping | Header short read, or callback walk hit the 4096 hard limit with no NULL terminator | Per‑file | +| **TLS_CALLBACK_RVA_INVALID** *(v0.7.6)* | A resolved callback target (`AddressOfCallBacks − ImageBase`) cannot form a valid RVA or does not map to any section | Callback VA yielding a negative RVA, or an RVA covered by no section | Per‑file | + +## TLS SUB‑REASONS + +### TLS_DIRECTORY_TRUNCATED + +| Sub‑reason | Meaning | +|------------|---------| +| header_decode | The fixed IMAGE_TLS_DIRECTORY could not be read or unpacked; unrecoverable, all later checks are skipped | +| callback_array | A parser truncation tag surfaced while walking the callback array (the `region` key names the tag) | + +### TLS_CALLBACK_RVA_INVALID + +Parser resolution tombstones (callback array unresolvable, `callbacks = []`): + +| Sub‑reason | Meaning | +|------------|---------| +| tls_image_base_unavailable | ImageBase unavailable, so VA → RVA conversion is impossible | +| tls_callbacks_va_below_image_base | `AddressOfCallBacks` lies below ImageBase | + +Per-target failures (one issue per target, capped at 16; the true count is +always in `invalid_callback_count`): + +| Sub‑reason | Meaning | +|------------|---------| +| image_base_unavailable | Callbacks were resolved but ImageBase is not an int | +| below_image_base | A callback VA lies below ImageBase, yielding a negative RVA | +| not_mapped | A resolved callback RVA falls inside no section | --- @@ -103,13 +200,31 @@ --- -## ** RESOURCE ANOMALIES** +## CERTIFICATE TABLE ANOMALIES + +*Added in v0.7.6. Raw structural truth from the `pe_certificates` struct-level decoder, which walks the WIN_CERTIFICATE array from the file bytes. `DATA_DIRECTORY[4].VirtualAddress` is treated as a **file offset**, not an RVA — the certificate table is appended to the file and never mapped into the image. Placement/overlap has a single owner here to avoid double-counting with the RVA-graph backbone; the signature symmetry checks above interpret trust facts on top.* + +| Reason Code | What Triggers It | Example Pattern | Scope | +|------------|------------------|-----------------|--------| +| **CERTIFICATE_OFFSET_INSIDE_IMAGE** | The certificate table's file offset falls **before** the on-disk end of any section (`offset < image_raw_end`), i.e. it is not genuinely appended after the mapped image | Table offset = 0x3000, but `.rsrc` raw data ends at 0x5000 | Per‑file | +| **CERTIFICATE_TABLE_MALFORMED** | Top-level decode failure, or a WIN_CERTIFICATE entry surfaced a truncation tag (`sub_reason: "truncation"`) — e.g. `dwLength` runs past the file end, or the 8-byte header could not be read on the QWORD entry alignment | `dwLength` claims 0x900 bytes but only 0x40 remain in the file | Per‑file / Per‑certificate | + +### CERTIFICATE_TABLE_MALFORMED sub‑reasons + +| Sub‑reason | Meaning | +|------------|---------| +| top_level_decode | The parser could not decode the security directory at all; short-circuits before every other signature check | +| truncation | A WIN_CERTIFICATE entry surfaced a truncation tag (the `region` key names it) | + +--- + +## **RESOURCE ANOMALIES** ### **Resource Directory Anomalies** | Reason Code | What Triggers It | Example Pattern | Scope | |------------|------------------|-----------------|--------| -| **RESOURCE_DIRECTORY_OUT_OF_BOUNDS** | A resource directory RVA/size lies outside the `.rsrc` section or outside `SizeOfImage` | Directory RVA = `0x90000000`, `.rsrc` ends at `0x400000` | Per‑file | +| **RESOURCE_DIRECTORY_OUT_OF_BOUNDS** | A resource directory's `rva + size` does not lie wholly inside the `.rsrc` section. Two cases reach this: the **root** directory lies outside `.rsrc` (`depth` = 0), or a **subdirectory** starts inside `.rsrc` but its extent overflows the end (`depth` ≥ 1). A subdirectory lying wholly outside is reported by the parent as `RESOURCE_ENTRY_OUT_OF_BOUNDS` instead, so the two never double-count. `SizeOfImage` is not consulted — `.rsrc` bounds are authoritative here | Root directory RVA = `0x90000000` while `.rsrc` spans `0x1000–0x3000`; or a Name directory at `0x2FF8` with size 24 | Per‑directory | | **RESOURCE_DIRECTORY_LOOP** | Recursive directory traversal detects a cycle (malformed or malicious resource tree) | Directory A → B → A | Per‑file | | **RESOURCE_DIRECTORY_ZERO_LENGTH** | A resource directory exists but has zero length or no valid entries | RVA = `0x3000`, size = `0` | Per‑file | @@ -123,9 +238,9 @@ | Reason Code | What Triggers It | Example Pattern | Scope | |------------|------------------|-----------------|--------| -| **RESOURCE_ENTRY_OUT_OF_BOUNDS** | A resource entry points to a data entry outside the `.rsrc` section or outside `SizeOfImage` | Entry RVA = `0x80000000` | Per‑file | +| **RESOURCE_ENTRY_OUT_OF_BOUNDS** | A resource directory entry points to a **subdirectory** whose RVA lies outside the `.rsrc` section. Out-of-bounds *data* entries are reported as `RESOURCE_DATA_OUT_OF_BOUNDS`, not here. The target's own size is not considered at this point — a subdirectory that starts inside `.rsrc` but overflows the end is caught by `RESOURCE_DIRECTORY_OUT_OF_BOUNDS` when it is descended into | Type directory entry points to a Name directory at RVA `0x80000000` | Per‑file | | **RESOURCE_DATA_OUT_OF_BOUNDS** | Resource data block lies outside the file or outside the `.rsrc` section | Data offset = `0x1F0000`, file size = `0x1E0000` | Per‑file | -| **RESOURCE_DATA_OVERLAPS_OTHER_DATA** | Two resource data blobs overlap in raw or virtual space | Data A: `0x2000–0x2400`, Data B: `0x2300–0x2500` | Per‑file | +| **RESOURCE_DATA_OVERLAPS_OTHER_DATA** | A resource data blob spans the overlay start, or its raw or virtual extent intersects a section other than `.rsrc`. Blob-versus-blob comparison is **not** performed | Data at raw `0x2000–0x2400` intersects `.text` raw range | Per-file *(one issue per check; the raw-overlap and VA-overlap loops each stop at the first intersecting section, so a blob crossing several sections reports once per check, not once per section)* | ### Resource Version‑Info Anomalies @@ -138,6 +253,27 @@ *Note: absence of an RT_VERSION resource is not treated as a structural anomaly — many legitimate binary types (kernel drivers, MSI helpers, cross‑compiled artefacts) omit version‑info entirely.* +### RESOURCE_VERSIONINFO_INVALID_HEADER sub‑reasons + +| Sub‑reason | Meaning | +|------------|---------| +| placement | The VS_VERSIONINFO blob does not lie wholly inside `.rsrc` | +| undecoded | The parser could not decode the envelope; short-circuits the FIXEDINFO / STRINGFILEINFO / VARFILEINFO checks | +| szkey_mismatch | `szKey` is not "VS_VERSION_INFO" | +| length_inconsistent | `wLength` disagrees with the buffer size | + +### RESOURCE_VERSIONINFO_INVALID_FIXEDINFO sub‑reasons + +| Sub‑reason | Meaning | +|------------|---------| +| parse_failed | VS_FIXEDFILEINFO is absent AND the parser recorded a `fixed_file_info*` error. A legitimate omission (`wValueLength == 0`) is not flagged | +| signature | `dwSignature` is not `0xFEEF04BD` | +| struct_version | `dwStrucVersion` is not `0x00010000` | + +`RESOURCE_VERSIONINFO_INVALID_STRINGFILEINFO` and +`RESOURCE_VERSIONINFO_INVALID_VARFILEINFO` carry no sub‑reason; the parser's +tags are passed through verbatim in an `errors` list. + ### **Resource String‑Table Anomalies** | Reason Code | What Triggers It | Example Pattern | Scope | @@ -152,13 +288,13 @@ |------------|------------------|-----------------|--------| | **ENTROPY_HIGH_SECTION** | Section entropy ≥ 7.5 and size ≥ 1 KB | `.text` entropy = 7.9 | Per‑section | | **ENTROPY_HIGH_OVERLAY** | Overlay entropy ≥ 7.5 and size ≥ 1 KB | Overlay entropy = 7.8 | Per‑file | -| **ENTROPY_UNIFORM_ACROSS_SECTIONS** | All sections have high entropy with very low variance | Mean = 7.7, stddev = 0.05 | Per‑file | -| **ENTROPY_VERY_LOW_SECTION** | Large section with entropy ≤ 0.2 (zero‑filled / padding abuse) | `.data` entropy = 0.03 | Per‑section | -| **ENTROPY_HIGH_RESOURCES** | Resource directory entropy ≥ 7.5 | `.rsrc` entropy = 7.9 | Per‑region | -| **ENTROPY_HIGH_RELOCATIONS** | Relocation table entropy ≥ 7.5 | `.reloc` entropy = 7.8 | Per‑region | -| **ENTROPY_HIGH_IMPORTS** | Import table entropy ≥ 7.5 | Import blob entropy = 7.7 | Per‑region | -| **ENTROPY_HIGH_TLS** | TLS directory entropy ≥ 7.5 | TLS entropy = 7.9 | Per‑region | -| **ENTROPY_HIGH_CERTIFICATE** | Certificate blob entropy ≥ 7.5 | WIN_CERTIFICATE entropy = 7.8 | Per‑region | +| **ENTROPY_UNIFORM_ACROSS_SECTIONS** | Mean entropy ≥ 7.5 **and** standard deviation ≤ 0.15, computed across sections whose raw size is ≥ 1 KB. Requires at least two such sections; smaller sections are excluded from the sample and can neither trigger nor prevent the finding. A single low-entropy section drags the mean below the threshold, so the check short-circuits before variance is considered | Mean = 7.7, stddev = 0.05 | Per‑file | +| **ENTROPY_VERY_LOW_SECTION** | Section entropy ≤ 0.2 **and** raw size ≥ 16 KB. The size floor is deliberately far higher than the 1 KB used by the high-entropy checks, to avoid flagging ordinary small padding sections | `.data` entropy = 0.03 | Per‑section | +| **ENTROPY_HIGH_RESOURCES** | Resource directory entropy ≥ 7.5 and region size ≥ 1 KB | `.rsrc` entropy = 7.9 | Per‑region | +| **ENTROPY_HIGH_RELOCATIONS** | Relocation table entropy ≥ 7.5 and region size ≥ 1 KB | `.reloc` entropy = 7.8 | Per‑region | +| **ENTROPY_HIGH_IMPORTS** | Import table entropy ≥ 7.5 and region size ≥ 1 KB | Import blob entropy = 7.7 | Per‑region | +| **ENTROPY_HIGH_TLS** | TLS directory entropy ≥ 7.5 and region size ≥ 1 KB | TLS entropy = 7.9 | Per‑region | +| **ENTROPY_HIGH_CERTIFICATE** | Certificate blob entropy ≥ 7.5 and region size ≥ 1 KB | WIN_CERTIFICATE entropy = 7.8 | Per‑region | --- @@ -173,6 +309,28 @@ | **LOAD_CONFIG_COOKIE_IN_OVERLAY** | Security cookie maps to a raw offset ≥ overlay start | Cookie raw offset = 0x6000, overlay starts at 0x5800 | Per‑directory | | **LOAD_CONFIG_SEH_INVALID** | SEH table is missing, unmapped, out of range, or overlaps overlay; or SEHCount > 0 but SEHTableRVA = 0 | SEHCount = 4, SEHTableRVA = 0 | Per‑directory | +## LOAD CONFIG SUB‑REASONS + +Note `unmapped` is emitted by **both** codes below. A consumer must pair the +sub‑reason with its parent code to identify which check fired — the two are +not distinguishable by sub‑reason alone. + +### LOAD_CONFIG_COOKIE_INVALID + +| Sub‑reason | Meaning | +|------------|---------| +| unmapped | The security-cookie RVA maps to no section | +| non_writable_section | The cookie maps to a section without `MEM_WRITE` | + +### LOAD_CONFIG_SEH_INVALID + +| Sub‑reason | Meaning | +|------------|---------| +| missing_table_rva | `SEHCount > 0` but `SEHTableRVA` is absent or zero | +| out_of_range | `SEHTableRVA + (SEHCount × 4)` exceeds SizeOfImage | +| unmapped | The SEH table RVA maps to no section | +| in_overlay | The SEH table's raw offset lies at or past the overlay start | + --- ## EXPORT ANOMALIES @@ -203,7 +361,7 @@ ## EXPORT SUB‑REASONS -Several export reason codes carry a reason field in their details payload that narrows the pathology. The full taxonomy: +Several export reason codes carry a `sub_reason` field in their details payload that narrows the pathology. The full taxonomy: ### EXPORT_DIRECTORY_INVALID_HEADER @@ -217,7 +375,7 @@ Several export reason codes carry a reason field in their details payload that n ### EXPORT_TABLE_TRUNCATED -The table field (not reason) identifies the affected sub‑table: +The table field (not sub_reason) identifies the affected sub‑table: | table value | Meaning | |-------------|---------| @@ -301,7 +459,7 @@ Priority‑resolved: ## DELAY‑LOAD IMPORT SUB‑REASONS -Most delay‑load reason codes carry a reason field in their details payload that narrows the pathology. The full taxonomy: +Most delay‑load reason codes carry a `sub_reason` field in their details payload that narrows the pathology. The full taxonomy: ### DELAY_IMPORT_DIRECTORY_INVALID_HEADER @@ -311,7 +469,7 @@ Most delay‑load reason codes carry a reason field in their details payload tha ### DELAY_IMPORT_TABLE_TRUNCATED -The table field (not reason) identifies the affected sub‑table: +The table field (not sub_reason) identifies the affected sub‑table: | table value | Meaning | |-------------|---------| @@ -326,14 +484,14 @@ The table field (not reason) identifies the affected sub‑table: ### DELAY_IMPORT_DESCRIPTOR_INVALID -Carries both table and reason in details: +Carries both `table` and `sub_reason` in details: -| table reason (priority order) | Meaning | +| table + sub-reason (priority order) | Meaning | |-------------------------------|---------| -| int int_rva_zero | INT RVA is zero despite the descriptor being non‑terminating | -| int int_truncated, int_read_failed, int_max_exceeded, int_unpack_failed | Sub‑table read or parse failure (also surfaces via DELAY_IMPORT_TABLE_TRUNCATED) | -| iat iat_rva_zero | IAT RVA is zero | -| iat iat_truncated, iat_read_failed, iat_max_exceeded, iat_unpack_failed | Same as INT | +| int: int_rva_zero | INT RVA is zero despite the descriptor being non‑terminating | +| int: int_truncated, int_read_failed, int_max_exceeded, int_unpack_failed | Sub‑table read or parse failure (also surfaces via DELAY_IMPORT_TABLE_TRUNCATED) | +| iat: iat_rva_zero | IAT RVA is zero | +| iat: iat_truncated, iat_read_failed, iat_max_exceeded, iat_unpack_failed | Same as INT | ### DELAY_IMPORT_DLL_NAME_INVALID @@ -373,6 +531,186 @@ Priority‑resolved: --- +## **RELOCATION ANOMALIES** + +*Added in v0.7.6 (validator §2.13). Backed by the `pe_relocations` struct-level decoder over `IMAGE_BASE_RELOCATION` blocks. Directory placement/bounds remain owned by the RVA-graph backbone; these codes cover block-stream and per-entry structural truth. `IMAGE_REL_BASED_ABSOLUTE` (type 0) padding entries are never flagged, and absence of a relocation directory is not a defect.* + +| Reason Code | What Triggers It | Example Pattern | Scope | +|------------|------------------|-----------------|--------| +| **RELOCATION_DIRECTORY_INVALID_HEADER** | Top-level decode failure: the directory placement could not be resolved or the first block header was unrecoverable | Directory at an RVA `pe.get_data` cannot resolve | Per‑file | +| **RELOCATION_TABLE_TRUNCATED** | A block's declared `SizeOfBlock` extends past the directory's declared end, or the entry region could not be fully read | Block claims 0x200 bytes but only 0x40 remain before directory end | Per‑file | +| **RELOCATION_BLOCK_MALFORMED** | A block is structurally invalid: `SizeOfBlock` below the 8-byte header minimum, not aligned to the 2-byte entry stride, or a non-advancing size that would stall the walk | `SizeOfBlock = 0`, or `SizeOfBlock = 7` | Per‑block *(priority-resolved sub-reason)* | +| **RELOCATION_ENTRY_RVA_INVALID** | A non-ABSOLUTE entry's target (`page_rva + offset`) does not map to any section | Entry target = 0x9000 with no covering section | Per‑entry *(count always reported in details even when emission is capped)* | + +## RELOCATION SUB‑REASONS + +### RELOCATION_DIRECTORY_INVALID_HEADER + +| Sub‑reason | Meaning | +|------------|---------| +| top_level_decode | The parser could not complete top-level decoding; short-circuits the block and entry checks | + +### RELOCATION_BLOCK_MALFORMED + +Priority‑resolved; the first matching tag wins: + +| Sub‑reason | Meaning | +|------------|---------| +| size_of_block_too_small | `SizeOfBlock` below the 8-byte header minimum | +| size_of_block_not_word_aligned | `SizeOfBlock` not a multiple of the 2-byte entry stride | +| entry_count_exceeds_max | Declared entry count exceeded the parser's hard limit | + +### RELOCATION_TABLE_TRUNCATED + +The `region` field (not `table`, and not `sub_reason`) names the truncated +region. + +--- + +## **DEBUG DIRECTORY ANOMALIES** + +*Added in v0.7.6 (validator §2.14). Backed by the `pe_debug` struct-level decoder over the fixed-stride 28-byte `IMAGE_DEBUG_DIRECTORY` array and the CodeView (RSDS / NB10) records it references. Directory placement/bounds remain owned by the RVA-graph backbone. Absence of a debug directory is not a defect, and entries reachable only via a raw file pointer (no `AddressOfRawData`) are not flagged for mapping.* + +| Reason Code | What Triggers It | Example Pattern | Scope | +|------------|------------------|-----------------|--------| +| **DEBUG_DIRECTORY_INVALID_HEADER** | Top-level decode failure: the directory placement could not be resolved or the first entry was unrecoverable | Directory at an RVA `pe.get_data` cannot resolve | Per‑file | +| **DEBUG_TABLE_TRUNCATED** | The declared directory size is not a whole multiple of the 28-byte entry stride, or the entry array could not be fully read | Directory size = 0x2A (one and a half entries) | Per‑file | +| **DEBUG_DIRECTORY_ENTRY_MALFORMED** | An entry could not be unpacked, its CodeView blob could not be read, or the CodeView record was malformed / of an unrecognised signature | 28-byte entry short read, or CodeView signature neither `RSDS` nor `NB10` | Per‑entry *(priority-resolved sub-reason)* | +| **DEBUG_ENTRY_RVA_INVALID** | An entry's `AddressOfRawData` region does not map to any section | Debug data RVA = 0x9000 with no covering section | Per‑entry | + +## DEBUG SUB‑REASONS + +### DEBUG_DIRECTORY_INVALID_HEADER + +| Sub‑reason | Meaning | +|------------|---------| +| top_level_decode | The parser could not complete top-level decoding; short-circuits the truncation and entry checks | + +### DEBUG_DIRECTORY_ENTRY_MALFORMED + +Priority‑resolved; the first matching tag wins: + +| Sub‑reason | Meaning | +|------------|---------| +| entry_unpack_failed | The 28-byte IMAGE_DEBUG_DIRECTORY entry could not be unpacked | +| codeview_read_failed | `pe.get_data` raised when reading the CodeView blob | +| codeview_too_short | The CodeView record was shorter than its minimum | +| codeview_rsds_truncated | An RSDS record was truncated | +| codeview_nb10_truncated | An NB10 record was truncated | +| codeview_signature_unknown | The CodeView signature was neither `RSDS` nor `NB10` | +| pdb_path_unterminated | The PDB path had no NUL terminator within the scan length | +| pdb_path_non_ascii | The PDB path decoded but contains non-printable bytes | + +### DEBUG_TABLE_TRUNCATED + +The `region` field (not `table`, and not `sub_reason`) names the truncated +region. + +--- + +## **EXCEPTION (.pdata) DIRECTORY ANOMALIES** + +*Added in v0.7.6.1 (validator §2.15). Backed by the `pe_exception` struct-level decoder — the deep semantic validator for the x64 `RUNTIME_FUNCTION` table (12-byte entries → `UNWIND_INFO` in `.xdata`) and the ARM/ARM64 8-byte `.pdata` record walk. Directory placement/bounds remain owned by the RVA-graph backbone; these codes cover the sorted function table and its unwind references. Absence of an exception directory is not a defect (x86 images carry no `.pdata`; x64/ARM images may legitimately omit it).* + +### Exception Directory Anomalies + +| Reason Code | What Triggers It | Example Pattern | Scope | +|------------|------------------|-----------------|--------| +| **EXCEPTION_DIRECTORY_INVALID_HEADER** | Top-level parser decode failure; presence of any top-level error short-circuits all further checks | Directory at an RVA `pe.get_data` cannot resolve | Per‑file | +| **EXCEPTION_DIRECTORY_OUT_OF_BOUNDS** | The directory's `rva + size` extends past SizeOfImage | RVA = 0xF0000, size = 0x4000, SizeOfImage = 0xF2000 | Per‑file | +| **EXCEPTION_DIRECTORY_UNALIGNED** | The directory RVA is not DWORD-aligned (`RUNTIME_FUNCTION` entries must be DWORD-aligned) | RVA = 0x2001 | Per‑file | +| **EXCEPTION_DIRECTORY_SIZE_NOT_MULTIPLE** | The directory Size is not a whole multiple of the per-entry stride (12 for amd64, 8 for arm) | size = 25, entry_size = 12 (remainder 1) | Per‑file | +| **EXCEPTION_TABLE_TRUNCATED** | A parser truncation tag surfaced while walking the counted entry array | Declared count exceeds the readable region; a partial trailing entry | Per‑file *(one issue per tag; `table` field names the cause)* | +| **EXCEPTION_UNSUPPORTED_MACHINE** | The directory is present on a machine whose `.pdata` format is not deep-parsed (x86, IA-64, unknown). Reported once; the function walk is skipped | `IMAGE_FILE_MACHINE_I386` with a non-empty exception directory | Per‑file | + +### Exception Function-Table Entry Anomalies + +| Reason Code | What Triggers It | Example Pattern | Scope | +|------------|------------------|-----------------|--------| +| **EXCEPTION_ENTRY_INVALID** | A per-entry parser error tag surfaced (unreadable / unpackable entry, or a zeroed mandatory RVA). Skips the cross-entry checks for that entry | `begin_rva = 0`, or the 12-byte entry could not be unpacked | Per‑entry *(priority-resolved sub-reason)* | +| **EXCEPTION_FUNCTION_RANGE_INVALID** | `BeginAddress >= EndAddress` (empty or inverted range). `EndAddress` is the RVA of the first byte past the function, so a well-formed entry has begin < end | begin = 0x1050, end = 0x1050 (empty); or begin > end (inverted) | Per‑entry *(amd64 only; arm records carry no EndAddress)* | +| **EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS** | One or more of begin / end / unwind RVA fall outside the mapped image | begin_rva = 0x99000, SizeOfImage = 0x40000 | Per‑entry *(`fields` lists the offending RVA names)* | +| **EXCEPTION_ENTRIES_NOT_SORTED** | A `BeginAddress` is lower than the previous entry's. The loader binary-searches this table, so an out-of-order entry silently loses its unwind data at runtime | Entry N begins at 0x1010 after entry N−1 began at 0x1050 | Per‑entry | +| **EXCEPTION_FUNCTION_OVERLAP** | A (sorted) entry's `BeginAddress` falls inside the previous entry's `[begin, end)` range | Entry N begins at 0x1040 while entry N−1 spans 0x1000–0x1050 | Per‑entry | + +### Exception Unwind-Info Anomalies (AMD64 UNWIND_INFO) + +| Reason Code | What Triggers It | Example Pattern | Scope | +|------------|------------------|-----------------|--------| +| **EXCEPTION_UNWIND_INFO_UNALIGNED** | A non-zero `UnwindInfoAddress` is not DWORD-aligned (`UNWIND_INFO` must be DWORD-aligned) | unwind_info_rva = 0x3021 | Per‑entry | +| **EXCEPTION_UNWIND_INFO_INVALID** | The UNWIND_INFO decode surfaced a pathology, or its Version is not 1/2/3, or Flags carry bits outside the known mask (EHANDLER \| UHANDLER \| CHAININFO \| LARGE) | version = 5; or flags = 0x10 (reserved bit set) | Per‑entry *(priority-resolved sub-reason)* | +| **EXCEPTION_UNWIND_CHAIN_INVALID** | An entry sets `UNW_FLAG_CHAININFO` but its chained target is missing, unaligned, out of bounds, or self-referential | chained_rva = 0; or chained_rva == this entry's own unwind_info_rva | Per‑entry *(priority-resolved sub-reason)* | + +## EXCEPTION DIRECTORY SUB‑REASONS + +Several exception reason codes carry a `sub_reason` (or `table` / `fields`) field in their details payload that narrows the pathology. The full taxonomy: + +### EXCEPTION_DIRECTORY_INVALID_HEADER + +| Sub‑reason | Meaning | +|------------|---------| +| top_level_decode | The parser could not complete top‑level decoding of the exception directory | + +### EXCEPTION_TABLE_TRUNCATED + +The `table` field (not `sub_reason`) identifies the truncation cause: + +| table value | Meaning | +|------------|---------| +| exception_table_ragged_tail | Declared directory size is not a whole multiple of the entry stride; the partial trailing entry is not decoded | +| exception_table_max_exceeded | Declared entry count exceeded the hard limit (2^20) and was clamped | +| exception_entry_read_failed | pe.get_data raised while reading an entry | +| exception_entry_truncated | An entry's fixed-size structure was short | + +### EXCEPTION_ENTRY_INVALID + +Priority‑resolved; the first matching tag wins: + +| Sub‑reason | Meaning | +|------------|---------| +| entry_truncated | The entry's fixed-size structure was short | +| entry_read_failed | pe.get_data raised when reading the entry | +| entry_unpack_failed | struct.unpack failed on the entry bytes | +| begin_rva_zero | BeginAddress was zero | +| end_rva_zero | EndAddress was zero (amd64) | +| unwind_rva_zero | UnwindInfoAddress was zero (amd64) / xdata RVA was zero (arm unpacked) | + +### EXCEPTION_UNWIND_INFO_INVALID + +Priority‑resolved: + +| Sub‑reason | Meaning | +|------------|---------| +| unwind_read_failed | pe.get_data raised when reading the UNWIND_INFO header | +| unwind_truncated | The 4-byte UNWIND_INFO header was short | +| unwind_unpack_failed | struct.unpack failed on the header bytes | +| unwind_version_invalid | Version field was not 1, 2, or 3 | +| unwind_flags_reserved_bits | Flags carried bits outside EHANDLER \| UHANDLER \| CHAININFO \| LARGE | +| unwind_codes_truncated | The trailing chained RUNTIME_FUNCTION could not be read past the unwind-code array | + +### EXCEPTION_UNWIND_CHAIN_INVALID + +Priority‑resolved: + +| Sub‑reason | Meaning | +|------------|---------| +| chain_target_missing | UNW_FLAG_CHAININFO set but the chained RVA is absent or zero | +| chain_target_unaligned | Chained RVA is not DWORD-aligned | +| chain_target_out_of_bounds | Chained RVA falls outside SizeOfImage | +| chain_self_reference | Chained RVA equals the entry's own UnwindInfoAddress | + +### EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS + +The `fields` list (not `reason`) names each RVA that fell outside the image: + +| fields value | Meaning | +|------------|---------| +| begin_rva | BeginAddress < 0 or ≥ SizeOfImage | +| end_rva | EndAddress < 0 or > SizeOfImage | +| unwind_info_rva | UnwindInfoAddress (non-zero) < 0 or ≥ SizeOfImage | + +--- + ## **PACKER HEURISTICS (Interpretation Layer)** | Reason Code | What Triggers It | Example Pattern | Scope | diff --git a/docs/specs/structural-validation-deterministic-heuristics.md b/docs/specs/structural-validation-deterministic-heuristics.md index adbf88b..73019b0 100644 --- a/docs/specs/structural-validation-deterministic-heuristics.md +++ b/docs/specs/structural-validation-deterministic-heuristics.md @@ -35,6 +35,16 @@ Together, they form a comprehensive, deterministic structural model across the c Some structural metadata extracted by parsers is **producer-facing**: it exists to enable validators and heuristics, not to be exposed via the public IOC schema. Examples include the export and delay-load structural details. Other structural metadata is **consumer-facing** and intended for public exposure: version-info string fields are an example of this, planned for promotion in a future release. +> **Placement ownership.** Where a directory's `(rva, size)` placement is +> validated is a deliberate single-owner decision. The RVA-graph backbone +> (§2.5) owns placement for every directory it can interpret as an RVA. +> Subsystem validators that additionally assert placement locally — exports, +> delay-load, exception — do so because the check is cheap and keeps the +> validator self-contained. Their codes are disjoint identifiers, so a directory +> overrunning `SizeOfImage` surfaces under both the backbone's `DATA_DIRECTORY_OUT_OF_RANGE` +> and the subsystem's own code — two distinct labels for one fact, not one code +> emitted twice. Relocations, debug and the security directory defer entirely. + --- # **2.1 Entropy Validator** @@ -121,7 +131,7 @@ This validator enforces: - Directories must not map into overlay data. - Zero‑length sections are invalid mapping targets. -This validator is the backbone of structural correctness for imports, exports, resources, relocations, and TLS directories. The security directory (index 4) is deliberately excluded from all RVA-based checks here; its VirtualAddress is a file offset, not an RVA, and its placement is owned by the signature validator (§2.7), so the two never double-count. +This validator is the backbone of structural correctness for imports, exports, resources, relocations, and TLS directories. The security directory (index 4) is deliberately excluded from all RVA-based checks here; its VirtualAddress is a file offset, not an RVA, and its placement is owned by the signature validator (§2.7), so the two never double-count. The relocation (§2.13) and debug (§2.14) directories likewise defer their placement checks to this validator. --- @@ -164,7 +174,7 @@ This validator enforces: This ensures the Authenticode block is structurally valid before any trust decisions are made. -**v0.7.6 structural decoder.** The certificate subsystem is now backed by a pure `struct`-level decoder (pe_certificates) that walks the `WIN_CERTIFICATE` array independently of pefile's `DIRECTORY_ENTRY_SECURITY` interpretation. The decoder treats `DATA_DIRECTORY[4].VirtualAddress` as a *file offset*, not an RVA, and reads from the raw file bytes, since the certificate table is appended to the file and never mapped into the image. It extracts each entry's revision, type, and length, decodes on the 8-byte (QWORD) entry alignment, and records the structural fact of whether the table offset falls before the on-disk end of any section (`overlaps_image`). This decoder establishes raw structural truth via two new reason codes: `CERTIFICATE_OFFSET_INSIDE_IMAGE` (the table offset falls before the on-disk end of any section) and `CERTIFICATE_TABLE_MALFORMED` (top-level decode failure or a truncation tag surfaced with reason: "truncation"). The placement/overlap fact has a single owner to avoid double-counting with the RVA-graph backbone, and the signature validator continues to interpret the trust-facing symmetry above it. +**v0.7.6 structural decoder.** The certificate subsystem is now backed by a pure `struct`-level decoder (pe_certificates) that walks the `WIN_CERTIFICATE` array independently of pefile's `DIRECTORY_ENTRY_SECURITY` interpretation. The decoder treats `DATA_DIRECTORY[4].VirtualAddress` as a *file offset*, not an RVA, and reads from the raw file bytes, since the certificate table is appended to the file and never mapped into the image. It extracts each entry's revision, type, and length, decodes on the 8-byte (QWORD) entry alignment, and records the structural fact of whether the table offset falls before the on-disk end of any section (`overlaps_image`). This decoder establishes raw structural truth via two new reason codes: `CERTIFICATE_OFFSET_INSIDE_IMAGE` (the table offset falls before the on-disk end of any section) and `CERTIFICATE_TABLE_MALFORMED` (top-level decode failure or a truncation tag surfaced with sub_reason: "truncation"). The placement/overlap fact has a single owner to avoid double-counting with the RVA-graph backbone, and the signature validator continues to interpret the trust-facing symmetry above it. --- @@ -207,8 +217,8 @@ This closes one of the most subtle structural attack surfaces in the PE format. --- -# 2.10 Version‑Info Validator -## Validates the structural integrity of the VS_VERSIONINFO blob extracted from the RT_VERSION resource. +# **2.10 Version‑Info Validator** +### Validates the structural integrity of the VS_VERSIONINFO blob extracted from the RT_VERSION resource. This validator performs: @@ -235,8 +245,8 @@ Version‑info is a high‑signal forensic surface: CompanyName, OriginalFilenam --- -# 2.11 Exports Validator -## Validates the structural integrity of the PE export table extracted by parser_exports. +# **2.11 Exports Validator** +### Validates the structural integrity of the PE export table extracted by parser_exports. This validator performs: @@ -258,7 +268,7 @@ This ensures that for any given malformed export table, the validator produces t --- -## 2.12 Delay-Load Imports Validator +# **2.12 Delay-Load Imports Validator** ### Validates the structural integrity of the PE delay-load import directory and its descriptor array. @@ -295,14 +305,14 @@ The delay-load parser is implemented as a pure `struct`-level decoder over `pe.g --- -## 2.13 Relocations Validator +# **2.13 Relocations Validator** ### Validates the structural integrity of the PE base-relocation table extracted by pe_relocations. This validator performs: - Top-level decode failure detection and short-circuit for unrecoverable directory placement. -- Relocation directory placement within `SizeOfImage`. +- Directory placement is **not** re-checked here — `DATA_DIRECTORY_OUT_OF_RANGE` from the RVA-graph backbone (§2.5) owns it, so the two never double-count. - Truncation reporting across the block array and per-block entry regions. - Per-block structural validation: `SizeOfBlock` below the 8-byte header minimum, `SizeOfBlock` not aligned to the WORD entry stride, and declared entry counts exceeding the per-block ceiling. - Per-entry relocation-target validation: each non-`ABSOLUTE` entry's `page_rva + offset` must map to a real section. @@ -318,18 +328,18 @@ The relocation parser is implemented as a pure `struct`-level decoder over `pe.g - Each entry is decoded by masking `(word >> 12) & 0xF` for the type and `word & 0x0FFF` for the offset; the target RVA is derived as `page_rva + offset` by fixed arithmetic, never by inference. - The readable entry region is clamped to the declared directory end so a block advertising a size past the directory cannot over-read; the shortfall is reported as a truncation tag rather than a partial read. -The validator then maps these structural states to a small, well-defined set of reason codes (`RELOCATION_DIRECTORY_INVALID_HEADER`, `RELOCATION_DIRECTORY_OUT_OF_BOUNDS`, `RELOCATION_TABLE_TRUNCATED`, `RELOCATION_BLOCK_MALFORMED`, `RELOCATION_ENTRY_RVA_INVALID`), which downstream heuristics and IOC consumers can rely on as a stable contract. Per-block malformations are priority-resolved so a block carrying several defects emits one deterministic sub-reason, and the count of invalid entry targets is always reported in the issue details even when the per-entry emission is capped. +The validator then maps these structural states to a small, well-defined set of reason codes (`RELOCATION_DIRECTORY_INVALID_HEADER`, `RELOCATION_TABLE_TRUNCATED`, `RELOCATION_BLOCK_MALFORMED`, `RELOCATION_ENTRY_RVA_INVALID`), which downstream heuristics and IOC consumers can rely on as a stable contract. Per-block malformations are priority-resolved so a block carrying several defects emits one deterministic sub-reason, and the count of invalid entry targets is always reported in the issue details even when the per-entry emission is capped. --- -## 2.14 Debug Directory Validator +# **2.14 Debug Directory Validator** ### Validates the structural integrity of the PE debug directory extracted by pe_debug. This validator performs: - Top-level decode failure detection and short-circuit for unrecoverable directory placement. -- Debug directory placement within `SizeOfImage`. +- Directory placement is **not** re-checked here — `DATA_DIRECTORY_OUT_OF_RANGE` from the RVA-graph backbone (§2.5) owns it, so the two never double-count. - Truncation reporting across the fixed-size entry array, including non-entry-aligned directory sizes. - Per-entry structural validation: entry unpack failure, CodeView blob read failure, and malformed or unrecognised CodeView records. - Per-entry data-region validation: each entry's `AddressOfRawData` region must map to a real section. @@ -346,7 +356,33 @@ The debug parser is implemented as a pure `struct`-level decoder over both `pe.g - The RSDS (PDB 7.0) and NB10 (PDB 2.0) records are decoded against their fixed header layouts; the GUID is formatted in the canonical mixed-endian symbol-server form (Data1/2/3 little-endian, Data4 big-endian) by fixed arithmetic, not library formatting. - The PDB path scan is bounded (512 bytes); an absent terminator emits a deterministic tombstone tag rather than an unbounded read, and non-ASCII bytes are reported rather than silently normalised. -The validator then maps these structural states to a small, well-defined set of reason codes (`DEBUG_DIRECTORY_INVALID_HEADER`, `DEBUG_DIRECTORY_OUT_OF_BOUNDS`, `DEBUG_TABLE_TRUNCATED`, `DEBUG_DIRECTORY_ENTRY_MALFORMED`, `DEBUG_ENTRY_RVA_INVALID`), which downstream heuristics and IOC consumers can rely on as a stable contract. Per-entry malformations are priority-resolved so an entry carrying several defects emits one deterministic sub-reason. The PDB path is a high-signal forensic surface; build paths routinely leak project names, usernames, and toolchain layout, so deterministic extraction is a prerequisite for treating it as a reliable triage signal. +The validator then maps these structural states to a small, well-defined set of reason codes (`DEBUG_DIRECTORY_INVALID_HEADER`, `DEBUG_TABLE_TRUNCATED`, `DEBUG_DIRECTORY_ENTRY_MALFORMED`, `DEBUG_ENTRY_RVA_INVALID`), which downstream heuristics and IOC consumers can rely on as a stable contract. Per-entry malformations are priority-resolved so an entry carrying several defects emits one deterministic sub-reason. The PDB path is a high-signal forensic surface; build paths routinely leak project names, usernames, and toolchain layout, so deterministic extraction is a prerequisite for treating it as a reliable triage signal. + +--- + +# **2.15 Exception Directory Validator** + +### Validates the structural integrity of the PE exception (`.pdata`) directory extracted by `pe_exception`. + +This validator performs: +- Top-level decode failure detection and short-circuit for unrecoverable directory placement. +- Exception directory placement within SizeOfImage, DWORD-alignment of the directory RVA, and Size being a whole multiple of the entry stride. +- Architecture gating: deep parsing is defined for AMD64 (12-byte `RUNTIME_FUNCTION` → `UNWIND_INFO`) and ARM/ARM64 (8-byte packed/`.xdata` records); any other machine (notably x86, which carries no `.pdata`) is reported once as unsupported and the function walk is skipped, so no spurious per-entry codes fire. +- A single ascending pass over the function table: per-entry parser-error resolution, begin/end/unwind RVA bounds, function-range validity (`begin < end`), **sortedness** (ascending `BeginAddress`), and adjacent-range overlap. +- Per-entry `UNWIND_INFO` semantics (AMD64): DWORD-alignment of `UnwindInfoAddress`, Version validity (1/2/3), reserved-flag-bit detection, and chained-unwind target sanity (missing / unaligned / out-of-bounds / self-referential). + +Absence of an exception directory is not treated as a structural defect — most 32-bit images carry no `.pdata` at all (their SEH is stack-based), and a 64-bit or ARM image may legitimately omit it. Placement/bounds of the directory itself are **also** asserted by the RVA-graph backbone (§2.5); this validator repeats the check locally so it stands alone, and a directory overrunning `SizeOfImage` will surface under both codes. They are disjoint identifiers for the same fact rather than a double-count of one code. + +The exception directory is a quietly divergence-prone surface because its interpretation is entirely machine-dependent and its correctness hinges on an invariant the raw bytes do not enforce. Two properties make general-purpose `.pdata` parsers prone to inconsistent output: the record layout is architecture-specific — AMD64 uses a 12-byte `RUNTIME_FUNCTION` with an explicit `EndAddress` and an `.xdata` `UNWIND_INFO` pointer, whereas ARM/ARM64 uses an 8-byte record with no `EndAddress` and a 2-bit Flag selecting between a packed-unwind word and an `.xdata` pointer, so a parser that assumes one layout mis-reads the other while the bytes are identical; and the table is a *counted* array (directory `Size` ÷ stride) that the loader **binary-searches**, requiring entries to be sorted ascending by `BeginAddress` — an invariant that malformed or adversarial binaries routinely violate, causing a function to silently lose its unwind/exception data at runtime even though every byte is present on disk. + +The exception parser is implemented as a pure struct-level decoder over `pe.get_data`-acquired byte buffers: +- The machine type is read once from `FILE_HEADER.Machine` and routed to the AMD64 or ARM(64) decoder; unsupported machines short-circuit to a single `EXCEPTION_UNSUPPORTED_MACHINE` fact with an empty function list. +- The 12-byte `RUNTIME_FUNCTION` (` **Output contract.** Each structural issue surfaces as a +> `pe_structure_anomaly` detection whose `metadata.reason` is the validator's +> parent reason code. Where a validator narrows the pathology, the detail +> appears alongside it under **`sub_reason`** — never under `reason`, which +> the emission layer reserves for the parent code. Truncation codes name the +> affected region in a `table` or `region` key instead. Sub-reasons are +> priority-resolved: an entry carrying several malformations reports exactly +> one, chosen by a fixed documented order, so the output is stable rather than +> dependent on which defect the parser noticed first. +> +> This separation is what makes structural findings machine-consumable: the +> parent code is the stable contract, the sub-reason is diagnostic detail, and +> neither can shadow the other. + Heuristics never contradict validators. They only interpret validated truth. diff --git a/iocx/analysis/heuristics.py b/iocx/analysis/heuristics.py index e53b3e5..795cd9f 100644 --- a/iocx/analysis/heuristics.py +++ b/iocx/analysis/heuristics.py @@ -52,7 +52,9 @@ def _det(value: str, reason: str, metadata: Optional[Dict[str, Any]] = None) -> start=0, end=0, category="pe_heuristic", - metadata={"reason": reason, **(metadata or {})}, + # `reason` is written LAST so a caller-supplied payload key of the same + # name cannot overwrite the parent reason code. Do not reorder. + metadata={**(metadata or {}), "reason": reason}, ) @@ -205,6 +207,8 @@ def _analyse_structural(analysis: Dict[str, Any]) -> List[Detection]: continue metadata = {**details} + if "reason" in metadata: + metadata["sub_reason"] = metadata.pop("reason") out.append(_det( "pe_structure_anomaly", diff --git a/iocx/engine.py b/iocx/engine.py index 6bfaeff..ced38ce 100644 --- a/iocx/engine.py +++ b/iocx/engine.py @@ -20,6 +20,7 @@ from .parsers.pe_debug import build_debug_structure from .parsers.pe_certificates import build_certificate_structure from .parsers.pe_tls import build_tls_structure +from .parsers.pe_exception import build_exception_structure from .detectors import all_detectors from .models import Detection, PluginContext from .plugins.loader import PluginLoader @@ -177,6 +178,7 @@ def _pipeline_pe(self, path: str) -> Dict[str, Any]: self._internal_metadata["debug_struct"] = build_debug_structure(pe) self._internal_metadata["certificate_struct"] = build_certificate_structure(pe) self._internal_metadata["tls_struct"] = build_tls_structure(pe) + self._internal_metadata["exception_struct"] = build_exception_structure(pe) self._internal_metadata.update(extract_optional_header_metadata(pe)) internal: InternalMetadata = self._internal_metadata structural = run_structural_validators(internal, metadata, analysis_dict) diff --git a/iocx/parsers/pe_exception.py b/iocx/parsers/pe_exception.py new file mode 100644 index 0000000..0ecfd02 --- /dev/null +++ b/iocx/parsers/pe_exception.py @@ -0,0 +1,450 @@ +# Copyright (c) 2026 MalX Labs and contributors +# SPDX-License-Identifier: MPL-2.0 + +""" +Deterministic structural extraction of the PE exception (.pdata) directory. + +Independent of pefile's DIRECTORY_ENTRY_EXCEPTION interpretation. +Pefile is used only to: + - Locate the exception directory (RVA, size) + - Determine the target machine via FILE_HEADER.Machine + - Resolve RVAs to file offsets via pe.get_data + +The function table is a *counted* array (directory Size / entry stride), not a +zero-terminated one — unlike the delay-load descriptor array. Entry layout is +architecture-specific: + + AMD64 (x64): 12-byte RUNTIME_FUNCTION { BeginAddress, EndAddress, + UnwindInfoAddress }, all 32-bit image-relative RVAs. The + UnwindInfoAddress points at UNWIND_INFO in .xdata, which we + decode for version/flags/prolog/code-count and chained-unwind. + + ARM64/ARMNT: 8-byte record { Function Start RVA, word1 }. The low 2 bits of + word1 are a Flag: Flag==0 → the upper 30 bits (word1 & ~3) are + an RVA to an .xdata record; Flag!=0 → word1 carries packed + unwind data (no .xdata). There is NO EndAddress field, so + end_rva is left None and the range/overlap checks in the + validator naturally no-op for these entries. We do not decode + .xdata / packed unwind semantics here (out of current scope). + + I386 / other: x86 carries no .pdata; any present directory on an + unsupported machine is reported via arch="unsupported" and the + function walk is skipped so no spurious per-entry codes fire. + +All structural fields are decoded from the raw bytes. Never raises; decode +failures produce tombstone tags in per-entry `errors`, per-unwind `errors`, +and the table-level `truncations` / `errors` lists. + +Output contract (consumed by validators.exception_table.validate_exception_table +and documented as ExceptionStruct in iocx.schemas.internal_schema): + + None - no exception directory present (not an error) + { + "rva": int, "size": int, + "machine": Optional[int], # IMAGE_FILE_MACHINE_* + "arch": str, # "amd64" | "arm64" | "arm" | "unsupported" + "entry_size": int, # 12 (amd64) | 8 (arm) | 0 (unsupported) + "errors": List[str], # top-level decode tags + "truncations": List[str], # per-table truncation tags + "functions": List[dict], # see _decode_* below + } +""" + +from __future__ import annotations + +import struct +from typing import Any, Dict, List, Optional, Tuple + +# IMAGE_DIRECTORY_ENTRY_EXCEPTION = 3 +_EXCEPTION_DIRECTORY_INDEX = 3 + +# Entry strides. +_AMD64_ENTRY_SIZE = 12 # RUNTIME_FUNCTION: 3 x DWORD +_ARM_ENTRY_SIZE = 8 # ARM(64) .pdata record: 2 x DWORD + +# IMAGE_FILE_MACHINE_* — only the values we need to route on. +_MACHINE_I386 = 0x014C +_MACHINE_AMD64 = 0x8664 +_MACHINE_IA64 = 0x0200 +_MACHINE_ARM = 0x01C0 +_MACHINE_ARMNT = 0x01C4 # ARM Thumb-2 (Windows on ARM 32-bit) +_MACHINE_ARM64 = 0xAA64 +_MACHINE_ARM64EC = 0xA641 # ARM64EC — uses the ARM64 .pdata table format + +# UNWIND_INFO (AMD64) constants. +_DWORD = 4 +_VALID_UNWIND_VERSIONS = (1, 2, 3) +_UNW_FLAG_CHAININFO = 0x04 +_UNW_FLAG_KNOWN_MASK = 0x0F # EHANDLER|UHANDLER|CHAININFO|LARGE(V3) +_UNWIND_HEADER_SIZE = 4 +_RUNTIME_FUNCTION_SIZE = 12 # trailing chained RUNTIME_FUNCTION + +# ARM word1 flag mask (low 2 bits) and the xdata-RVA mask (upper 30 bits). +_ARM_FLAG_MASK = 0x3 +_ARM_XDATA_RVA_MASK = ~0x3 & 0xFFFFFFFF + +# Hard limit on function-table entries to defend against a bogus directory +# Size claiming an absurd count. Real .pdata tables are large but bounded; +# 2**20 entries (~12 MiB of .pdata on x64) is already far beyond real images. +_MAX_FUNCTIONS = 1 << 20 + + +def build_exception_structure(pe) -> Optional[Dict[str, Any]]: + """ + Locate and structurally decode the PE exception (.pdata) directory. + + Returns None if no exception directory is present. Otherwise returns a + dict per the module docstring contract. Never raises; decode failures + produce tombstone entries in `errors` / `truncations` and per-entry tags. + """ + placement = _locate_exception_directory(pe) + if placement is None: + return None + + rva, size = placement + machine = _read_machine(pe) + arch = _classify_arch(machine) + entry_size = _entry_size_for_arch(arch) + + truncations: List[str] = [] + errors: List[str] = [] + + if arch == "unsupported" or entry_size == 0: + # Directory present on a machine we don't deep-parse. Report the + # placement so the validator can raise EXCEPTION_UNSUPPORTED_MACHINE; + # emit no function entries. + return { + "rva": rva, + "size": size, + "machine": machine, + "arch": arch, + "entry_size": entry_size, + "functions": [], + "truncations": truncations, + "errors": errors, + } + + functions = _read_function_table( + pe, rva, size, arch, entry_size, truncations, errors, + ) + + return { + "rva": rva, + "size": size, + "machine": machine, + "arch": arch, + "entry_size": entry_size, + "functions": functions, + "truncations": truncations, + "errors": errors, + } + + +# ================================================================= +# Locator / machine routing +# ================================================================= + +def _locate_exception_directory(pe) -> Optional[Tuple[int, int]]: + """Return (rva, size) of the exception directory, or None if absent.""" + try: + data_dir = pe.OPTIONAL_HEADER.DATA_DIRECTORY[_EXCEPTION_DIRECTORY_INDEX] + rva = int(data_dir.VirtualAddress) + size = int(data_dir.Size) + except (AttributeError, IndexError, ValueError, TypeError): + return None + + if rva == 0 or size == 0: + return None + + return (rva, size) + + +def _read_machine(pe) -> Optional[int]: + """Read FILE_HEADER.Machine, or None if unavailable.""" + try: + return int(pe.FILE_HEADER.Machine) + except (AttributeError, ValueError, TypeError): + return None + + +def _classify_arch(machine: Optional[int]) -> str: + """ + Map a machine type to a deep-parse architecture class. + + amd64 → 12-byte RUNTIME_FUNCTION with UNWIND_INFO. + arm64/arm → 8-byte record (packed or .xdata pointer). + unsupported → x86, IA-64, unknown, or unreadable. + """ + if machine == _MACHINE_AMD64: + return "amd64" + if machine in (_MACHINE_ARM64, _MACHINE_ARM64EC): + return "arm64" + if machine in (_MACHINE_ARM, _MACHINE_ARMNT): + return "arm" + return "unsupported" + + +def _entry_size_for_arch(arch: str) -> int: + """Per-entry stride in bytes for a given arch class (0 = unsupported).""" + if arch == "amd64": + return _AMD64_ENTRY_SIZE + if arch in ("arm64", "arm"): + return _ARM_ENTRY_SIZE + return 0 + + +# ================================================================= +# Function-table walk +# ================================================================= + +def _read_function_table( + pe, + base_rva: int, + declared_size: int, + arch: str, + entry_size: int, + truncations: List[str], + errors: List[str], +) -> List[Dict[str, Any]]: + """ + Walk the counted array of function-table entries. + + Unlike the delay-import descriptor array, .pdata has no zero terminator: + the entry count is declared_size // entry_size. We clamp to _MAX_FUNCTIONS + and note when the declared size isn't a whole multiple of the stride (a + ragged tail) — the directory-level validator also flags the size, but the + parser records the truncation so a partial trailing entry never gets + half-decoded. + """ + functions: List[Dict[str, Any]] = [] + + count = declared_size // entry_size + remainder = declared_size % entry_size + if remainder != 0: + truncations.append("exception_table_ragged_tail") + + if count > _MAX_FUNCTIONS: + truncations.append("exception_table_max_exceeded") + count = _MAX_FUNCTIONS + + pos = base_rva + for index in range(count): + try: + raw = bytes(pe.get_data(pos, entry_size)) + except Exception: + truncations.append("exception_entry_read_failed") + break + + if len(raw) < entry_size: + truncations.append("exception_entry_truncated") + break + + if arch == "amd64": + entry = _decode_amd64_entry(pe, raw, index) + else: # arm64 / arm + entry = _decode_arm_entry(raw, index, arch) + + functions.append(entry) + pos += entry_size + + return functions + + +# ================================================================= +# AMD64 entry + UNWIND_INFO +# ================================================================= + +def _decode_amd64_entry(pe, buf: bytes, index: int) -> Dict[str, Any]: + """ + Decode one 12-byte RUNTIME_FUNCTION and its referenced UNWIND_INFO. + + Emits per-entry error tags consumed by the validator's + _ENTRY_ERROR_PRIORITY (begin_rva_zero / end_rva_zero / unwind_rva_zero / + entry_unpack_failed). + """ + errors: List[str] = [] + try: + begin_rva, end_rva, unwind_info_rva = struct.unpack_from(" Dict[str, Any]: + """ + Decode the 4-byte UNWIND_INFO header (+ chained RUNTIME_FUNCTION if + UNW_FLAG_CHAININFO is set). + + Header byte 0: bits[2:0] Version, bits[7:3] Flags. + Header byte 1: SizeOfProlog. Byte 2: CountOfUnwindCodes. + Byte 3: FrameRegister (low nibble) + scaled offset (high nibble). + + Full decode is defined for V1/V2. V3 (APX preview) reuses this header + layout for Version/Flags but repacks the trailing payload differently, so + we surface Version/Flags and stop there rather than mis-decode the chain — + an honest "recognised, not deeply parsed" outcome. Emits tags consumed by + the validator's _UNWIND_ERROR_PRIORITY. + """ + errors: List[str] = [] + + try: + header = bytes(pe.get_data(rva, _UNWIND_HEADER_SIZE)) + except Exception: + return _unwind_result(errors=["unwind_read_failed"]) + + if len(header) < _UNWIND_HEADER_SIZE: + return _unwind_result(errors=["unwind_truncated"]) + + try: + b0, size_of_prolog, count_of_codes, _frame = struct.unpack_from( + "> 3) & 0x1F + + if version not in _VALID_UNWIND_VERSIONS: + errors.append("unwind_version_invalid") + if (flags & ~_UNW_FLAG_KNOWN_MASK) != 0: + errors.append("unwind_flags_reserved_bits") + + is_chained = False + chained_rva: Optional[int] = None + + # Chain resolution is only reliable for the classic V1/V2 payload layout. + if version in (1, 2) and (flags & _UNW_FLAG_CHAININFO): + is_chained = True + # Unwind codes are USHORT[]; the array is padded to an even count. + padded_codes = (count_of_codes + 1) & ~1 + chain_off = _UNWIND_HEADER_SIZE + padded_codes * 2 + try: + rf = bytes(pe.get_data(rva + chain_off, _RUNTIME_FUNCTION_SIZE)) + except Exception: + errors.append("unwind_codes_truncated") + rf = b"" + if len(rf) < _RUNTIME_FUNCTION_SIZE: + if "unwind_codes_truncated" not in errors: + errors.append("unwind_codes_truncated") + else: + # Trailing structure is a RUNTIME_FUNCTION whose UnwindInfoAddress + # points at the *primary* fragment's UNWIND_INFO. + try: + _cb, _ce, chained_rva = struct.unpack_from(" Dict[str, Any]: + """Build the unwind sub-dict with a stable key set.""" + return { + "version": version, + "flags": flags, + "size_of_prolog": size_of_prolog, + "count_of_codes": count_of_codes, + "is_chained": is_chained, + "chained_rva": chained_rva, + "errors": errors or [], + } + + +# ================================================================= +# ARM / ARM64 entry +# ================================================================= + +def _decode_arm_entry(buf: bytes, index: int, arch: str) -> Dict[str, Any]: + """ + Decode one 8-byte ARM(64) .pdata record. + + word0 = Function Start RVA. word1 low 2 bits = Flag: + Flag == 0 → (word1 & ~3) is an RVA to an .xdata record (unpacked). + Flag != 0 → word1 is packed unwind data (no .xdata pointer). + + There is no EndAddress, so end_rva is None (the validator's range/overlap + checks require both endpoints and therefore skip these entries). We do not + decode .xdata / packed unwind bodies here; unwind is left None so only the + structural table checks (bounds, sortedness, alignment) apply. + """ + errors: List[str] = [] + try: + begin_rva, word1 = struct.unpack_from(" List[Tuple[int, int]]: def rva_in_any_section( rva: Optional[int], analysis: Dict[str, Any], + size_of_image: Optional[int] = None, ) -> Optional[bool]: """ True if `rva` falls inside any section's virtual extent. @@ -82,13 +83,14 @@ def rva_in_any_section( return True return False - return region_within_image(rva, 0, analysis.get("size_of_image")) + return region_within_image(rva, 0, size_of_image) def region_in_any_section( rva: Optional[int], size: Optional[int], analysis: Dict[str, Any], + size_of_image: Optional[int] = None, ) -> Optional[bool]: """ True if the whole region [rva, rva+size) fits inside a single section. @@ -108,4 +110,4 @@ def region_in_any_section( return True return False - return region_within_image(rva, span, analysis.get("size_of_image")) + return region_within_image(rva, span, size_of_image) diff --git a/iocx/validators/debug.py b/iocx/validators/debug.py index a95874c..c77cca0 100644 --- a/iocx/validators/debug.py +++ b/iocx/validators/debug.py @@ -26,11 +26,12 @@ DEBUG_ENTRY_RVA_INVALID """ -from typing import Any, Dict, List +from typing import Any, Dict, List, Optional from iocx.reason_codes import ReasonCodes from iocx.validators.schema import StructuralIssue from iocx.schemas.internal_schema import InternalMetadata +from iocx.schemas.public_metadata import PublicMetadata from iocx.schemas.analysis import AnalysisDict from .decorators import depends_on from ._directory_invariants import region_in_any_section @@ -48,20 +49,24 @@ ] -@depends_on("internal", "analysis") -def validate_debug(metadata: InternalMetadata, +@depends_on("internal", "metadata", "analysis") +def validate_debug(internal: InternalMetadata, + metadata: PublicMetadata, analysis: AnalysisDict) -> List[StructuralIssue]: issues: List[StructuralIssue] = [] - debug = metadata.get("debug_struct") + debug = internal.get("debug_struct") if debug is None: return issues # no debug directory — not a defect + opt = metadata.get("optional_header") or {} + size_of_image = opt.get("size_of_image") + # ---- Top-level decode failures short-circuit ---- if debug.get("errors"): issues.append(StructuralIssue( issue=ReasonCodes.DEBUG_DIRECTORY_INVALID_HEADER, - details={"reason": "top_level_decode", + details={"sub_reason": "top_level_decode", "errors": list(debug["errors"])}, )) return issues @@ -69,7 +74,7 @@ def validate_debug(metadata: InternalMetadata, # NOTE: directory placement is intentionally NOT checked here — it is # owned by rva_graph. See module docstring. _validate_truncations(debug, issues) - _validate_entries(debug, analysis, issues) + _validate_entries(debug, analysis, size_of_image, issues) return issues @@ -94,6 +99,7 @@ def _validate_truncations(debug: Dict[str, Any], def _validate_entries(debug: Dict[str, Any], analysis: AnalysisDict, + size_of_image: Optional[int], issues: List[StructuralIssue]) -> None: """Emit per-entry malformation and data-region RVA issues.""" for entry in debug.get("entries", []) or []: @@ -107,14 +113,15 @@ def _validate_entries(debug: Dict[str, Any], details={"index": index, "type": entry.get("type"), "type_name": entry.get("type_name"), - "reason": reason}, + "sub_reason": reason}, )) - _validate_entry_rva(entry, analysis, issues) + _validate_entry_rva(entry, analysis, size_of_image, issues) def _validate_entry_rva(entry: Dict[str, Any], analysis: AnalysisDict, + size_of_image: Optional[int], issues: List[StructuralIssue]) -> None: """ Flag a debug entry whose AddressOfRawData region does not map to any @@ -128,7 +135,7 @@ def _validate_entry_rva(entry: Dict[str, Any], if not addr_raw: return - mapped = region_in_any_section(addr_raw, size_of_data, analysis) + mapped = region_in_any_section(addr_raw, size_of_data, analysis, size_of_image) if mapped is False: issues.append(StructuralIssue( issue=ReasonCodes.DEBUG_ENTRY_RVA_INVALID, diff --git a/iocx/validators/delay_imports.py b/iocx/validators/delay_imports.py index 8006c62..0a7f02e 100644 --- a/iocx/validators/delay_imports.py +++ b/iocx/validators/delay_imports.py @@ -30,7 +30,7 @@ from iocx.reason_codes import ReasonCodes from iocx.validators.schema import StructuralIssue from iocx.schemas.internal_schema import InternalMetadata -from iocx.schemas.analysis import AnalysisDict +from iocx.schemas.public_metadata import PublicMetadata from .decorators import depends_on @@ -73,22 +73,23 @@ ] -@depends_on("internal", "analysis") -def validate_delay_imports(metadata: InternalMetadata, - analysis: AnalysisDict) -> List[StructuralIssue]: +@depends_on("internal", "metadata") +def validate_delay_imports(internal: InternalMetadata, metadata: PublicMetadata) -> List[StructuralIssue]: issues: List[StructuralIssue] = [] - di = metadata.get("delay_import_struct") + di = internal.get("delay_import_struct") if di is None: return issues # no delay-load directory — not a defect - size_of_image = analysis.get("size_of_image") + opt = metadata.get("optional_header") or {} + + size_of_image = opt.get("size_of_image") # ---- Top-level decode failures short-circuit ---- if di.get("errors"): issues.append(StructuralIssue( issue=ReasonCodes.DELAY_IMPORT_DIRECTORY_INVALID_HEADER, - details={"reason": "top_level_decode", + details={"sub_reason": "top_level_decode", "errors": list(di["errors"])}, )) return issues @@ -113,7 +114,7 @@ def _validate_placement(di: Dict[str, Any], rva = di.get("rva") size = di.get("size") or 0 - # Skip placement check if the analysis layer didn't populate + # Skip placement check if the metadata layer didn't populate # size_of_image. This shouldn't happen in normal operation — if it # does, an upstream bug needs investigating, not a placement issue. if rva is None or size_of_image is None: @@ -183,7 +184,7 @@ def _validate_descriptors(di: Dict[str, Any], details={"index": index, "dll_name_rva": descriptor.get("dll_name_rva"), "dll_name": descriptor.get("dll_name"), - "reason": dll_name_reason}, + "sub_reason": dll_name_reason}, )) # ---- INT/IAT table-level errors ---- @@ -197,7 +198,7 @@ def _validate_descriptors(di: Dict[str, Any], issue=ReasonCodes.DELAY_IMPORT_DESCRIPTOR_INVALID, details={"index": index, "table": "int", - "reason": int_reason, + "sub_reason": int_reason, "int_rva": descriptor.get("int_rva")}, )) @@ -209,7 +210,7 @@ def _validate_descriptors(di: Dict[str, Any], issue=ReasonCodes.DELAY_IMPORT_DESCRIPTOR_INVALID, details={"index": index, "table": "iat", - "reason": iat_reason, + "sub_reason": iat_reason, "iat_rva": descriptor.get("iat_rva")}, )) @@ -255,7 +256,7 @@ def _validate_import_entries(descriptor: Dict[str, Any], "ordinal": entry.get("ordinal"), "name": entry.get("name"), "name_rva": entry.get("name_rva"), - "reason": reason, + "sub_reason": reason, }, )) diff --git a/iocx/validators/entrypoint.py b/iocx/validators/entrypoint.py index 78dc9df..b55ce47 100644 --- a/iocx/validators/entrypoint.py +++ b/iocx/validators/entrypoint.py @@ -161,13 +161,13 @@ def validate_entrypoint(metadata: PublicMetadata, analysis: AnalysisDict) -> Lis if isinstance(vs, int) and vs == 0: issues.append(StructuralIssue( issue=ReasonCodes.ENTRYPOINT_IN_TRUNCATED_REGION, - details={"entry_point": ep, "section": name, "reason": "zero_length_section"}, + details={"entry_point": ep, "section": name, "sub_reason": "zero_length_section"}, )) elif isinstance(va, int) and isinstance(vs, int) and ep >= va + vs: # Only emit the "beyond_virtual_size" variant if we didn't already flag zero-length issues.append(StructuralIssue( issue=ReasonCodes.ENTRYPOINT_IN_TRUNCATED_REGION, - details={"entry_point": ep, "section": name, "reason": "beyond_virtual_size"}, + details={"entry_point": ep, "section": name, "sub_reason": "beyond_virtual_size"}, )) # --- D. EP must not point into overlays (RVA → file offset) --- diff --git a/iocx/validators/exception_table.py b/iocx/validators/exception_table.py new file mode 100644 index 0000000..deac360 --- /dev/null +++ b/iocx/validators/exception_table.py @@ -0,0 +1,459 @@ +# Copyright (c) 2026 MalX Labs and contributors +# SPDX-License-Identifier: MPL-2.0 + +""" +Validate the exception (.pdata) directory structure — the deep semantic +counterpart to the generic RVA/placement checks in ``rva_graph``. + +``rva_graph`` already treats IMAGE_DIRECTORY_ENTRY_EXCEPTION like any other +data directory (bounds, section mapping, overlap). This validator owns the +*semantic* truth of that directory: the sorted function table of +RUNTIME_FUNCTION entries and the UNWIND_INFO structures they reference. + +Absence of an exception directory is NOT a structural defect — most 32-bit +(x86 / I386) images carry no .pdata at all (their SEH is stack-based), and a +64-bit image may legitimately omit it. We only emit codes when the directory +is present and structurally malformed. + +Scope / grounding +----------------- +Deep parsing here is defined for table-based exception handling only: + * AMD64 (x64): array of 12-byte RUNTIME_FUNCTION entries, DWORD-aligned, + each { BeginAddress, EndAddress, UnwindInfoAddress } as + 32-bit image-relative RVAs, SORTED ascending by + BeginAddress, referencing UNWIND_INFO in .xdata. + * ARM64 / ARM: also table-based (ARM_RUNTIME_FUNCTION, packed/unpacked + unwind). Structural table checks below apply; UNWIND_INFO + version/flag checks are AMD64-specific and are skipped. +Any other machine (notably I386) is reported once as unsupported-for-deep-parse +and the semantic walk is skipped — we do not manufacture false positives on +directories we cannot interpret. + +Facts grounded against Microsoft's x64 exception-handling documentation and +the PE/COFF spec: + - RUNTIME_FUNCTION is 12 bytes and must be DWORD aligned; all three fields + are 32-bit image-relative RVAs. Entries are sorted ascending by + BeginAddress and stored in .pdata of a PE32+ image. + - EndAddress is the RVA of the first byte *past* the function, so a + well-formed entry has BeginAddress < EndAddress. + - UNWIND_INFO must be DWORD aligned; Version is 1 (V1/V2) or 3 (APX / V3 + preview). Flags are a mask of UNW_FLAG_EHANDLER(0x1), UNW_FLAG_UHANDLER + (0x2), UNW_FLAG_CHAININFO(0x4), plus UNW_FLAG_LARGE(0x8) in V3. + +Determinism +----------- +Every heuristic is a pure function of the parsed structure. Emission order is +fixed: directory-level checks, then a single ascending pass over the function +table (index order), then per-entry unwind checks. Per-entry pathologies are +collapsed to a single first-matching reason via ``_first_matching`` over a +fixed priority list. No set iteration governs what or when we emit. Same file +in → same StructuralIssue sequence out. + +Reason codes emitted: + EXCEPTION_DIRECTORY_INVALID_HEADER top-level decode failure + EXCEPTION_DIRECTORY_OUT_OF_BOUNDS directory rva+size exceeds SizeOfImage + EXCEPTION_DIRECTORY_UNALIGNED directory rva not DWORD-aligned + EXCEPTION_DIRECTORY_SIZE_NOT_MULTIPLE size not a multiple of entry stride + EXCEPTION_TABLE_TRUNCATED parser truncation tag(s) + EXCEPTION_UNSUPPORTED_MACHINE present but arch not deep-parseable + EXCEPTION_ENTRY_INVALID per-entry parser error (priority-resolved) + EXCEPTION_FUNCTION_RANGE_INVALID begin >= end (empty/inverted range) + EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS begin/end/unwind RVA outside image + EXCEPTION_ENTRIES_NOT_SORTED BeginAddress not ascending (loader-visible) + EXCEPTION_FUNCTION_OVERLAP adjacent function ranges overlap + EXCEPTION_UNWIND_INFO_UNALIGNED UnwindInfoAddress not DWORD-aligned + EXCEPTION_UNWIND_INFO_INVALID unwind decode/version/flags anomaly (priority-resolved) + EXCEPTION_UNWIND_CHAIN_INVALID chained-unwind target invalid / cycle / depth + +Parser contract (metadata["exception_struct"], populated by pe_exception): + { + "rva": int, "size": int, # directory VirtualAddress / Size + "machine": Optional[int], # IMAGE_FILE_MACHINE_* + "arch": str, # "amd64" | "arm64" | "arm" | "unsupported" + "entry_size": int, # 12 for amd64 + "errors": List[str], # top-level decode tags + "truncations": List[str], # per-table truncation tags + "functions": [ + { + "index": int, + "begin_rva": Optional[int], + "end_rva": Optional[int], + "unwind_info_rva": Optional[int], # amd64 + "errors": List[str], # per-entry parser tags + "unwind": { # amd64 only, optional + "version": Optional[int], + "flags": Optional[int], + "size_of_prolog": Optional[int], + "count_of_codes": Optional[int], + "is_chained": bool, + "chained_rva": Optional[int], + "errors": List[str], # per-unwind parser tags + } | None, + }, ... + ], + } +""" + +from typing import Any, Dict, List, Optional + +from iocx.reason_codes import ReasonCodes +from iocx.validators.schema import StructuralIssue +from iocx.schemas.internal_schema import InternalMetadata +from iocx.schemas.public_metadata import PublicMetadata +from .decorators import depends_on + + +# Architectures for which the RUNTIME_FUNCTION table walk is defined. x86 +# (I386) carries no .pdata, so a present directory there is reported once as +# unsupported rather than semantically walked. +_TABLE_ARCHS = ("amd64", "arm64", "arm") + +# AMD64 RUNTIME_FUNCTION stride and required alignment (DWORD). +_AMD64_ENTRY_SIZE = 12 +_DWORD = 4 + +# Valid UNWIND_INFO version numbers: 1 (V1), 2 (V2), 3 (APX / V3 preview). +_VALID_UNWIND_VERSIONS = (1, 2, 3) + +# Known UNW_FLAG bits: EHANDLER(0x1) | UHANDLER(0x2) | CHAININFO(0x4) | +# LARGE(0x8, V3). Any bit outside this mask is a reserved-bit anomaly. +_UNW_FLAG_KNOWN_MASK = 0x0F +_UNW_FLAG_CHAININFO = 0x04 + +# Guard against pathological / hostile chain graphs during the chain walk. +_MAX_CHAIN_DEPTH = 32 + + +# Priority-resolved sub-reasons for per-entry table pathologies. +# First-matching wins for deterministic emission. +_ENTRY_ERROR_PRIORITY = [ + "entry_truncated", + "entry_read_failed", + "entry_unpack_failed", + "begin_rva_zero", + "end_rva_zero", + "unwind_rva_zero", +] + +_UNWIND_ERROR_PRIORITY = [ + "unwind_read_failed", + "unwind_truncated", + "unwind_unpack_failed", + "unwind_version_invalid", + "unwind_flags_reserved_bits", + "unwind_codes_truncated", +] + + +@depends_on("internal", "metadata") +def validate_exception_table(internal: InternalMetadata, + metadata: PublicMetadata) -> List[StructuralIssue]: + issues: List[StructuralIssue] = [] + + ex = internal.get("exception_struct") + if ex is None: + return issues # no exception directory — not a defect + + opt = metadata.get("optional_header") or {} + size_of_image = opt.get("size_of_image") + + # ---- Top-level decode failures short-circuit ---- + if ex.get("errors"): + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_DIRECTORY_INVALID_HEADER, + details={"sub_reason": "top_level_decode", + "errors": list(ex["errors"])}, + )) + return issues + + _validate_directory(ex, size_of_image, issues) + _validate_truncations(ex, issues) + + # ---- Architecture gate ---- + # Deep table/unwind semantics are only defined for table-based archs. + # Report once and stop rather than emit spurious per-entry noise. + arch = ex.get("arch") + if arch not in _TABLE_ARCHS: + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_UNSUPPORTED_MACHINE, + details={"arch": arch, "machine": ex.get("machine")}, + )) + return issues + + _validate_function_table(ex, size_of_image, issues) + return issues + + +# ================================================================= +# Directory-level validation +# ================================================================= + +def _validate_directory(ex: Dict[str, Any], + size_of_image: Optional[int], + issues: List[StructuralIssue]) -> None: + """ + Directory placement, alignment, and stride consistency. + + RUNTIME_FUNCTION entries must be DWORD aligned, so the directory RVA + itself must be DWORD aligned, and Size must be a whole multiple of the + per-entry stride (12 bytes on amd64). + """ + rva = ex.get("rva") + size = ex.get("size") or 0 + entry_size = ex.get("entry_size") or _AMD64_ENTRY_SIZE + + if rva is None: + return + + # Placement within the mapped image. Mirrors the delay-import placement + # check; rva_graph owns the generic version but we assert the semantic + # invariant locally so this validator stands alone. + if size_of_image is not None and rva + size > size_of_image: + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_DIRECTORY_OUT_OF_BOUNDS, + details={"rva": rva, "size": size, + "size_of_image": size_of_image}, + )) + + if rva % _DWORD != 0: + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_DIRECTORY_UNALIGNED, + details={"rva": rva, "alignment": _DWORD}, + )) + + if entry_size > 0 and size % entry_size != 0: + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_DIRECTORY_SIZE_NOT_MULTIPLE, + details={"size": size, "entry_size": entry_size, + "remainder": size % entry_size}, + )) + + +def _validate_truncations(ex: Dict[str, Any], + issues: List[StructuralIssue]) -> None: + """ + Map parser truncation tags to a single reason code with structured + details. One issue per truncated table so the consumer sees one issue + per truncation. + """ + for tag in ex.get("truncations", []) or []: + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_TABLE_TRUNCATED, + details={"table": tag}, + )) + + +# ================================================================= +# Function-table validation (the semantic core) +# ================================================================= + +def _validate_function_table(ex: Dict[str, Any], + size_of_image: Optional[int], + issues: List[StructuralIssue]) -> None: + """ + Single ascending pass over the RUNTIME_FUNCTION array. + + Per entry we check, in fixed order: + 1. parser error tags → EXCEPTION_ENTRY_INVALID + 2. begin/end/unwind RVA bounds → EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS + 3. range validity (begin < end) → EXCEPTION_FUNCTION_RANGE_INVALID + 4. sortedness vs. previous begin → EXCEPTION_ENTRIES_NOT_SORTED + 5. overlap vs. previous end → EXCEPTION_FUNCTION_OVERLAP + 6. unwind alignment + semantics → EXCEPTION_UNWIND_INFO_* codes + + Sortedness (4) is the loader-visible invariant: ntdll locates a function's + unwind data by binary search over this table, so an out-of-order entry + means a real function silently "loses" its exception/unwind data at + runtime even though every byte is present on disk. + """ + functions = ex.get("functions", []) or [] + + prev_begin: Optional[int] = None + prev_end: Optional[int] = None + + for entry in functions: + index = entry.get("index") + + # ---- 1. Parser-level per-entry errors (priority-resolved) ---- + entry_errors = entry.get("errors", []) or [] + entry_reason = _first_matching(entry_errors, _ENTRY_ERROR_PRIORITY) + if entry_reason != "unknown": + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_ENTRY_INVALID, + details={"index": index, "sub_reason": entry_reason}, + )) + # A structurally unreadable entry can't feed the cross-entry + # invariants below; skip it but keep walking the table. + continue + + begin = entry.get("begin_rva") + end = entry.get("end_rva") + unwind_rva = entry.get("unwind_info_rva") + + # ---- 2. RVA bounds ---- + oob_fields = _out_of_bounds_fields(begin, end, unwind_rva, + size_of_image) + if oob_fields: + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS, + details={"index": index, + "begin_rva": begin, "end_rva": end, + "unwind_info_rva": unwind_rva, + "fields": oob_fields, + "size_of_image": size_of_image}, + )) + + # ---- 3. Range validity ---- + if isinstance(begin, int) and isinstance(end, int) and begin >= end: + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_FUNCTION_RANGE_INVALID, + details={"index": index, "begin_rva": begin, "end_rva": end, + "empty": begin == end}, + )) + + # ---- 4. Sortedness (ascending BeginAddress) ---- + if isinstance(begin, int) and prev_begin is not None \ + and begin < prev_begin: + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_ENTRIES_NOT_SORTED, + details={"index": index, "begin_rva": begin, + "prev_begin_rva": prev_begin}, + )) + + # ---- 5. Overlap with previous function range ---- + # Only meaningful when this entry starts at/after the previous one + # (i.e. the table is locally sorted); an unsorted pair is already + # flagged above and we don't double-count it as an overlap. + if isinstance(begin, int) and prev_end is not None \ + and prev_begin is not None and begin >= prev_begin \ + and begin < prev_end: + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_FUNCTION_OVERLAP, + details={"index": index, "begin_rva": begin, + "prev_end_rva": prev_end}, + )) + + # ---- 6. Unwind-info semantics (amd64) ---- + _validate_unwind_info(index, entry.get("unwind"), unwind_rva, + size_of_image, issues) + + # Advance the ascending cursor only on sane, sorted begins so a single + # wild entry doesn't poison every subsequent comparison. + if isinstance(begin, int) and (prev_begin is None or begin >= prev_begin): + prev_begin = begin + if isinstance(end, int): + prev_end = end + + +def _validate_unwind_info(index: Optional[int], + unwind: Optional[Dict[str, Any]], + unwind_rva: Optional[int], + size_of_image: Optional[int], + issues: List[StructuralIssue]) -> None: + """ + UNWIND_INFO alignment, decode, version, flags, and chain sanity. + + AMD64-specific. On ARM/ARM64 the parser omits ``unwind`` (packed unwind + has no comparable version/flags byte), so these checks are skipped. + """ + # Alignment is checkable from the RVA alone, independent of decode. + if isinstance(unwind_rva, int) and unwind_rva != 0 \ + and unwind_rva % _DWORD != 0: + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_UNWIND_INFO_UNALIGNED, + details={"index": index, "unwind_info_rva": unwind_rva, + "alignment": _DWORD}, + )) + + if unwind is None: + return + + # Parser-reported decode pathologies + derived version/flag anomalies, + # collapsed to a single first-matching reason for determinism. + unwind_errors = list(unwind.get("errors", []) or []) + + version = unwind.get("version") + if isinstance(version, int) and version not in _VALID_UNWIND_VERSIONS \ + and "unwind_version_invalid" not in unwind_errors: + unwind_errors.append("unwind_version_invalid") + + flags = unwind.get("flags") + if isinstance(flags, int) and (flags & ~_UNW_FLAG_KNOWN_MASK) != 0 \ + and "unwind_flags_reserved_bits" not in unwind_errors: + unwind_errors.append("unwind_flags_reserved_bits") + + unwind_reason = _first_matching(unwind_errors, _UNWIND_ERROR_PRIORITY) + if unwind_reason != "unknown": + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_UNWIND_INFO_INVALID, + details={"index": index, "sub_reason": unwind_reason, + "version": version, "flags": flags, + "unwind_info_rva": unwind_rva}, + )) + + # ---- Chained unwind info ---- + # If UNW_FLAG_CHAININFO is set the trailing shared field is an + # image-relative pointer to the *primary* RUNTIME_FUNCTION. A missing, + # out-of-bounds, self-referential, or unaligned target is malformed. + is_chained = bool(unwind.get("is_chained")) or \ + (isinstance(flags, int) and (flags & _UNW_FLAG_CHAININFO) != 0) + if is_chained: + chained_rva = unwind.get("chained_rva") + reason = _chain_target_reason(chained_rva, unwind_rva, size_of_image) + if reason is not None: + issues.append(StructuralIssue( + issue=ReasonCodes.EXCEPTION_UNWIND_CHAIN_INVALID, + details={"index": index, "chained_rva": chained_rva, + "unwind_info_rva": unwind_rva, "sub_reason": reason}, + )) + + +# ================================================================= +# Helpers +# ================================================================= + +def _out_of_bounds_fields(begin: Optional[int], end: Optional[int], + unwind_rva: Optional[int], + size_of_image: Optional[int]) -> List[str]: + """ + Return the fixed-order list of RVA field names that fall outside the + mapped image. Empty when everything is in bounds or bounds are unknown. + """ + if not isinstance(size_of_image, int): + return [] + out: List[str] = [] + if isinstance(begin, int) and (begin < 0 or begin >= size_of_image): + out.append("begin_rva") + if isinstance(end, int) and (end < 0 or end > size_of_image): + out.append("end_rva") + if isinstance(unwind_rva, int) and unwind_rva != 0 \ + and (unwind_rva < 0 or unwind_rva >= size_of_image): + out.append("unwind_info_rva") + return out + + +def _chain_target_reason(chained_rva: Optional[int], + unwind_rva: Optional[int], + size_of_image: Optional[int]) -> Optional[str]: + """ + Classify a chained-unwind pointer. Returns a sub-reason string when the + target is malformed, else None. Ordered so the first applicable wins. + """ + if not isinstance(chained_rva, int) or chained_rva == 0: + return "chain_target_missing" + if chained_rva % _DWORD != 0: + return "chain_target_unaligned" + if isinstance(size_of_image, int) and \ + (chained_rva < 0 or chained_rva >= size_of_image): + return "chain_target_out_of_bounds" + if isinstance(unwind_rva, int) and chained_rva == unwind_rva: + return "chain_self_reference" + return None + + +def _first_matching(errors: List[str], candidates: List[str]) -> str: + """Return the first error tag from `candidates` that appears in `errors`.""" + for c in candidates: + if c in errors: + return c + return "unknown" diff --git a/iocx/validators/exports.py b/iocx/validators/exports.py index fb5b8eb..8094957 100644 --- a/iocx/validators/exports.py +++ b/iocx/validators/exports.py @@ -26,7 +26,7 @@ from iocx.reason_codes import ReasonCodes from iocx.validators.schema import StructuralIssue from iocx.schemas.internal_schema import InternalMetadata -from iocx.schemas.analysis import AnalysisDict +from iocx.schemas.public_metadata import PublicMetadata from .decorators import depends_on @@ -46,22 +46,23 @@ ] -@depends_on("internal", "analysis") -def validate_exports(metadata: InternalMetadata, analysis: AnalysisDict) -> List[StructuralIssue]: +@depends_on("internal", "metadata") +def validate_exports(internal: InternalMetadata, metadata: PublicMetadata) -> List[StructuralIssue]: issues: List[StructuralIssue] = [] - exp = metadata.get("export_struct") + exp = internal.get("export_struct") if exp is None: return issues - size_of_image = analysis.get("size_of_image") + opt = metadata.get("optional_header") or {} + size_of_image = opt.get("size_of_image") # If the parser couldn't decode the header at all, the rest of the # validation can't run meaningfully. if exp.get("errors"): issues.append(StructuralIssue( issue=ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER, - details={"reason": "top_level_decode", + details={"sub_reason": "top_level_decode", "errors": list(exp["errors"])}, )) return issues @@ -94,7 +95,7 @@ def _validate_placement(exp, size_of_image, issues): rva = exp.get("rva") size = exp.get("size") or 0 - # Skip placement check if the analysis layer didn't populate + # Skip placement check if the metadata layer didn't populate # size_of_image. This shouldn't happen in normal operation. If it # does, an upstream bug needs investigating, not a placement issue. if rva is None or size_of_image is None: @@ -153,28 +154,28 @@ def _validate_header_consistency(exp: Dict[str, Any], if num_funcs > 0 and addr_funcs == 0: issues.append(StructuralIssue( issue=ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER, - details={"reason": "eat_rva_zero_with_nonzero_count", + details={"sub_reason": "eat_rva_zero_with_nonzero_count", "NumberOfFunctions": num_funcs}, )) if num_names > 0 and addr_names == 0: issues.append(StructuralIssue( issue=ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER, - details={"reason": "enpt_rva_zero_with_nonzero_count", + details={"sub_reason": "enpt_rva_zero_with_nonzero_count", "NumberOfNames": num_names}, )) if num_names > 0 and addr_name_ord == 0: issues.append(StructuralIssue( issue=ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER, - details={"reason": "eot_rva_zero_with_nonzero_count", + details={"sub_reason": "eot_rva_zero_with_nonzero_count", "NumberOfNames": num_names}, )) if num_names > num_funcs: issues.append(StructuralIssue( issue=ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER, - details={"reason": "num_names_exceeds_num_functions", + details={"sub_reason": "num_names_exceeds_num_functions", "NumberOfNames": num_names, "NumberOfFunctions": num_funcs}, )) @@ -210,7 +211,7 @@ def _validate_name_pointers(exp: Dict[str, Any], issue=ReasonCodes.EXPORT_NAME_RVA_INVALID, details={"index": index, "name_rva": entry.get("name_rva"), - "reason": rva_reason}, + "sub_reason": rva_reason}, )) # ---- Name encoding: one issue per entry, priority-resolved ---- @@ -222,7 +223,7 @@ def _validate_name_pointers(exp: Dict[str, Any], issue=ReasonCodes.EXPORT_NAME_NOT_ASCII, details={"index": index, "name": entry.get("name"), - "reason": encoding_reason}, + "sub_reason": encoding_reason}, )) # ---- Ordinal index bounds ---- @@ -230,7 +231,7 @@ def _validate_name_pointers(exp: Dict[str, Any], issues.append(StructuralIssue( issue=ReasonCodes.EXPORT_NAME_ORDINAL_INDEX_INVALID, details={"index": index, - "reason": "missing"}, + "sub_reason": "missing"}, )) elif "ordinal_index_out_of_range" in entry_errors: issues.append(StructuralIssue( @@ -238,7 +239,7 @@ def _validate_name_pointers(exp: Dict[str, Any], details={"index": index, "ordinal_index": entry.get("ordinal_index"), "num_functions": num_funcs, - "reason": "out_of_range"}, + "sub_reason": "out_of_range"}, )) @@ -307,7 +308,7 @@ def _validate_functions(exp, header, size_of_image, issues): if max_ordinal > 0xFFFF: issues.append(StructuralIssue( issue=ReasonCodes.EXPORT_ORDINAL_OUT_OF_RANGE, - details={"reason": "max_exceeds_u16", + details={"sub_reason": "max_exceeds_u16", "base": base, "num_functions": num_funcs, "max_ordinal": max_ordinal}, @@ -324,13 +325,13 @@ def _validate_functions(exp, header, size_of_image, issues): issues.append(StructuralIssue( issue=ReasonCodes.EXPORT_FORWARDER_MALFORMED, details={"index": index, "ordinal": ordinal, - "reason": "unreadable"}, + "sub_reason": "unreadable"}, )) elif not entry.get("forwarder_valid"): issues.append(StructuralIssue( issue=ReasonCodes.EXPORT_FORWARDER_MALFORMED, details={"index": index, "ordinal": ordinal, - "forwarder": forwarder, "reason": "format"}, + "forwarder": forwarder, "sub_reason": "format"}, )) continue @@ -345,7 +346,7 @@ def _validate_functions(exp, header, size_of_image, issues): details={"index": index, "ordinal": ordinal, "address_rva": address_rva, "size_of_image": size_of_image, - "reason": "exceeds_image"}, + "sub_reason": "exceeds_image"}, )) diff --git a/iocx/validators/load_config_directory.py b/iocx/validators/load_config_directory.py index d6ef490..d125a82 100644 --- a/iocx/validators/load_config_directory.py +++ b/iocx/validators/load_config_directory.py @@ -154,7 +154,7 @@ def validate_load_config_directory(internal: InternalMetadata, metadata: PublicM if not mapped: issues.append(StructuralIssue( issue=ReasonCodes.LOAD_CONFIG_COOKIE_INVALID, - details={"cookie_rva": cookie_rva, "reason": "unmapped"}, + details={"cookie_rva": cookie_rva, "sub_reason": "unmapped"}, )) else: cookie_raw, sec = mapped @@ -169,7 +169,7 @@ def validate_load_config_directory(internal: InternalMetadata, metadata: PublicM "cookie_rva": cookie_rva, "section": sec.get("name"), "characteristics": characteristics, - "reason": "non_writable_section", + "sub_reason": "non_writable_section", }, )) @@ -196,7 +196,7 @@ def validate_load_config_directory(internal: InternalMetadata, metadata: PublicM details={ "seh_table_rva": seh_table_rva, "seh_count": seh_count, - "reason": "missing_table_rva", + "sub_reason": "missing_table_rva", }, )) else: @@ -210,7 +210,7 @@ def validate_load_config_directory(internal: InternalMetadata, metadata: PublicM "seh_table_rva": seh_table_rva, "seh_count": seh_count, "size_of_image": size_of_image, - "reason": "out_of_range", + "sub_reason": "out_of_range", }, )) else: @@ -221,7 +221,7 @@ def validate_load_config_directory(internal: InternalMetadata, metadata: PublicM details={ "seh_table_rva": seh_table_rva, "seh_count": seh_count, - "reason": "unmapped", + "sub_reason": "unmapped", }, )) else: @@ -234,7 +234,7 @@ def validate_load_config_directory(internal: InternalMetadata, metadata: PublicM "seh_count": seh_count, "seh_raw": seh_raw, "overlay_offset": overlay_offset, - "reason": "in_overlay", + "sub_reason": "in_overlay", }, )) diff --git a/iocx/validators/optional_header.py b/iocx/validators/optional_header.py index cfeefec..fcd3627 100644 --- a/iocx/validators/optional_header.py +++ b/iocx/validators/optional_header.py @@ -116,7 +116,7 @@ def validate_optional_header(internalMetadata: InternalMetadata, metadata: Publi if not _is_power_of_two(section_alignment): issues.append(StructuralIssue( issue=ReasonCodes.OPTIONAL_HEADER_INVALID_SECTION_ALIGNMENT, - details={"section_alignment": section_alignment, "reason": "not_power_of_two"}, + details={"section_alignment": section_alignment, "sub_reason": "not_power_of_two"}, )) # --------------------------------------------------------- @@ -126,14 +126,14 @@ def validate_optional_header(internalMetadata: InternalMetadata, metadata: Publi if not _is_power_of_two(file_alignment): issues.append(StructuralIssue( issue=ReasonCodes.OPTIONAL_HEADER_INVALID_FILE_ALIGNMENT, - details={"file_alignment": file_alignment, "reason": "not_power_of_two"}, + details={"file_alignment": file_alignment, "sub_reason": "not_power_of_two"}, )) # Microsoft recommends 512–64K if file_alignment < 512 or file_alignment > 65536: issues.append(StructuralIssue( issue=ReasonCodes.OPTIONAL_HEADER_INVALID_FILE_ALIGNMENT, - details={"file_alignment": file_alignment, "reason": "out_of_range"}, + details={"file_alignment": file_alignment, "sub_reason": "out_of_range"}, )) # --------------------------------------------------------- diff --git a/iocx/validators/relocations.py b/iocx/validators/relocations.py index 9cdf2e7..dfd21fa 100644 --- a/iocx/validators/relocations.py +++ b/iocx/validators/relocations.py @@ -25,11 +25,12 @@ RELOCATION_ENTRY_RVA_INVALID """ -from typing import Any, Dict, List +from typing import Any, Dict, List, Optional from iocx.reason_codes import ReasonCodes from iocx.validators.schema import StructuralIssue from iocx.schemas.internal_schema import InternalMetadata +from iocx.schemas.public_metadata import PublicMetadata from iocx.schemas.analysis import AnalysisDict from .decorators import depends_on from ._directory_invariants import rva_in_any_section @@ -48,20 +49,24 @@ _MAX_ENTRY_ISSUES_PER_BLOCK = 8 -@depends_on("internal", "analysis") -def validate_relocations(metadata: InternalMetadata, +@depends_on("internal", "metadata", "analysis") +def validate_relocations(internal: InternalMetadata, + metadata: PublicMetadata, analysis: AnalysisDict) -> List[StructuralIssue]: issues: List[StructuralIssue] = [] - reloc = metadata.get("relocation_struct") + reloc = internal.get("relocation_struct") if reloc is None: return issues # no relocation directory — not a defect + opt = metadata.get("optional_header") or {} + size_of_image = opt.get("size_of_image") + # ---- Top-level decode failures short-circuit ---- if reloc.get("errors"): issues.append(StructuralIssue( issue=ReasonCodes.RELOCATION_DIRECTORY_INVALID_HEADER, - details={"reason": "top_level_decode", + details={"sub_reason": "top_level_decode", "errors": list(reloc["errors"])}, )) return issues @@ -69,7 +74,7 @@ def validate_relocations(metadata: InternalMetadata, # NOTE: directory placement is intentionally NOT checked here — it is # owned by rva_graph. See module docstring. _validate_truncations(reloc, issues) - _validate_blocks(reloc, analysis, issues) + _validate_blocks(reloc, analysis, size_of_image, issues) return issues @@ -94,6 +99,7 @@ def _validate_truncations(reloc: Dict[str, Any], def _validate_blocks(reloc: Dict[str, Any], analysis: AnalysisDict, + size_of_image: Optional[int], issues: List[StructuralIssue]) -> None: """Emit per-block structural issues and per-entry RVA issues.""" for block in reloc.get("blocks", []) or []: @@ -107,14 +113,15 @@ def _validate_blocks(reloc: Dict[str, Any], details={"index": index, "page_rva": block.get("page_rva"), "size_of_block": block.get("size_of_block"), - "reason": reason}, + "sub_reason": reason}, )) - _validate_block_entries(block, analysis, issues) + _validate_block_entries(block, analysis, size_of_image, issues) def _validate_block_entries(block: Dict[str, Any], analysis: AnalysisDict, + size_of_image: Optional[int], issues: List[StructuralIssue]) -> None: """ Flag relocation entries whose target RVA does not map to any section. @@ -131,7 +138,7 @@ def _validate_block_entries(block: Dict[str, Any], if entry.get("type") == 0: # IMAGE_REL_BASED_ABSOLUTE — padding continue target_rva = entry.get("rva") - mapped = rva_in_any_section(target_rva, analysis) + mapped = rva_in_any_section(target_rva, analysis, size_of_image) if mapped is False: invalid_rvas.append(target_rva) diff --git a/iocx/validators/resources.py b/iocx/validators/resources.py index bac625e..d66c0bf 100644 --- a/iocx/validators/resources.py +++ b/iocx/validators/resources.py @@ -8,6 +8,7 @@ from iocx.schemas.analysis import AnalysisDict from .decorators import depends_on + @depends_on("internal", "analysis") def validate_resources(metadata: InternalMetadata, analysis: AnalysisDict) -> List[StructuralIssue]: issues: List[StructuralIssue] = [] @@ -56,8 +57,24 @@ def validate_directory(dir_node: Dict[str, Any], depth: int = 0) -> None: rva = dir_node["rva"] size = dir_node["size"] - # Skip if the directory is not inside .rsrc + # The directory node itself must fit wholly inside .rsrc. + # + # Two distinct malformations reach here: + # * the ROOT directory lies outside .rsrc entirely, or + # * a CHILD starts inside .rsrc but its extent overflows the end - + # the caller's entry check tests target_rva WITHOUT a size, so such + # a child passes there and is caught here. + # + # A child lying wholly outside is already reported by the caller as + # RESOURCE_ENTRY_OUT_OF_BOUNDS, and its `continue` prevents descent, so + # the two codes never double-count. `depth` distinguishes the cases: + # depth 0 is the root, deeper values are the overflow case. if not rva_in_rsrc(rva, size): + issues.append(StructuralIssue( + issue=ReasonCodes.RESOURCE_DIRECTORY_OUT_OF_BOUNDS, + details={"rva": rva, "size": size, "depth": depth, + "rsrc_start": rsrc_va, "rsrc_end": rsrc_va + rsrc_vs}, + )) return entries = dir_node["entries"] @@ -81,7 +98,7 @@ def validate_directory(dir_node: Dict[str, Any], depth: int = 0) -> None: return visited_dirs.add(rva) - # --- language layer (depth 2) must use integer LCIDs --- + # Language layer (depth 2) must use integer LCIDs: # Per PE spec, the Type → Name → Language tree's deepest directory # layer is keyed by language ID. Named entries here are malformed. if depth == 2: @@ -105,17 +122,17 @@ def validate_directory(dir_node: Dict[str, Any], depth: int = 0) -> None: )) continue - validate_directory(target, depth + 1) # <-- depth bumped + validate_directory(target, depth + 1) # <-- depth bumped continue - # --- data entries should only appear at depth 2 (Language layer) --- + # Data entries should only appear at depth 2 (Language layer): # A data leaf at depth 0 or 1 means the tree shape violates the # Type → Name → Language hierarchy. if depth != 2: issues.append(StructuralIssue( issue=ReasonCodes.RESOURCE_DATA_AT_INVALID_DEPTH, details={"rva": rva, "depth": depth, - "data_rva": entry["data_rva"]}, + "data_rva": entry["data_rva"]}, )) # ------------------------------ @@ -161,6 +178,7 @@ def validate_directory(dir_node: Dict[str, Any], depth: int = 0) -> None: for sec in sections: if sec is rsrc_section: continue + if raw_overlaps_section(data_raw, data_size, sec): issues.append(StructuralIssue( issue=ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA, @@ -172,6 +190,7 @@ def validate_directory(dir_node: Dict[str, Any], depth: int = 0) -> None: for sec in sections: if sec is rsrc_section: continue + if va_overlaps_section(data_rva, data_size, sec): issues.append(StructuralIssue( issue=ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA, diff --git a/iocx/validators/rva_graph.py b/iocx/validators/rva_graph.py index cd1d7d4..b35c0cf 100644 --- a/iocx/validators/rva_graph.py +++ b/iocx/validators/rva_graph.py @@ -11,6 +11,10 @@ # No directories are strictly required to be non-zero. +# +# The set is intentionally empty: the DATA_DIRECTORY_ZERO_SIZE_UNEXPECTED +# branch below is therefore unreachable in production and exists so a future +# policy change ("directory X must be present") needs only an entry here. REQUIRED_NONZERO_DIRS: set[str] = set() @@ -21,7 +25,7 @@ # is a category error and double-counts with the certificate parser / # signature validator, which own that directory's placement truth (validated # against file_size, e.g. certificate_offset_past_eof). Exclude it from ALL -# RVA-based checks — both the per-directory loop and the overlap detection. +# RVA-based checks - both the per-directory loop and the overlap detection. _SECURITY_DIRECTORY_INDEX = 4 _SECURITY_DIRECTORY_NAME = "IMAGE_DIRECTORY_ENTRY_SECURITY" @@ -38,6 +42,9 @@ def _is_security_directory(d: Dict[str, Any]) -> bool: def validate_rva_graph(metadata: PublicMetadata, analysis: AnalysisDict) -> List[StructuralIssue]: issues: List[StructuralIssue] = [] + # Directory source: the analysis layer wins, falling back to metadata. + # Note an EMPTY analysis list is falsy and therefore falls through to + # metadata as well. dirs = analysis.get("data_directories") or metadata.get("data_directories") or [] opt = metadata.get("optional_header") or {} sections = analysis.get("sections", []) or [] @@ -51,15 +58,12 @@ def validate_rva_graph(metadata: PublicMetadata, analysis: AnalysisDict) -> List # Build section ranges section_ranges = [] - zero_length_sections = set() for sec in sections: va = sec.get("virtual_address") vs = sec.get("virtual_size") name = sec.get("name") if isinstance(va, int) and isinstance(vs, int): section_ranges.append((va, va + vs, name)) - if vs == 0: - zero_length_sections.add(name) # --------------------------------------------------------- # Directory validation @@ -111,6 +115,8 @@ def validate_rva_graph(metadata: PublicMetadata, analysis: AnalysisDict) -> List continue # 4) Directory in headers + # Deliberately does NOT short-circuit: a directory can lie inside the + # headers AND be unmapped / out-of-range, and both are worth reporting. if isinstance(size_of_headers, int) and rva < size_of_headers: issues.append(StructuralIssue( issue=ReasonCodes.DATA_DIRECTORY_IN_HEADERS, @@ -143,7 +149,12 @@ def validate_rva_graph(metadata: PublicMetadata, analysis: AnalysisDict) -> List raw_offset = base_raw + (rva - va_start) - # RVA→raw consistency check + # RVA→raw consistency check: + # NOTE raw_size defaults to 0, so a section without one + # has an empty raw range and ANY directory mapping into it + # will mismatch. On mismatch we `continue` rather than + # break, so a later section covering the same VA can still + # resolve raw_offset. sec_raw_start = base_raw sec_raw_end = base_raw + sec.get("raw_size", 0) @@ -151,29 +162,32 @@ def validate_rva_graph(metadata: PublicMetadata, analysis: AnalysisDict) -> List issues.append(StructuralIssue( issue=ReasonCodes.DATA_DIRECTORY_RAW_MISMATCH, details={ - "directory": name, - "rva": rva, + "directory": name, "rva": rva, "raw_offset": raw_offset, "section": sec_name, "section_raw_start": sec_raw_start, - "section_raw_end": sec_raw_end, + "section_raw_end": sec_raw_end }, )) continue break - # raw mapping safety — skip overlay check if raw_offset invalid - if raw_offset is None: - continue - - if raw_offset >= overlay_offset: + # Raw mapping safety - skip ONLY the overlay check when raw_offset could + # not be resolved (the RVA maps to no section, or the matching section has + # no raw_address). + # + # Deliberately scoped: this previously used a bare `continue`, which also + # skipped the section-mapping checks below, so the mere presence of an + # overlay suppressed DATA_DIRECTORY_NOT_MAPPED_TO_SECTION. + if raw_offset is not None and raw_offset >= overlay_offset: issues.append(StructuralIssue( issue=ReasonCodes.DATA_DIRECTORY_IN_OVERLAY, details={"directory": name, "rva": rva, "raw_offset": raw_offset}, )) # 7) Skip mapping if directory lands on a zero-length section + # (va_start == rva and the section has no extent). zero_length_hit = False for va_start, va_end, sec_name in section_ranges: if va_start == rva and va_start == va_end: @@ -183,7 +197,8 @@ def validate_rva_graph(metadata: PublicMetadata, analysis: AnalysisDict) -> List if zero_length_hit: continue - # 8) Section mapping + # 8) Section mapping (half-open interval: a directory ending exactly at + # a section start does not count as spanning it). mapped_sections = [] for va_start, va_end, sec_name in section_ranges: if rva < va_end and (rva + size) > va_start: @@ -202,6 +217,9 @@ def validate_rva_graph(metadata: PublicMetadata, analysis: AnalysisDict) -> List # --------------------------------------------------------- # Directory overlap detection + # + # Runs independently of the per-directory loop above: a directory skipped + # there (e.g. zero size) is still compared here. # --------------------------------------------------------- for i in range(len(dirs)): a = dirs[i] @@ -212,6 +230,7 @@ def validate_rva_graph(metadata: PublicMetadata, analysis: AnalysisDict) -> List rva_a = a.get("rva") size_a = a.get("size") + if not isinstance(rva_a, int) or not isinstance(size_a, int): continue end_a = rva_a + size_a diff --git a/iocx/validators/sections.py b/iocx/validators/sections.py index 6a4a27c..970a736 100644 --- a/iocx/validators/sections.py +++ b/iocx/validators/sections.py @@ -152,17 +152,17 @@ def validate_sections(metadata: PublicMetadata, analysis: AnalysisDict) -> List[ if has_code and not readable: issues.append(StructuralIssue( issue=ReasonCodes.SECTION_FLAGS_INCONSISTENT, - details={"section": name, "reason": "code_without_read"}, + details={"section": name, "sub_reason": "code_without_read"}, )) if writable and not readable: issues.append(StructuralIssue( issue=ReasonCodes.SECTION_FLAGS_INCONSISTENT, - details={"section": name, "reason": "write_without_read"}, + details={"section": name, "sub_reason": "write_without_read"}, )) if executable and not readable: issues.append(StructuralIssue( issue=ReasonCodes.SECTION_FLAGS_INCONSISTENT, - details={"section": name, "reason": "exec_without_read"}, + details={"section": name, "sub_reason": "exec_without_read"}, )) # --------------------------------------------------------- diff --git a/iocx/validators/signature.py b/iocx/validators/signature.py index 7739613..ddf65d4 100644 --- a/iocx/validators/signature.py +++ b/iocx/validators/signature.py @@ -72,7 +72,7 @@ def validate_signature(internal: InternalMetadata, if cert_struct is not None and cert_struct.get("errors"): issues.append(StructuralIssue( issue=ReasonCodes.CERTIFICATE_TABLE_MALFORMED, - details={"reason": "top_level_decode", + details={"sub_reason": "top_level_decode", "errors": list(cert_struct["errors"])}, )) return issues @@ -113,7 +113,7 @@ def validate_signature(internal: InternalMetadata, for tag in cert_struct.get("truncations", []) or []: issues.append(StructuralIssue( issue=ReasonCodes.CERTIFICATE_TABLE_MALFORMED, - details={"reason": "truncation", "region": tag}, + details={"sub_reason": "truncation", "region": tag}, )) # --------------------------------------------------------- diff --git a/iocx/validators/tls.py b/iocx/validators/tls.py index 67f4828..6232189 100644 --- a/iocx/validators/tls.py +++ b/iocx/validators/tls.py @@ -119,7 +119,7 @@ def validate_tls(internal: InternalMetadata, if header_errs: issues.append(StructuralIssue( issue=ReasonCodes.TLS_DIRECTORY_TRUNCATED, - details={"reason": "header_decode", "errors": header_errs}, + details={"sub_reason": "header_decode", "errors": header_errs}, )) return issues @@ -129,7 +129,7 @@ def validate_tls(internal: InternalMetadata, for tag in tls.get("truncations", []) or []: issues.append(StructuralIssue( issue=ReasonCodes.TLS_DIRECTORY_TRUNCATED, - details={"reason": "callback_array", "region": tag}, + details={"sub_reason": "callback_array", "region": tag}, )) # --------------------------------------------------------- @@ -262,7 +262,7 @@ def _validate_callback_targets(tls: Dict[str, Any], for tag in sorted(set(errors) & _CALLBACK_RESOLUTION_ERROR_TAGS): issues.append(StructuralIssue( issue=ReasonCodes.TLS_CALLBACK_RVA_INVALID, - details={"reason": tag}, + details={"sub_reason": tag}, )) callbacks = tls.get("callbacks") or [] @@ -276,7 +276,7 @@ def _validate_callback_targets(tls: Dict[str, Any], # Cannot convert VA -> RVA; the parser records this too. One issue. issues.append(StructuralIssue( issue=ReasonCodes.TLS_CALLBACK_RVA_INVALID, - details={"reason": "image_base_unavailable", + details={"sub_reason": "image_base_unavailable", "callback_count": len(callbacks)}, )) return @@ -287,11 +287,11 @@ def _validate_callback_targets(tls: Dict[str, Any], continue rva = va - image_base if rva < 0: - invalid.append({"callback_va": va, "reason": "below_image_base"}) + invalid.append({"callback_va": va, "sub_reason": "below_image_base"}) continue if _map_rva_to_section(sections, rva) is None: invalid.append({"callback_va": va, "callback_rva": rva, - "reason": "not_mapped"}) + "sub_reason": "not_mapped"}) if not invalid: return diff --git a/iocx/validators/version_info.py b/iocx/validators/version_info.py index 995607f..03481dd 100644 --- a/iocx/validators/version_info.py +++ b/iocx/validators/version_info.py @@ -27,7 +27,7 @@ def validate_version_info(metadata: InternalMetadata, analysis: AnalysisDict) -> if vi is None: return issues # no RT_VERSION present — not a defect - sections = analysis["sections"] + sections = analysis.get("sections") or [] rsrc_section = next( (s for s in sections if s["name"].lower() == ".rsrc"), None, @@ -42,26 +42,26 @@ def validate_version_info(metadata: InternalMetadata, analysis: AnalysisDict) -> if not (rsrc_va <= rva and rva + size <= rsrc_va + rsrc_vs): issues.append(StructuralIssue( issue=ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER, - details={"reason": "placement", "rva": rva, "size": size}, + details={"sub_reason": "placement", "rva": rva, "size": size}, )) # ---- Top-level header ---- if not vi.get("decoded"): issues.append(StructuralIssue( issue=ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER, - details={"reason": "undecoded", "errors": vi.get("errors", [])}, + details={"sub_reason": "undecoded", "errors": vi.get("errors", [])}, )) return issues # nothing further to validate if not vi.get("header_ok"): issues.append(StructuralIssue( issue=ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER, - details={"reason": "szkey_mismatch"}, + details={"sub_reason": "szkey_mismatch"}, )) if not vi.get("length_consistent"): issues.append(StructuralIssue( issue=ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER, - details={"reason": "length_inconsistent"}, + details={"sub_reason": "length_inconsistent"}, )) # ---- VS_FIXEDFILEINFO ---- @@ -72,7 +72,7 @@ def validate_version_info(metadata: InternalMetadata, analysis: AnalysisDict) -> if any(e.startswith("fixed_file_info") for e in vi.get("errors", [])): issues.append(StructuralIssue( issue=ReasonCodes.RESOURCE_VERSIONINFO_INVALID_FIXEDINFO, - details={"reason": "parse_failed", + details={"sub_reason": "parse_failed", "errors": [e for e in vi["errors"] if e.startswith("fixed_file_info")]}, )) @@ -80,13 +80,13 @@ def validate_version_info(metadata: InternalMetadata, analysis: AnalysisDict) -> if not ffi.get("signature_ok"): issues.append(StructuralIssue( issue=ReasonCodes.RESOURCE_VERSIONINFO_INVALID_FIXEDINFO, - details={"reason": "signature", + details={"sub_reason": "signature", "signature": ffi.get("signature")}, )) if not ffi.get("struct_version_ok"): issues.append(StructuralIssue( issue=ReasonCodes.RESOURCE_VERSIONINFO_INVALID_FIXEDINFO, - details={"reason": "struct_version", + details={"sub_reason": "struct_version", "struct_version": ffi.get("struct_version")}, )) diff --git a/pyproject.toml b/pyproject.toml index cb6a652..a489d82 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "iocx" -version = "0.7.6" +version = "0.7.6.1" description = "A deterministic, high‑performance static‑analysis engine that extracts high‑signal IOCs from PE binaries, text, and logs — built for SOC automation and modern threat‑analysis pipelines." authors = [ { name = "MalX Labs" } diff --git a/tests/contract/snapshots/layer3_adversarial/corrupted_data_directories.full.json b/tests/contract/snapshots/layer3_adversarial/corrupted_data_directories.full.json index 6c26f1a..fd5e9ac 100644 --- a/tests/contract/snapshots/layer3_adversarial/corrupted_data_directories.full.json +++ b/tests/contract/snapshots/layer3_adversarial/corrupted_data_directories.full.json @@ -193,10 +193,55 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "top_level_decode", + "reason": "certificate_table_malformed", "errors": [ "certificate_offset_past_eof" - ] + ], + "sub_reason": "top_level_decode" + } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_directory_out_of_bounds", + "rva": 12032, + "size": 8192, + "size_of_image": 12288 + } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_directory_size_not_multiple", + "size": 8192, + "entry_size": 12, + "remainder": 8 + } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_table_truncated", + "table": "exception_table_ragged_tail" + } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_table_truncated", + "table": "exception_entry_read_failed" } } ] diff --git a/tests/contract/snapshots/layer3_adversarial/crypto_entropy_payload.full.json b/tests/contract/snapshots/layer3_adversarial/crypto_entropy_payload.full.json index 204ca37..c3062e6 100644 --- a/tests/contract/snapshots/layer3_adversarial/crypto_entropy_payload.full.json +++ b/tests/contract/snapshots/layer3_adversarial/crypto_entropy_payload.full.json @@ -669,8 +669,9 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "unmapped", - "cookie_rva": 5368721472 + "reason": "load_config_cookie_invalid", + "cookie_rva": 5368721472, + "sub_reason": "unmapped" } } ] diff --git a/tests/contract/snapshots/layer3_adversarial/directory_raw_mismatch.full.json b/tests/contract/snapshots/layer3_adversarial/directory_raw_mismatch.full.json index 74b2bc7..2e953f3 100644 --- a/tests/contract/snapshots/layer3_adversarial/directory_raw_mismatch.full.json +++ b/tests/contract/snapshots/layer3_adversarial/directory_raw_mismatch.full.json @@ -176,6 +176,38 @@ "rva": 6144, "raw_offset": 2560 } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_directory_size_not_multiple", + "size": 256, + "entry_size": 12, + "remainder": 4 + } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_table_truncated", + "table": "exception_table_ragged_tail" + } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_table_truncated", + "table": "exception_entry_truncated" + } } ] } diff --git a/tests/contract/snapshots/layer3_adversarial/fixture_007_entrypoint_zero_length_section.full.json b/tests/contract/snapshots/layer3_adversarial/fixture_007_entrypoint_zero_length_section.full.json index 082e5a4..46463ec 100644 --- a/tests/contract/snapshots/layer3_adversarial/fixture_007_entrypoint_zero_length_section.full.json +++ b/tests/contract/snapshots/layer3_adversarial/fixture_007_entrypoint_zero_length_section.full.json @@ -160,9 +160,10 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "zero_length_section", + "reason": "entrypoint_in_truncated_region", "entry_point": 4096, - "section": ".text" + "section": ".text", + "sub_reason": "zero_length_section" } } ] diff --git a/tests/contract/snapshots/layer3_adversarial/fixture_027_opt_section_alignment_invalid.full.json b/tests/contract/snapshots/layer3_adversarial/fixture_027_opt_section_alignment_invalid.full.json index a41d54e..8f92fba 100644 --- a/tests/contract/snapshots/layer3_adversarial/fixture_027_opt_section_alignment_invalid.full.json +++ b/tests/contract/snapshots/layer3_adversarial/fixture_027_opt_section_alignment_invalid.full.json @@ -204,8 +204,9 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "not_power_of_two", - "section_alignment": 384 + "reason": "optional_header_invalid_section_alignment", + "section_alignment": 384, + "sub_reason": "not_power_of_two" } }, { diff --git a/tests/contract/snapshots/layer3_adversarial/fixture_028_opt_file_alignment_invalid.full.json b/tests/contract/snapshots/layer3_adversarial/fixture_028_opt_file_alignment_invalid.full.json index d9cb6f4..89f2daa 100644 --- a/tests/contract/snapshots/layer3_adversarial/fixture_028_opt_file_alignment_invalid.full.json +++ b/tests/contract/snapshots/layer3_adversarial/fixture_028_opt_file_alignment_invalid.full.json @@ -231,8 +231,9 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "not_power_of_two", - "file_alignment": 768 + "reason": "optional_header_invalid_file_alignment", + "file_alignment": 768, + "sub_reason": "not_power_of_two" } } ] diff --git a/tests/contract/snapshots/layer3_adversarial/franken_malformed_pe.full.json b/tests/contract/snapshots/layer3_adversarial/franken_malformed_pe.full.json index ea0ba88..7030fea 100644 --- a/tests/contract/snapshots/layer3_adversarial/franken_malformed_pe.full.json +++ b/tests/contract/snapshots/layer3_adversarial/franken_malformed_pe.full.json @@ -287,6 +287,38 @@ "rva": 0, "size": 256 } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_directory_size_not_multiple", + "size": 512, + "entry_size": 12, + "remainder": 8 + } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_table_truncated", + "table": "exception_table_ragged_tail" + } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_table_truncated", + "table": "exception_entry_read_failed" + } } ] } diff --git a/tests/contract/snapshots/layer3_adversarial/franken_malformed_pe.pe32.full.json b/tests/contract/snapshots/layer3_adversarial/franken_malformed_pe.pe32.full.json index 176c9ef..8a29915 100644 --- a/tests/contract/snapshots/layer3_adversarial/franken_malformed_pe.pe32.full.json +++ b/tests/contract/snapshots/layer3_adversarial/franken_malformed_pe.pe32.full.json @@ -287,6 +287,29 @@ "rva": 0, "size": 256 } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_directory_size_not_multiple", + "size": 512, + "entry_size": 12, + "remainder": 8 + } + }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "exception_unsupported_machine", + "arch": "unsupported", + "machine": 332 + } } ] } diff --git a/tests/contract/snapshots/layer3_adversarial/franken_url_domain_ip.full.json b/tests/contract/snapshots/layer3_adversarial/franken_url_domain_ip.full.json index 0116f28..2cb4449 100644 --- a/tests/contract/snapshots/layer3_adversarial/franken_url_domain_ip.full.json +++ b/tests/contract/snapshots/layer3_adversarial/franken_url_domain_ip.full.json @@ -717,8 +717,9 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "unmapped", - "cookie_rva": 5368725760 + "reason": "load_config_cookie_invalid", + "cookie_rva": 5368725760, + "sub_reason": "unmapped" } } ] diff --git a/tests/contract/snapshots/layer3_adversarial/invalid_optional_header.full.json b/tests/contract/snapshots/layer3_adversarial/invalid_optional_header.full.json index 0e4caa2..ac20c08 100644 --- a/tests/contract/snapshots/layer3_adversarial/invalid_optional_header.full.json +++ b/tests/contract/snapshots/layer3_adversarial/invalid_optional_header.full.json @@ -191,7 +191,8 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "top_level_decode", + "reason": "export_directory_invalid_header", + "sub_reason": "top_level_decode", "errors": [ "header_read_failed" ] diff --git a/tests/contract/snapshots/layer3_adversarial/invalid_optional_header.pe32.full.json b/tests/contract/snapshots/layer3_adversarial/invalid_optional_header.pe32.full.json index 99f66c6..5dd69d1 100644 --- a/tests/contract/snapshots/layer3_adversarial/invalid_optional_header.pe32.full.json +++ b/tests/contract/snapshots/layer3_adversarial/invalid_optional_header.pe32.full.json @@ -243,6 +243,18 @@ "size_of_image": 512 } }, + { + "value": "pe_structure_anomaly", + "start": 0, + "end": 0, + "category": "pe_heuristic", + "metadata": { + "reason": "export_directory_out_of_bounds", + "rva": 4096, + "size": 512, + "size_of_image": 512 + } + }, { "value": "pe_structure_anomaly", "start": 0, diff --git a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_cookie_in_overlay.full.json b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_cookie_in_overlay.full.json index 7333839..b300fe8 100644 --- a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_cookie_in_overlay.full.json +++ b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_cookie_in_overlay.full.json @@ -152,10 +152,11 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "non_writable_section", + "reason": "load_config_cookie_invalid", "cookie_rva": 12800, "section": ".rdata", - "characteristics": 1073741888 + "characteristics": 1073741888, + "sub_reason": "non_writable_section" } }, { diff --git a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_cookie_invalid.full.json b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_cookie_invalid.full.json index 0cbad97..824a04d 100644 --- a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_cookie_invalid.full.json +++ b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_cookie_invalid.full.json @@ -152,8 +152,9 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "unmapped", - "cookie_rva": 2415919104 + "reason": "load_config_cookie_invalid", + "cookie_rva": 2415919104, + "sub_reason": "unmapped" } } ] diff --git a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_guard_cf_inconsistent.full.json b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_guard_cf_inconsistent.full.json index 91e0a97..08bac37 100644 --- a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_guard_cf_inconsistent.full.json +++ b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_guard_cf_inconsistent.full.json @@ -165,10 +165,11 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "non_writable_section", + "reason": "load_config_cookie_invalid", "cookie_rva": 13568, "section": ".rdata", - "characteristics": 1073741888 + "characteristics": 1073741888, + "sub_reason": "non_writable_section" } }, { diff --git a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_seh_invalid.full.json b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_seh_invalid.full.json index f41c478..7ba654a 100644 --- a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_seh_invalid.full.json +++ b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_seh_invalid.full.json @@ -152,10 +152,11 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "non_writable_section", + "reason": "load_config_cookie_invalid", "cookie_rva": 13568, "section": ".rdata", - "characteristics": 1073741888 + "characteristics": 1073741888, + "sub_reason": "non_writable_section" } }, { @@ -176,9 +177,10 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "missing_table_rva", + "reason": "load_config_seh_invalid", "seh_table_rva": 0, - "seh_count": 4 + "seh_count": 4, + "sub_reason": "missing_table_rva" } } ] diff --git a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_size_exceeds_section.full.json b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_size_exceeds_section.full.json index e843293..5e61dd2 100644 --- a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_size_exceeds_section.full.json +++ b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_size_exceeds_section.full.json @@ -165,8 +165,9 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "unmapped", - "cookie_rva": 13568 + "reason": "load_config_cookie_invalid", + "cookie_rva": 13568, + "sub_reason": "unmapped" } } ] diff --git a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_size_too_small.full.json b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_size_too_small.full.json index e3093c4..c9e55e4 100644 --- a/tests/contract/snapshots/layer3_adversarial/load_config_malformed_size_too_small.full.json +++ b/tests/contract/snapshots/layer3_adversarial/load_config_malformed_size_too_small.full.json @@ -164,10 +164,11 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "non_writable_section", + "reason": "load_config_cookie_invalid", "cookie_rva": 13568, "section": ".rdata", - "characteristics": 1073741888 + "characteristics": 1073741888, + "sub_reason": "non_writable_section" } }, { diff --git a/tests/contract/snapshots/layer3_adversarial/malformed_domain.full.json b/tests/contract/snapshots/layer3_adversarial/malformed_domain.full.json index 4f175ff..a8d430a 100644 --- a/tests/contract/snapshots/layer3_adversarial/malformed_domain.full.json +++ b/tests/contract/snapshots/layer3_adversarial/malformed_domain.full.json @@ -695,8 +695,9 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "unmapped", - "cookie_rva": 5368721536 + "reason": "load_config_cookie_invalid", + "cookie_rva": 5368721536, + "sub_reason": "unmapped" } } ] diff --git a/tests/contract/snapshots/layer3_adversarial/malformed_ip.full.json b/tests/contract/snapshots/layer3_adversarial/malformed_ip.full.json index 8376f85..933aa6f 100644 --- a/tests/contract/snapshots/layer3_adversarial/malformed_ip.full.json +++ b/tests/contract/snapshots/layer3_adversarial/malformed_ip.full.json @@ -701,8 +701,9 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "unmapped", - "cookie_rva": 5368721600 + "reason": "load_config_cookie_invalid", + "cookie_rva": 5368721600, + "sub_reason": "unmapped" } } ] diff --git a/tests/contract/snapshots/layer3_adversarial/malformed_url.full.json b/tests/contract/snapshots/layer3_adversarial/malformed_url.full.json index 1fb0a08..bcaf223 100644 --- a/tests/contract/snapshots/layer3_adversarial/malformed_url.full.json +++ b/tests/contract/snapshots/layer3_adversarial/malformed_url.full.json @@ -699,8 +699,9 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "unmapped", - "cookie_rva": 5368721536 + "reason": "load_config_cookie_invalid", + "cookie_rva": 5368721536, + "sub_reason": "unmapped" } } ] diff --git a/tests/contract/snapshots/layer3_adversarial/string_obfuscation_tricks.full.json b/tests/contract/snapshots/layer3_adversarial/string_obfuscation_tricks.full.json index fb53e75..21978d8 100644 --- a/tests/contract/snapshots/layer3_adversarial/string_obfuscation_tricks.full.json +++ b/tests/contract/snapshots/layer3_adversarial/string_obfuscation_tricks.full.json @@ -692,8 +692,9 @@ "end": 0, "category": "pe_heuristic", "metadata": { - "reason": "unmapped", - "cookie_rva": 5368721472 + "reason": "load_config_cookie_invalid", + "cookie_rva": 5368721472, + "sub_reason": "unmapped" } } ] diff --git a/tests/integration/test_crypto_entropy_payload.py b/tests/integration/test_crypto_entropy_payload.py index de4d714..d802ec9 100644 --- a/tests/integration/test_crypto_entropy_payload.py +++ b/tests/integration/test_crypto_entropy_payload.py @@ -68,8 +68,7 @@ def test_crypto_entropy_payload_heuristics(crypto_payload_result): # These anomalies are expected for this test binary assert structural_reasons == { - "load_config_guard_cf_inconsistent", - "unmapped", + "load_config_guard_cf_inconsistent", "load_config_cookie_invalid" } diff --git a/tests/integration/test_franken_malformed_pe.py b/tests/integration/test_franken_malformed_pe.py index de8cc9e..5822033 100644 --- a/tests/integration/test_franken_malformed_pe.py +++ b/tests/integration/test_franken_malformed_pe.py @@ -45,11 +45,11 @@ def test_franken_expected_heuristics(franken_result): "data_directory_zero_rva_nonzero_size", "section_raw_misaligned", "section_overlap", - "section_raw_overlap" + "section_raw_overlap", + "exception_directory_size_not_multiple", + "exception_table_truncated" } - print(heur) - assert heur == expected @pytest.mark.integration diff --git a/tests/integration/test_string_obfuscation_tricks.py b/tests/integration/test_string_obfuscation_tricks.py index 9fa5cb2..3b7d22d 100644 --- a/tests/integration/test_string_obfuscation_tricks.py +++ b/tests/integration/test_string_obfuscation_tricks.py @@ -67,7 +67,7 @@ def test_string_obfuscation_heuristics(string_obfuscation_tricks_result): # These anomalies are expected for this test binary assert structural_reasons == { "load_config_guard_cf_inconsistent", - "unmapped", + "load_config_cookie_invalid", } diff --git a/tests/unit/analysis/test_reason_codes.py b/tests/unit/analysis/test_reason_codes.py new file mode 100644 index 0000000..1fb4214 --- /dev/null +++ b/tests/unit/analysis/test_reason_codes.py @@ -0,0 +1,378 @@ +# Copyright (c) 2026 MalX Labs and contributors +# SPDX-License-Identifier: MPL-2.0 + +""" +Regression tests for the heuristic reason-code contract. + +BACKGROUND +---------- +`_det()` builds its metadata as ``{"reason": reason, **(metadata or {})}``, so a +payload key literally named ``reason`` silently OVERWRITES the parent reason +code. Because ``_analyse_structural`` forwards each validator's ``details`` +verbatim, any validator emitting ``details={"reason": ...}`` lost its parent +code in the output. + +The bug was invisible for a long time: the clobbered output still *looked* +structured (a plausible string in a ``reason`` field), and +``analysis["structural"]`` is never copied into the result, so no unclobbered +view existed to compare against. Eleven documented reason codes had never once +appeared in output. + +These tests lock the contract at both ends: + + * SOURCE - no validator may put a top-level ``reason`` key in ``details``. + * OUTPUT - every ``pe_structure_anomaly`` reason must be a real ReasonCode. + +Either test alone would have caught the original defect. + +DESIGN NOTE +----------- +The membership assertion is deliberately scoped to ``pe_structure_anomaly`` +detections. The behavioural heuristics (anti-debug, import anomalies) use their +own vocabulary - ``anti_debug_api_import``, ``rwx_section``, +``large_import_table`` etc. - which is intentionally NOT part of ReasonCodes. +Those are pinned by an explicit allowlist instead, so adding a new behavioural +reason is a conscious act rather than an accident. +""" + +import pytest + +from iocx.reason_codes import ReasonCodes +from iocx.analysis.heuristics import analyse_pe_heuristics + + +# -------------------------------------------------------------------------- +# Helpers +# -------------------------------------------------------------------------- + +def _reason_code_values() -> set: + """Every string value declared on ReasonCodes.""" + return { + v for k, v in vars(ReasonCodes).items() + if not k.startswith("_") and isinstance(v, str) + } + + +def _as_dicts(detections): + """Normalise Detection objects (or dicts) to plain dicts.""" + out = [] + for d in detections: + if isinstance(d, dict): + out.append(d) + else: + out.append({ + "value": d.value, + "category": d.category, + "metadata": d.metadata, + }) + return out + + +# Behavioural heuristic reasons that are intentionally NOT ReasonCodes members. +# Adding to this set should be a deliberate decision, not a silent drift. +BEHAVIOURAL_REASONS = { + "anti_debug_api_import", + "timing_api_import", + "rwx_section", + "large_import_table", + "high_ordinal_import_ratio", + "uncommon_dll_for_gui_subsystem", +} + + +# -------------------------------------------------------------------------- +# Fixtures +# -------------------------------------------------------------------------- + +@pytest.fixture +def structural_payload(): + """ + One issue per validator that historically carried a colliding ``reason`` + key, plus controls that never did. Mirrors the real + ``analysis["structural"]`` shape: {category: [ {issue, details}, ... ]}. + """ + return { + "load_config_directory": [ + {"issue": ReasonCodes.LOAD_CONFIG_SEH_INVALID, + "details": {"seh_table_rva": 0, "seh_count": 4, + "sub_reason": "missing_table_rva"}}, + {"issue": ReasonCodes.LOAD_CONFIG_COOKIE_INVALID, + "details": {"cookie_rva": 0x3500, "section": ".rdata", + "sub_reason": "non_writable_section"}}, + # control: never had a colliding key + {"issue": ReasonCodes.LOAD_CONFIG_COOKIE_IN_OVERLAY, + "details": {"cookie_rva": 0x3500, "cookie_raw": 0xB00, + "overlay_offset": 0x694}}, + ], + "tls": [ + {"issue": ReasonCodes.TLS_DIRECTORY_TRUNCATED, + "details": {"sub_reason": "header_decode", "errors": ["x"]}}, + {"issue": ReasonCodes.TLS_CALLBACK_RVA_INVALID, + "details": {"callback_va": 0x401000, + "sub_reason": "below_image_base", + "invalid_callback_count": 1}}, + ], + "sections": [ + {"issue": ReasonCodes.SECTION_FLAGS_INCONSISTENT, + "details": {"section": ".text", + "sub_reason": "code_without_read"}}, + {"issue": ReasonCodes.SECTION_RWX, + "details": {"section": ".text", "characteristics": 0xE0000020}}, + ], + "entrypoint": [ + {"issue": ReasonCodes.ENTRYPOINT_IN_TRUNCATED_REGION, + "details": {"entry_point": 0x1000, "section": ".text", + "sub_reason": "zero_length_section"}}, + ], + "signature": [ + {"issue": ReasonCodes.CERTIFICATE_TABLE_MALFORMED, + "details": {"sub_reason": "truncation", "region": "cert_blob"}}, + ], + "version_info": [ + {"issue": ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER, + "details": {"sub_reason": "szkey_mismatch"}}, + ], + "exception_table": [ + # uses "table", never collided - must be untouched + {"issue": ReasonCodes.EXCEPTION_TABLE_TRUNCATED, + "details": {"table": "exception_entry_truncated"}}, + ], + } + + +@pytest.fixture +def legacy_structural_payload(): + """ + An UNMIGRATED validator payload that still uses the reserved ``reason`` + key. + + This is the adversarial case: it proves the emission layer defends itself + rather than relying on every validator having been migrated. Without it the + output tests only assert the already-fixed state and would pass even + against the original clobbering code. + """ + return { + "load_config_directory": [ + {"issue": ReasonCodes.LOAD_CONFIG_SEH_INVALID, + "details": {"seh_table_rva": 0, "seh_count": 4, + "reason": "missing_table_rva"}}, # legacy key + ], + } + + +@pytest.fixture +def analysis(structural_payload): + return { + "sections": [], + "extended": [], + "obfuscation": [], + "structural": structural_payload, + } + + +# ========================================================================== +# OUTPUT CONTRACT +# ========================================================================== + +def test_structural_reasons_are_declared_reason_codes(analysis): + """ + THE regression test. Every ``pe_structure_anomaly`` reason must be a + declared ReasonCode. + + Under the clobber bug this fails loudly: the emitted reasons were payload + strings such as ``missing_table_rva`` and ``below_image_base``, which are + sub-reasons and are not ReasonCodes members. + """ + valid = _reason_code_values() + + offenders = [] + for det in _as_dicts(analyse_pe_heuristics({}, analysis)): + if det["value"] != "pe_structure_anomaly": + continue + reason = det["metadata"].get("reason") + if reason not in valid: + offenders.append(reason) + + assert not offenders, ( + "pe_structure_anomaly emitted reason(s) that are not declared in " + f"ReasonCodes: {sorted(set(offenders))}. This usually means a " + "validator put a top-level 'reason' key in its details payload, which " + "_det() merges OVER the parent reason code. Use 'sub_reason' instead." + ) + + +def test_every_structural_parent_code_survives(analysis, structural_payload): + """ + Stronger form: the emitted reason must equal the *specific* parent code the + validator raised, one-for-one and in order. + """ + expected = [ + issue["issue"] + for issues in structural_payload.values() + for issue in issues + ] + emitted = [ + d["metadata"]["reason"] + for d in _as_dicts(analyse_pe_heuristics({}, analysis)) + if d["value"] == "pe_structure_anomaly" + ] + + assert emitted == expected, ( + "Structural parent codes were altered in transit.\n" + f" expected: {expected}\n" + f" emitted : {emitted}" + ) + + +def test_sub_reason_is_preserved_alongside_parent(analysis): + """ + The parent must not win by *destroying* the sub-reason. Both survive. + + This guards against the naive 'fix' of reordering the merge so the parent + overwrites the payload - that trades one information loss for another. + """ + by_reason = { + d["metadata"]["reason"]: d["metadata"] + for d in _as_dicts(analyse_pe_heuristics({}, analysis)) + if d["value"] == "pe_structure_anomaly" + } + + md = by_reason[ReasonCodes.LOAD_CONFIG_SEH_INVALID] + assert md.get("sub_reason") == "missing_table_rva" + # payload fields must survive intact too + assert md.get("seh_table_rva") == 0 + assert md.get("seh_count") == 4 + + +def test_no_sub_reason_key_invented_when_none_existed(analysis): + """ + Issues whose details never carried a colliding key must be untouched - no + spurious ``sub_reason`` added. + """ + by_reason = { + d["metadata"]["reason"]: d["metadata"] + for d in _as_dicts(analyse_pe_heuristics({}, analysis)) + if d["value"] == "pe_structure_anomaly" + } + + assert "sub_reason" not in by_reason[ReasonCodes.LOAD_CONFIG_COOKIE_IN_OVERLAY] + assert "sub_reason" not in by_reason[ReasonCodes.SECTION_RWX] + # 'table' is a distinct key and must be passed through unchanged + assert by_reason[ReasonCodes.EXCEPTION_TABLE_TRUNCATED]["table"] == \ + "exception_entry_truncated" + + +def test_legacy_reason_key_cannot_clobber_parent(legacy_structural_payload): + """ + THE bug, reproduced exactly. + + A validator that has not been migrated still emits ``details={"reason": + ...}``. The pipeline must survive that: the parent code wins and the legacy + value is preserved under ``sub_reason``. + + Against the original code this fails with + ``reason == "missing_table_rva"``. + """ + analysis = {"sections": [], "extended": [], "obfuscation": [], + "structural": legacy_structural_payload} + + dets = [d for d in _as_dicts(analyse_pe_heuristics({}, analysis)) + if d["value"] == "pe_structure_anomaly"] + assert len(dets) == 1 + md = dets[0]["metadata"] + + assert md["reason"] == ReasonCodes.LOAD_CONFIG_SEH_INVALID, ( + "A legacy 'reason' detail key overwrote the parent reason code. The " + "emission layer must re-key it to 'sub_reason' before merging." + ) + assert md.get("sub_reason") == "missing_table_rva", ( + "The parent code was preserved by DESTROYING the sub-reason. Both must " + "survive - re-key the collision, do not simply reorder the merge." + ) + assert md["seh_count"] == 4 # payload intact + + +def test_behavioural_reasons_match_allowlist(): + """ + Behavioural heuristics use a vocabulary outside ReasonCodes. Pin it, so a + new reason string is a deliberate change rather than silent drift. + """ + metadata = { + "import_details": [ + {"dll": "kernel32.dll", "function": "IsDebuggerPresent"}, + {"dll": "kernel32.dll", "function": "GetTickCount"}, + ], + } + analysis = { + "sections": [{"name": ".text", "characteristics": 0xE0000020, + "entropy": 0.0, "raw_size": 512}], + "extended": [], + } + + emitted = { + d["metadata"]["reason"] + for d in _as_dicts(analyse_pe_heuristics(metadata, analysis)) + if d["value"] != "pe_structure_anomaly" + } + + known = BEHAVIOURAL_REASONS | _reason_code_values() + assert emitted <= known, ( + f"Unrecognised behavioural reason(s): {sorted(emitted - known)}. " + "Add to BEHAVIOURAL_REASONS if intentional." + ) + + +# ========================================================================== +# SOURCE CONTRACT +# ========================================================================== + +def test_no_validator_emits_a_top_level_reason_detail(): + """ + Static guard at the source. No validator may use ``reason`` as a top-level + key in ``details`` - that is the reserved name the emission layer owns. + Use ``sub_reason``. + + Catches the defect at authoring time, before any file is analysed, and + covers validators that no fixture happens to exercise. + """ + import pathlib + import re + + validators_dir = pathlib.Path( + "iocx/validators" + ) + if not validators_dir.is_dir(): # pragma: no cover - path guard + pytest.skip(f"validators directory not found at {validators_dir}") + + pattern = re.compile(r'["\']reason["\']\s*:') + + offenders = [] + for path in sorted(validators_dir.glob("*.py")): + for lineno, line in enumerate(path.read_text().splitlines(), 1): + if pattern.search(line): + offenders.append(f"{path.name}:{lineno}: {line.strip()}") + + assert not offenders, ( + "Validators must not use a top-level 'reason' key in details " + "(it is overwritten by the emission layer). Use 'sub_reason'.\n " + + "\n ".join(offenders) + ) + + +@pytest.mark.parametrize("colliding_key", ["reason"]) +def test_det_parent_is_not_overwritable(colliding_key): + """ + Directly pin ``_det``'s merge behaviour: a caller-supplied payload must not + be able to overwrite the parent reason code. + """ + from iocx.analysis.heuristics import _det + + det = _det("pe_structure_anomaly", + ReasonCodes.LOAD_CONFIG_SEH_INVALID, + {colliding_key: "attacker_controlled"}) + + metadata = det.metadata if hasattr(det, "metadata") else det["metadata"] + assert metadata["reason"] == ReasonCodes.LOAD_CONFIG_SEH_INVALID, ( + f"_det allowed a payload key '{colliding_key}' to overwrite the parent " + "reason code." + ) diff --git a/tests/unit/parsers/test_pe_exception.py b/tests/unit/parsers/test_pe_exception.py new file mode 100644 index 0000000..c879649 --- /dev/null +++ b/tests/unit/parsers/test_pe_exception.py @@ -0,0 +1,864 @@ +# Copyright (c) 2026 MalX Labs and contributors +# SPDX-License-Identifier: MPL-2.0 + +""" +Unit tests for pe_exception.build_exception_structure. + +Strategy: +- The parser touches pefile only via three surfaces: OPTIONAL_HEADER. + DATA_DIRECTORY[3], FILE_HEADER.Machine, and pe.get_data(rva, length). + FakePE below implements exactly those, over a flat {rva: bytes} map, so + every branch is driven from real bytes without building a PE on disk. +- Tests assert on the documented output contract: the top-level key set, the + arch/entry_size routing, the per-entry dicts, and the tombstone tags in + `errors` / `truncations`. + +get_data semantics: pefile returns a SHORT buffer when a read runs past the +end of a section rather than raising, and raises for an unmapped RVA. FakePE +reproduces both, because the parser distinguishes them +(exception_entry_read_failed vs exception_entry_truncated). + +Contract note: these tombstone tags are the parser's half of a contract with +validators.exception_table, whose _ENTRY_ERROR_PRIORITY and +_UNWIND_ERROR_PRIORITY lists consume them. TestValidatorContract at the end +pins the exact vocabulary so the two cannot drift apart silently. +""" + +from __future__ import annotations + +import struct +from typing import Any, Dict, List, Optional + +import pytest + +from iocx.parsers.pe_exception import build_exception_structure + + +# Machine values the parser routes on +M_I386 = 0x014C +M_IA64 = 0x0200 +M_ARM = 0x01C0 +M_ARMNT = 0x01C4 +M_AMD64 = 0x8664 +M_ARM64 = 0xAA64 +M_ARM64EC = 0xA641 + +DIR_RVA = 0x1000 +UW_RVA = 0x3000 + + +# ================================================================= +# Test doubles +# ================================================================= + +class _Dir: + def __init__(self, va, size): + self.VirtualAddress = va + self.Size = size + + +class _OptionalHeader: + def __init__(self, dirs): + self.DATA_DIRECTORY = dirs + + +class _FileHeader: + def __init__(self, machine): + self.Machine = machine + + +class FakePE: + """ + Minimal pefile stand-in exposing only what the parser uses. + + `mem` is a flat {base_rva: bytes} map. get_data mirrors pefile: a read + inside a mapped blob but running past its end returns a SHORT buffer; a + read at an unmapped RVA raises. + """ + + def __init__(self, machine: Any = M_AMD64, + exc: Optional[tuple] = (DIR_RVA, 24), + mem: Optional[Dict[int, bytes]] = None, + dirs_len: int = 16, + omit_optional_header: bool = False, + omit_file_header: bool = False): + dirs = [_Dir(0, 0) for _ in range(dirs_len)] + if exc is not None and dirs_len > 3: + dirs[3] = _Dir(*exc) + if not omit_optional_header: + self.OPTIONAL_HEADER = _OptionalHeader(dirs) + if not omit_file_header: + self.FILE_HEADER = _FileHeader(machine) + self._mem = mem or {} + + def get_data(self, rva: int, length: int) -> bytes: + for base, blob in self._mem.items(): + if base <= rva < base + len(blob): + off = rva - base + return blob[off:off + length] # may be short, like pefile + raise ValueError(f"unmapped rva 0x{rva:X}") + + +# ================================================================= +# Byte builders +# ================================================================= + +def rf(begin: int, end: int, unwind: int) -> bytes: + """AMD64 RUNTIME_FUNCTION: 3 x DWORD.""" + return struct.pack(" bytes: + """ARM(64) .pdata record: 2 x DWORD.""" + return struct.pack(" bytes: + """ + UNWIND_INFO: byte0 = version(bits 2:0) | flags(bits 7:3), then prolog, + count, frame; then the even-padded USHORT code array; then optionally a + trailing RUNTIME_FUNCTION for the chained case. + """ + b0 = (version & 0x07) | ((flags & 0x1F) << 3) + out = bytes([b0, prolog & 0xFF, count & 0xFF, 0]) + padded = (count + 1) & ~1 + out += b"\x00" * (padded * 2) + if chain is not None: + out += struct.pack(" bytes: + """Two sorted amd64 entries, both pointing at a valid unwind blob.""" + return rf(0x2000, 0x2050, UW_RVA) + rf(0x2060, 0x20B0, UW_RVA) + + +def _amd64_pe(entries: bytes, size: Optional[int] = None, + unwind: Optional[bytes] = None, + unwind_rva: int = UW_RVA, **kw) -> FakePE: + mem = {DIR_RVA: entries} + if unwind is not None: + mem[unwind_rva] = unwind + return FakePE(machine=M_AMD64, exc=(DIR_RVA, size or len(entries)), + mem=mem, **kw) + + +def _first_unwind(unwind: bytes, unwind_rva: int = UW_RVA) -> Dict[str, Any]: + """Decode a single entry pointing at `unwind`; return its unwind dict.""" + pe = _amd64_pe(rf(0x2000, 0x2050, unwind_rva), size=12, + unwind=unwind, unwind_rva=unwind_rva) + return build_exception_structure(pe)["functions"][0]["unwind"] + + +# ================================================================= +# Absence +# ================================================================= + +class TestAbsence: + """Absence of an exception directory returns None - never an error.""" + + def test_no_optional_header_returns_none(self): + assert build_exception_structure( + FakePE(omit_optional_header=True)) is None + + def test_short_directory_array_returns_none(self): + """IndexError on DATA_DIRECTORY[3] is swallowed.""" + assert build_exception_structure(FakePE(dirs_len=3)) is None + + def test_zero_rva_returns_none(self): + assert build_exception_structure(FakePE(exc=(0, 24))) is None + + def test_zero_size_returns_none(self): + assert build_exception_structure(FakePE(exc=(DIR_RVA, 0))) is None + + def test_both_zero_returns_none(self): + assert build_exception_structure(FakePE(exc=(0, 0))) is None + + def test_non_int_directory_fields_return_none(self): + """A ValueError/TypeError from int() is swallowed.""" + assert build_exception_structure( + FakePE(exc=("not-an-int", 24))) is None + + +# ================================================================= +# Machine reading and arch routing +# ================================================================= + +class TestMachineAndArch: + + @pytest.mark.parametrize("machine,arch,entry_size", [ + (M_AMD64, "amd64", 12), + (M_ARM64, "arm64", 8), + (M_ARM64EC, "arm64", 8), # ARM64EC shares the ARM64 table format + (M_ARM, "arm", 8), + (M_ARMNT, "arm", 8), + (M_I386, "unsupported", 0), + (M_IA64, "unsupported", 0), + (0xDEAD, "unsupported", 0), + ]) + def test_arch_classification(self, machine, arch, entry_size): + pe = FakePE(machine=machine, exc=(DIR_RVA, 24), + mem={DIR_RVA: b"\x00" * 24}) + result = build_exception_structure(pe) + assert result["machine"] == machine + assert result["arch"] == arch + assert result["entry_size"] == entry_size + + def test_missing_file_header_is_unsupported(self): + pe = FakePE(exc=(DIR_RVA, 12), mem={DIR_RVA: rf(0x2000, 0x2050, 0)}, + omit_file_header=True) + result = build_exception_structure(pe) + assert result["machine"] is None + assert result["arch"] == "unsupported" + + def test_non_int_machine_is_unsupported(self): + pe = FakePE(machine="nope", exc=(DIR_RVA, 12), + mem={DIR_RVA: rf(0x2000, 0x2050, 0)}) + result = build_exception_structure(pe) + assert result["machine"] is None + assert result["arch"] == "unsupported" + + def test_unsupported_machine_skips_the_walk(self): + """ + A present directory on a non-table arch must report placement but + decode no entries, so the validator emits one code rather than + spurious per-entry noise. + """ + pe = FakePE(machine=M_I386, exc=(DIR_RVA, 24), + mem={DIR_RVA: _control_table()}) + result = build_exception_structure(pe) + assert result["functions"] == [] + assert result["truncations"] == [] + assert result["errors"] == [] + assert result["rva"] == DIR_RVA and result["size"] == 24 + + +# ================================================================= +# Output contract +# ================================================================= + +class TestOutputContract: + + def test_top_level_key_set_is_stable(self): + expected = {"rva", "size", "machine", "arch", "entry_size", + "functions", "truncations", "errors"} + pe = _amd64_pe(_control_table(), unwind=unwind_bytes()) + assert set(build_exception_structure(pe)) == expected + + def test_unsupported_path_has_same_key_set(self): + """The early return must not produce a different shape.""" + expected = {"rva", "size", "machine", "arch", "entry_size", + "functions", "truncations", "errors"} + pe = FakePE(machine=M_I386, exc=(DIR_RVA, 24), + mem={DIR_RVA: b"\x00" * 24}) + assert set(build_exception_structure(pe)) == expected + + def test_amd64_entry_key_set(self): + expected = {"index", "begin_rva", "end_rva", "unwind_info_rva", + "unwind", "errors"} + pe = _amd64_pe(rf(0x2000, 0x2050, UW_RVA), size=12, + unwind=unwind_bytes()) + assert set(build_exception_structure(pe)["functions"][0]) == expected + + def test_arm_entry_key_set(self): + """ARM entries add is_packed / packed_data.""" + expected = {"index", "begin_rva", "end_rva", "unwind_info_rva", + "unwind", "is_packed", "packed_data", "errors"} + pe = FakePE(machine=M_ARM64, exc=(DIR_RVA, 8), + mem={DIR_RVA: arm_rec(0x2000, 0x4000)}) + assert set(build_exception_structure(pe)["functions"][0]) == expected + + def test_unwind_key_set(self): + expected = {"version", "flags", "size_of_prolog", "count_of_codes", + "is_chained", "chained_rva", "errors"} + assert set(_first_unwind(unwind_bytes())) == expected + + def test_indices_are_sequential(self): + entries = b"".join(rf(0x2000 + i * 0x100, 0x2050 + i * 0x100, 0) + for i in range(4)) + pe = _amd64_pe(entries, size=48) + result = build_exception_structure(pe) + assert [f["index"] for f in result["functions"]] == [0, 1, 2, 3] + + def test_json_serialisable(self): + import json + pe = _amd64_pe(_control_table(), unwind=unwind_bytes(flags=0x01)) + json.dumps(build_exception_structure(pe)) # must not raise + + def test_never_raises_on_hostile_input(self): + """ + Random bytes at every surface must produce a struct, not an exception. + """ + import os + for seed in range(20): + blob = os.urandom(64) + pe = FakePE(machine=M_AMD64, exc=(DIR_RVA, 48), + mem={DIR_RVA: blob, UW_RVA: blob}) + result = build_exception_structure(pe) + assert isinstance(result, dict) + + +# ================================================================= +# Counted-array walk +# ================================================================= + +class TestFunctionTableWalk: + + def test_entry_count_derived_from_size_and_stride(self): + pe = _amd64_pe(_control_table(), size=24) + assert len(build_exception_structure(pe)["functions"]) == 2 + + def test_size_smaller_than_data_limits_the_walk(self): + """The declared Size is authoritative, not the available bytes.""" + pe = _amd64_pe(_control_table(), size=12) + assert len(build_exception_structure(pe)["functions"]) == 1 + + def test_arm_stride_is_eight(self): + entries = arm_rec(0x2000, 0x4000) + arm_rec(0x2100, 0x4100) + pe = FakePE(machine=M_ARM64, exc=(DIR_RVA, 16), mem={DIR_RVA: entries}) + assert len(build_exception_structure(pe)["functions"]) == 2 + + def test_ragged_tail_tagged_and_partial_entry_not_decoded(self): + pe = _amd64_pe(_control_table(), size=25) + result = build_exception_structure(pe) + assert result["truncations"] == ["exception_table_ragged_tail"] + assert len(result["functions"]) == 2 # the partial 3rd is not decoded + + def test_unmapped_entry_tagged_read_failed(self): + """Declaring more entries than are mapped raises inside get_data.""" + pe = _amd64_pe(_control_table(), size=36) + result = build_exception_structure(pe) + assert result["truncations"] == ["exception_entry_read_failed"] + assert len(result["functions"]) == 2 + + def test_short_read_tagged_entry_truncated(self): + """A read running past the mapped blob returns short, not an error.""" + entries = rf(0x2000, 0x2050, 0) + b"\x00\x00\x00" + pe = _amd64_pe(entries, size=24) + result = build_exception_structure(pe) + assert result["truncations"] == ["exception_entry_truncated"] + assert len(result["functions"]) == 1 + + def test_read_failed_and_truncated_are_distinct(self): + """ + The two tags mean different things - unmapped vs short - and the + validator surfaces both under EXCEPTION_TABLE_TRUNCATED. Pin that they + do not collapse into one. + """ + unmapped = build_exception_structure( + _amd64_pe(_control_table(), size=36))["truncations"] + short = build_exception_structure( + _amd64_pe(rf(0x2000, 0x2050, 0) + b"\x00", size=24))["truncations"] + assert unmapped == ["exception_entry_read_failed"] + assert short == ["exception_entry_truncated"] + + def test_max_functions_clamp_tagged(self): + """ + A bogus Size claiming more than 2**20 entries is clamped and tagged + before the walk begins. + """ + size = ((1 << 20) + 1) * 12 + pe = _amd64_pe(rf(0x2000, 0x2050, 0), size=size) + result = build_exception_structure(pe) + assert "exception_table_max_exceeded" in result["truncations"] + + def test_empty_walk_when_size_below_one_entry(self): + pe = _amd64_pe(_control_table(), size=8) + result = build_exception_structure(pe) + assert result["functions"] == [] + assert result["truncations"] == ["exception_table_ragged_tail"] + + +# ================================================================= +# AMD64 entry decode +# ================================================================= + +class TestAmd64EntryDecode: + + def test_fields_decoded_little_endian(self): + pe = _amd64_pe(rf(0x11223344, 0x55667788, 0x99AABBCC), size=12) + f = build_exception_structure(pe)["functions"][0] + assert f["begin_rva"] == 0x11223344 + assert f["end_rva"] == 0x55667788 + assert f["unwind_info_rva"] == 0x99AABBCC + + def test_clean_entry_has_no_errors(self): + pe = _amd64_pe(rf(0x2000, 0x2050, UW_RVA), size=12, + unwind=unwind_bytes()) + assert build_exception_structure(pe)["functions"][0]["errors"] == [] + + @pytest.mark.parametrize("begin,end,unwind,tags", [ + (0, 0x2050, UW_RVA, ["begin_rva_zero"]), + (0x2000, 0, UW_RVA, ["end_rva_zero"]), + (0x2000, 0x2050, 0, ["unwind_rva_zero"]), + (0, 0, 0, ["begin_rva_zero", "end_rva_zero", "unwind_rva_zero"]), + ]) + def test_zero_field_tags(self, begin, end, unwind, tags): + pe = _amd64_pe(rf(begin, end, unwind), size=12, + unwind=unwind_bytes()) + assert build_exception_structure(pe)["functions"][0]["errors"] == tags + + def test_zero_unwind_rva_skips_unwind_decode(self): + """No pointer means no .xdata read; unwind stays None.""" + pe = _amd64_pe(rf(0x2000, 0x2050, 0), size=12) + assert build_exception_structure(pe)["functions"][0]["unwind"] is None + + def test_nonzero_unwind_rva_triggers_decode(self): + pe = _amd64_pe(rf(0x2000, 0x2050, UW_RVA), size=12, + unwind=unwind_bytes()) + assert build_exception_structure(pe)["functions"][0]["unwind"] is not None + + +# ================================================================= +# UNWIND_INFO decode +# ================================================================= + +class TestUnwindInfoDecode: + + def test_header_fields_decoded(self): + u = _first_unwind(unwind_bytes(version=1, flags=0x01, prolog=0x12, + count=3)) + assert u["version"] == 1 + assert u["flags"] == 0x01 + assert u["size_of_prolog"] == 0x12 + assert u["count_of_codes"] == 3 + + def test_byte0_bit_packing(self): + """version occupies bits[2:0]; flags bits[7:3].""" + u = _first_unwind(unwind_bytes(version=3, flags=0x0F)) + assert u["version"] == 3 + assert u["flags"] == 0x0F + + @pytest.mark.parametrize("version", [1, 2, 3]) + def test_valid_versions_untagged(self, version): + assert _first_unwind(unwind_bytes(version=version))["errors"] == [] + + @pytest.mark.parametrize("version", [0, 4, 5, 6, 7]) + def test_invalid_versions_tagged(self, version): + u = _first_unwind(unwind_bytes(version=version)) + assert "unwind_version_invalid" in u["errors"] + + @pytest.mark.parametrize("flags", [0x00, 0x01, 0x02, 0x08, 0x0B]) + def test_known_flag_bits_untagged(self, flags): + assert _first_unwind(unwind_bytes(flags=flags))["errors"] == [] + + @pytest.mark.parametrize("flags", [0x10, 0x18, 0x1F]) + def test_reserved_flag_bits_tagged(self, flags): + u = _first_unwind(unwind_bytes(flags=flags)) + assert "unwind_flags_reserved_bits" in u["errors"] + + def test_unmapped_unwind_rva_tagged_read_failed(self): + pe = _amd64_pe(rf(0x2000, 0x2050, 0x9000), size=12) + u = build_exception_structure(pe)["functions"][0]["unwind"] + assert u["errors"] == ["unwind_read_failed"] + assert u["version"] is None # nothing decoded + + def test_short_header_tagged_truncated(self): + pe = _amd64_pe(rf(0x2000, 0x2050, UW_RVA), size=12, + unwind=b"\x01\x04") # 2 of 4 bytes + u = build_exception_structure(pe)["functions"][0]["unwind"] + assert u["errors"] == ["unwind_truncated"] + + def test_version_and_flags_tags_can_coexist(self): + u = _first_unwind(unwind_bytes(version=5, flags=0x10)) + assert u["errors"] == ["unwind_version_invalid", + "unwind_flags_reserved_bits"] + + +# ================================================================= +# Chained unwind +# ================================================================= + +class TestChainedUnwind: + + def test_chain_resolved_for_v1(self): + u = _first_unwind(unwind_bytes(version=1, flags=0x04, count=0, + chain=(0x2000, 0x2050, 0x4000))) + assert u["is_chained"] is True + assert u["chained_rva"] == 0x4000 + + def test_chain_resolved_for_v2(self): + u = _first_unwind(unwind_bytes(version=2, flags=0x04, count=0, + chain=(0x2000, 0x2050, 0x4000))) + assert u["is_chained"] is True + assert u["chained_rva"] == 0x4000 + + @pytest.mark.parametrize("count,expected_offset", [ + (0, 4), (1, 8), (2, 8), (3, 12), (4, 12), + ]) + def test_chain_offset_skips_even_padded_code_array(self, count, + expected_offset): + """ + Unwind codes are USHORT[] padded to an even count, so the trailing + RUNTIME_FUNCTION sits at 4 + ((count+1) & ~1) * 2. + """ + u = _first_unwind(unwind_bytes(version=1, flags=0x04, count=count, + chain=(0, 0, 0x4444))) + assert u["chained_rva"] == 0x4444 + assert 4 + (((count + 1) & ~1) * 2) == expected_offset + + def test_missing_trailing_record_tagged(self): + """CHAININFO set but no trailing RUNTIME_FUNCTION present.""" + u = _first_unwind(unwind_bytes(version=1, flags=0x04, count=0)) + assert u["errors"] == ["unwind_codes_truncated"] + assert u["is_chained"] is True + assert u["chained_rva"] is None + + def test_short_trailing_record_tagged(self): + blob = unwind_bytes(version=1, flags=0x04, count=0) + b"\x00" * 6 + u = _first_unwind(blob) + assert u["errors"] == ["unwind_codes_truncated"] + + def test_v3_recognised_but_chain_not_resolved(self): + """ + V3 (APX preview) repacks the payload, so the parser surfaces + version/flags and declines to follow the chain rather than + mis-decoding it. + """ + u = _first_unwind(unwind_bytes(version=3, flags=0x04, count=0, + chain=(0, 0, 0x4000))) + assert u["version"] == 3 + assert u["flags"] == 0x04 + assert u["is_chained"] is False + assert u["chained_rva"] is None + assert u["errors"] == [] # recognised, not an error + + def test_no_chain_flag_means_no_resolution(self): + u = _first_unwind(unwind_bytes(version=1, flags=0x01, count=0, + chain=(0, 0, 0x4000))) + assert u["is_chained"] is False + assert u["chained_rva"] is None + + def test_chain_and_version_tags_coexist(self): + """A reserved-bit flag alongside CHAININFO still resolves the chain.""" + u = _first_unwind(unwind_bytes(version=1, flags=0x04 | 0x10, count=0, + chain=(0, 0, 0x4000))) + assert "unwind_flags_reserved_bits" in u["errors"] + assert u["chained_rva"] == 0x4000 + + +# ================================================================= +# ARM / ARM64 entry decode +# ================================================================= + +class TestArmEntryDecode: + + def _arm(self, begin: int, word1: int, machine: int = M_ARM64): + pe = FakePE(machine=machine, exc=(DIR_RVA, 8), + mem={DIR_RVA: arm_rec(begin, word1)}) + return build_exception_structure(pe)["functions"][0] + + def test_flag_zero_yields_xdata_rva(self): + f = self._arm(0x2000, 0x4000) + assert f["is_packed"] is False + assert f["unwind_info_rva"] == 0x4000 + assert f["packed_data"] is None + + @pytest.mark.parametrize("flag", [1, 2, 3]) + def test_nonzero_flag_yields_packed_data(self, flag): + word1 = 0x0AB10000 | flag + f = self._arm(0x2000, word1) + assert f["is_packed"] is True + assert f["unwind_info_rva"] is None # no .xdata pointer to check + assert f["packed_data"] == word1 + + def test_xdata_rva_is_dword_aligned(self): + """ + The .xdata pointer is taken as (word1 & ~3). + + Note the mask is NOT observable from the output: this branch is only + reached when Flag == 0, which by definition means the low 2 bits are + already clear, so masked and unmasked values are always identical. + Removing the mask is therefore a behaviour-preserving change and this + test cannot detect it - verified by mutation. The mask is defensive, + and what IS observable is that the resulting RVA is DWORD-aligned. + """ + assert self._arm(0x2000, 0x4000)["unwind_info_rva"] % 4 == 0 + assert self._arm(0x2000, 0x8004)["unwind_info_rva"] % 4 == 0 + + def test_zero_xdata_rva_tagged(self): + f = self._arm(0x2000, 0x0) + assert f["errors"] == ["unwind_rva_zero"] + + def test_packed_entry_never_tags_unwind_rva_zero(self): + """Packed records have no .xdata pointer, so the tag must not fire.""" + f = self._arm(0x2000, 0x00000001) + assert f["errors"] == [] + assert f["unwind_info_rva"] is None + + def test_begin_zero_tagged(self): + assert self._arm(0, 0x4000)["errors"] == ["begin_rva_zero"] + + def test_end_rva_always_none(self): + """ARM(64) .pdata carries no EndAddress field.""" + assert self._arm(0x2000, 0x4000)["end_rva"] is None + assert self._arm(0x2000, 0x1)["end_rva"] is None + + def test_unwind_always_none(self): + """.xdata / packed bodies are out of scope for this parser.""" + assert self._arm(0x2000, 0x4000)["unwind"] is None + + @pytest.mark.parametrize("machine", [M_ARM64, M_ARM64EC, M_ARM, M_ARMNT]) + def test_all_arm_machines_decode_identically(self, machine): + """ + ARM64EC and ARM32 use the same 8-byte record, so the decode must not + vary by machine - only the arch label does. + """ + f = self._arm(0x2000, 0x4000, machine=machine) + assert f["begin_rva"] == 0x2000 + assert f["unwind_info_rva"] == 0x4000 + assert f["is_packed"] is False + + +# ================================================================= +# Parser -> validator contract +# ================================================================= + +class TestValidatorContract: + """ + The tombstone vocabulary is a contract with + validators.exception_table, whose priority lists consume these exact + strings. A tag renamed here without updating the validator would silently + stop being reported. + """ + + ENTRY_TAGS = {"entry_truncated", "entry_read_failed", "entry_unpack_failed", + "begin_rva_zero", "end_rva_zero", "unwind_rva_zero"} + UNWIND_TAGS = {"unwind_read_failed", "unwind_truncated", + "unwind_unpack_failed", "unwind_version_invalid", + "unwind_flags_reserved_bits", "unwind_codes_truncated"} + TRUNCATION_TAGS = {"exception_table_ragged_tail", + "exception_table_max_exceeded", + "exception_entry_read_failed", + "exception_entry_truncated"} + + def test_entry_tags_are_in_the_agreed_vocabulary(self): + pe = _amd64_pe(rf(0, 0, 0), size=12) + tags = set(build_exception_structure(pe)["functions"][0]["errors"]) + assert tags <= self.ENTRY_TAGS + + def test_arm_entry_tags_are_in_the_agreed_vocabulary(self): + pe = FakePE(machine=M_ARM64, exc=(DIR_RVA, 8), + mem={DIR_RVA: arm_rec(0, 0)}) + tags = set(build_exception_structure(pe)["functions"][0]["errors"]) + assert tags <= self.ENTRY_TAGS + + @pytest.mark.parametrize("blob,expected", [ + (b"\x01\x04", "unwind_truncated"), + (unwind_bytes(version=5), "unwind_version_invalid"), + (unwind_bytes(flags=0x10), "unwind_flags_reserved_bits"), + (unwind_bytes(version=1, flags=0x04), "unwind_codes_truncated"), + ]) + def test_unwind_tags_are_in_the_agreed_vocabulary(self, blob, expected): + u = _first_unwind(blob) + assert expected in u["errors"] + assert set(u["errors"]) <= self.UNWIND_TAGS + + def test_unmapped_unwind_tag_in_vocabulary(self): + pe = _amd64_pe(rf(0x2000, 0x2050, 0x9000), size=12) + u = build_exception_structure(pe)["functions"][0]["unwind"] + assert set(u["errors"]) <= self.UNWIND_TAGS + + @pytest.mark.parametrize("size,mem_key,expected", [ + (25, DIR_RVA, "exception_table_ragged_tail"), + (36, DIR_RVA, "exception_entry_read_failed"), + ]) + def test_truncation_tags_are_in_the_agreed_vocabulary(self, size, mem_key, + expected): + pe = _amd64_pe(_control_table(), size=size) + truncations = build_exception_structure(pe)["truncations"] + assert expected in truncations + assert set(truncations) <= self.TRUNCATION_TAGS + + def test_top_level_errors_stays_empty_on_recoverable_faults(self): + """ + The parser records recoverable faults per-entry or per-table; the + top-level `errors` list drives the validator's short-circuit and must + not be populated by ordinary decode problems. + """ + pe = _amd64_pe(rf(0, 0, 0) + b"\x00\x00", size=25) + assert build_exception_structure(pe)["errors"] == [] + + +# ================================================================= +# Defensive unpack branches +# ================================================================= + +class TestDefensiveUnpackBranches: + """ + Four `except struct.error` clauses that are UNREACHABLE through the public + API. Each is preceded by an explicit length guard: + + _decode_amd64_entry <- `if len(raw) < entry_size: break` in the walk + _decode_arm_entry <- same guard + _decode_unwind_info <- `if len(header) < _UNWIND_HEADER_SIZE: return` + (chain unpack) <- `if len(rf) < _RUNTIME_FUNCTION_SIZE: tag` + + Verified: a get_data returning a LONGER buffer than requested, or a + bytearray, still unpacks cleanly, so no realistic pefile behaviour reaches + them. They exist so a future refactor that weakens a guard degrades to a + tombstone instead of raising - the parser's "never raises" contract. + + The two entry branches are reached by calling the private helper directly + with a short buffer. The two unwind branches are reachable only by + injecting a struct.error, since the guard and the unpack read the same + buffer; mock.patch is used deliberately rather than contorting the input. + """ + + def test_amd64_entry_short_buffer_yields_tombstone(self): + from iocx.parsers.pe_exception import _decode_amd64_entry + entry = _decode_amd64_entry(None, b"\x00" * 11, index=7) + assert entry["errors"] == ["entry_unpack_failed"] + assert entry["index"] == 7 + assert entry["begin_rva"] is None + assert entry["end_rva"] is None + assert entry["unwind_info_rva"] is None + assert entry["unwind"] is None + + def test_amd64_tombstone_key_set_matches_normal_entry(self): + """ + The tombstone must be shape-compatible with a decoded entry, or the + validator's .get() calls would silently see missing fields. + """ + from iocx.parsers.pe_exception import _decode_amd64_entry + tombstone = _decode_amd64_entry(None, b"\x00" * 11, index=0) + pe = _amd64_pe(rf(0x2000, 0x2050, UW_RVA), size=12, + unwind=unwind_bytes()) + normal = build_exception_structure(pe)["functions"][0] + assert set(tombstone) == set(normal) + + def test_arm_entry_short_buffer_yields_tombstone(self): + from iocx.parsers.pe_exception import _decode_arm_entry + entry = _decode_arm_entry(b"\x00" * 7, index=9, arch="arm64") + assert entry["errors"] == ["entry_unpack_failed"] + assert entry["index"] == 9 + assert entry["begin_rva"] is None + assert entry["end_rva"] is None + assert entry["unwind_info_rva"] is None + assert entry["unwind"] is None + assert entry["is_packed"] is None + + def test_arm_tombstone_key_set_matches_normal_entry(self): + from iocx.parsers.pe_exception import _decode_arm_entry + tombstone = _decode_arm_entry(b"\x00" * 7, index=0, arch="arm64") + pe = FakePE(machine=M_ARM64, exc=(DIR_RVA, 8), + mem={DIR_RVA: arm_rec(0x2000, 0x4000)}) + normal = build_exception_structure(pe)["functions"][0] + # the tombstone omits packed_data, which is only meaningful when a + # Flag was actually decoded + assert set(tombstone) | {"packed_data"} == set(normal) + + def test_unwind_header_unpack_failure_yields_tombstone(self): + """ + Header unpack cannot fail past the length guard, so inject the error + to prove the handler degrades rather than propagating. + """ + import struct as _struct + from unittest import mock + from iocx.parsers.pe_exception import _decode_unwind_info + + real = _struct.unpack_from + + def fail_header(fmt, *args, **kwargs): + if fmt == " SizeOfImage bound check - assert rva_in_any_section(0x500, {"size_of_image": 0x1000}) is True + # no sections -> SizeOfImage bound check (explicit 3rd argument) + assert rva_in_any_section(0x500, {}, 0x1000) is True def test_fallback_out_of_size_of_image_false(self): - assert rva_in_any_section(0x2000, {"size_of_image": 0x1000}) is False + assert rva_in_any_section(0x2000, {}, 0x1000) is False def test_no_sections_and_no_size_of_image_none(self): assert rva_in_any_section(0x500, {}) is None def test_empty_sections_list_uses_fallback(self): # [] is falsy -> fallback path, not the section loop - assert rva_in_any_section(0x500, {"sections": [], - "size_of_image": 0x1000}) is True + assert rva_in_any_section(0x500, {"sections": []}, 0x1000) is True + + def test_size_of_image_in_analysis_is_ignored(self): + """ + REGRESSION GUARD. SizeOfImage must come from the explicit argument. + A stale analysis["size_of_image"] must NOT be consulted - if it were, + this would return True and the production dead-path would be back. + """ + assert rva_in_any_section(0x500, {"size_of_image": 0x1000}) is None + + def test_sections_take_precedence_over_size_of_image(self): + """Section geometry is authoritative when present.""" + analysis = {"sections": [{"rva": 0x1000, "virtual_size": 0x10}]} + # 0x9000 is inside SizeOfImage but outside every section + assert rva_in_any_section(0x9000, analysis, 0x100000) is False # ================================================================= @@ -167,8 +205,7 @@ def test_empty_sections_list_uses_fallback(self): class TestRegionInAnySection: def test_none_rva_returns_none(self): - assert region_in_any_section(None, 0x10, - {"size_of_image": 0x1000}) is None + assert region_in_any_section(None, 0x10, {}, 0x1000) is None def test_whole_region_fits_true(self): analysis = {"sections": [{"rva": 0x1000, "virtual_size": 0x1000}]} @@ -214,20 +251,28 @@ def test_spans_two_sections_not_single_false(self): assert region_in_any_section(0x1080, 0x100, analysis) is False def test_fallback_within_size_of_image_true(self): - assert region_in_any_section(0x100, 0x10, - {"size_of_image": 0x1000}) is True + assert region_in_any_section(0x100, 0x10, {}, 0x1000) is True def test_fallback_out_of_size_of_image_false(self): - assert region_in_any_section(0xFF0, 0x100, - {"size_of_image": 0x1000}) is False + assert region_in_any_section(0xFF0, 0x100, {}, 0x1000) is False def test_no_sections_and_no_size_of_image_none(self): assert region_in_any_section(0x100, 0x10, {}) is None def test_empty_sections_list_uses_fallback(self): + assert region_in_any_section(0x100, 0x10, {"sections": []}, + 0x1000) is True + + def test_size_of_image_in_analysis_is_ignored(self): + """ + REGRESSION GUARD - companion to the rva_in_any_section case above. + """ assert region_in_any_section(0x100, 0x10, - {"sections": [], - "size_of_image": 0x1000}) is True + {"size_of_image": 0x1000}) is None + + def test_sections_take_precedence_over_size_of_image(self): + analysis = {"sections": [{"rva": 0x1000, "virtual_size": 0x10}]} + assert region_in_any_section(0x9000, 0x10, analysis, 0x100000) is False # ================================================================= @@ -245,12 +290,33 @@ def test_tri_state_values(self): assert region_in_any_section(0x9000, 0x10, secs) is False assert region_in_any_section(None, 0x10, secs) is None + def test_tri_state_on_fallback_path(self): + """ + The fallback path must express the same tri-state. Without this, the + None case is only ever exercised via the section path. + """ + assert rva_in_any_section(0x500, {}, 0x1000) is True + assert rva_in_any_section(0x2000, {}, 0x1000) is False + assert rva_in_any_section(0x500, {}) is None + assert region_in_any_section(0x500, 0x10, {}, 0x1000) is True + assert region_in_any_section(0x2000, 0x10, {}, 0x1000) is False + assert region_in_any_section(0x500, 0x10, {}) is None + + def test_size_of_image_defaults_preserve_two_arg_call(self): + """ + Back-compat: the SizeOfImage parameter is optional, so pre-existing + two-/three-argument call sites keep working (degrading to None on the + fallback path rather than raising). + """ + secs = {"sections": [{"rva": 0x1000, "virtual_size": 0x1000}]} + assert rva_in_any_section(0x1500, secs) is True + assert region_in_any_section(0x1000, 0x10, secs) is True + def test_no_mutation_of_analysis(self): - analysis = {"sections": [{"rva": 0x1000, "virtual_size": 0x100}], - "size_of_image": 0x1000} + analysis = {"sections": [{"rva": 0x1000, "virtual_size": 0x100}]} import copy snapshot = copy.deepcopy(analysis) - rva_in_any_section(0x1050, analysis) - region_in_any_section(0x1050, 0x10, analysis) + rva_in_any_section(0x1050, analysis, 0x1000) + region_in_any_section(0x1050, 0x10, analysis, 0x1000) region_within_image(0x100, 0x10, 0x1000) assert analysis == snapshot # side-effect-free diff --git a/tests/unit/validators/test_validator_debug.py b/tests/unit/validators/test_validator_debug.py index f82adca..0d7f391 100644 --- a/tests/unit/validators/test_validator_debug.py +++ b/tests/unit/validators/test_validator_debug.py @@ -6,10 +6,21 @@ Strategy: - Input is the debug_struct dict produced by parser pe_debug, carried under - metadata["debug_struct"] (the dispatcher binds it via @depends_on with the - first positional arg named `metadata` receiving `internal`). + internal["debug_struct"]. - Build dicts directly to isolate validator logic from parser behaviour. - Tests assert on emitted REASONCODES and the details payload. + +Layer note: the validator is @depends_on("internal", "metadata", "analysis"), +so it takes THREE positional arguments. SizeOfImage is read from +metadata["optional_header"]["size_of_image"] and threaded explicitly into the +_directory_invariants helpers; section geometry comes from analysis["sections"]. +Keeping those in separate fixtures is deliberate - see +test_no_sections_falls_back_to_size_of_image. + +Details note: priority-resolved sub-reasons are carried in a "sub_reason" key. +The key "reason" is reserved by the heuristics emission layer, which merges +details over its own reason field - a details["reason"] would overwrite the +parent reason code. """ from __future__ import annotations @@ -26,11 +37,26 @@ # Input builders # ================================================================= +def _make_metadata(size_of_image: Optional[int] = 0x100000) -> Dict[str, Any]: + """ + Public-metadata layer. SizeOfImage lives under optional_header; it is NOT + part of the analysis layer. + """ + return {"optional_header": {"size_of_image": size_of_image}} + + def _make_analysis( - size_of_image: Optional[int] = 0x100000, sections: Optional[List[Dict[str, Any]]] = None, ) -> Dict[str, Any]: - analysis: Dict[str, Any] = {"size_of_image": size_of_image} + """ + Analysis layer. Carries section geometry only. + + Deliberately does NOT accept a size_of_image argument: the previous fixture + injected one here, which made the SizeOfImage fallback path appear to work + in tests while it was dead in production (analysis never carries that key). + Use _make_metadata for SizeOfImage. + """ + analysis: Dict[str, Any] = {} if sections is not None: analysis["sections"] = sections return analysis @@ -75,8 +101,12 @@ def _make_debug( "truncations": truncations or [], "errors": errors or []} -def _run(debug: Optional[Dict[str, Any]], analysis: Dict[str, Any]): - return validate_debug({"debug_struct": debug}, analysis) +def _run(debug: Optional[Dict[str, Any]], + analysis: Dict[str, Any], + metadata: Optional[Dict[str, Any]] = None): + if metadata is None: + metadata = _make_metadata() + return validate_debug({"debug_struct": debug}, metadata, analysis) def _codes(issues) -> List: @@ -96,7 +126,7 @@ def test_none_struct_no_issues(self): assert _run(None, _make_analysis()) == [] def test_missing_key_no_issues(self): - assert validate_debug({}, _make_analysis()) == [] + assert validate_debug({}, _make_metadata(), _make_analysis()) == [] # ================================================================= @@ -109,7 +139,7 @@ def test_errors_emit_invalid_header(self): issues = _run(debug, _make_analysis()) assert _codes(issues) == [ReasonCodes.DEBUG_DIRECTORY_INVALID_HEADER] d = _details_for(issues, ReasonCodes.DEBUG_DIRECTORY_INVALID_HEADER)[0] - assert d["reason"] == "top_level_decode" + assert d["sub_reason"] == "top_level_decode" assert d["errors"] == ["entry_unpack_failed"] def test_short_circuit_skips_entries_and_truncations(self): @@ -138,6 +168,8 @@ def test_one_issue_per_tag(self): def test_region_detail_preserved(self): debug = _make_debug(truncations=["debug_entry_read_failed"]) issues = _run(debug, _make_analysis()) + # "region" is a distinct key and was never subject to the reason + # collision, so it is unchanged by the sub_reason migration. assert _details_for(issues, ReasonCodes.DEBUG_TABLE_TRUNCATED)[0] == { "region": "debug_entry_read_failed"} @@ -154,7 +186,7 @@ def test_single_reason_flagged(self): issues = _run(debug, _make_analysis()) assert ReasonCodes.DEBUG_DIRECTORY_ENTRY_MALFORMED in _codes(issues) d = _details_for(issues, ReasonCodes.DEBUG_DIRECTORY_ENTRY_MALFORMED)[0] - assert d["reason"] == "codeview_signature_unknown" + assert d["sub_reason"] == "codeview_signature_unknown" assert d["index"] == 0 and d["type_name"] == "CODEVIEW" def test_priority_first_match_wins(self): @@ -165,7 +197,7 @@ def test_priority_first_match_wins(self): issues = _run(debug, _make_analysis()) malformed = _details_for(issues, ReasonCodes.DEBUG_DIRECTORY_ENTRY_MALFORMED) assert len(malformed) == 1 - assert malformed[0]["reason"] == "entry_unpack_failed" + assert malformed[0]["sub_reason"] == "entry_unpack_failed" def test_unknown_error_not_flagged(self): # an error tag not in the priority list -> no malformation issue @@ -219,9 +251,48 @@ def test_missing_size_defaults_zero(self): assert _details_for(issues, ReasonCodes.DEBUG_ENTRY_RVA_INVALID)[0]["size_of_data"] == 0 def test_no_sections_falls_back_to_size_of_image(self): + """ + With no section geometry, the check falls back to a SizeOfImage bound. + + SizeOfImage must come from the METADATA layer. The previous version of + this test put it in `analysis`, which the helpers used to read - so the + fallback passed here while being dead in production, where `analysis` + never carries that key. + """ + debug = _make_debug(entries=[_make_entry( + address_of_raw_data=0x200000, size_of_data=0x10, errors=[])]) + issues = _run(debug, _make_analysis(), + metadata=_make_metadata(size_of_image=0x100000)) + assert _codes(issues) == [ReasonCodes.DEBUG_ENTRY_RVA_INVALID] + + def test_no_sections_in_bounds_not_flagged(self): + """Counterpart: the fallback must not false-positive on a valid RVA.""" + debug = _make_debug(entries=[_make_entry( + address_of_raw_data=0x2000, size_of_data=0x10, errors=[])]) + issues = _run(debug, _make_analysis(), + metadata=_make_metadata(size_of_image=0x100000)) + assert ReasonCodes.DEBUG_ENTRY_RVA_INVALID not in _codes(issues) + + def test_no_sections_and_no_size_of_image_skips_check(self): + """ + With neither section geometry nor SizeOfImage the check is unknowable + and must be skipped rather than guessed. Pins the absent-optional-header + case explicitly so it is asserted on purpose. + """ debug = _make_debug(entries=[_make_entry( address_of_raw_data=0x200000, size_of_data=0x10, errors=[])]) - issues = _run(debug, _make_analysis(size_of_image=0x100000)) + issues = _run(debug, _make_analysis(), metadata={}) + assert ReasonCodes.DEBUG_ENTRY_RVA_INVALID not in _codes(issues) + + def test_sections_take_precedence_over_size_of_image(self): + """ + When section geometry is present it is authoritative: an RVA inside + SizeOfImage but outside every section is still flagged. + """ + debug = _make_debug(entries=[_make_entry( + address_of_raw_data=0x9000, size_of_data=0x10, errors=[])]) + issues = _run(debug, _make_analysis(sections=_tiny_section()), + metadata=_make_metadata(size_of_image=0x100000)) assert _codes(issues) == [ReasonCodes.DEBUG_ENTRY_RVA_INVALID] @@ -257,7 +328,8 @@ def test_malformed_and_rva_invalid_same_entry(self): class TestOutputContract: def test_dependency_contract(self): - assert getattr(validate_debug, "_depends_on") == ("internal", "analysis") + assert getattr(validate_debug, "_depends_on") == ( + "internal", "metadata", "analysis") def test_issue_shape(self): debug = _make_debug(entries=[_make_entry( @@ -278,6 +350,32 @@ def test_json_serializable(self): issues = _run(debug, _make_analysis()) json.dumps([i for i in issues]) + def test_no_details_payload_uses_reserved_reason_key(self): + """ + "reason" is reserved by the heuristics emission layer: _det builds + metadata as {"reason": parent, **details}, so a details["reason"] would + overwrite the parent reason code. Validators must use "sub_reason". + + Exercises every non-short-circuit emission path at once. + """ + debug = _make_debug( + truncations=["debug_entry_truncated"], + entries=[_make_entry(address_of_raw_data=0x9000, size_of_data=0x10, + errors=["codeview_signature_unknown"])]) + issues = _run(debug, _make_analysis(sections=_tiny_section())) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + + def test_top_level_decode_avoids_reserved_reason_key(self): + """The short-circuit path is unreachable above; pin it separately.""" + debug = _make_debug(errors=["entry_unpack_failed"]) + issues = _run(debug, _make_analysis()) + assert issues + assert "reason" not in issues[0]["details"] + # ================================================================= # Determinism diff --git a/tests/unit/validators/test_validator_delay_imports.py b/tests/unit/validators/test_validator_delay_imports.py index f3dcdcc..b4b8e03 100644 --- a/tests/unit/validators/test_validator_delay_imports.py +++ b/tests/unit/validators/test_validator_delay_imports.py @@ -8,6 +8,15 @@ - Input is the delay_import_struct dict produced by parser_delay_imports. - Build dicts directly to isolate validator logic from parser behaviour. - Tests assert on emitted REASONCODES and the details payload. + +Layer note: the validator is @depends_on("internal", "metadata"), so the second +positional argument is the PUBLIC METADATA layer, not the analysis layer. +SizeOfImage is read from metadata["optional_header"]["size_of_image"]. + +Details note: priority-resolved sub-reasons are carried in a "sub_reason" key. +The key "reason" is reserved by the heuristics emission layer, which merges +details over its own reason field - a details["reason"] would overwrite the +parent reason code. """ from __future__ import annotations @@ -27,8 +36,12 @@ _NOT_PROVIDED = object() -def _make_analysis(size_of_image: Optional[int] = 0x100000) -> Dict[str, Any]: - return {"size_of_image": size_of_image} +def _make_metadata(size_of_image: Optional[int] = 0x100000) -> Dict[str, Any]: + """ + Public-metadata layer. SizeOfImage lives under optional_header; passing + None models an optional header present but missing the field. + """ + return {"optional_header": {"size_of_image": size_of_image}} def _make_entry( @@ -124,11 +137,11 @@ def _details_for(issues, code) -> List[Dict[str, Any]]: class TestAbsence: def test_no_delay_import_struct_returns_no_issues(self): - assert validate_delay_imports({}, _make_analysis()) == [] + assert validate_delay_imports({}, _make_metadata()) == [] def test_explicit_none_returns_no_issues(self): assert validate_delay_imports( - {"delay_import_struct": None}, _make_analysis(), + {"delay_import_struct": None}, _make_metadata(), ) == [] @@ -142,7 +155,7 @@ def test_errors_present_emits_invalid_header_and_returns_early(self): di = _make_di(errors=["header_read_failed"]) di["truncations"] = ["delay_import_descriptor_truncated"] issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) codes = _codes(issues) assert ReasonCodes.DELAY_IMPORT_DIRECTORY_INVALID_HEADER in codes @@ -150,7 +163,7 @@ def test_errors_present_emits_invalid_header_and_returns_early(self): assert ReasonCodes.DELAY_IMPORT_TABLE_TRUNCATED not in codes details = _details_for(issues, ReasonCodes.DELAY_IMPORT_DIRECTORY_INVALID_HEADER) - assert details[0]["reason"] == "top_level_decode" + assert details[0]["sub_reason"] == "top_level_decode" # ================================================================= @@ -163,7 +176,7 @@ def test_in_bounds_no_issue(self): di = _make_di(rva=0x1000, size=64) issues = validate_delay_imports( {"delay_import_struct": di}, - _make_analysis(size_of_image=0x100000), + _make_metadata(size_of_image=0x100000), ) assert ReasonCodes.DELAY_IMPORT_DIRECTORY_OUT_OF_BOUNDS not in _codes(issues) @@ -171,7 +184,7 @@ def test_extends_past_image_flagged(self): di = _make_di(rva=0xFFFF0, size=0x200) issues = validate_delay_imports( {"delay_import_struct": di}, - _make_analysis(size_of_image=0x100000), + _make_metadata(size_of_image=0x100000), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_DIRECTORY_OUT_OF_BOUNDS) @@ -182,10 +195,24 @@ def test_silent_when_size_of_image_missing(self): di = _make_di(rva=0xFFFF0, size=0x200) issues = validate_delay_imports( {"delay_import_struct": di}, - _make_analysis(size_of_image=None), + _make_metadata(size_of_image=None), ) assert ReasonCodes.DELAY_IMPORT_DIRECTORY_OUT_OF_BOUNDS not in _codes(issues) + def test_silent_when_optional_header_absent(self): + """ + The metadata layer may omit optional_header entirely. The placement + check must skip rather than raise. + + Regression guard: while `optional_header` was absent the OUT_OF_BOUNDS + check could never fire, so the two tests above passed vacuously. This + test pins the absent-header case explicitly so that behaviour is + asserted on purpose rather than by accident. + """ + di = _make_di(rva=0xFFFF0, size=0x200) + issues = validate_delay_imports({"delay_import_struct": di}, {}) + assert ReasonCodes.DELAY_IMPORT_DIRECTORY_OUT_OF_BOUNDS not in _codes(issues) + # ================================================================= # Truncations @@ -196,17 +223,19 @@ class TestTruncations: def test_no_truncations_no_issues(self): di = _make_di(truncations=[]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) assert ReasonCodes.DELAY_IMPORT_TABLE_TRUNCATED not in _codes(issues) def test_single_truncation_emits_one_issue(self): di = _make_di(truncations=["delay_import_descriptor_truncated"]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_TABLE_TRUNCATED) assert len(details) == 1 + # "table" is a distinct key and was never subject to the reason + # collision, so it is unchanged by the sub_reason migration. assert details[0]["table"] == "delay_import_descriptor_truncated" def test_multiple_truncations_emit_separate_issues(self): @@ -216,7 +245,7 @@ def test_multiple_truncations_emit_separate_issues(self): "iat_truncated", ]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_TABLE_TRUNCATED) assert len(details) == 3 @@ -232,7 +261,7 @@ def test_clean_descriptor_no_issues(self): d = _make_descriptor() di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) assert issues == [] @@ -240,7 +269,7 @@ def test_v0_attributes_flagged(self): d = _make_descriptor(attributes=0, attributes_v1=False) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_ATTRIBUTES_LEGACY_VA_MODE) @@ -252,22 +281,22 @@ def test_dll_name_rva_zero_flagged(self): d = _make_descriptor(errors=["dll_name_rva_zero"]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_DLL_NAME_INVALID) assert len(details) == 1 - assert details[0]["reason"] == "dll_name_rva_zero" + assert details[0]["sub_reason"] == "dll_name_rva_zero" def test_dll_name_priority_resolution(self): """dll_name_rva_zero wins over read_failed when both are present.""" d = _make_descriptor(errors=["read_failed", "dll_name_rva_zero"]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_DLL_NAME_INVALID) assert len(details) == 1 - assert details[0]["reason"] == "dll_name_rva_zero" + assert details[0]["sub_reason"] == "dll_name_rva_zero" def test_dll_name_not_printable_flagged(self): d = _make_descriptor( @@ -277,40 +306,40 @@ def test_dll_name_not_printable_flagged(self): ) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_DLL_NAME_INVALID) - assert details[0]["reason"] == "dll_name_not_printable" + assert details[0]["sub_reason"] == "dll_name_not_printable" assert details[0]["dll_name"] == "kernel\x0132.dll" def test_int_rva_zero_flagged(self): d = _make_descriptor(errors=["int_rva_zero"], int_rva=0) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_DESCRIPTOR_INVALID) # Find the INT-specific one int_details = [d for d in details if d["table"] == "int"] assert len(int_details) == 1 - assert int_details[0]["reason"] == "int_rva_zero" + assert int_details[0]["sub_reason"] == "int_rva_zero" def test_iat_rva_zero_flagged(self): d = _make_descriptor(errors=["iat_rva_zero"], iat_rva=0) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_DESCRIPTOR_INVALID) iat_details = [d for d in details if d["table"] == "iat"] assert len(iat_details) == 1 - assert iat_details[0]["reason"] == "iat_rva_zero" + assert iat_details[0]["sub_reason"] == "iat_rva_zero" def test_int_iat_mismatch_flagged(self): d = _make_descriptor(errors=["int_iat_length_mismatch"]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_INT_IAT_MISMATCH) assert len(details) == 1 @@ -323,7 +352,7 @@ def test_no_double_emission_per_descriptor(self): ) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) dll_issues = [ i for i in issues @@ -342,7 +371,7 @@ def test_entry_with_unknown_error_tag_skipped(self): d = _make_descriptor(imports=[e]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) # No DELAY_IMPORT_ENTRY_INVALID should fire — the unknown tag is # not in the priority list, so the validator skips emission rather @@ -361,7 +390,7 @@ def test_clean_entry_no_issues(self): d = _make_descriptor(imports=[e]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) assert ReasonCodes.DELAY_IMPORT_ENTRY_INVALID not in _codes(issues) @@ -373,11 +402,11 @@ def test_ordinal_zero_flagged(self): d = _make_descriptor(imports=[e]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_ENTRY_INVALID) assert len(details) == 1 - assert details[0]["reason"] == "ordinal_zero" + assert details[0]["sub_reason"] == "ordinal_zero" assert details[0]["is_ordinal"] is True assert details[0]["ordinal"] == 0 @@ -386,20 +415,20 @@ def test_int_entry_missing_flagged(self): d = _make_descriptor(imports=[e]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_ENTRY_INVALID) - assert details[0]["reason"] == "int_entry_missing" + assert details[0]["sub_reason"] == "int_entry_missing" def test_int_entry_zero_flagged(self): e = _make_entry(errors=["int_entry_zero"]) d = _make_descriptor(imports=[e]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_ENTRY_INVALID) - assert details[0]["reason"] == "int_entry_zero" + assert details[0]["sub_reason"] == "int_entry_zero" def test_name_unterminated_flagged(self): e = _make_entry( @@ -409,10 +438,10 @@ def test_name_unterminated_flagged(self): d = _make_descriptor(imports=[e]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_ENTRY_INVALID) - assert details[0]["reason"] == "name_unterminated" + assert details[0]["sub_reason"] == "name_unterminated" def test_name_not_printable_flagged(self): e = _make_entry( @@ -422,10 +451,10 @@ def test_name_not_printable_flagged(self): d = _make_descriptor(imports=[e]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_ENTRY_INVALID) - assert details[0]["reason"] == "name_not_printable" + assert details[0]["sub_reason"] == "name_not_printable" assert details[0]["name"] == "Foo\x01Bar" def test_entry_priority_resolution(self): @@ -434,10 +463,10 @@ def test_entry_priority_resolution(self): d = _make_descriptor(imports=[e]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_ENTRY_INVALID) - assert details[0]["reason"] == "int_entry_missing" + assert details[0]["sub_reason"] == "int_entry_missing" def test_multiple_bad_entries_each_flagged(self): e1 = _make_entry(index=0, errors=["int_entry_missing"]) @@ -446,7 +475,7 @@ def test_multiple_bad_entries_each_flagged(self): d = _make_descriptor(imports=[e1, e2, e3]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) details = _details_for(issues, ReasonCodes.DELAY_IMPORT_ENTRY_INVALID) assert len(details) == 2 @@ -465,7 +494,7 @@ def test_v0_with_cascading_dll_errors_emits_both(self): ) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) codes = set(_codes(issues)) assert ReasonCodes.DELAY_IMPORT_ATTRIBUTES_LEGACY_VA_MODE in codes @@ -476,7 +505,7 @@ def test_multiple_descriptors_each_independently_validated(self): d2 = _make_descriptor(index=1, attributes=0, attributes_v1=False) di = _make_di(descriptors=[d1, d2]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) codes = set(_codes(issues)) assert ReasonCodes.DELAY_IMPORT_DLL_NAME_INVALID in codes @@ -491,13 +520,13 @@ class TestOutputContract: def test_returns_list(self): result = validate_delay_imports( - {"delay_import_struct": _make_di()}, _make_analysis(), + {"delay_import_struct": _make_di()}, _make_metadata(), ) assert isinstance(result, list) def test_clean_returns_empty_list(self): result = validate_delay_imports( - {"delay_import_struct": _make_di()}, _make_analysis(), + {"delay_import_struct": _make_di()}, _make_metadata(), ) assert result == [] @@ -505,13 +534,44 @@ def test_each_issue_has_issue_and_details(self): d = _make_descriptor(errors=["dll_name_rva_zero"]) di = _make_di(descriptors=[d]) issues = validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) for issue in issues: assert "issue" in issue assert "details" in issue assert isinstance(issue["details"], dict) + def test_no_details_payload_uses_reserved_reason_key(self): + """ + "reason" is reserved by the heuristics emission layer: _det builds + metadata as {"reason": parent, **details}, so a details["reason"] + would overwrite the parent reason code. Validators must use + "sub_reason". + + Exercises every emission path in this validator at once. + """ + d = _make_descriptor( + attributes=0, attributes_v1=False, + errors=[ + "dll_name_rva_zero", "int_rva_zero", "iat_rva_zero", + "int_iat_length_mismatch", + ], + imports=[_make_entry(errors=["ordinal_zero"])], + ) + di = _make_di( + rva=0xFFFF0, size=0x200, + truncations=["delay_import_descriptor_truncated"], + descriptors=[d], + ) + issues = validate_delay_imports( + {"delay_import_struct": di}, _make_metadata(size_of_image=0x100000), + ) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + # ================================================================= # Determinism @@ -532,11 +592,11 @@ def test_repeated_validation_produces_identical_issues(self): truncations=["delay_import_descriptor_truncated"], descriptors=[d], ) - metadata = {"delay_import_struct": di} - analysis = _make_analysis() + internal = {"delay_import_struct": di} + metadata = _make_metadata() results = [ - validate_delay_imports(metadata, analysis) for _ in range(20) + validate_delay_imports(internal, metadata) for _ in range(20) ] for r in results[1:]: assert r == results[0] @@ -548,7 +608,7 @@ def test_priority_resolution_deterministic(self): di = _make_di(descriptors=[d]) results = [ validate_delay_imports( - {"delay_import_struct": di}, _make_analysis(), + {"delay_import_struct": di}, _make_metadata(), ) for _ in range(20) ] @@ -557,4 +617,4 @@ def test_priority_resolution_deterministic(self): # Confirm priority winner details = _details_for(results[0], ReasonCodes.DELAY_IMPORT_DLL_NAME_INVALID) - assert details[0]["reason"] == "dll_name_rva_zero" + assert details[0]["sub_reason"] == "dll_name_rva_zero" diff --git a/tests/unit/validators/test_validator_entropy_ext.py b/tests/unit/validators/test_validator_entropy_ext.py new file mode 100644 index 0000000..9a1765b --- /dev/null +++ b/tests/unit/validators/test_validator_entropy_ext.py @@ -0,0 +1,619 @@ +# Copyright (c) 2026 MalX Labs and contributors +# SPDX-License-Identifier: MPL-2.0 + +""" +Unit tests for iocx.validators.entropy.validate_entropy. + +Layer note: the validator is @depends_on("metadata", "analysis") and takes TWO +positional arguments. It reads only the analysis layer; the metadata argument +is accepted but unused. + +Threshold note: every check here is a numeric comparison against a module +constant, so the tests below pin the BOUNDARY (>= / <=) rather than just +asserting a code appears for an obviously-extreme value. Presence-only tests +cannot detect a threshold drifting by one, or a `>=` becoming `>`. + +Float note: `stddev` is computed, so the UNIFORM_STDDEV_THRESHOLD boundary is +not cleanly testable at exactly 0.15 - two entropies whose spread is nominally +0.15 produce 0.15000000000000036 and fail the `<=`. The tests use values +clearly inside and outside the threshold and document the boundary rather than +asserting a flaky equality. +""" + +from __future__ import annotations + +from typing import Any, Dict, List + +import pytest + +from iocx.validators.entropy import ( + validate_entropy, + HIGH_ENTROPY_THRESHOLD, + LOW_ENTROPY_THRESHOLD, + MIN_SECTION_SIZE_FOR_ENTROPY, + MIN_SECTION_SIZE_FOR_LOW_ENTROPY, + MIN_OVERLAY_SIZE_FOR_ENTROPY, + UNIFORM_STDDEV_THRESHOLD, +) +from iocx.reason_codes import ReasonCodes + + +# ================================================================= +# Helpers +# ================================================================= + +def _section(name: str = ".text", entropy: Any = 5.0, + raw_size: Any = 2000) -> Dict[str, Any]: + return {"name": name, "entropy": entropy, "raw_size": raw_size} + + +def _run(**analysis) -> List[Dict[str, Any]]: + return validate_entropy({}, analysis) + + +def make_issue_list(result) -> List[str]: + return [i["issue"] for i in result] + + +def _details_for(issues, code) -> List[Dict[str, Any]]: + return [i["details"] for i in issues if i["issue"] == code] + + +# ================================================================= +# Input tolerance +# ================================================================= + +class TestInputTolerance: + + def test_missing_sections_key(self): + assert _run() == [] + + def test_none_sections_value(self): + assert _run(sections=None) == [] + + def test_empty_sections_list(self): + assert _run(sections=[]) == [] + + @pytest.mark.parametrize("entropy", ["bad", None, [], {}]) + def test_non_numeric_entropy_skipped(self, entropy): + assert _run(sections=[_section(entropy=entropy)]) == [] + + @pytest.mark.parametrize("raw_size", ["bad", None, 2000.5, []]) + def test_non_int_raw_size_skipped(self, raw_size): + """raw_size must be an int - a float is rejected, not coerced.""" + assert _run(sections=[_section(entropy=8.0, raw_size=raw_size)]) == [] + + def test_bool_entropy_accepted_as_numeric(self): + """ + bool is a subclass of int, so isinstance(True, (int, float)) passes. + Documents the behaviour rather than asserting it is desirable. + """ + issues = _run(sections=[_section(entropy=True, raw_size=20000)]) + # True -> 1.0, which is above LOW and below HIGH, so nothing fires + assert issues == [] + + def test_missing_name_defaults_to_empty_string(self): + issues = _run(sections=[{"entropy": 8.0, "raw_size": 2000}]) + assert _details_for(issues, ReasonCodes.ENTROPY_HIGH_SECTION)[0]["section"] == "" + + def test_none_name_defaults_to_empty_string(self): + issues = _run(sections=[_section(name=None, entropy=8.0)]) + assert _details_for(issues, ReasonCodes.ENTROPY_HIGH_SECTION)[0]["section"] == "" + + def test_int_entropy_converted_to_float(self): + issues = _run(sections=[_section(entropy=8, raw_size=2000)]) + d = _details_for(issues, ReasonCodes.ENTROPY_HIGH_SECTION)[0] + assert isinstance(d["entropy"], float) + assert d["entropy"] == 8.0 + + +# ================================================================= +# High-entropy sections +# ================================================================= + +class TestHighEntropySection: + + def test_high_entropy_flagged(self): + issues = _run(sections=[_section(entropy=8.0)]) + assert make_issue_list(issues) == [ReasonCodes.ENTROPY_HIGH_SECTION] + + def test_entropy_exactly_at_threshold_flagged(self): + """The comparison is `>=`, so the threshold itself fires.""" + issues = _run(sections=[_section(entropy=HIGH_ENTROPY_THRESHOLD)]) + assert ReasonCodes.ENTROPY_HIGH_SECTION in make_issue_list(issues) + + def test_entropy_just_below_threshold_not_flagged(self): + issues = _run(sections=[_section(entropy=HIGH_ENTROPY_THRESHOLD - 0.01)]) + assert issues == [] + + def test_size_exactly_at_minimum_flagged(self): + issues = _run(sections=[ + _section(entropy=8.0, raw_size=MIN_SECTION_SIZE_FOR_ENTROPY)]) + assert ReasonCodes.ENTROPY_HIGH_SECTION in make_issue_list(issues) + + def test_size_just_below_minimum_not_flagged(self): + """ + Small sections are excluded entirely - a 1023-byte section with + entropy 8.0 is not reported, and is also absent from the uniform + sample. + """ + issues = _run(sections=[ + _section(entropy=8.0, raw_size=MIN_SECTION_SIZE_FOR_ENTROPY - 1)]) + assert issues == [] + + def test_details_payload(self): + issues = _run(sections=[_section(".upx0", 7.9, 4096)]) + assert _details_for(issues, ReasonCodes.ENTROPY_HIGH_SECTION)[0] == { + "section": ".upx0", "entropy": 7.9, "raw_size": 4096} + + def test_each_high_section_flagged_separately(self): + issues = _run(sections=[_section(".a", 8.0), _section(".b", 8.1)]) + high = _details_for(issues, ReasonCodes.ENTROPY_HIGH_SECTION) + assert [d["section"] for d in high] == [".a", ".b"] + + +# ================================================================= +# Very-low-entropy sections +# ================================================================= + +class TestVeryLowEntropySection: + + def test_very_low_entropy_flagged(self): + issues = _run(sections=[_section(entropy=0.1, raw_size=20000)]) + assert make_issue_list(issues) == [ReasonCodes.ENTROPY_VERY_LOW_SECTION] + + def test_entropy_exactly_at_threshold_flagged(self): + """The comparison is `<=`, so the threshold itself fires.""" + issues = _run(sections=[ + _section(entropy=LOW_ENTROPY_THRESHOLD, raw_size=20000)]) + assert ReasonCodes.ENTROPY_VERY_LOW_SECTION in make_issue_list(issues) + + def test_entropy_just_above_threshold_not_flagged(self): + issues = _run(sections=[ + _section(entropy=LOW_ENTROPY_THRESHOLD + 0.01, raw_size=20000)]) + assert issues == [] + + def test_size_exactly_at_low_minimum_flagged(self): + issues = _run(sections=[ + _section(entropy=0.1, raw_size=MIN_SECTION_SIZE_FOR_LOW_ENTROPY)]) + assert ReasonCodes.ENTROPY_VERY_LOW_SECTION in make_issue_list(issues) + + def test_size_just_below_low_minimum_not_flagged(self): + """ + The low-entropy check uses a much larger minimum (16 KB) than the high + check (1 KB): a 16383-byte zero-filled section is deliberately ignored. + """ + issues = _run(sections=[ + _section(entropy=0.1, + raw_size=MIN_SECTION_SIZE_FOR_LOW_ENTROPY - 1)]) + assert issues == [] + + def test_low_and_high_minimums_are_different(self): + """ + Pin the asymmetry: a section between the two minimums participates in + the high check and the uniform sample, but never the low check. + """ + assert MIN_SECTION_SIZE_FOR_LOW_ENTROPY > MIN_SECTION_SIZE_FOR_ENTROPY + issues = _run(sections=[_section(entropy=0.0, raw_size=2000)]) + assert issues == [] + + def test_details_payload(self): + issues = _run(sections=[_section(".bss", 0.03, 32768)]) + assert _details_for(issues, ReasonCodes.ENTROPY_VERY_LOW_SECTION)[0] == { + "section": ".bss", "entropy": 0.03, "raw_size": 32768} + + def test_high_and_low_are_mutually_exclusive(self): + """No entropy value can satisfy both thresholds.""" + assert LOW_ENTROPY_THRESHOLD < HIGH_ENTROPY_THRESHOLD + + +# ================================================================= +# Overlay entropy +# ================================================================= + +class TestOverlayEntropy: + + def test_high_overlay_flagged(self): + issues = _run(sections=[], overlay={"entropy": 8.0, "size": 2000}) + assert make_issue_list(issues) == [ReasonCodes.ENTROPY_HIGH_OVERLAY] + + def test_entropy_exactly_at_threshold_flagged(self): + issues = _run(sections=[], + overlay={"entropy": HIGH_ENTROPY_THRESHOLD, "size": 2000}) + assert ReasonCodes.ENTROPY_HIGH_OVERLAY in make_issue_list(issues) + + def test_entropy_just_below_threshold_not_flagged(self): + issues = _run(sections=[], + overlay={"entropy": HIGH_ENTROPY_THRESHOLD - 0.01, + "size": 2000}) + assert issues == [] + + def test_size_exactly_at_minimum_flagged(self): + issues = _run(sections=[], + overlay={"entropy": 8.0, + "size": MIN_OVERLAY_SIZE_FOR_ENTROPY}) + assert ReasonCodes.ENTROPY_HIGH_OVERLAY in make_issue_list(issues) + + def test_size_just_below_minimum_not_flagged(self): + issues = _run(sections=[], + overlay={"entropy": 8.0, + "size": MIN_OVERLAY_SIZE_FOR_ENTROPY - 1}) + assert issues == [] + + @pytest.mark.parametrize("overlay", [None, "not-a-dict", [], 42]) + def test_non_dict_overlay_skipped(self, overlay): + assert _run(sections=[], overlay=overlay) == [] + + def test_missing_overlay_key_skipped(self): + assert _run(sections=[]) == [] + + @pytest.mark.parametrize("entropy,size", [ + ("bad", 2000), (8.0, "bad"), (None, 2000), (8.0, None), + ]) + def test_malformed_overlay_fields_skipped(self, entropy, size): + assert _run(sections=[], + overlay={"entropy": entropy, "size": size}) == [] + + def test_details_payload(self): + issues = _run(sections=[], overlay={"entropy": 7.85, "size": 4096}) + assert _details_for(issues, ReasonCodes.ENTROPY_HIGH_OVERLAY)[0] == { + "entropy": 7.85, "size": 4096} + + def test_overlay_does_not_join_the_uniform_sample(self): + """ + Only sections feed `entropies`; a high overlay must not make a + single-section file look uniform. + """ + issues = _run(sections=[_section(entropy=7.6)], + overlay={"entropy": 7.6, "size": 2000}) + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS not in make_issue_list(issues) + + +# ================================================================= +# Region entropy +# ================================================================= + +class TestRegionEntropy: + + REGIONS = [ + ("resources", ReasonCodes.ENTROPY_HIGH_RESOURCES), + ("relocations", ReasonCodes.ENTROPY_HIGH_RELOCATIONS), + ("imports", ReasonCodes.ENTROPY_HIGH_IMPORTS), + ("tls", ReasonCodes.ENTROPY_HIGH_TLS), + ("certificate", ReasonCodes.ENTROPY_HIGH_CERTIFICATE), + ] + + @pytest.mark.parametrize("region,reason", REGIONS) + def test_each_region_maps_to_its_own_code(self, region, reason): + issues = _run(region_entropy={region: {"entropy": 8.0, "size": 2000}}) + assert make_issue_list(issues) == [reason] + + @pytest.mark.parametrize("region,reason", REGIONS) + def test_region_threshold_boundary(self, region, reason): + at = _run(region_entropy={ + region: {"entropy": HIGH_ENTROPY_THRESHOLD, "size": 2000}}) + below = _run(region_entropy={ + region: {"entropy": HIGH_ENTROPY_THRESHOLD - 0.01, "size": 2000}}) + assert reason in make_issue_list(at) + assert below == [] + + @pytest.mark.parametrize("region,reason", REGIONS) + def test_region_size_boundary(self, region, reason): + at = _run(region_entropy={ + region: {"entropy": 8.0, "size": MIN_SECTION_SIZE_FOR_ENTROPY}}) + below = _run(region_entropy={ + region: {"entropy": 8.0, "size": MIN_SECTION_SIZE_FOR_ENTROPY - 1}}) + assert reason in make_issue_list(at) + assert below == [] + + def test_all_regions_can_fire_together(self): + issues = _run(region_entropy={ + r: {"entropy": 8.0, "size": 2000} for r, _ in self.REGIONS}) + assert set(make_issue_list(issues)) == {c for _, c in self.REGIONS} + + def test_emission_order_follows_region_map(self): + """Dict literal order is insertion order, so emission is deterministic.""" + issues = _run(region_entropy={ + r: {"entropy": 8.0, "size": 2000} for r, _ in reversed(self.REGIONS)}) + assert make_issue_list(issues) == [c for _, c in self.REGIONS] + + def test_region_below_threshold_continues_to_next_region(self): + """ + A region present but under threshold must not emit, and must not stop + later regions being evaluated (the loop-back branch). + """ + issues = _run(region_entropy={ + "resources": {"entropy": 1.0, "size": 2000}, # below threshold + "tls": {"entropy": 8.0, "size": 2000}, # above + }) + assert make_issue_list(issues) == [ReasonCodes.ENTROPY_HIGH_TLS] + + def test_unknown_region_ignored(self): + assert _run(region_entropy={"bogus": {"entropy": 8.0, "size": 2000}}) == [] + + @pytest.mark.parametrize("value", [None, "not-a-dict", [], 42]) + def test_non_dict_region_info_skipped(self, value): + assert _run(region_entropy={"resources": value}) == [] + + @pytest.mark.parametrize("entropy,size", [ + ("bad", 2000), (8.0, "bad"), (None, 2000), (8.0, None), (8.0, 2000.5), + ]) + def test_malformed_region_fields_skipped(self, entropy, size): + """ + The info dict is present but its fields fail the type check. The loop + must continue to the next region rather than emitting or raising. + """ + issues = _run(region_entropy={ + "resources": {"entropy": entropy, "size": size}, + "tls": {"entropy": 8.0, "size": 2000}, + }) + assert make_issue_list(issues) == [ReasonCodes.ENTROPY_HIGH_TLS] + + def test_missing_region_entropy_key(self): + assert _run(sections=[]) == [] + + def test_none_region_entropy_value(self): + assert _run(region_entropy=None) == [] + + def test_details_payload(self): + issues = _run(region_entropy={"tls": {"entropy": 7.7, "size": 8192}}) + assert _details_for(issues, ReasonCodes.ENTROPY_HIGH_TLS)[0] == { + "entropy": 7.7, "size": 8192} + + def test_region_does_not_join_the_uniform_sample(self): + issues = _run(sections=[_section(entropy=7.6)], + region_entropy={"resources": {"entropy": 7.6, + "size": 2000}}) + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS not in make_issue_list(issues) + + +# ================================================================= +# Uniform entropy across sections +# ================================================================= + +class TestUniformEntropy: + """ + Requires BOTH mean >= HIGH_ENTROPY_THRESHOLD and + stddev <= UNIFORM_STDDEV_THRESHOLD, over sections large enough to have + joined the sample. + """ + + def test_uniform_high_entropy_flagged(self): + issues = _run(sections=[ + _section(".text", 7.60), _section(".data", 7.62), + _section(".rdata", 7.58)]) + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS in make_issue_list(issues) + + def test_requires_at_least_two_sections(self): + issues = _run(sections=[_section(".text", 7.6)]) + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS not in make_issue_list(issues) + + def test_two_sections_is_enough(self): + issues = _run(sections=[_section(".a", 7.6), _section(".b", 7.61)]) + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS in make_issue_list(issues) + + def test_mean_just_below_threshold_not_flagged(self): + """Identical entropies give stddev 0, so only the mean gate applies.""" + e = HIGH_ENTROPY_THRESHOLD - 0.01 + issues = _run(sections=[_section(".a", e), _section(".b", e)]) + assert issues == [] + + def test_mean_exactly_at_threshold_flagged(self): + e = HIGH_ENTROPY_THRESHOLD + issues = _run(sections=[_section(".a", e), _section(".b", e)]) + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS in make_issue_list(issues) + + def test_stddev_clearly_inside_threshold_flagged(self): + """ + Spread of 0.14 -> stddev 0.14, comfortably inside 0.15. + + The exact boundary is NOT asserted: a nominal spread of 0.15 computes + as 0.15000000000000036 and fails the `<=`. Testing equality there + would pin float error rather than intent. + """ + issues = _run(sections=[_section(".a", 7.60), _section(".b", 7.88)]) + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS in make_issue_list(issues) + + def test_stddev_clearly_outside_threshold_not_flagged(self): + issues = _run(sections=[_section(".a", 7.6), _section(".b", 8.0)]) + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS not in make_issue_list(issues) + + def test_low_outlier_defeats_uniformity_via_mean(self): + """ + A single low section drags the mean below the gate, so the check + short-circuits before stddev is even considered. This is why a mixed + fixture does not report uniformity. + """ + issues = _run(sections=[ + _section(".a", 8.0), _section(".b", 0.1, 20000), + _section(".c", 7.6)]) + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS not in make_issue_list(issues) + + def test_small_sections_excluded_from_the_sample(self): + """ + A sub-1 KB section never joins `entropies`, so it can neither trigger + nor prevent uniformity. + """ + with_small = _run(sections=[ + _section(".a", 7.6), _section(".b", 7.61), + _section(".tiny", 0.0, 512)]) + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS in make_issue_list(with_small) + + def test_two_small_sections_cannot_reach_the_gate(self): + issues = _run(sections=[ + _section(".a", 7.6, 512), _section(".b", 7.61, 512)]) + assert issues == [] + + def test_details_payload_reports_computed_statistics(self): + issues = _run(sections=[_section(".a", 7.60), _section(".b", 7.62)]) + d = _details_for(issues, + ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS)[0] + assert d["count"] == 2 + assert d["mean_entropy"] == pytest.approx(7.61) + assert d["stddev_entropy"] == pytest.approx(0.01) + + def test_population_stddev_not_sample_stddev(self): + """ + Variance divides by N, not N-1. For [7.5, 7.9] that is 0.2, not the + 0.2828 a sample stddev would give - which would fail the threshold and + change the verdict. + """ + issues = _run(sections=[_section(".a", 7.5), _section(".b", 7.9)]) + # population stddev 0.2 > 0.15, so no uniformity + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS not in make_issue_list(issues) + # and the co-fired high sections confirm both were in the sample + assert make_issue_list(issues).count(ReasonCodes.ENTROPY_HIGH_SECTION) == 2 + + def test_uniform_co_fires_with_per_section_high(self): + """ + Uniformly high sections trip BOTH the per-section code (once each) and + the aggregate code - they are independent facts. + """ + issues = _run(sections=[_section(".a", 7.6), _section(".b", 7.62)]) + codes = make_issue_list(issues) + assert codes.count(ReasonCodes.ENTROPY_HIGH_SECTION) == 2 + assert codes.count(ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS) == 1 + + +# ================================================================= +# Combined scenarios +# ================================================================= + +class TestCombinedScenarios: + + def test_clean_file_emits_nothing(self): + issues = _run(sections=[_section(".text", 5.0), _section(".data", 4.0)]) + assert issues == [] + + def test_all_independent_paths_can_fire_together(self): + issues = _run( + sections=[_section(".text", 8.0), _section(".data", 0.1, 20000)], + overlay={"entropy": 8.0, "size": 2000}, + region_entropy={"resources": {"entropy": 8.0, "size": 2000}}, + ) + codes = set(make_issue_list(issues)) + assert codes == { + ReasonCodes.ENTROPY_HIGH_SECTION, + ReasonCodes.ENTROPY_VERY_LOW_SECTION, + ReasonCodes.ENTROPY_HIGH_OVERLAY, + ReasonCodes.ENTROPY_HIGH_RESOURCES, + } + + def test_packed_binary_profile(self): + """A UPX-like image: uniformly high sections plus a high overlay.""" + issues = _run( + sections=[_section(".upx0", 7.90, 4096), + _section(".upx1", 7.95, 8192)], + overlay={"entropy": 7.99, "size": 4096}, + ) + codes = make_issue_list(issues) + assert codes.count(ReasonCodes.ENTROPY_HIGH_SECTION) == 2 + assert ReasonCodes.ENTROPY_HIGH_OVERLAY in codes + assert ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS in codes + + +# ================================================================= +# Output contract +# ================================================================= + +class TestOutputContract: + + def test_dependency_contract(self): + assert getattr(validate_entropy, "_depends_on") == ("metadata", "analysis") + + def test_returns_list(self): + assert isinstance(_run(sections=[_section()]), list) + + def test_each_issue_has_issue_and_details(self): + issues = _run(sections=[_section(entropy=8.0)], + overlay={"entropy": 8.0, "size": 2000}) + for issue in issues: + assert set(issue) == {"issue", "details"} + assert isinstance(issue["issue"], str) + assert isinstance(issue["details"], dict) + + def test_no_details_payload_uses_reserved_reason_key(self): + """ + "reason" is reserved by the heuristics emission layer: _det builds + metadata as {"reason": parent, **details}, so a details["reason"] would + overwrite the parent reason code. This validator uses no sub-reasons, + so the guard simply pins that none is ever introduced. + """ + issues = _run( + sections=[_section(".a", 8.0), _section(".b", 0.1, 20000), + _section(".c", 7.9)], + overlay={"entropy": 8.0, "size": 2000}, + region_entropy={"resources": {"entropy": 8.0, "size": 2000}}, + ) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + + def test_json_serialisable(self): + import json + issues = _run( + sections=[_section(".a", 7.6), _section(".b", 7.62)], + overlay={"entropy": 8.0, "size": 2000}, + region_entropy={"imports": {"entropy": 8.0, "size": 2000}}, + ) + json.dumps(issues) # must not raise + + def test_analysis_is_not_mutated(self): + import copy + analysis = { + "sections": [_section(".a", 8.0), _section(".b", 7.6)], + "overlay": {"entropy": 8.0, "size": 2000}, + "region_entropy": {"tls": {"entropy": 8.0, "size": 2000}}, + } + snapshot = copy.deepcopy(analysis) + validate_entropy({}, analysis) + assert analysis == snapshot + + def test_metadata_argument_is_unused(self): + """ + The validator declares a metadata dependency but reads nothing from + it. Pin that, so a future change that starts using it is a conscious + one. + """ + sections = [_section(entropy=8.0)] + assert validate_entropy({}, {"sections": sections}) == \ + validate_entropy({"optional_header": {"size_of_image": 0x1000}}, + {"sections": sections}) + + +# ================================================================= +# Determinism +# ================================================================= + +class TestDeterminism: + + def test_repeated_validation_is_identical(self): + import json + analysis = { + "sections": [_section(".a", 8.0), _section(".b", 0.1, 20000), + _section(".c", 7.6)], + "overlay": {"entropy": 8.0, "size": 2000}, + "region_entropy": {r: {"entropy": 8.0, "size": 2000} + for r in ("resources", "tls", "imports")}, + } + first = json.dumps(validate_entropy({}, analysis), sort_keys=True) + for _ in range(20): + assert json.dumps(validate_entropy({}, analysis), + sort_keys=True) == first + + def test_emission_order_is_stable(self): + """Sections first (in order), then overlay, then regions, then uniform.""" + issues = _run( + sections=[_section(".a", 7.6), _section(".b", 7.62)], + overlay={"entropy": 8.0, "size": 2000}, + region_entropy={"resources": {"entropy": 8.0, "size": 2000}}, + ) + assert make_issue_list(issues) == [ + ReasonCodes.ENTROPY_HIGH_SECTION, + ReasonCodes.ENTROPY_HIGH_SECTION, + ReasonCodes.ENTROPY_HIGH_OVERLAY, + ReasonCodes.ENTROPY_HIGH_RESOURCES, + ReasonCodes.ENTROPY_UNIFORM_ACROSS_SECTIONS, + ] diff --git a/tests/unit/validators/test_validator_entrypoint.py b/tests/unit/validators/test_validator_entrypoint.py index aab9ccf..100b8de 100644 --- a/tests/unit/validators/test_validator_entrypoint.py +++ b/tests/unit/validators/test_validator_entrypoint.py @@ -9,6 +9,14 @@ def make_issue_list(result): return [i["issue"] for i in result] +def _has(issues, code, sub_reason=None): + """True if an issue with `code` (and optionally `sub_reason`) was emitted.""" + return any( + i["issue"] == code + and (sub_reason is None or i["details"].get("sub_reason") == sub_reason) + for i in issues + ) + # --------------------------------------------------------- # 1) No extended header → early return @@ -107,20 +115,19 @@ def test_entrypoint_out_of_bounds_beyond_image(): # --------------------------------------------------------- def test_entrypoint_section_not_executable(): - metadata = {"optional_header": {}} analysis = { "extended": [{"value": "header", "metadata": {"entry_point": 150}}], - "sections": [ - { - "name": ".text", - "virtual_address": 100, - "virtual_size": 100, - "characteristics": 0, # not executable - } - ], + "sections": [{ + "name": ".text", "virtual_address": 100, "virtual_size": 100, + # CNT_CODE (not executable) isolates this check: without it the + # `not has_code and not executable` branch also fires + # ENTRYPOINT_IN_NON_CODE_SECTION. + "characteristics": 0x00000020, + }], } - issues = validate_entrypoint(metadata, analysis) - assert ReasonCodes.ENTRYPOINT_SECTION_NOT_EXECUTABLE in make_issue_list(issues) + issues = validate_entrypoint({"optional_header": {}}, analysis) + assert len(issues) == 1 + assert issues[0]["issue"] == ReasonCodes.ENTRYPOINT_SECTION_NOT_EXECUTABLE # --------------------------------------------------------- @@ -128,20 +135,18 @@ def test_entrypoint_section_not_executable(): # --------------------------------------------------------- def test_entrypoint_in_non_code_section(): - metadata = {"optional_header": {}} analysis = { "extended": [{"value": "header", "metadata": {"entry_point": 150}}], - "sections": [ - { - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 100, - "characteristics": 0, # not code - } - ], + "sections": [{ + # EXECUTE|READ keeps ENTRYPOINT_SECTION_NOT_EXECUTABLE quiet; the + # ".rsrc" name alone drives this check. + "name": ".rsrc", "virtual_address": 100, "virtual_size": 100, + "characteristics": 0x20000000 | 0x40000000, + }], } - issues = validate_entrypoint(metadata, analysis) - assert ReasonCodes.ENTRYPOINT_IN_NON_CODE_SECTION in make_issue_list(issues) + issues = validate_entrypoint({"optional_header": {}}, analysis) + assert len(issues) == 1 + assert issues[0]["issue"] == ReasonCodes.ENTRYPOINT_IN_NON_CODE_SECTION # --------------------------------------------------------- @@ -186,7 +191,8 @@ def test_entrypoint_zero_length_section(): } issues = validate_entrypoint(metadata, analysis) - assert ReasonCodes.ENTRYPOINT_IN_TRUNCATED_REGION in make_issue_list(issues) + assert _has(issues, ReasonCodes.ENTRYPOINT_IN_TRUNCATED_REGION, + "zero_length_section") # --------------------------------------------------------- @@ -210,7 +216,8 @@ def test_entrypoint_beyond_virtual_size(): } issues = validate_entrypoint(metadata, analysis) - assert ReasonCodes.ENTRYPOINT_IN_TRUNCATED_REGION in make_issue_list(issues) + assert _has(issues, ReasonCodes.ENTRYPOINT_IN_TRUNCATED_REGION, + "beyond_virtual_size") # --------------------------------------------------------- @@ -265,3 +272,35 @@ def test_map_rva_to_file_offset_return_none(): # EP outside VA range → no match → return None result = _map_rva_to_file_offset(sections, 999) assert result is None + + +# -------------------------------------------------------------------- +# 13) Contract and reason key tests +# -------------------------------------------------------------------- + + +def test_dependency_contract(): + assert getattr(validate_entrypoint, "_depends_on") == ("metadata", "analysis") + + +def test_no_details_payload_uses_reserved_reason_key(): + """ + "reason" is reserved by the heuristics emission layer: _det builds metadata + as {"reason": parent, **details}, so a details["reason"] would overwrite + the parent reason code. Validators must use "sub_reason". + """ + metadata = {"optional_header": {"size_of_headers": 300, + "size_of_image": 0x2000}} + analysis = { + "overlay_offset": 50, + "extended": [{"value": "header", "metadata": {"entry_point": 0}}], + "sections": [{"name": ".rsrc", "virtual_address": 0, "virtual_size": 0, + "raw_address": 100, "raw_size": 50, + "characteristics": 0x02000000}], + } + issues = validate_entrypoint(metadata, analysis) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) diff --git a/tests/unit/validators/test_validator_exception_dir.py b/tests/unit/validators/test_validator_exception_dir.py new file mode 100644 index 0000000..6f93ba7 --- /dev/null +++ b/tests/unit/validators/test_validator_exception_dir.py @@ -0,0 +1,1005 @@ +# Copyright (c) 2026 MalX Labs and contributors +# SPDX-License-Identifier: MPL-2.0 + +""" +Unit tests for iocx.validators.exception_table.validate_exception_table. + +Strategy: +- Input is the exception_struct dict produced by parser pe_exception, carried + under internal["exception_struct"]. +- Build dicts directly to isolate validator logic from parser behaviour. +- Tests assert on emitted REASONCODES and the details payload. + +Layer note: the validator is @depends_on("internal", "metadata") and takes TWO +positional arguments. SizeOfImage is read from +metadata["optional_header"]["size_of_image"] - it is NOT part of the analysis +layer, and the validator never receives one. + +Details note: priority-resolved sub-reasons are carried in a "sub_reason" key. +The key "reason" is reserved by the heuristics emission layer, which merges +details over its own reason field - a details["reason"] would overwrite the +parent reason code. + +Fixture note: every fixture below has been verified to emit exactly ONE issue +(or none, for controls) unless a multi-issue outcome is asserted deliberately. +""" + +from __future__ import annotations + +from typing import Any, Dict, List, Optional + +import pytest + +from iocx.reason_codes import ReasonCodes +from iocx.validators.exception_table import validate_exception_table + + +SIZE_OF_IMAGE = 0x5000 + + +# ================================================================= +# Input builders +# ================================================================= + +def _make_metadata(size_of_image: Optional[int] = SIZE_OF_IMAGE) -> Dict[str, Any]: + """ + Public-metadata layer. SizeOfImage lives under optional_header; passing + None models an optional header present but missing the field. + """ + return {"optional_header": {"size_of_image": size_of_image}} + + +def _make_unwind( + version: Optional[int] = 1, + flags: Optional[int] = 0, + size_of_prolog: Optional[int] = 4, + count_of_codes: Optional[int] = 0, + is_chained: bool = False, + chained_rva: Optional[int] = None, + errors: Optional[List[str]] = None, +) -> Dict[str, Any]: + """AMD64 UNWIND_INFO sub-dict. Defaults are a clean V1 record.""" + return { + "version": version, + "flags": flags, + "size_of_prolog": size_of_prolog, + "count_of_codes": count_of_codes, + "is_chained": is_chained, + "chained_rva": chained_rva, + "errors": errors or [], + } + + +def _make_entry( + index: int = 0, + begin_rva: Optional[int] = 0x1000, + end_rva: Optional[int] = 0x1050, + unwind_info_rva: Optional[int] = 0x4100, + unwind: Any = "DEFAULT", + errors: Optional[List[str]] = None, +) -> Dict[str, Any]: + """AMD64 RUNTIME_FUNCTION entry. `unwind="DEFAULT"` builds a clean record.""" + return { + "index": index, + "begin_rva": begin_rva, + "end_rva": end_rva, + "unwind_info_rva": unwind_info_rva, + "unwind": _make_unwind() if unwind == "DEFAULT" else unwind, + "errors": errors or [], + } + + +def _make_arm_entry( + index: int = 0, + begin_rva: Optional[int] = 0x1500, + unwind_info_rva: Optional[int] = 0x4100, + is_packed: bool = False, + errors: Optional[List[str]] = None, +) -> Dict[str, Any]: + """ + ARM/ARM64 8-byte record. Carries NO EndAddress, so end_rva is None and the + validator's range/overlap checks correctly no-op. Packed records carry no + .xdata pointer. + """ + return { + "index": index, + "begin_rva": begin_rva, + "end_rva": None, + "unwind_info_rva": None if is_packed else unwind_info_rva, + "unwind": None, + "is_packed": is_packed, + "packed_data": 0x0AB10001 if is_packed else None, + "errors": errors or [], + } + + +def _make_ex( + rva: Optional[int] = 0x4000, + size: int = 24, + machine: Optional[int] = 0x8664, + arch: str = "amd64", + entry_size: int = 12, + functions: Optional[List[Dict[str, Any]]] = None, + truncations: Optional[List[str]] = None, + errors: Optional[List[str]] = None, +) -> Dict[str, Any]: + return { + "rva": rva, + "size": size, + "machine": machine, + "arch": arch, + "entry_size": entry_size, + "functions": functions or [], + "truncations": truncations or [], + "errors": errors or [], + } + + +def _control_functions() -> List[Dict[str, Any]]: + """Two sorted, non-overlapping, in-bounds entries with clean unwind info.""" + return [ + _make_entry(0, 0x1000, 0x1050, 0x4100), + _make_entry(1, 0x1060, 0x10B0, 0x4110), + ] + + +def _run(ex: Optional[Dict[str, Any]], + metadata: Optional[Dict[str, Any]] = None): + if metadata is None: + metadata = _make_metadata() + return validate_exception_table({"exception_struct": ex}, metadata) + + +def _codes(issues) -> List: + return [i["issue"] for i in issues] + + +def _details_for(issues, code) -> List[Dict[str, Any]]: + return [i["details"] for i in issues if i["issue"] == code] + + +def _has(issues, code, sub_reason=None) -> bool: + """True if an issue with `code` (and optionally `sub_reason`) was emitted.""" + return any( + i["issue"] == code + and (sub_reason is None or i["details"].get("sub_reason") == sub_reason) + for i in issues + ) + + +# ================================================================= +# Absence +# ================================================================= + +class TestAbsence: + """Absence of an exception directory is never a structural defect.""" + + def test_no_exception_struct_returns_no_issues(self): + assert validate_exception_table({}, _make_metadata()) == [] + + def test_explicit_none_returns_no_issues(self): + assert _run(None) == [] + + +# ================================================================= +# Controls +# ================================================================= + +class TestControls: + """Well-formed tables emit nothing. These anchor every anomaly below.""" + + def test_amd64_control_emits_nothing(self): + assert _run(_make_ex(functions=_control_functions())) == [] + + def test_arm64_xdata_control_emits_nothing(self): + ex = _make_ex(machine=0xAA64, arch="arm64", entry_size=8, size=16, + functions=[_make_arm_entry(0, 0x1500), + _make_arm_entry(1, 0x1600)]) + assert _run(ex) == [] + + def test_arm64_packed_control_emits_nothing(self): + ex = _make_ex(machine=0xAA64, arch="arm64", entry_size=8, size=16, + functions=[_make_arm_entry(0, 0x1500, is_packed=True), + _make_arm_entry(1, 0x1600, is_packed=True)]) + assert _run(ex) == [] + + def test_arm64ec_control_emits_nothing(self): + """ARM64EC (0xA641) routes through the same arm64 walk.""" + ex = _make_ex(machine=0xA641, arch="arm64", entry_size=8, size=16, + functions=[_make_arm_entry(0, 0x1500), + _make_arm_entry(1, 0x1600)]) + assert _run(ex) == [] + + +# ================================================================= +# Top-level decode short-circuit +# ================================================================= + +class TestTopLevelDecodeFailure: + + def test_errors_emit_invalid_header(self): + ex = _make_ex(errors=["directory_read_failed"]) + issues = _run(ex) + assert _codes(issues) == [ReasonCodes.EXCEPTION_DIRECTORY_INVALID_HEADER] + d = _details_for(issues, ReasonCodes.EXCEPTION_DIRECTORY_INVALID_HEADER)[0] + assert d["sub_reason"] == "top_level_decode" + assert d["errors"] == ["directory_read_failed"] + + def test_short_circuit_skips_all_later_checks(self): + """ + A top-level error must suppress directory, truncation, arch and + function-table checks - every one of which this fixture would + otherwise trip. + """ + ex = _make_ex( + rva=0x4001, # would trip UNALIGNED + size=25, # would trip SIZE_NOT_MULTIPLE + errors=["boom"], + truncations=["exception_entry_truncated"], + functions=[_make_entry(0, 0, 0x1050, 0x4100, + errors=["begin_rva_zero"])], + ) + issues = _run(ex) + assert _codes(issues) == [ReasonCodes.EXCEPTION_DIRECTORY_INVALID_HEADER] + + +# ================================================================= +# Directory-level checks +# ================================================================= + +class TestDirectoryPlacement: + + def test_unaligned_directory_rva_flagged(self): + ex = _make_ex(rva=0x4001, functions=_control_functions()) + issues = _run(ex) + assert len(issues) == 1 + assert issues[0]["issue"] == ReasonCodes.EXCEPTION_DIRECTORY_UNALIGNED + assert issues[0]["details"]["rva"] == 0x4001 + assert issues[0]["details"]["alignment"] == 4 + + def test_aligned_directory_rva_not_flagged(self): + ex = _make_ex(rva=0x4000, functions=_control_functions()) + assert ReasonCodes.EXCEPTION_DIRECTORY_UNALIGNED not in _codes(_run(ex)) + + def test_size_not_multiple_of_entry_stride_flagged(self): + ex = _make_ex(size=25, functions=_control_functions()) + issues = _run(ex) + assert len(issues) == 1 + d = issues[0]["details"] + assert issues[0]["issue"] == ReasonCodes.EXCEPTION_DIRECTORY_SIZE_NOT_MULTIPLE + assert d["entry_size"] == 12 + assert d["remainder"] == 1 + + def test_size_multiple_of_stride_not_flagged(self): + ex = _make_ex(size=24, functions=_control_functions()) + assert ReasonCodes.EXCEPTION_DIRECTORY_SIZE_NOT_MULTIPLE not in _codes(_run(ex)) + + def test_arm_stride_of_eight_respected(self): + """A size of 16 is valid for arm (stride 8) though not for amd64.""" + ex = _make_ex(machine=0xAA64, arch="arm64", entry_size=8, size=16, + functions=[_make_arm_entry(0, 0x1500)]) + assert ReasonCodes.EXCEPTION_DIRECTORY_SIZE_NOT_MULTIPLE not in _codes(_run(ex)) + + def test_directory_out_of_bounds_flagged(self): + # rva 0x4FF8 + size 12 = 0x5004 > SizeOfImage 0x5000 + ex = _make_ex(rva=0x4FF8, size=12, + functions=[_make_entry(0, 0x1000, 0x1050, 0x4100)]) + issues = _run(ex) + assert len(issues) == 1 + d = issues[0]["details"] + assert issues[0]["issue"] == ReasonCodes.EXCEPTION_DIRECTORY_OUT_OF_BOUNDS + assert d["rva"] == 0x4FF8 + assert d["size_of_image"] == SIZE_OF_IMAGE + + def test_directory_exactly_at_boundary_not_flagged(self): + # rva + size == SizeOfImage is inclusive + ex = _make_ex(rva=0x4FF4, size=12, + functions=[_make_entry(0, 0x1000, 0x1050, 0x4100)]) + assert ReasonCodes.EXCEPTION_DIRECTORY_OUT_OF_BOUNDS not in _codes(_run(ex)) + + def test_rva_none_skips_directory_checks(self): + ex = _make_ex(rva=None, size=25, functions=[]) + issues = _run(ex) + assert ReasonCodes.EXCEPTION_DIRECTORY_UNALIGNED not in _codes(issues) + assert ReasonCodes.EXCEPTION_DIRECTORY_SIZE_NOT_MULTIPLE not in _codes(issues) + assert issues == [] + + +# ================================================================= +# Layer sourcing (SizeOfImage) +# ================================================================= + +class TestSizeOfImageLayer: + """ + SizeOfImage must be read from metadata["optional_header"]. Reading it from + the analysis layer - where it does not exist - silently disabled every + bounds check in production while unit tests that supplied it there passed. + """ + + def test_out_of_bounds_fires_with_metadata_layer(self): + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x99000, 0x99050, 0x4100)]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS) + + def test_size_of_image_at_top_level_is_ignored(self): + """ + REGRESSION GUARD. A stale analysis-shaped dict must NOT be consulted; + if it were, the bounds check would fire and the production dead-path + would be back. + """ + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x99000, 0x99050, 0x4100)]) + issues = _run(ex, metadata={"size_of_image": SIZE_OF_IMAGE}) + assert ReasonCodes.EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS not in _codes(issues) + + def test_absent_optional_header_skips_bounds_checks(self): + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x99000, 0x99050, 0x4100)]) + issues = _run(ex, metadata={}) + assert ReasonCodes.EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS not in _codes(issues) + assert ReasonCodes.EXCEPTION_DIRECTORY_OUT_OF_BOUNDS not in _codes(issues) + + def test_none_size_of_image_skips_bounds_checks(self): + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x99000, 0x99050, 0x4100)]) + issues = _run(ex, metadata=_make_metadata(size_of_image=None)) + assert ReasonCodes.EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS not in _codes(issues) + + +# ================================================================= +# Truncations +# ================================================================= + +class TestTruncations: + + def test_single_tag_emits_one_issue(self): + ex = _make_ex(truncations=["exception_entry_truncated"], + functions=_control_functions()) + issues = _run(ex) + assert len(issues) == 1 + # "table" is a distinct key and was never subject to the reason + # collision, so it is unchanged by the sub_reason migration. + assert issues[0]["details"] == {"table": "exception_entry_truncated"} + + def test_multiple_tags_emit_one_issue_each_in_order(self): + ex = _make_ex(truncations=["exception_table_ragged_tail", + "exception_entry_read_failed"], + functions=_control_functions()) + issues = _run(ex) + tables = [d["table"] for d in + _details_for(issues, ReasonCodes.EXCEPTION_TABLE_TRUNCATED)] + assert tables == ["exception_table_ragged_tail", + "exception_entry_read_failed"] + + def test_no_truncations_emits_nothing(self): + ex = _make_ex(truncations=[], functions=_control_functions()) + assert ReasonCodes.EXCEPTION_TABLE_TRUNCATED not in _codes(_run(ex)) + + +# ================================================================= +# Architecture gate +# ================================================================= + +class TestArchitectureGate: + + def test_unsupported_machine_reported_once(self): + ex = _make_ex(machine=0x014C, arch="unsupported", entry_size=0, + functions=[]) + issues = _run(ex) + assert len(issues) == 1 + d = issues[0]["details"] + assert issues[0]["issue"] == ReasonCodes.EXCEPTION_UNSUPPORTED_MACHINE + assert d["arch"] == "unsupported" + assert d["machine"] == 0x014C + + def test_unsupported_machine_skips_function_walk(self): + """ + The walk must be skipped rather than producing spurious per-entry + codes on a directory we cannot interpret. + """ + ex = _make_ex(machine=0x014C, arch="unsupported", entry_size=0, + functions=[_make_entry(0, 0, 0x1050, 0x4100, + errors=["begin_rva_zero"])]) + issues = _run(ex) + assert _codes(issues) == [ReasonCodes.EXCEPTION_UNSUPPORTED_MACHINE] + + @pytest.mark.parametrize("machine,arch", [ + (0x8664, "amd64"), + (0xAA64, "arm64"), + (0xA641, "arm64"), # ARM64EC + (0x01C4, "arm"), # ARMNT + ]) + def test_table_archs_are_walked(self, machine, arch): + """Every table-based arch reaches the function walk.""" + entry_size = 12 if arch == "amd64" else 8 + if arch == "amd64": + funcs = [_make_entry(0, 0x1900, 0x1950, 0x4100), + _make_entry(1, 0x1800, 0x1850, 0x4110)] + size = 24 + else: + funcs = [_make_arm_entry(0, 0x1900), _make_arm_entry(1, 0x1800)] + size = 16 + ex = _make_ex(machine=machine, arch=arch, entry_size=entry_size, + size=size, functions=funcs) + assert _has(_run(ex), ReasonCodes.EXCEPTION_ENTRIES_NOT_SORTED) + + +# ================================================================= +# Per-entry parser errors +# ================================================================= + +class TestEntryInvalid: + + def test_begin_rva_zero_flagged(self): + ex = _make_ex(functions=[ + _make_entry(0, 0x1000, 0x1050, 0x4100), + _make_entry(1, 0, 0x10B0, 0x4110, errors=["begin_rva_zero"]), + ]) + issues = _run(ex) + assert len(issues) == 1 + assert _has(issues, ReasonCodes.EXCEPTION_ENTRY_INVALID, "begin_rva_zero") + assert _details_for(issues, ReasonCodes.EXCEPTION_ENTRY_INVALID)[0]["index"] == 1 + + @pytest.mark.parametrize("tag", [ + "entry_truncated", "entry_read_failed", "entry_unpack_failed", + "begin_rva_zero", "end_rva_zero", "unwind_rva_zero", + ]) + def test_each_priority_tag_resolves(self, tag): + ex = _make_ex(size=12, functions=[ + _make_entry(0, None, None, None, unwind=None, errors=[tag])]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_ENTRY_INVALID, tag) + + def test_priority_first_match_wins(self): + """entry_truncated outranks begin_rva_zero and unwind_rva_zero.""" + ex = _make_ex(size=12, functions=[_make_entry( + 0, None, None, None, unwind=None, + errors=["begin_rva_zero", "entry_truncated", "unwind_rva_zero"])]) + issues = _run(ex) + assert len(issues) == 1 + assert _has(issues, ReasonCodes.EXCEPTION_ENTRY_INVALID, "entry_truncated") + + def test_unknown_tag_not_flagged(self): + """A future parser tag not in the priority list is skipped silently.""" + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, errors=["future_tag_not_recognised"])]) + assert ReasonCodes.EXCEPTION_ENTRY_INVALID not in _codes(_run(ex)) + + def test_invalid_entry_skips_cross_entry_checks(self): + """ + A structurally unreadable entry cannot feed sortedness/overlap, so it + is skipped rather than producing misleading follow-on codes. + """ + ex = _make_ex(functions=[ + _make_entry(0, 0x1900, 0x1950, 0x4100), + # begin 0 would look "unsorted" against 0x1900 if not skipped + _make_entry(1, 0, 0x1850, 0x4110, errors=["begin_rva_zero"]), + ]) + issues = _run(ex) + assert _codes(issues) == [ReasonCodes.EXCEPTION_ENTRY_INVALID] + + def test_multiple_bad_entries_each_flagged(self): + ex = _make_ex(size=36, functions=[ + _make_entry(0, 0x1000, 0x1050, 0x4100), + _make_entry(1, 0, 0x10B0, 0x4110, errors=["begin_rva_zero"]), + _make_entry(2, 0x10C0, 0, 0x4120, errors=["end_rva_zero"]), + ]) + assert len(_details_for(_run(ex), ReasonCodes.EXCEPTION_ENTRY_INVALID)) == 2 + + +# ================================================================= +# Function range +# ================================================================= + +class TestFunctionRange: + + def test_begin_equals_end_flagged_as_empty(self): + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x1500, 0x1500, 0x4100)]) + issues = _run(ex) + assert len(issues) == 1 + d = issues[0]["details"] + assert issues[0]["issue"] == ReasonCodes.EXCEPTION_FUNCTION_RANGE_INVALID + assert d["empty"] is True + + def test_begin_greater_than_end_flagged_as_inverted(self): + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x1500, 0x1400, 0x4100)]) + issues = _run(ex) + assert _has(issues, ReasonCodes.EXCEPTION_FUNCTION_RANGE_INVALID) + d = _details_for(issues, ReasonCodes.EXCEPTION_FUNCTION_RANGE_INVALID)[0] + assert d["empty"] is False + + def test_valid_range_not_flagged(self): + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x1000, 0x1050, 0x4100)]) + assert ReasonCodes.EXCEPTION_FUNCTION_RANGE_INVALID not in _codes(_run(ex)) + + def test_arm_entries_skip_range_check(self): + """ARM records carry no EndAddress, so the check must no-op.""" + ex = _make_ex(machine=0xAA64, arch="arm64", entry_size=8, size=8, + functions=[_make_arm_entry(0, 0x1500)]) + assert ReasonCodes.EXCEPTION_FUNCTION_RANGE_INVALID not in _codes(_run(ex)) + + +# ================================================================= +# RVA bounds +# ================================================================= + +class TestFunctionRvaBounds: + + def test_begin_and_end_out_of_bounds_listed_in_fields(self): + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x99000, 0x99050, 0x4100)]) + issues = _run(ex) + assert len(issues) == 1 + d = issues[0]["details"] + assert issues[0]["issue"] == ReasonCodes.EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS + # "fields" is a distinct key naming each offending RVA + assert d["fields"] == ["begin_rva", "end_rva"] + assert d["size_of_image"] == SIZE_OF_IMAGE + + def test_unwind_rva_out_of_bounds_listed(self): + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x1000, 0x1050, 0x99000, + unwind=None)]) + d = _details_for(_run(ex), + ReasonCodes.EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS)[0] + assert d["fields"] == ["unwind_info_rva"] + + def test_zero_unwind_rva_not_bounds_checked(self): + """UnwindInfoAddress of 0 is absent, not out of bounds.""" + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x1000, 0x1050, 0, + unwind=None)]) + issues = _run(ex) + assert ReasonCodes.EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS not in _codes(issues) + + def test_end_rva_may_equal_size_of_image(self): + """ + EndAddress is one past the function, so end == SizeOfImage is legal + while begin == SizeOfImage is not. + """ + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x4FF0, SIZE_OF_IMAGE, 0x4100)]) + assert ReasonCodes.EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS not in _codes(_run(ex)) + + def test_begin_at_size_of_image_flagged(self): + ex = _make_ex(size=12, + functions=[_make_entry(0, SIZE_OF_IMAGE, SIZE_OF_IMAGE, + 0x4100)]) + d = _details_for(_run(ex), + ReasonCodes.EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS)[0] + assert "begin_rva" in d["fields"] + + +# ================================================================= +# Sortedness — the headline heuristic +# ================================================================= + +class TestSortedness: + """ + The loader binary-searches .pdata, so BeginAddress must ascend. An + out-of-order entry silently loses its unwind data at runtime while every + byte is present on disk. + """ + + def test_descending_begin_flagged(self): + ex = _make_ex(functions=[ + _make_entry(0, 0x1900, 0x1950, 0x4100), + _make_entry(1, 0x1800, 0x1850, 0x4110), + ]) + issues = _run(ex) + assert len(issues) == 1 + d = issues[0]["details"] + assert issues[0]["issue"] == ReasonCodes.EXCEPTION_ENTRIES_NOT_SORTED + assert d["begin_rva"] == 0x1800 + assert d["prev_begin_rva"] == 0x1900 + assert d["index"] == 1 + + def test_ascending_begin_not_flagged(self): + assert ReasonCodes.EXCEPTION_ENTRIES_NOT_SORTED not in _codes( + _run(_make_ex(functions=_control_functions()))) + + def test_equal_begin_not_flagged(self): + """Ascending is non-strict; equal begins are an overlap, not disorder.""" + ex = _make_ex(functions=[ + _make_entry(0, 0x1000, 0x1050, 0x4100), + _make_entry(1, 0x1000, 0x1050, 0x4110), + ]) + assert ReasonCodes.EXCEPTION_ENTRIES_NOT_SORTED not in _codes(_run(ex)) + + def test_wild_entry_does_not_poison_the_cursor(self): + """ + The ascending cursor only advances on sane, sorted begins, so a single + out-of-order entry must not lower the bar for everything after it. + + Entry 2 is chosen to sit BETWEEN the wild value (0x0900) and the true + cursor (0x1000). A correct cursor still holds 0x1000, so entry 2 is + also unsorted -> 2 issues. A cursor that advanced onto the wild entry + would hold 0x0900, making entry 2 look fine -> 1 issue. A fixture + above both values (e.g. 0x1100) cannot tell the two apart. + """ + ex = _make_ex(size=36, functions=[ + _make_entry(0, 0x1000, 0x1050, 0x4100), + _make_entry(1, 0x0900, 0x0950, 0x4110), # the wild one + _make_entry(2, 0x0950, 0x0990, 0x4120), # between wild and cursor + ]) + issues = _run(ex) + not_sorted = _details_for(issues, ReasonCodes.EXCEPTION_ENTRIES_NOT_SORTED) + assert len(not_sorted) == 2 + assert [d["index"] for d in not_sorted] == [1, 2] + # both compared against the UNMOVED cursor, not the wild entry + assert all(d["prev_begin_rva"] == 0x1000 for d in not_sorted) + + def test_sortedness_applies_to_arm(self): + ex = _make_ex(machine=0xAA64, arch="arm64", entry_size=8, size=16, + functions=[_make_arm_entry(0, 0x1600), + _make_arm_entry(1, 0x1500)]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_ENTRIES_NOT_SORTED) + + def test_sortedness_applies_to_arm_packed(self): + ex = _make_ex(machine=0xAA64, arch="arm64", entry_size=8, size=16, + functions=[_make_arm_entry(0, 0x1600, is_packed=True), + _make_arm_entry(1, 0x1500, is_packed=True)]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_ENTRIES_NOT_SORTED) + + def test_sortedness_applies_to_arm64ec(self): + """ARM64EC uses the same walk, so the invariant holds there too.""" + ex = _make_ex(machine=0xA641, arch="arm64", entry_size=8, size=16, + functions=[_make_arm_entry(0, 0x1600), + _make_arm_entry(1, 0x1500)]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_ENTRIES_NOT_SORTED) + + +# ================================================================= +# Overlap +# ================================================================= + +class TestFunctionOverlap: + + def test_overlapping_ranges_flagged(self): + ex = _make_ex(functions=[ + _make_entry(0, 0x1800, 0x1880, 0x4100), + _make_entry(1, 0x1840, 0x18C0, 0x4110), # begins inside entry 0 + ]) + issues = _run(ex) + assert len(issues) == 1 + d = issues[0]["details"] + assert issues[0]["issue"] == ReasonCodes.EXCEPTION_FUNCTION_OVERLAP + assert d["begin_rva"] == 0x1840 + assert d["prev_end_rva"] == 0x1880 + + def test_adjacent_ranges_not_flagged(self): + """prev_end is exclusive: begin == prev_end is adjacency, not overlap.""" + ex = _make_ex(functions=[ + _make_entry(0, 0x1000, 0x1050, 0x4100), + _make_entry(1, 0x1050, 0x10A0, 0x4110), + ]) + assert ReasonCodes.EXCEPTION_FUNCTION_OVERLAP not in _codes(_run(ex)) + + def test_unsorted_pair_not_double_counted_as_overlap(self): + """ + An unsorted pair is reported once as NOT_SORTED; the overlap check is + gated on begin >= prev_begin so it does not also fire. + """ + ex = _make_ex(functions=[ + _make_entry(0, 0x1900, 0x1950, 0x4100), + _make_entry(1, 0x1800, 0x1850, 0x4110), + ]) + issues = _run(ex) + assert ReasonCodes.EXCEPTION_FUNCTION_OVERLAP not in _codes(issues) + assert _codes(issues) == [ReasonCodes.EXCEPTION_ENTRIES_NOT_SORTED] + + def test_arm_entries_skip_overlap_check(self): + """No EndAddress means no overlap can be computed.""" + ex = _make_ex(machine=0xAA64, arch="arm64", entry_size=8, size=16, + functions=[_make_arm_entry(0, 0x1800), + _make_arm_entry(1, 0x1840)]) + assert _run(ex) == [] + + +# ================================================================= +# Unwind info (AMD64) +# ================================================================= + +class TestUnwindInfo: + + def test_unaligned_unwind_rva_flagged(self): + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x1000, 0x1050, 0x4101)]) + issues = _run(ex) + assert len(issues) == 1 + d = issues[0]["details"] + assert issues[0]["issue"] == ReasonCodes.EXCEPTION_UNWIND_INFO_UNALIGNED + assert d["unwind_info_rva"] == 0x4101 + assert d["alignment"] == 4 + + def test_zero_unwind_rva_not_alignment_checked(self): + ex = _make_ex(size=12, + functions=[_make_entry(0, 0x1000, 0x1050, 0, + unwind=None)]) + assert ReasonCodes.EXCEPTION_UNWIND_INFO_UNALIGNED not in _codes(_run(ex)) + + @pytest.mark.parametrize("version", [1, 2, 3]) + def test_valid_versions_not_flagged(self, version): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, unwind=_make_unwind(version=version))]) + assert ReasonCodes.EXCEPTION_UNWIND_INFO_INVALID not in _codes(_run(ex)) + + @pytest.mark.parametrize("version", [0, 4, 5, 7]) + def test_invalid_versions_flagged(self, version): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, unwind=_make_unwind(version=version))]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_UNWIND_INFO_INVALID, + "unwind_version_invalid") + + @pytest.mark.parametrize("flags", [0x00, 0x01, 0x02, 0x04, 0x08, 0x0F]) + def test_known_flag_bits_not_flagged(self, flags): + """EHANDLER | UHANDLER | CHAININFO | LARGE are all legal.""" + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + # a chained blob needs a valid target, else the chain check fires + unwind=_make_unwind(flags=flags, chained_rva=0x4200))]) + assert ReasonCodes.EXCEPTION_UNWIND_INFO_INVALID not in _codes(_run(ex)) + + @pytest.mark.parametrize("flags", [0x10, 0x20, 0x80]) + def test_reserved_flag_bits_flagged(self, flags): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, unwind=_make_unwind(flags=flags))]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_UNWIND_INFO_INVALID, + "unwind_flags_reserved_bits") + + @pytest.mark.parametrize("tag", [ + "unwind_read_failed", "unwind_truncated", "unwind_unpack_failed", + "unwind_version_invalid", "unwind_flags_reserved_bits", + "unwind_codes_truncated", + ]) + def test_each_unwind_priority_tag_resolves(self, tag): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(version=None, flags=None, errors=[tag]))]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_UNWIND_INFO_INVALID, tag) + + def test_unwind_priority_first_match_wins(self): + """ + unwind_read_failed outranks the later tags, including the version and + flag anomalies the validator derives itself. + """ + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(version=5, flags=0x10, + errors=["unwind_codes_truncated", + "unwind_read_failed"]))]) + issues = _run(ex) + invalid = _details_for(issues, ReasonCodes.EXCEPTION_UNWIND_INFO_INVALID) + assert len(invalid) == 1 + assert invalid[0]["sub_reason"] == "unwind_read_failed" + + def test_derived_version_error_not_duplicated(self): + """ + The validator appends unwind_version_invalid only if absent, so a + parser that already reported it produces one issue, not two. + """ + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(version=5, errors=["unwind_version_invalid"]))]) + assert len(_details_for(_run(ex), + ReasonCodes.EXCEPTION_UNWIND_INFO_INVALID)) == 1 + + def test_arm_entries_have_no_unwind_checks(self): + """ARM records carry unwind=None, so the decode checks must no-op.""" + ex = _make_ex(machine=0xAA64, arch="arm64", entry_size=8, size=8, + functions=[_make_arm_entry(0, 0x1500)]) + assert ReasonCodes.EXCEPTION_UNWIND_INFO_INVALID not in _codes(_run(ex)) + + +# ================================================================= +# Chained unwind +# ================================================================= + +class TestUnwindChain: + + def test_valid_chain_target_not_flagged(self): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(flags=0x04, is_chained=True, + chained_rva=0x4200))]) + assert ReasonCodes.EXCEPTION_UNWIND_CHAIN_INVALID not in _codes(_run(ex)) + + def test_missing_chain_target_flagged(self): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(flags=0x04, is_chained=True, chained_rva=0))]) + issues = _run(ex) + assert len(issues) == 1 + assert _has(issues, ReasonCodes.EXCEPTION_UNWIND_CHAIN_INVALID, + "chain_target_missing") + + def test_none_chain_target_flagged_as_missing(self): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(flags=0x04, is_chained=True, + chained_rva=None))]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_UNWIND_CHAIN_INVALID, + "chain_target_missing") + + def test_unaligned_chain_target_flagged(self): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(flags=0x04, is_chained=True, + chained_rva=0x4201))]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_UNWIND_CHAIN_INVALID, + "chain_target_unaligned") + + def test_out_of_bounds_chain_target_flagged(self): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(flags=0x04, is_chained=True, + chained_rva=0x99000))]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_UNWIND_CHAIN_INVALID, + "chain_target_out_of_bounds") + + def test_self_referential_chain_target_flagged(self): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(flags=0x04, is_chained=True, + chained_rva=0x4100))]) # == its own unwind rva + assert _has(_run(ex), ReasonCodes.EXCEPTION_UNWIND_CHAIN_INVALID, + "chain_self_reference") + + def test_chain_detected_from_flag_without_is_chained(self): + """ + The chain check triggers on UNW_FLAG_CHAININFO even when the parser + did not set is_chained, so a partial parse still gets validated. + """ + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(flags=0x04, is_chained=False, + chained_rva=0))]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_UNWIND_CHAIN_INVALID, + "chain_target_missing") + + def test_chain_detected_from_is_chained_without_flag(self): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(flags=0, is_chained=True, chained_rva=0))]) + assert _has(_run(ex), ReasonCodes.EXCEPTION_UNWIND_CHAIN_INVALID, + "chain_target_missing") + + def test_unchained_entry_skips_chain_check(self): + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x1000, 0x1050, 0x4100, + unwind=_make_unwind(flags=0, is_chained=False, chained_rva=None))]) + assert ReasonCodes.EXCEPTION_UNWIND_CHAIN_INVALID not in _codes(_run(ex)) + + +# ================================================================= +# Combined scenarios +# ================================================================= + +class TestCombinedAnomalies: + + def test_directory_and_entry_anomalies_emit_independently(self): + ex = _make_ex(rva=0x4001, size=25, functions=[ + _make_entry(0, 0x1900, 0x1950, 0x4100), + _make_entry(1, 0x1800, 0x1850, 0x4110), + ]) + codes = set(_codes(_run(ex))) + assert ReasonCodes.EXCEPTION_DIRECTORY_UNALIGNED in codes + assert ReasonCodes.EXCEPTION_DIRECTORY_SIZE_NOT_MULTIPLE in codes + assert ReasonCodes.EXCEPTION_ENTRIES_NOT_SORTED in codes + + def test_one_entry_can_raise_several_distinct_codes(self): + """Range, bounds and unwind faults are independent facts.""" + ex = _make_ex(size=12, functions=[_make_entry( + 0, 0x99000, 0x99000, 0x4101, + unwind=_make_unwind(version=5))]) + codes = set(_codes(_run(ex))) + assert ReasonCodes.EXCEPTION_FUNCTION_RVA_OUT_OF_BOUNDS in codes + assert ReasonCodes.EXCEPTION_FUNCTION_RANGE_INVALID in codes + assert ReasonCodes.EXCEPTION_UNWIND_INFO_UNALIGNED in codes + assert ReasonCodes.EXCEPTION_UNWIND_INFO_INVALID in codes + + +# ================================================================= +# Output contract +# ================================================================= + +class TestOutputContract: + + def test_dependency_contract(self): + assert getattr(validate_exception_table, "_depends_on") == ( + "internal", "metadata") + + def test_returns_list(self): + assert isinstance(_run(_make_ex(functions=_control_functions())), list) + + def test_each_issue_has_issue_and_details(self): + ex = _make_ex(rva=0x4001, functions=_control_functions()) + for issue in _run(ex): + assert set(issue) == {"issue", "details"} + assert isinstance(issue["issue"], str) + assert isinstance(issue["details"], dict) + + def test_json_serializable(self): + import json + ex = _make_ex(rva=0x4001, size=25, + truncations=["exception_entry_truncated"], + functions=[_make_entry(0, 0x99000, 0x99000, 0x4101, + unwind=_make_unwind(version=5))]) + json.dumps(_run(ex)) # must not raise + + def test_no_details_payload_uses_reserved_reason_key(self): + """ + "reason" is reserved by the heuristics emission layer: _det builds + metadata as {"reason": parent, **details}, so a details["reason"] would + overwrite the parent reason code. Validators must use "sub_reason". + + Exercises the directory, truncation, entry, range, bounds, sortedness, + unwind and chain paths together. + """ + ex = _make_ex(rva=0x4001, size=25, + truncations=["exception_entry_truncated"], + functions=[ + _make_entry(0, 0x1900, 0x1950, 0x4100, + unwind=_make_unwind(version=5)), + _make_entry(1, 0x1800, 0x1800, 0x4101, + unwind=_make_unwind(flags=0x04, + is_chained=True, + chained_rva=0)), + _make_entry(2, 0, 0, 0, unwind=None, + errors=["begin_rva_zero"]), + ]) + issues = _run(ex) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + + def test_top_level_decode_avoids_reserved_reason_key(self): + """The short-circuit path is unreachable above; pin it separately.""" + issues = _run(_make_ex(errors=["boom"])) + assert issues + assert "reason" not in issues[0]["details"] + + def test_unsupported_machine_avoids_reserved_reason_key(self): + """This path also returns early and is mutually exclusive.""" + issues = _run(_make_ex(machine=0x014C, arch="unsupported", + entry_size=0)) + assert issues + assert "reason" not in issues[0]["details"] + + +# ================================================================= +# Determinism +# ================================================================= + +class TestDeterminism: + + def test_repeated_validation_produces_identical_issues(self): + import json + ex = _make_ex(rva=0x4001, size=25, + truncations=["exception_entry_truncated", + "exception_table_ragged_tail"], + functions=[ + _make_entry(0, 0x1900, 0x1950, 0x4100, + unwind=_make_unwind(version=5)), + _make_entry(1, 0x1800, 0x1800, 0x4101, + unwind=_make_unwind(flags=0x04, + is_chained=True, + chained_rva=0)), + ]) + results = [_run(ex) for _ in range(20)] + first = json.dumps(results[0], sort_keys=True) + for r in results[1:]: + assert json.dumps(r, sort_keys=True) == first + + def test_priority_resolution_deterministic(self): + ex = _make_ex(size=12, functions=[_make_entry( + 0, None, None, None, unwind=None, + errors=["unwind_rva_zero", "entry_truncated", "begin_rva_zero"])]) + results = [_run(ex) for _ in range(20)] + for r in results[1:]: + assert r == results[0] + assert results[0][0]["details"]["sub_reason"] == "entry_truncated" diff --git a/tests/unit/validators/test_validator_exports.py b/tests/unit/validators/test_validator_exports.py index 4b4daa7..59b8d62 100644 --- a/tests/unit/validators/test_validator_exports.py +++ b/tests/unit/validators/test_validator_exports.py @@ -9,6 +9,15 @@ - Build dicts directly; isolate validator logic from parser behaviour. - Each test asserts on the set of REASONCODES emitted and the details payload. + +Layer note: the validator is @depends_on("internal", "metadata"), so the second +positional argument is the PUBLIC METADATA layer, not the analysis layer. +SizeOfImage is read from metadata["optional_header"]["size_of_image"]. + +Details note: priority-resolved sub-reasons are carried in a "sub_reason" key. +The key "reason" is reserved by the heuristics emission layer, which merges +details over its own reason field - a details["reason"] would overwrite the +parent reason code. """ from __future__ import annotations @@ -28,8 +37,12 @@ _NOT_PROVIDED = object() -def _make_analysis(size_of_image: Optional[int] = 0x100000) -> Dict[str, Any]: - return {"size_of_image": size_of_image} +def _make_metadata(size_of_image: Optional[int] = 0x100000) -> Dict[str, Any]: + """ + Public-metadata layer. SizeOfImage lives under optional_header; passing + None models an optional header present but missing the field. + """ + return {"optional_header": {"size_of_image": size_of_image}} def _make_header( @@ -133,10 +146,10 @@ def _details_for(issues, code) -> List[Dict[str, Any]]: class TestAbsence: def test_no_export_struct_returns_no_issues(self): - assert validate_exports({}, _make_analysis()) == [] + assert validate_exports({}, _make_metadata()) == [] def test_explicit_none_returns_no_issues(self): - assert validate_exports({"export_struct": None}, _make_analysis()) == [] + assert validate_exports({"export_struct": None}, _make_metadata()) == [] # ================================================================= @@ -149,12 +162,12 @@ def test_errors_present_emits_invalid_returns_early(self): exp = _make_exp(errors=["header_read_failed"]) # Add other issues that should NOT be emitted due to early return exp["truncations"] = ["eat_truncated"] - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) codes = _codes(issues) assert ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER in codes assert ReasonCodes.EXPORT_TABLE_TRUNCATED not in codes details = _details_for(issues, ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER) - assert details[0]["reason"] == "top_level_decode" + assert details[0]["sub_reason"] == "top_level_decode" assert details[0]["errors"] == ["header_read_failed"] @@ -166,27 +179,41 @@ class TestPlacement: def test_in_bounds_no_issue(self): exp = _make_exp(rva=0x1000, size=200) - analysis = _make_analysis(size_of_image=0x100000) - issues = validate_exports({"export_struct": exp}, analysis) + metadata = _make_metadata(size_of_image=0x100000) + issues = validate_exports({"export_struct": exp}, metadata) assert ReasonCodes.EXPORT_DIRECTORY_OUT_OF_BOUNDS not in _codes(issues) def test_extends_past_image_flagged(self): exp = _make_exp(rva=0xFFF00, size=0x200) - analysis = _make_analysis(size_of_image=0x100000) - issues = validate_exports({"export_struct": exp}, analysis) + metadata = _make_metadata(size_of_image=0x100000) + issues = validate_exports({"export_struct": exp}, metadata) details = _details_for(issues, ReasonCodes.EXPORT_DIRECTORY_OUT_OF_BOUNDS) assert len(details) == 1 assert details[0]["rva"] == 0xFFF00 def test_silent_when_size_of_image_missing(self): exp = _make_exp(rva=0xFFF00, size=0x200) - analysis = _make_analysis(size_of_image=None) - issues = validate_exports({"export_struct": exp}, analysis) + metadata = _make_metadata(size_of_image=None) + issues = validate_exports({"export_struct": exp}, metadata) + assert ReasonCodes.EXPORT_DIRECTORY_OUT_OF_BOUNDS not in _codes(issues) + + def test_silent_when_optional_header_absent(self): + """ + The metadata layer may omit optional_header entirely. The placement + check must skip rather than raise. + + Regression guard: while the fixture supplied size_of_image at the top + level of the wrong layer, `optional_header` was always absent, so the + OUT_OF_BOUNDS check could never fire and the sibling "silent" tests + passed vacuously. This pins the absent-header case on purpose. + """ + exp = _make_exp(rva=0xFFF00, size=0x200) + issues = validate_exports({"export_struct": exp}, {}) assert ReasonCodes.EXPORT_DIRECTORY_OUT_OF_BOUNDS not in _codes(issues) def test_silent_when_rva_none(self): exp = _make_exp(rva=None, size=0) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_DIRECTORY_OUT_OF_BOUNDS not in _codes(issues) @@ -198,19 +225,21 @@ class TestTruncations: def test_no_truncations_no_issues(self): exp = _make_exp(truncations=[]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_TABLE_TRUNCATED not in _codes(issues) def test_single_truncation_emits_one_issue(self): exp = _make_exp(truncations=["eat_truncated"]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_TABLE_TRUNCATED) assert len(details) == 1 + # "table" is a distinct key and was never subject to the reason + # collision, so it is unchanged by the sub_reason migration. assert details[0]["table"] == "eat_truncated" def test_multiple_truncations_emit_separate_issues(self): exp = _make_exp(truncations=["eat_truncated", "enpt_truncated", "eot_truncated"]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_TABLE_TRUNCATED) assert len(details) == 3 tables = [d["table"] for d in details] @@ -226,39 +255,39 @@ class TestHeaderConsistency: def test_clean_header_no_issues(self): header = _make_header(num_functions=2, num_names=2) exp = _make_exp(header=header) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER not in _codes(issues) def test_eat_rva_zero_with_nonzero_count_flagged(self): header = _make_header(num_functions=5, addr_functions=0) exp = _make_exp(header=header) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER) - reasons = [d["reason"] for d in details] + reasons = [d["sub_reason"] for d in details] assert "eat_rva_zero_with_nonzero_count" in reasons def test_enpt_rva_zero_with_nonzero_count_flagged(self): header = _make_header(num_functions=5, num_names=3, addr_names=0) exp = _make_exp(header=header) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER) - reasons = [d["reason"] for d in details] + reasons = [d["sub_reason"] for d in details] assert "enpt_rva_zero_with_nonzero_count" in reasons def test_eot_rva_zero_with_nonzero_count_flagged(self): header = _make_header(num_functions=5, num_names=3, addr_name_ordinals=0) exp = _make_exp(header=header) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER) - reasons = [d["reason"] for d in details] + reasons = [d["sub_reason"] for d in details] assert "eot_rva_zero_with_nonzero_count" in reasons def test_num_names_exceeds_num_functions_flagged(self): header = _make_header(num_functions=2, num_names=5) exp = _make_exp(header=header) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER) - reasons = [d["reason"] for d in details] + reasons = [d["sub_reason"] for d in details] assert "num_names_exceeds_num_functions" in reasons def test_multiple_consistency_failures_emit_multiple_issues(self): @@ -268,21 +297,21 @@ def test_multiple_consistency_failures_emit_multiple_issues(self): addr_functions=0, # zero with count ) exp = _make_exp(header=header) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER) assert len(details) >= 2 def test_header_none_skips_consistency_check(self): exp = _make_exp(header=None, truncations=["export_directory_header"]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) # Should emit truncation but not consistency issues assert ReasonCodes.EXPORT_TABLE_TRUNCATED in _codes(issues) consistency = [ d for d in _details_for(issues, ReasonCodes.EXPORT_DIRECTORY_INVALID_HEADER) - if d.get("reason") in {"eat_rva_zero_with_nonzero_count", - "enpt_rva_zero_with_nonzero_count", - "eot_rva_zero_with_nonzero_count", - "num_names_exceeds_num_functions"} + if d.get("sub_reason") in {"eat_rva_zero_with_nonzero_count", + "enpt_rva_zero_with_nonzero_count", + "eot_rva_zero_with_nonzero_count", + "num_names_exceeds_num_functions"} ] assert consistency == [] @@ -299,7 +328,7 @@ def test_clean_name_pointer_no_issues(self): header=_make_header(num_functions=1, num_names=1), name_pointers=[np], ) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_NAME_RVA_INVALID not in _codes(issues) assert ReasonCodes.EXPORT_NAME_NOT_ASCII not in _codes(issues) assert ReasonCodes.EXPORT_NAME_ORDINAL_INDEX_INVALID not in _codes(issues) @@ -307,34 +336,34 @@ def test_clean_name_pointer_no_issues(self): def test_name_rva_zero_flagged(self): np = _make_name_pointer(errors=["name_rva_zero"]) exp = _make_exp(name_pointers=[np]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_NAME_RVA_INVALID) assert len(details) == 1 - assert details[0]["reason"] == "name_rva_zero" + assert details[0]["sub_reason"] == "name_rva_zero" def test_name_rva_priority_resolution(self): """name_rva_missing wins over read_failed when both are present.""" np = _make_name_pointer(errors=["read_failed", "name_rva_missing"]) exp = _make_exp(name_pointers=[np]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_NAME_RVA_INVALID) assert len(details) == 1 - assert details[0]["reason"] == "name_rva_missing" + assert details[0]["sub_reason"] == "name_rva_missing" def test_unterminated_flagged_with_correct_reason(self): np = _make_name_pointer(errors=["unterminated"]) exp = _make_exp(name_pointers=[np]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_NAME_RVA_INVALID) - assert details[0]["reason"] == "unterminated" + assert details[0]["sub_reason"] == "unterminated" def test_non_ascii_flagged_with_correct_reason(self): np = _make_name_pointer(name="caf\ufffd", errors=["non_ascii"]) exp = _make_exp(name_pointers=[np]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_NAME_NOT_ASCII) assert len(details) == 1 - assert details[0]["reason"] == "non_ascii" + assert details[0]["sub_reason"] == "non_ascii" def test_name_not_printable_ascii_flagged(self): np = _make_name_pointer( @@ -343,10 +372,10 @@ def test_name_not_printable_ascii_flagged(self): errors=["name_not_printable_ascii"], ) exp = _make_exp(name_pointers=[np]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_NAME_NOT_ASCII) assert len(details) == 1 - assert details[0]["reason"] == "name_not_printable_ascii" + assert details[0]["sub_reason"] == "name_not_printable_ascii" def test_name_encoding_priority_resolution(self): """non_ascii wins over name_not_printable_ascii.""" @@ -354,18 +383,18 @@ def test_name_encoding_priority_resolution(self): errors=["name_not_printable_ascii", "non_ascii"], ) exp = _make_exp(name_pointers=[np]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_NAME_NOT_ASCII) assert len(details) == 1 - assert details[0]["reason"] == "non_ascii" + assert details[0]["sub_reason"] == "non_ascii" def test_ordinal_index_missing_flagged(self): np = _make_name_pointer(errors=["ordinal_index_missing"]) exp = _make_exp(name_pointers=[np]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_NAME_ORDINAL_INDEX_INVALID) assert len(details) == 1 - assert details[0]["reason"] == "missing" + assert details[0]["sub_reason"] == "missing" def test_ordinal_index_out_of_range_flagged(self): np = _make_name_pointer( @@ -376,10 +405,10 @@ def test_ordinal_index_out_of_range_flagged(self): header=_make_header(num_functions=5, num_names=1), name_pointers=[np], ) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_NAME_ORDINAL_INDEX_INVALID) assert len(details) == 1 - assert details[0]["reason"] == "out_of_range" + assert details[0]["sub_reason"] == "out_of_range" assert details[0]["ordinal_index"] == 99 assert details[0]["num_functions"] == 5 @@ -389,7 +418,7 @@ def test_no_double_emission_per_entry(self): errors=["name_rva_zero", "name_rva_missing", "read_failed"], ) exp = _make_exp(name_pointers=[np]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) rva_issues = [ i for i in issues if i["issue"] == ReasonCodes.EXPORT_NAME_RVA_INVALID @@ -413,7 +442,7 @@ def test_sorted_names_no_issue(self): header=_make_header(num_functions=3, num_names=3), name_pointers=nps, ) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_NAME_POINTER_TABLE_UNSORTED not in _codes(issues) def test_unsorted_names_flagged(self): @@ -425,11 +454,14 @@ def test_unsorted_names_flagged(self): header=_make_header(num_functions=2, num_names=2), name_pointers=nps, ) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_NAME_POINTER_TABLE_UNSORTED) assert len(details) == 1 + # This code carries no sub_reason: its details were never subject to + # the reason collision, so the ENPT ordering payload is unchanged. assert details[0]["name_count"] == 2 assert details[0]["first_violation_index"] == 1 + assert "sub_reason" not in details[0] def test_unreadable_name_skips_ordering_check(self): nps = [ @@ -437,12 +469,12 @@ def test_unreadable_name_skips_ordering_check(self): _make_name_pointer(index=1, name="Zeta"), ] exp = _make_exp(name_pointers=nps) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_NAME_POINTER_TABLE_UNSORTED not in _codes(issues) def test_empty_name_pointers_no_issue(self): exp = _make_exp(name_pointers=[]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_NAME_POINTER_TABLE_UNSORTED not in _codes(issues) @@ -458,23 +490,23 @@ def test_clean_function_no_issues(self): header=_make_header(num_functions=1), functions=[fn], ) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_FUNCTION_RVA_INVALID not in _codes(issues) def test_max_ordinal_exceeds_u16_flagged(self): header = _make_header(base=0xFFF0, num_functions=32) exp = _make_exp(header=header, functions=[]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_ORDINAL_OUT_OF_RANGE) assert len(details) == 1 - assert details[0]["reason"] == "max_exceeds_u16" + assert details[0]["sub_reason"] == "max_exceeds_u16" assert details[0]["base"] == 0xFFF0 assert details[0]["max_ordinal"] == 0xFFF0 + 31 def test_max_ordinal_in_range_no_issue(self): header = _make_header(base=1, num_functions=100) exp = _make_exp(header=header) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_ORDINAL_OUT_OF_RANGE not in _codes(issues) def test_function_rva_within_image_no_issue(self): @@ -485,7 +517,7 @@ def test_function_rva_within_image_no_issue(self): ) issues = validate_exports( {"export_struct": exp}, - _make_analysis(size_of_image=0x100000), + _make_metadata(size_of_image=0x100000), ) assert ReasonCodes.EXPORT_FUNCTION_RVA_INVALID not in _codes(issues) @@ -497,11 +529,11 @@ def test_function_rva_beyond_image_flagged(self): ) issues = validate_exports( {"export_struct": exp}, - _make_analysis(size_of_image=0x100000), + _make_metadata(size_of_image=0x100000), ) details = _details_for(issues, ReasonCodes.EXPORT_FUNCTION_RVA_INVALID) assert len(details) == 1 - assert details[0]["reason"] == "exceeds_image" + assert details[0]["sub_reason"] == "exceeds_image" assert details[0]["address_rva"] == 0x200000 def test_zero_rva_function_skipped(self): @@ -511,7 +543,7 @@ def test_zero_rva_function_skipped(self): header=_make_header(num_functions=1), functions=[fn], ) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_FUNCTION_RVA_INVALID not in _codes(issues) def test_none_rva_function_skipped(self): @@ -520,7 +552,7 @@ def test_none_rva_function_skipped(self): header=_make_header(num_functions=1), functions=[fn], ) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_FUNCTION_RVA_INVALID not in _codes(issues) def test_function_rva_check_silent_when_size_of_image_missing(self): @@ -531,8 +563,22 @@ def test_function_rva_check_silent_when_size_of_image_missing(self): ) issues = validate_exports( {"export_struct": exp}, - _make_analysis(size_of_image=None), + _make_metadata(size_of_image=None), + ) + assert ReasonCodes.EXPORT_FUNCTION_RVA_INVALID not in _codes(issues) + + def test_function_rva_check_silent_when_optional_header_absent(self): + """ + Companion to the test above: with no optional_header at all the check + must skip rather than raise. Pins the case that previously made the + "silent" test pass vacuously. + """ + fn = _make_function(address_rva=0x200000) + exp = _make_exp( + header=_make_header(num_functions=1), + functions=[fn], ) + issues = validate_exports({"export_struct": exp}, {}) assert ReasonCodes.EXPORT_FUNCTION_RVA_INVALID not in _codes(issues) @@ -553,7 +599,7 @@ def test_valid_forwarder_no_issue(self): header=_make_header(num_functions=1), functions=[fn], ) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_FORWARDER_MALFORMED not in _codes(issues) def test_unreadable_forwarder_flagged(self): @@ -567,10 +613,10 @@ def test_unreadable_forwarder_flagged(self): header=_make_header(num_functions=1), functions=[fn], ) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_FORWARDER_MALFORMED) assert len(details) == 1 - assert details[0]["reason"] == "unreadable" + assert details[0]["sub_reason"] == "unreadable" def test_malformed_forwarder_format_flagged(self): fn = _make_function( @@ -583,10 +629,10 @@ def test_malformed_forwarder_format_flagged(self): header=_make_header(num_functions=1), functions=[fn], ) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) details = _details_for(issues, ReasonCodes.EXPORT_FORWARDER_MALFORMED) assert len(details) == 1 - assert details[0]["reason"] == "format" + assert details[0]["sub_reason"] == "format" assert details[0]["forwarder"] == "NoDotInThisString" def test_forwarder_skips_function_rva_check(self): @@ -602,7 +648,7 @@ def test_forwarder_skips_function_rva_check(self): header=_make_header(num_functions=1), functions=[fn], ) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) assert ReasonCodes.EXPORT_FUNCTION_RVA_INVALID not in _codes(issues) @@ -622,8 +668,8 @@ def test_multiple_pathology_classes_emit_independently(self): _make_name_pointer(errors=["name_rva_zero"]), ], ) - analysis = _make_analysis(size_of_image=0x100000) - issues = validate_exports({"export_struct": exp}, analysis) + metadata = _make_metadata(size_of_image=0x100000) + issues = validate_exports({"export_struct": exp}, metadata) codes = set(_codes(issues)) assert ReasonCodes.EXPORT_DIRECTORY_OUT_OF_BOUNDS in codes assert ReasonCodes.EXPORT_TABLE_TRUNCATED in codes @@ -636,23 +682,67 @@ def test_multiple_pathology_classes_emit_independently(self): class TestOutputContract: + def test_dependency_contract(self): + assert getattr(validate_exports, "_depends_on") == ("internal", "metadata") + def test_returns_list(self): - result = validate_exports({"export_struct": _make_exp()}, _make_analysis()) + result = validate_exports({"export_struct": _make_exp()}, _make_metadata()) assert isinstance(result, list) def test_clean_exports_return_empty_list(self): - result = validate_exports({"export_struct": _make_exp()}, _make_analysis()) + result = validate_exports({"export_struct": _make_exp()}, _make_metadata()) assert result == [] def test_each_issue_has_issue_and_details(self): np = _make_name_pointer(errors=["name_rva_zero"]) exp = _make_exp(name_pointers=[np]) - issues = validate_exports({"export_struct": exp}, _make_analysis()) + issues = validate_exports({"export_struct": exp}, _make_metadata()) for issue in issues: assert "issue" in issue assert "details" in issue assert isinstance(issue["details"], dict) + def test_no_details_payload_uses_reserved_reason_key(self): + """ + "reason" is reserved by the heuristics emission layer: _det builds + metadata as {"reason": parent, **details}, so a details["reason"] would + overwrite the parent reason code. Validators must use "sub_reason". + + Exercises every non-short-circuit emission path at once: placement, + truncation, all four header-consistency branches, both name-pointer + pathology classes, ordinal-index, ordinal range, forwarder and + function-RVA. + """ + exp = _make_exp( + rva=0xFFF00, size=0x200, + header=_make_header(base=0xFFF0, num_functions=32, num_names=40, + addr_functions=0, addr_names=0, + addr_name_ordinals=0), + truncations=["eat_truncated"], + name_pointers=[_make_name_pointer( + errors=["name_rva_zero", "non_ascii", "ordinal_index_missing"], + name=None, name_valid=False)], + functions=[ + _make_function(index=0, address_rva=0x200000), + _make_function(index=1, address_rva=0x1050, is_forwarder=True, + forwarder=None, forwarder_valid=False), + ], + ) + issues = validate_exports({"export_struct": exp}, + _make_metadata(size_of_image=0x100000)) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + + def test_top_level_decode_avoids_reserved_reason_key(self): + """The short-circuit path is unreachable above; pin it separately.""" + exp = _make_exp(errors=["header_read_failed"]) + issues = validate_exports({"export_struct": exp}, _make_metadata()) + assert issues + assert "reason" not in issues[0]["details"] + # ================================================================= # Determinism @@ -676,10 +766,10 @@ def test_repeated_validation_produces_identical_issues(self): name_pointers=[np], functions=[fn], ) - metadata = {"export_struct": exp} - analysis = _make_analysis() + internal = {"export_struct": exp} + metadata = _make_metadata() - results = [validate_exports(metadata, analysis) for _ in range(20)] + results = [validate_exports(internal, metadata) for _ in range(20)] for r in results[1:]: assert r == results[0] @@ -689,11 +779,11 @@ def test_priority_resolution_deterministic(self): ) exp = _make_exp(name_pointers=[np]) results = [ - validate_exports({"export_struct": exp}, _make_analysis()) + validate_exports({"export_struct": exp}, _make_metadata()) for _ in range(20) ] for r in results[1:]: assert r == results[0] # Confirm priority winner details = _details_for(results[0], ReasonCodes.EXPORT_NAME_RVA_INVALID) - assert details[0]["reason"] == "name_rva_missing" + assert details[0]["sub_reason"] == "name_rva_missing" diff --git a/tests/unit/validators/test_validator_load_config_dir.py b/tests/unit/validators/test_validator_load_config_dir.py index b498488..475941b 100644 --- a/tests/unit/validators/test_validator_load_config_dir.py +++ b/tests/unit/validators/test_validator_load_config_dir.py @@ -1,6 +1,25 @@ # Copyright (c) 2026 MalX Labs and contributors # SPDX-License-Identifier: MPL-2.0 +""" +Unit tests for iocx.validators.load_config_directory. + +Layer note: the validator is @depends_on("internal", "metadata", "analysis") +and takes THREE positional arguments. SizeOfImage is read from +metadata["optional_header"]["size_of_image"]. + +Details note: sub-reasons are carried in a "sub_reason" key. The key "reason" +is reserved by the heuristics emission layer, which merges details over its own +reason field - a details["reason"] would overwrite the parent reason code. + +AMBIGUITY CAUTION: this validator shares sub-reason strings across two parent +codes - "unmapped" is emitted by BOTH LOAD_CONFIG_COOKIE_INVALID and +LOAD_CONFIG_SEH_INVALID. Asserting on the sub-reason alone is therefore not +sufficient to identify which check fired; always pair it with the parent code +(as the `any(... and ...)` assertions below do). Under the reason collision the +parent was overwritten and these two were indistinguishable in output. +""" + import pytest from iocx.validators.load_config_directory import validate_load_config_directory, _map_rva_to_raw from iocx.reason_codes import ReasonCodes @@ -8,7 +27,7 @@ def _base(): """Return minimal valid metadata structures.""" - internal = {"optional_header_magic": 0x10B} # PE32 + internal = {"optional_header_magic": 0x10B} # PE32 metadata = {"optional_header": {"size_of_image": 0x2000}} analysis = { "data_directories": [ @@ -20,7 +39,7 @@ def _base(): "virtual_address": 0x1000, "virtual_size": 0x1000, "raw_address": 0x200, - "characteristics": 0x80000000, # writable + "characteristics": 0x80000000, # writable } ], "overlay_offset": None, @@ -29,6 +48,15 @@ def _base(): return internal, metadata, analysis +def _has(issues, code, sub_reason=None): + """True if an issue with `code` (and optionally `sub_reason`) was emitted.""" + return any( + i["issue"] == code + and (sub_reason is None or i["details"].get("sub_reason") == sub_reason) + for i in issues + ) + + # --------------------------------------------------------- # 1. Directory missing → no issues # --------------------------------------------------------- @@ -44,7 +72,7 @@ def test_no_directory(): # --------------------------------------------------------- def test_load_config_too_small(): internal, metadata, analysis = _base() - analysis["data_directories"][0]["size"] = 0x20 # < 0x48 + analysis["data_directories"][0]["size"] = 0x20 # < 0x48 issues = validate_load_config_directory(internal, metadata, analysis) assert any(i["issue"] == ReasonCodes.LOAD_CONFIG_TOO_SMALL for i in issues) @@ -80,9 +108,11 @@ def test_guard_cf_inconsistent(): # --------------------------------------------------------- def test_cookie_unmapped(): internal, metadata, analysis = _base() - analysis["load_config"]["security_cookie_rva"] = 0x5000 # outside section + analysis["load_config"]["security_cookie_rva"] = 0x5000 # outside section issues = validate_load_config_directory(internal, metadata, analysis) - assert any(i["issue"] == ReasonCodes.LOAD_CONFIG_COOKIE_INVALID for i in issues) + # Pair code + sub_reason: "unmapped" is also emitted by SEH_INVALID, so the + # code alone would not distinguish which check fired. + assert _has(issues, ReasonCodes.LOAD_CONFIG_COOKIE_INVALID, "unmapped") # --------------------------------------------------------- @@ -90,14 +120,11 @@ def test_cookie_unmapped(): # --------------------------------------------------------- def test_cookie_non_writable(): internal, metadata, analysis = _base() - analysis["sections"][0]["characteristics"] = 0 # not writable + analysis["sections"][0]["characteristics"] = 0 # not writable analysis["load_config"]["security_cookie_rva"] = 0x1000 issues = validate_load_config_directory(internal, metadata, analysis) - assert any( - i["issue"] == ReasonCodes.LOAD_CONFIG_COOKIE_INVALID - and i["details"]["reason"] == "non_writable_section" - for i in issues - ) + assert _has(issues, ReasonCodes.LOAD_CONFIG_COOKIE_INVALID, + "non_writable_section") # --------------------------------------------------------- @@ -105,7 +132,7 @@ def test_cookie_non_writable(): # --------------------------------------------------------- def test_cookie_in_overlay(): internal, metadata, analysis = _base() - analysis["overlay_offset"] = 0x300 # raw offset threshold + analysis["overlay_offset"] = 0x300 # raw offset threshold analysis["load_config"]["security_cookie_rva"] = 0x1100 issues = validate_load_config_directory(internal, metadata, analysis) assert any(i["issue"] == ReasonCodes.LOAD_CONFIG_COOKIE_IN_OVERLAY for i in issues) @@ -118,7 +145,8 @@ def test_seh_missing_table_rva(): internal, metadata, analysis = _base() analysis["load_config"].update({"seh_count": 3, "seh_table_rva": None}) issues = validate_load_config_directory(internal, metadata, analysis) - assert any(i["issue"] == ReasonCodes.LOAD_CONFIG_SEH_INVALID for i in issues) + # SEH_INVALID has four distinct sub-reasons; pin which one fired. + assert _has(issues, ReasonCodes.LOAD_CONFIG_SEH_INVALID, "missing_table_rva") # --------------------------------------------------------- @@ -128,11 +156,7 @@ def test_seh_out_of_range(): internal, metadata, analysis = _base() analysis["load_config"].update({"seh_count": 1000, "seh_table_rva": 0x1800}) issues = validate_load_config_directory(internal, metadata, analysis) - assert any( - i["issue"] == ReasonCodes.LOAD_CONFIG_SEH_INVALID - and i["details"]["reason"] == "out_of_range" - for i in issues - ) + assert _has(issues, ReasonCodes.LOAD_CONFIG_SEH_INVALID, "out_of_range") # --------------------------------------------------------- @@ -142,11 +166,7 @@ def test_seh_unmapped(): internal, metadata, analysis = _base() analysis["load_config"].update({"seh_count": 1, "seh_table_rva": 0x800}) issues = validate_load_config_directory(internal, metadata, analysis) - assert any( - i["issue"] == ReasonCodes.LOAD_CONFIG_SEH_INVALID - and i["details"]["reason"] == "unmapped" - for i in issues - ) + assert _has(issues, ReasonCodes.LOAD_CONFIG_SEH_INVALID, "unmapped") # --------------------------------------------------------- @@ -157,11 +177,37 @@ def test_seh_in_overlay(): analysis["overlay_offset"] = 0x300 analysis["load_config"].update({"seh_count": 1, "seh_table_rva": 0x1100}) issues = validate_load_config_directory(internal, metadata, analysis) - assert any( - i["issue"] == ReasonCodes.LOAD_CONFIG_SEH_INVALID - and i["details"]["reason"] == "in_overlay" - for i in issues - ) + assert _has(issues, ReasonCodes.LOAD_CONFIG_SEH_INVALID, "in_overlay") + + +# --------------------------------------------------------- +# 11b. The shared "unmapped" string is disambiguated by its parent +# --------------------------------------------------------- +def test_shared_unmapped_substring_is_disambiguated_by_parent(): + """ + "unmapped" is emitted by BOTH LOAD_CONFIG_COOKIE_INVALID and + LOAD_CONFIG_SEH_INVALID. The parent code is what separates them. + + This is precisely the ambiguity the reason collision created: with the + parent overwritten, a consumer saw the bare string "unmapped" and could + not tell a security-cookie fault from an SEH-table fault. + """ + # cookie unmapped, SEH clean + internal, metadata, analysis = _base() + analysis["load_config"].update({"security_cookie_rva": 0x5000, "seh_count": 0}) + cookie_issues = validate_load_config_directory(internal, metadata, analysis) + + # SEH unmapped, cookie clean + internal, metadata, analysis = _base() + analysis["load_config"].update({"security_cookie_rva": 0x1000, + "seh_count": 1, "seh_table_rva": 0x800}) + seh_issues = validate_load_config_directory(internal, metadata, analysis) + + assert _has(cookie_issues, ReasonCodes.LOAD_CONFIG_COOKIE_INVALID, "unmapped") + assert not _has(cookie_issues, ReasonCodes.LOAD_CONFIG_SEH_INVALID, "unmapped") + + assert _has(seh_issues, ReasonCodes.LOAD_CONFIG_SEH_INVALID, "unmapped") + assert not _has(seh_issues, ReasonCodes.LOAD_CONFIG_COOKIE_INVALID, "unmapped") # --------------------------------------------------------- @@ -228,7 +274,7 @@ def test_validate_load_config_no_directory(): internal = {"optional_header_magic": 0x10B} metadata = {"optional_header": {"size_of_image": 0x2000}} analysis = { - "data_directories": [], # no load config entry + "data_directories": [], # no load config entry "sections": [], "overlay_offset": None, "load_config": {}, @@ -236,7 +282,7 @@ def test_validate_load_config_no_directory(): issues = validate_load_config_directory(internal, metadata, analysis) - assert issues == [] # early return + assert issues == [] # early return def test_validate_load_config_invalid_rva_or_size(): @@ -253,4 +299,68 @@ def test_validate_load_config_invalid_rva_or_size(): issues = validate_load_config_directory(internal, metadata, analysis) - assert issues == [] # early return + assert issues == [] # early return + + +# --------------------------------------------------------- +# 14. Output contract +# --------------------------------------------------------- +def test_dependency_contract(): + assert getattr(validate_load_config_directory, "_depends_on") == ( + "internal", "metadata", "analysis") + + +def test_no_details_payload_uses_reserved_reason_key(): + """ + "reason" is reserved by the heuristics emission layer: _det builds metadata + as {"reason": parent, **details}, so a details["reason"] would overwrite + the parent reason code. Validators must use "sub_reason". + + Exercises the too-small, truncated, guard-CF, cookie and SEH paths together. + """ + internal, metadata, analysis = _base() + analysis["data_directories"][0]["size"] = 0x20 # too small + analysis["sections"][0]["characteristics"] = 0 # cookie non-writable + analysis["load_config"].update({ + "parsed_size": 0x10, # truncated + "guard_cf_check_function_pointer": 0x1234, # guard CF inconsistent + "guard_cf_function_count": 0, + "security_cookie_rva": 0x1000, + "seh_count": 1000, "seh_table_rva": 0x1800, # SEH out of range + }) + issues = validate_load_config_directory(internal, metadata, analysis) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + + +def test_unmapped_paths_avoid_reserved_reason_key(): + """The two 'unmapped' branches are mutually exclusive with the above.""" + internal, metadata, analysis = _base() + analysis["load_config"].update({"security_cookie_rva": 0x5000, + "seh_count": 1, "seh_table_rva": 0x800}) + issues = validate_load_config_directory(internal, metadata, analysis) + assert issues + assert all("reason" not in i["details"] for i in issues) + + +# --------------------------------------------------------- +# 15. Determinism +# --------------------------------------------------------- +def test_repeated_calls_identical(): + import json + internal, metadata, analysis = _base() + analysis["sections"][0]["characteristics"] = 0 + analysis["overlay_offset"] = 0x300 + analysis["load_config"].update({ + "parsed_size": 0x10, + "guard_cf_check_function_pointer": 0x1234, + "guard_cf_function_count": 0, + "security_cookie_rva": 0x1100, + "seh_count": 1, "seh_table_rva": 0x1100, + }) + a = validate_load_config_directory(internal, metadata, analysis) + b = validate_load_config_directory(internal, metadata, analysis) + assert json.dumps(a, sort_keys=True) == json.dumps(b, sort_keys=True) diff --git a/tests/unit/validators/test_validator_optional_header.py b/tests/unit/validators/test_validator_optional_header.py index 5361ffc..443f1db 100644 --- a/tests/unit/validators/test_validator_optional_header.py +++ b/tests/unit/validators/test_validator_optional_header.py @@ -9,6 +9,14 @@ def make_issue_list(result): return [i["issue"] for i in result] +def _has(issues, code, sub_reason=None): + """True if an issue with `code` (and optionally `sub_reason`) was emitted.""" + return any( + i["issue"] == code + and (sub_reason is None or i["details"].get("sub_reason") == sub_reason) + for i in issues + ) + # --------------------------------------------------------- # 1) SizeOfImage < max section end @@ -37,16 +45,20 @@ def test_optional_header_inconsistent_size_of_image(): def test_optional_header_invalid_size_of_headers_alignment(): metadata = { "optional_header": { - "size_of_headers": 300, - "file_alignment": 256, + # 768 is not a multiple of 512 -> alignment branch. + # fa=512 is a valid alignment, so no FILE_ALIGNMENT noise + # (the old fa=256 was below the 512 minimum and also fired). + "size_of_headers": 768, + "file_alignment": 512, } } analysis = {"sections": []} - internal = { - "data_directories_raw": [] - } + internal = {"data_directories_raw": []} issues = validate_optional_header(internal, metadata, analysis) - assert ReasonCodes.OPTIONAL_HEADER_INVALID_SIZE_OF_HEADERS in make_issue_list(issues) + assert issues == [issues[0]] # single anomaly + assert issues[0]["issue"] == ReasonCodes.OPTIONAL_HEADER_INVALID_SIZE_OF_HEADERS + assert "file_alignment" in issues[0]["details"] # alignment branch + assert "required_minimum" not in issues[0]["details"] # not the header_end branch # --------------------------------------------------------- @@ -56,17 +68,19 @@ def test_optional_header_invalid_size_of_headers_alignment(): def test_optional_header_invalid_size_of_headers_header_end(): metadata = { "optional_header": { - "size_of_headers": 200, - "file_alignment": 200, + # 1024 IS a multiple of 512, so the alignment branch stays quiet; + # only the header_end branch fires. + "size_of_headers": 1024, + "file_alignment": 512, }, - "header_end": 300, + "header_end": 2048, } analysis = {"sections": []} - internal = { - "data_directories_raw": [] - } + internal = {"data_directories_raw": []} issues = validate_optional_header(internal, metadata, analysis) - assert ReasonCodes.OPTIONAL_HEADER_INVALID_SIZE_OF_HEADERS in make_issue_list(issues) + assert len(issues) == 1 + assert issues[0]["issue"] == ReasonCodes.OPTIONAL_HEADER_INVALID_SIZE_OF_HEADERS + assert issues[0]["details"]["required_minimum"] == 2048 # header_end branch # --------------------------------------------------------- @@ -85,7 +99,10 @@ def test_optional_header_invalid_section_alignment_less_than_file_alignment(): "data_directories_raw": [] } issues = validate_optional_header(internal, metadata, analysis) - assert ReasonCodes.OPTIONAL_HEADER_INVALID_SECTION_ALIGNMENT in make_issue_list(issues) + assert _has(issues, ReasonCodes.OPTIONAL_HEADER_INVALID_SECTION_ALIGNMENT) + # the sa= fa (so the sa Dict[str, Any]: + """ + Public-metadata layer. SizeOfImage lives under optional_header; it is NOT + part of the analysis layer. + """ + return {"optional_header": {"size_of_image": size_of_image}} + + def _make_analysis( - size_of_image: Optional[int] = 0x100000, sections: Optional[List[Dict[str, Any]]] = None, ) -> Dict[str, Any]: - analysis: Dict[str, Any] = {"size_of_image": size_of_image} + """ + Analysis layer. Carries section geometry only. + + Deliberately does NOT accept a size_of_image argument: the previous fixture + injected one here, which made the SizeOfImage fallback path appear to work + in tests while it was dead in production (analysis never carries that key). + Use _make_metadata for SizeOfImage. + """ + analysis: Dict[str, Any] = {} if sections is not None: analysis["sections"] = sections return analysis @@ -86,8 +113,12 @@ def _make_reloc( "truncations": truncations or [], "errors": errors or []} -def _run(reloc: Optional[Dict[str, Any]], analysis: Dict[str, Any]): - return validate_relocations({"relocation_struct": reloc}, analysis) +def _run(reloc: Optional[Dict[str, Any]], + analysis: Dict[str, Any], + metadata: Optional[Dict[str, Any]] = None): + if metadata is None: + metadata = _make_metadata() + return validate_relocations({"relocation_struct": reloc}, metadata, analysis) def _codes(issues) -> List: @@ -107,7 +138,7 @@ def test_none_struct_no_issues(self): assert _run(None, _make_analysis()) == [] def test_missing_key_no_issues(self): - assert validate_relocations({}, _make_analysis()) == [] + assert validate_relocations({}, _make_metadata(), _make_analysis()) == [] # ================================================================= @@ -120,7 +151,7 @@ def test_errors_emit_invalid_header(self): issues = _run(reloc, _make_analysis()) assert _codes(issues) == [ReasonCodes.RELOCATION_DIRECTORY_INVALID_HEADER] details = _details_for(issues, ReasonCodes.RELOCATION_DIRECTORY_INVALID_HEADER)[0] - assert details["reason"] == "top_level_decode" + assert details["sub_reason"] == "top_level_decode" assert details["errors"] == ["block_header_unpack_failed_at_0"] def test_short_circuit_skips_blocks_and_truncations(self): @@ -152,6 +183,8 @@ def test_region_detail_preserved_in_order(self): reloc = _make_reloc(truncations=["relocation_entries_truncated", "relocation_block_read_failed"]) issues = _run(reloc, _make_analysis()) + # "region" is a distinct key and was never subject to the reason + # collision, so it is unchanged by the sub_reason migration. regions = [d["region"] for d in _details_for(issues, ReasonCodes.RELOCATION_TABLE_TRUNCATED)] assert regions == ["relocation_entries_truncated", @@ -169,7 +202,7 @@ def test_size_too_small_flagged(self): issues = _run(reloc, _make_analysis()) assert ReasonCodes.RELOCATION_BLOCK_MALFORMED in _codes(issues) d = _details_for(issues, ReasonCodes.RELOCATION_BLOCK_MALFORMED)[0] - assert d["reason"] == "size_of_block_too_small" + assert d["sub_reason"] == "size_of_block_too_small" assert d["index"] == 0 def test_priority_first_match_wins(self): @@ -179,7 +212,7 @@ def test_priority_first_match_wins(self): issues = _run(reloc, _make_analysis()) malformed = _details_for(issues, ReasonCodes.RELOCATION_BLOCK_MALFORMED) assert len(malformed) == 1 - assert malformed[0]["reason"] == "size_of_block_too_small" + assert malformed[0]["sub_reason"] == "size_of_block_too_small" def test_clean_block_no_issue(self): reloc = _make_reloc(blocks=[_make_block( @@ -228,10 +261,48 @@ def test_count_reported_and_capped(self): ReasonCodes.RELOCATION_ENTRY_RVA_INVALID)) def test_no_sections_falls_back_to_size_of_image(self): - # No sections -> region_within_image bound check against size_of_image + """ + No sections -> region_within_image bound check against SizeOfImage. + + SizeOfImage must come from the METADATA layer. The previous version of + this test put it in `analysis`, which the helpers used to read - so the + fallback passed here while being dead in production, where `analysis` + never carries that key. + """ reloc = _make_reloc(blocks=[_make_block(entries=[ _make_entry(rva=0x200000)])]) # beyond size_of_image - issues = _run(reloc, _make_analysis(size_of_image=0x100000)) + issues = _run(reloc, _make_analysis(), + metadata=_make_metadata(size_of_image=0x100000)) + assert _codes(issues) == [ReasonCodes.RELOCATION_ENTRY_RVA_INVALID] + + def test_no_sections_in_bounds_not_flagged(self): + """Counterpart: the fallback must not false-positive on a valid RVA.""" + reloc = _make_reloc(blocks=[_make_block(entries=[ + _make_entry(rva=0x2010)])]) + issues = _run(reloc, _make_analysis(), + metadata=_make_metadata(size_of_image=0x100000)) + assert ReasonCodes.RELOCATION_ENTRY_RVA_INVALID not in _codes(issues) + + def test_no_sections_and_no_size_of_image_skips_check(self): + """ + With neither section geometry nor SizeOfImage the check is unknowable + and must be skipped rather than guessed. Pins the absent-optional-header + case explicitly so it is asserted on purpose. + """ + reloc = _make_reloc(blocks=[_make_block(entries=[ + _make_entry(rva=0x200000)])]) + issues = _run(reloc, _make_analysis(), metadata={}) + assert ReasonCodes.RELOCATION_ENTRY_RVA_INVALID not in _codes(issues) + + def test_sections_take_precedence_over_size_of_image(self): + """ + When section geometry is present it is authoritative: a target inside + SizeOfImage but outside every section is still flagged. + """ + reloc = _make_reloc(blocks=[_make_block(entries=[ + _make_entry(rva=0x9000)])]) + issues = _run(reloc, _make_analysis(sections=_tiny_section()), + metadata=_make_metadata(size_of_image=0x100000)) assert _codes(issues) == [ReasonCodes.RELOCATION_ENTRY_RVA_INVALID] @@ -245,8 +316,8 @@ def test_out_of_bounds_directory_emits_no_placement_issue(self): # placement finding from this validator (rva_graph owns that). reloc = _make_reloc(rva=0x90000, size=0x2000, blocks=[_make_block(entries=[_make_entry(rva=0x1004)])]) - issues = _run(reloc, _make_analysis( - size_of_image=0x10000, sections=_whole_image_sections())) + issues = _run(reloc, _make_analysis(sections=_whole_image_sections()), + metadata=_make_metadata(size_of_image=0x10000)) assert issues == [] @@ -281,7 +352,8 @@ def test_malformed_block_plus_invalid_entries(self): class TestOutputContract: def test_dependency_contract(self): - assert getattr(validate_relocations, "_depends_on") == ("internal", "analysis") + assert getattr(validate_relocations, "_depends_on") == ( + "internal", "metadata", "analysis") def test_issue_shape(self): reloc = _make_reloc(blocks=[_make_block(entries=[ @@ -301,6 +373,32 @@ def test_json_serializable(self): issues = _run(reloc, _make_analysis()) json.dumps([i for i in issues]) # must not raise + def test_no_details_payload_uses_reserved_reason_key(self): + """ + "reason" is reserved by the heuristics emission layer: _det builds + metadata as {"reason": parent, **details}, so a details["reason"] would + overwrite the parent reason code. Validators must use "sub_reason". + + Exercises every non-short-circuit emission path at once. + """ + reloc = _make_reloc( + truncations=["relocation_entries_truncated"], + blocks=[_make_block(errors=["size_of_block_too_small"], + entries=[_make_entry(rva=0x9000)])]) + issues = _run(reloc, _make_analysis(sections=_tiny_section())) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + + def test_top_level_decode_avoids_reserved_reason_key(self): + """The short-circuit path is unreachable above; pin it separately.""" + reloc = _make_reloc(errors=["block_header_unpack_failed_at_0"]) + issues = _run(reloc, _make_analysis()) + assert issues + assert "reason" not in issues[0]["details"] + # ================================================================= # Determinism diff --git a/tests/unit/validators/test_validator_resources.py b/tests/unit/validators/test_validator_resources.py index f00ae11..19ec6d1 100644 --- a/tests/unit/validators/test_validator_resources.py +++ b/tests/unit/validators/test_validator_resources.py @@ -1,462 +1,739 @@ # Copyright (c) 2026 MalX Labs and contributors # SPDX-License-Identifier: MPL-2.0 +""" +Unit tests for iocx.validators.resources.validate_resources. + +Layer note: @depends_on("internal", "analysis"), TWO positional arguments. +The resource tree comes from internal["resources_struct"]; sections, file_size +and overlay_offset from analysis. Note the validator reads those three with +DIRECT SUBSCRIPTS (`analysis["sections"]`), so a missing key raises KeyError +rather than degrading - pinned below. + +Fixture note: a data leaf is only well-formed at depth 2 (the Language layer). +Any leaf hung directly off the root is at depth 0 and therefore ALSO trips +RESOURCE_DATA_AT_INVALID_DEPTH. Tests targeting a data-entry check must either +build a full Type -> Name -> Language tree or assert the full issue list, or +they silently exercise two codes while claiming one. + +The `_tree` helper below builds a correctly-shaped three-level tree so that +data-entry checks can be isolated. +""" + +from __future__ import annotations + +from typing import Any, Dict, List, Optional + import pytest + from iocx.validators.resources import validate_resources from iocx.reason_codes import ReasonCodes -def make_issue_list(result): +# .rsrc spans VA 0x1000..0x3000, raw 0x400..0x2400 +RSRC_VA = 0x1000 +RSRC_VS = 0x2000 +RSRC_RAW = 0x400 +RSRC_RAW_SIZE = 0x2000 +FILE_SIZE = 0x10000 +OVERLAY = 0xF000 + + +# ================================================================= +# Builders +# ================================================================= + +def _rsrc_section(va: Any = RSRC_VA, vs: Any = RSRC_VS, + raw: Any = RSRC_RAW, raw_size: Any = RSRC_RAW_SIZE) -> Dict[str, Any]: + return {"name": ".rsrc", "virtual_address": va, "virtual_size": vs, + "raw_address": raw, "raw_size": raw_size} + + +def _section(name: str, va: int, vs: int, raw: int, raw_size: int) -> Dict[str, Any]: + return {"name": name, "virtual_address": va, "virtual_size": vs, + "raw_address": raw, "raw_size": raw_size} + + +def _analysis(sections: Optional[List[Dict[str, Any]]] = None, + file_size: Any = FILE_SIZE, + overlay_offset: Any = OVERLAY) -> Dict[str, Any]: + return {"sections": sections if sections is not None else [_rsrc_section()], + "file_size": file_size, "overlay_offset": overlay_offset} + + +def _leaf(data_rva: Any = 0x1100, data_size: Any = 0x50, + raw_offset: Any = 0x500, name: Any = None, + entry_id: Any = 0x409) -> Dict[str, Any]: + """A data-entry (leaf) child.""" + return {"name": name, "id": entry_id, "is_directory": False, + "directory": None, "data_rva": data_rva, "data_size": data_size, + "raw_offset": raw_offset} + + +def _node(rva: int, size: int = 24, + entries: Optional[List[Dict[str, Any]]] = None) -> Dict[str, Any]: + """A directory NODE (the thing `directory` points at).""" + return {"rva": rva, "size": size, "entries": entries or []} + + +def _subdir(target: Dict[str, Any], name: Any = None, + entry_id: Any = 1) -> Dict[str, Any]: + """A directory-entry child pointing at `target`.""" + return {"name": name, "id": entry_id, "is_directory": True, + "directory": target, "data_rva": None, "data_size": None, + "raw_offset": None} + + +def _tree(leaves: List[Dict[str, Any]], + lang_rva: int = 0x1080) -> Dict[str, Any]: + """ + A correctly-shaped Type -> Name -> Language tree whose Language directory + (depth 2) holds `leaves`. Lets data-entry checks be tested without also + tripping RESOURCE_DATA_AT_INVALID_DEPTH. + """ + lang = _node(lang_rva, 24, leaves) + name = _node(0x1040, 24, [_subdir(lang)]) + return _node(0x1000, 24, [_subdir(name)]) + + +def _run(root: Dict[str, Any], + analysis: Optional[Dict[str, Any]] = None, + string_tables: Optional[List[Dict[str, Any]]] = None, + resources: Any = "DEFAULT") -> List[Dict[str, Any]]: + if resources == "DEFAULT": + resources = {"root": root} + if string_tables is not None: + resources["string_tables"] = string_tables + return validate_resources({"resources_struct": resources}, + analysis or _analysis()) + + +def make_issue_list(result) -> List[str]: return [i["issue"] for i in result] -def test_resources_no_resources_struct(): - metadata = {"resources_struct": None} - analysis = {} - issues = validate_resources(metadata, analysis) - assert issues == [] - - -def test_resources_no_rsrc_section(): - metadata = {"resources_struct": {"root": {}}} - analysis = { - "sections": [{"name": ".text"}], - "file_size": 1000, - "overlay_offset": 500, - } - issues = validate_resources(metadata, analysis) - assert issues == [] - - -def test_resources_zero_length_directory(): - metadata = { - "resources_struct": { - "root": {"rva": 100, "size": 0, "entries": []} - } - } - analysis = { - "sections": [{ - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 100, - "raw_address": 200, - "raw_size": 100, - }], - "file_size": 1000, - "overlay_offset": 500, - } - issues = validate_resources(metadata, analysis) - assert ReasonCodes.RESOURCE_DIRECTORY_ZERO_LENGTH in make_issue_list(issues) - - -def test_resources_directory_loop(): - loop = {"rva": 100, "size": 10, "entries": []} - loop["entries"] = [{"is_directory": True, "directory": loop}] - - metadata = {"resources_struct": {"root": loop}} - analysis = { - "sections": [{ - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 200, - "raw_address": 200, - "raw_size": 200, - }], - "file_size": 1000, - "overlay_offset": 500, - } - - issues = validate_resources(metadata, analysis) - assert ReasonCodes.RESOURCE_DIRECTORY_LOOP in make_issue_list(issues) - - -def test_resources_entry_out_of_bounds(): - metadata = { - "resources_struct": { - "root": { - "rva": 100, "size": 10, - "entries": [ - {"is_directory": True, - "directory": {"rva": 9999, "size": 10, "entries": []}} - ] - } - } - } - analysis = { - "sections": [{ - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 200, - "raw_address": 200, - "raw_size": 200, - }], - "file_size": 1000, - "overlay_offset": 500, - } - - issues = validate_resources(metadata, analysis) - assert ReasonCodes.RESOURCE_ENTRY_OUT_OF_BOUNDS in make_issue_list(issues) - - -def test_resources_zero_size_data(): - metadata = { - "resources_struct": { - "root": { - "rva": 100, "size": 10, - "entries": [ - {"is_directory": False, - "data_rva": 120, "data_size": 0, "raw_offset": 300} - ] - } - } - } - analysis = { - "sections": [{ - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 200, - "raw_address": 200, - "raw_size": 200, - }], - "file_size": 1000, - "overlay_offset": 500, - } - - issues = validate_resources(metadata, analysis) - assert ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS in make_issue_list(issues) - - -def test_resources_rva_out_of_bounds(): - metadata = { - "resources_struct": { - "root": { - "rva": 100, "size": 10, - "entries": [ - {"is_directory": False, - "data_rva": 9999, "data_size": 10, "raw_offset": 300} - ] - } - } - } - analysis = { - "sections": [{ - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 200, - "raw_address": 200, - "raw_size": 200, - }], - "file_size": 1000, - "overlay_offset": 500, - } - - issues = validate_resources(metadata, analysis) - assert ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS in make_issue_list(issues) - - -def test_resources_raw_out_of_bounds(): - metadata = { - "resources_struct": { - "root": { - "rva": 100, "size": 10, - "entries": [ - {"is_directory": False, - "data_rva": 120, "data_size": 50, "raw_offset": 980} - ] - } - } - } - analysis = { - "sections": [{ - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 200, - "raw_address": 200, - "raw_size": 200, - }], - "file_size": 1000, - "overlay_offset": 500, - } - - issues = validate_resources(metadata, analysis) - assert ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS in make_issue_list(issues) - - -def test_resources_overlay_overlap(): - metadata = { - "resources_struct": { - "root": { - "rva": 100, "size": 10, - "entries": [ - {"is_directory": False, - "data_rva": 120, "data_size": 100, "raw_offset": 450} - ] - } - } - } - analysis = { - "sections": [{ - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 300, - "raw_address": 200, - "raw_size": 300, - }], - "file_size": 1000, - "overlay_offset": 500, - } - - issues = validate_resources(metadata, analysis) - assert ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA in make_issue_list(issues) - - -def test_resources_raw_overlap_other_section(): - metadata = { - "resources_struct": { - "root": { - "rva": 100, "size": 10, - "entries": [ - {"is_directory": False, - "data_rva": 120, "data_size": 50, "raw_offset": 250} - ] - } - } - } - analysis = { - "sections": [ - { - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 300, - "raw_address": 200, - "raw_size": 300, - }, - { - "name": ".text", - "virtual_address": 1000, - "virtual_size": 100, - "raw_address": 240, - "raw_size": 20, - } - ], - "file_size": 1000, - "overlay_offset": 900, - } - - issues = validate_resources(metadata, analysis) - assert ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA in make_issue_list(issues) - - -def test_resources_va_overlap_other_section(): - metadata = { - "resources_struct": { - "root": { - "rva": 100, "size": 10, - "entries": [ - {"is_directory": False, - "data_rva": 150, "data_size": 50, "raw_offset": 250} - ] - } - } - } - analysis = { - "sections": [ - { - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 300, - "raw_address": 200, - "raw_size": 300, - }, - { - "name": ".text", - "virtual_address": 140, - "virtual_size": 100, - "raw_address": 500, - "raw_size": 100, - } - ], - "file_size": 1000, - "overlay_offset": 900, - } - - issues = validate_resources(metadata, analysis) - assert ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA in make_issue_list(issues) - - -def test_resources_string_table_corrupt(): - metadata = { - "resources_struct": { - "root": {"rva": 100, "size": 10, "entries": []}, - "string_tables": [ - {"rva": 9999, "size": 20} - ] - } - } - analysis = { - "sections": [{ - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 300, - "raw_address": 200, - "raw_size": 300, - }], - "file_size": 1000, - "overlay_offset": 500, - } - - issues = validate_resources(metadata, analysis) - assert ReasonCodes.RESOURCE_STRING_TABLE_CORRUPT in make_issue_list(issues) - - -def test_resources_directory_outside_rsrc_skips_validation(): - metadata = { - "resources_struct": { - "root": { - "rva": 9999, # OUTSIDE .rsrc VA range - "size": 10, - "entries": [] - } - } - } - - analysis = { - "sections": [ - { - "name": ".rsrc", - "virtual_address": 100, - "virtual_size": 200, # .rsrc covers VA 100–300 - "raw_address": 200, - "raw_size": 200, - } - ], - "file_size": 5000, - "overlay_offset": 4000, - } - - issues = validate_resources(metadata, analysis) - - # Because the directory is outside .rsrc, validate_directory() returns immediately - # → no issues should be produced - assert issues == [] - - -class TestLanguageLayerNamedEntry: +def _details_for(issues, code) -> List[Dict[str, Any]]: + return [i["details"] for i in issues if i["issue"] == code] + + +# ================================================================= +# Absence / early return +# ================================================================= + +class TestAbsence: + + def test_none_resources_struct(self): + assert validate_resources({"resources_struct": None}, {}) == [] + + def test_missing_resources_key(self): + assert validate_resources({}, {}) == [] + + @pytest.mark.parametrize("falsy", [{}, [], 0, False, ""]) + def test_any_falsy_resources_struct_returns_early(self, falsy): + """ + The guard is `if not resources`, so an EMPTY dict short-circuits too - + it never reaches the analysis subscripts. + """ + assert validate_resources({"resources_struct": falsy}, {}) == [] + + def test_no_rsrc_section_returns_early(self): + analysis = _analysis(sections=[_section(".text", 0x1000, 0x1000, + 0x400, 0x1000)]) + assert _run(_node(0x1000, 24, [_leaf()]), analysis) == [] + + def test_rsrc_match_is_case_insensitive(self): + analysis = _analysis(sections=[ + {"name": ".RSRC", "virtual_address": RSRC_VA, + "virtual_size": RSRC_VS, "raw_address": RSRC_RAW, + "raw_size": RSRC_RAW_SIZE}]) + issues = _run(_node(0x1000, 0, []), analysis) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DIRECTORY_ZERO_LENGTH] + + def test_first_rsrc_section_wins(self): + """`next(...)` takes the first match if a file declares two.""" + analysis = _analysis(sections=[ + _rsrc_section(va=0x1000, vs=0x2000), + _rsrc_section(va=0x9000, vs=0x1000)]) + # root at 0x1000 is inside the FIRST .rsrc only + assert _run(_node(0x1000, 0, []), analysis) != [] + + +class TestAnalysisContract: """ - Cover the depth-2 named-entry detection: a Language-layer directory - that contains a named entry instead of an integer LCID. + The validator uses direct subscripts for sections / file_size / + overlay_offset. These pin the current (strict) contract so a change to + `.get()` is a conscious one. """ - def _make_resources_with_named_language_entry(self) -> dict: + @pytest.mark.parametrize("missing", ["sections", "file_size", + "overlay_offset"]) + def test_missing_analysis_key_raises(self, missing): + analysis = _analysis() + del analysis[missing] + with pytest.raises(KeyError): + _run(_node(0x1000, 24, [_leaf()]), analysis) + + def test_missing_analysis_key_tolerated_when_no_resources(self): + """The early return happens before any analysis access.""" + assert validate_resources({"resources_struct": None}, {}) == [] + + +# ================================================================= +# Directory-level checks +# ================================================================= + +class TestDirectoryChecks: + + def test_directory_outside_rsrc_flagged(self): """ - Build a minimal resources_struct dict with this shape: - Type (depth 0) - └─ Name (depth 1) - └─ Language (depth 2) ← contains a named entry (anomaly) - └─ data leaf + A root directory outside .rsrc is reported rather than silently + skipped. depth 0 identifies it as the root case. + """ + issues = _run(_node(0x9999, 24, [_leaf()])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DIRECTORY_OUT_OF_BOUNDS] + d = _details_for(issues, + ReasonCodes.RESOURCE_DIRECTORY_OUT_OF_BOUNDS)[0] + assert d["rva"] == 0x9999 + assert d["depth"] == 0 + assert d["rsrc_start"] == RSRC_VA + assert d["rsrc_end"] == RSRC_VA + RSRC_VS + + def test_directory_partially_outside_rsrc_flagged(self): + """The check is rva_in_rsrc(rva, size) - the whole node must fit.""" + issues = _run(_node(RSRC_VA + RSRC_VS - 8, 24, [_leaf()])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DIRECTORY_OUT_OF_BOUNDS] + + def test_directory_exactly_filling_rsrc_is_validated(self): + """Boundary: rva + size == rsrc_end is inclusive.""" + issues = _run(_node(RSRC_VA + RSRC_VS - 24, 24, [_leaf(0x9999, 8)])) + assert issues != [] + + def test_zero_length_directory_flagged(self): + issues = _run(_node(0x1000, 0, [])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DIRECTORY_ZERO_LENGTH] + assert _details_for( + issues, ReasonCodes.RESOURCE_DIRECTORY_ZERO_LENGTH)[0] == {"rva": 0x1000} + + def test_zero_length_directory_stops_descent(self): + """Entries are not walked once the zero-length check fires.""" + issues = _run(_node(0x1000, 0, [_leaf(0x9999, 0x50)])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DIRECTORY_ZERO_LENGTH] + + def test_self_referential_loop_flagged(self): + loop = _node(0x1000, 24, []) + loop["entries"] = [_subdir(loop)] + issues = _run(loop) + assert ReasonCodes.RESOURCE_DIRECTORY_LOOP in make_issue_list(issues) + + def test_mutual_loop_flagged(self): + a = _node(0x1000, 24, []) + b = _node(0x1100, 24, []) + a["entries"] = [_subdir(b)] + b["entries"] = [_subdir(a)] + issues = _run(a) + assert ReasonCodes.RESOURCE_DIRECTORY_LOOP in make_issue_list(issues) + + def test_shared_subtree_reported_as_loop(self): + """ + `visited_dirs` is global rather than per-path, so a DAG - the same + directory legitimately reached by two branches - is indistinguishable + from a true cycle and is reported as a loop. + + Pinned as current behaviour: if shared subtrees are ever considered + valid, this is the test that must change. """ - # Data leaf at depth 3 — well-formed, in-bounds - data_leaf_entry = { - "name": None, - "id": 0x0409, - "is_directory": False, - "directory": None, - "data_rva": 0x1100, - "data_size": 100, - "raw_offset": 0x500, - } - - # Depth 2: Language directory containing a NAMED entry (the anomaly) - # plus a valid ID entry as a sibling, to prove the validator emits - # per-violation rather than per-directory. - lang_dir = { - "rva": 0x1080, - "size": 32, - "entries": [ - { - "name": "EN-US", # ← anomaly: named instead of LCID - "id": None, - "is_directory": True, - "directory": { - "rva": 0x1090, - "size": 24, - "entries": [data_leaf_entry], - }, - "data_rva": None, - "data_size": None, - "raw_offset": None, - }, - ], - } - - # Depth 1: Name directory - name_dir = { - "rva": 0x1040, - "size": 24, - "entries": [ - { - "name": None, - "id": 1, - "is_directory": True, - "directory": lang_dir, - "data_rva": None, - "data_size": None, - "raw_offset": None, - }, - ], - } - - # Depth 0: Type directory (root) - root = { - "rva": 0x1000, - "size": 24, - "entries": [ - { - "name": None, - "id": 16, # RT_VERSION - "is_directory": True, - "directory": name_dir, - "data_rva": None, - "data_size": None, - "raw_offset": None, - }, - ], - } - - return { - "root": root, - "string_tables": [], - } - - def _make_analysis(self) -> dict: - return { - "sections": [{ - "name": ".rsrc", - "virtual_address": 0x1000, - "virtual_size": 0x2000, - "raw_address": 0x400, - "raw_size": 0x2000, - }], - "file_size": 0x10000, - "overlay_offset": 0x9000, - } - - def test_named_language_entry_flagged(self): - from iocx.reason_codes import ReasonCodes - from iocx.validators.resources import validate_resources - - resources = self._make_resources_with_named_language_entry() - metadata = {"resources_struct": resources} - issues = validate_resources(metadata, self._make_analysis()) - - codes = [i["issue"] for i in issues] - assert ReasonCodes.RESOURCE_DIRECTORY_LANGUAGE_NOT_ID in codes - - # Confirm the details payload is shaped correctly - details = [ - i["details"] for i in issues - if i["issue"] == ReasonCodes.RESOURCE_DIRECTORY_LANGUAGE_NOT_ID + shared = _node(0x1200, 24, [_leaf()]) + root = _node(0x1000, 24, [ + _subdir(_node(0x1100, 24, [_subdir(shared)])), + _subdir(_node(0x1180, 24, [_subdir(shared)])), + ]) + assert ReasonCodes.RESOURCE_DIRECTORY_LOOP in make_issue_list(_run(root)) + + def test_loop_detection_keys_on_rva_not_identity(self): + """Two distinct nodes sharing an RVA collide in visited_dirs.""" + root = _node(0x1000, 24, [ + _subdir(_node(0x1100, 24, [])), + _subdir(_node(0x1100, 24, [])), # different object, same RVA + ]) + assert ReasonCodes.RESOURCE_DIRECTORY_LOOP in make_issue_list(_run(root)) + + def test_sibling_directories_at_distinct_rvas_are_fine(self): + root = _node(0x1000, 24, [ + _subdir(_node(0x1100, 24, [])), + _subdir(_node(0x1180, 24, [])), + ]) + assert _run(root) == [] + + + +# ================================================================= +# Language layer (depth 2) +# ================================================================= + +class TestLanguageLayer: + + def _lang_tree(self, lang_entries: List[Dict[str, Any]]) -> Dict[str, Any]: + lang = _node(0x1080, 24, lang_entries) + name = _node(0x1040, 24, [_subdir(lang)]) + return _node(0x1000, 24, [_subdir(name)]) + + def test_named_entry_at_language_layer_flagged(self): + root = self._lang_tree([ + _subdir(_node(0x1090, 24, [_leaf()]), name="EN-US", entry_id=None)]) + issues = _run(root) + assert ReasonCodes.RESOURCE_DIRECTORY_LANGUAGE_NOT_ID in make_issue_list(issues) + d = _details_for(issues, + ReasonCodes.RESOURCE_DIRECTORY_LANGUAGE_NOT_ID)[0] + assert d == {"rva": 0x1080, "name": "EN-US"} + + def test_integer_lcid_at_language_layer_is_fine(self): + root = self._lang_tree([_leaf(entry_id=0x0409)]) + assert ReasonCodes.RESOURCE_DIRECTORY_LANGUAGE_NOT_ID not in \ + make_issue_list(_run(root)) + + def test_each_named_entry_flagged_separately(self): + root = self._lang_tree([ + _subdir(_node(0x1090, 24, []), name="EN-US", entry_id=None), + _subdir(_node(0x10A0, 24, []), name="FR-FR", entry_id=None), + ]) + names = [d["name"] for d in _details_for( + _run(root), ReasonCodes.RESOURCE_DIRECTORY_LANGUAGE_NOT_ID)] + assert names == ["EN-US", "FR-FR"] + + def test_entry_with_both_name_and_id_not_flagged(self): + """The condition requires name set AND id None.""" + root = self._lang_tree([ + _subdir(_node(0x1090, 24, []), name="EN-US", entry_id=0x409)]) + assert ReasonCodes.RESOURCE_DIRECTORY_LANGUAGE_NOT_ID not in \ + make_issue_list(_run(root)) + + def test_named_entries_at_type_and_name_layers_are_legal(self): + """Only depth 2 requires integer IDs; named Types/Names are normal.""" + lang = _node(0x1080, 24, [_leaf()]) + name = _node(0x1040, 24, [_subdir(lang, name="MYNAME", entry_id=None)]) + root = _node(0x1000, 24, [_subdir(name, name="MYTYPE", entry_id=None)]) + assert ReasonCodes.RESOURCE_DIRECTORY_LANGUAGE_NOT_ID not in \ + make_issue_list(_run(root)) + + +# ================================================================= +# Data leaf depth +# ================================================================= + +class TestDataDepth: + + @pytest.mark.parametrize("depth", [0, 1]) + def test_leaf_above_language_layer_flagged(self, depth): + leaf = _leaf(0x1100, 0x50, 0x500) + node = _node(0x1080, 24, [leaf]) + for _ in range(depth): + node = _node(0x1000 + 0x40 * depth, 24, [_subdir(node)]) + root = node if depth else _node(0x1000, 24, [leaf]) + issues = _run(root) + assert ReasonCodes.RESOURCE_DATA_AT_INVALID_DEPTH in make_issue_list(issues) + + def test_leaf_at_language_layer_not_flagged(self): + issues = _run(_tree([_leaf(0x1100, 0x50, 0x500)])) + assert ReasonCodes.RESOURCE_DATA_AT_INVALID_DEPTH not in make_issue_list(issues) + assert issues == [] + + def test_leaf_below_language_layer_also_flagged(self): + """ + The check is `depth != 2`, so a leaf that is too DEEP is flagged as + well as one that is too shallow. + """ + deep = _node(0x1300, 24, [_leaf(0x1400, 0x50, 0x500)]) + root = _node(0x1000, 24, [_subdir( + _node(0x1100, 24, [_subdir(_node(0x1200, 24, [_subdir(deep)]))]))]) + issues = _run(root) + assert ReasonCodes.RESOURCE_DATA_AT_INVALID_DEPTH in make_issue_list(issues) + assert _details_for( + issues, ReasonCodes.RESOURCE_DATA_AT_INVALID_DEPTH)[0]["depth"] == 3 + + def test_invalid_depth_does_not_stop_data_validation(self): + """ + The depth check falls through: a misplaced leaf is STILL bounds-checked, + so both codes fire. + """ + issues = _run(_node(0x1000, 24, [_leaf(0x9999, 0x50, 0x500)])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DATA_AT_INVALID_DEPTH, + ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS] + + def test_details_payload(self): + issues = _run(_node(0x1000, 24, [_leaf(0x1100, 0x50, 0x500)])) + assert _details_for( + issues, ReasonCodes.RESOURCE_DATA_AT_INVALID_DEPTH)[0] == { + "rva": 0x1000, "depth": 0, "data_rva": 0x1100} + + +# ================================================================= +# Entry targets +# ================================================================= + +class TestEntryTargets: + + def test_subdirectory_outside_rsrc_flagged(self): + root = _node(0x1000, 24, [_subdir(_node(0x9999, 24, []))]) + issues = _run(root) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_ENTRY_OUT_OF_BOUNDS] + assert _details_for( + issues, ReasonCodes.RESOURCE_ENTRY_OUT_OF_BOUNDS)[0] == { + "directory_rva": 0x1000, "target_rva": 0x9999} + + def test_out_of_bounds_target_does_not_stop_siblings(self): + root = _node(0x1000, 24, [ + _subdir(_node(0x9999, 24, [])), + _subdir(_node(0x8888, 24, [])), + ]) + assert make_issue_list(_run(root)) == [ + ReasonCodes.RESOURCE_ENTRY_OUT_OF_BOUNDS, + ReasonCodes.RESOURCE_ENTRY_OUT_OF_BOUNDS] + + def test_overflowing_target_caught_by_the_child_guard(self): + """ + The entry check calls rva_in_rsrc(target_rva) with NO size, so a target + starting inside .rsrc but overflowing it passes there. It is caught by + the child's own guard instead - previously it vanished silently. + + depth 1 distinguishes this from the root case. + """ + target = _node(RSRC_VA + RSRC_VS - 8, 24, [_leaf(0x9999, 0x50)]) + issues = _run(_node(0x1000, 24, [_subdir(target)])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DIRECTORY_OUT_OF_BOUNDS] + assert _details_for( + issues, ReasonCodes.RESOURCE_DIRECTORY_OUT_OF_BOUNDS)[0]["depth"] == 1 + + def test_wholly_outside_target_still_reports_entry_code_only(self): + """ + Control: a child wholly outside is caught by the CALLER, whose + `continue` prevents descent - so the two codes never double-count. + """ + issues = _run(_node(0x1000, 24, [_subdir(_node(0x9999, 24, []))])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_ENTRY_OUT_OF_BOUNDS] + + +# ================================================================= +# Data-entry bounds +# ================================================================= + +class TestDataBounds: + """All fixtures use a correctly-shaped tree so DATA_AT_INVALID_DEPTH + does not confound the assertions.""" + + def test_wellformed_leaf_is_silent(self): + assert _run(_tree([_leaf(0x1100, 0x50, 0x500)])) == [] + + def test_zero_size_data_flagged(self): + issues = _run(_tree([_leaf(0x1100, 0, 0x500)])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS] + assert _details_for( + issues, ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS)[0] == { + "data_rva": 0x1100, "data_size": 0} + + def test_data_rva_outside_rsrc_flagged(self): + issues = _run(_tree([_leaf(0x9999, 0x50, 0x500)])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS] + + def test_data_overflowing_rsrc_end_flagged(self): + issues = _run(_tree([_leaf(RSRC_VA + RSRC_VS - 8, 0x50, 0x500)])) + assert ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS in make_issue_list(issues) + + def test_data_exactly_at_rsrc_end_is_fine(self): + """Boundary: data_rva + data_size == rsrc_end is inclusive.""" + assert _run(_tree([_leaf(RSRC_VA + RSRC_VS - 0x50, 0x50, 0x500)])) == [] + + def test_negative_raw_offset_flagged(self): + """The -1 sentinel from a failed RVA->offset lookup lands here.""" + issues = _run(_tree([_leaf(0x1100, 0x50, -1)])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS] + assert _details_for( + issues, ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS)[0]["data_raw"] == -1 + + def test_raw_past_file_size_flagged(self): + issues = _run(_tree([_leaf(0x1100, 0x50, FILE_SIZE - 8)])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS] + + def test_raw_exactly_at_file_size_is_fine(self): + """Boundary: data_raw + data_size == file_size is inclusive.""" + analysis = _analysis(overlay_offset=FILE_SIZE) + assert _run(_tree([_leaf(0x1100, 0x50, FILE_SIZE - 0x50)]), + analysis) == [] + + def test_bounds_checks_short_circuit_in_order(self): + """ + Zero size wins over an out-of-range RVA: only ONE code per leaf. + """ + issues = _run(_tree([_leaf(0x9999, 0, -1)])) + assert len(issues) == 1 + assert _details_for( + issues, ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS)[0] == { + "data_rva": 0x9999, "data_size": 0} + + def test_each_bad_leaf_reported(self): + issues = _run(_tree([_leaf(0x1100, 0, 0x500), + _leaf(0x9999, 0x50, 0x500), + _leaf(0x1200, 0x50, 0x500)])) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS, + ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS] + + +# ================================================================= +# Overlap detection +# ================================================================= + +class TestOverlapDetection: + + def test_data_spanning_overlay_start_flagged(self): + analysis = _analysis(overlay_offset=0x520) + issues = _run(_tree([_leaf(0x1100, 0x50, 0x500)]), analysis) + assert ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA in make_issue_list(issues) + assert _details_for( + issues, + ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA)[0]["overlay_offset"] == 0x520 + + def test_overlay_exactly_at_data_start_flagged(self): + """The check is `data_raw <= overlay < data_raw + size` - inclusive.""" + analysis = _analysis(overlay_offset=0x500) + assert ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA in make_issue_list( + _run(_tree([_leaf(0x1100, 0x50, 0x500)]), analysis)) + + def test_overlay_exactly_at_data_end_not_flagged(self): + """...and exclusive at the upper bound.""" + analysis = _analysis(overlay_offset=0x550) + assert _run(_tree([_leaf(0x1100, 0x50, 0x500)]), analysis) == [] + + def test_raw_overlap_with_other_section_flagged(self): + analysis = _analysis(sections=[ + _rsrc_section(), _section(".text", 0x8000, 0x100, 0x510, 0x20)]) + issues = _run(_tree([_leaf(0x1100, 0x50, 0x500)]), analysis) + d = _details_for(issues, ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA) + assert any(x.get("section") == ".text" and "data_raw" in x for x in d) + + def test_va_overlap_with_other_section_flagged(self): + analysis = _analysis(sections=[ + _rsrc_section(), _section(".text", 0x1120, 0x100, 0x9000, 0x100)]) + issues = _run(_tree([_leaf(0x1100, 0x50, 0x500)]), analysis) + d = _details_for(issues, ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA) + assert any(x.get("section") == ".text" and "data_rva" in x for x in d) + + def test_rsrc_section_excluded_from_both_overlap_checks(self): + """Resource data lives IN .rsrc; overlapping itself is not an anomaly.""" + assert _run(_tree([_leaf(0x1100, 0x50, 0x500)])) == [] + + def test_raw_overlap_breaks_after_first_section(self): + """One issue per leaf, not one per overlapping section.""" + analysis = _analysis(sections=[ + _rsrc_section(), + _section(".a", 0x8000, 0x100, 0x510, 0x20), + _section(".b", 0x8200, 0x100, 0x520, 0x20)]) + issues = _run(_tree([_leaf(0x1100, 0x50, 0x500)]), analysis) + raw_hits = [d for d in _details_for( + issues, ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA) + if "data_raw" in d] + assert len(raw_hits) == 1 + assert raw_hits[0]["section"] == ".a" + + def test_raw_and_va_overlaps_both_reported(self): + """They are independent loops, so a leaf can trip both.""" + analysis = _analysis(sections=[ + _rsrc_section(), _section(".text", 0x1120, 0x100, 0x510, 0x20)]) + issues = _run(_tree([_leaf(0x1100, 0x50, 0x500)]), analysis) + assert make_issue_list(issues).count( + ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA) == 2 + + def test_overlay_and_section_overlap_both_reported(self): + analysis = _analysis( + sections=[_rsrc_section(), + _section(".text", 0x8000, 0x100, 0x510, 0x20)], + overlay_offset=0x520) + assert make_issue_list(_run(_tree([_leaf(0x1100, 0x50, 0x500)]), + analysis)).count( + ReasonCodes.RESOURCE_DATA_OVERLAPS_OTHER_DATA) == 2 + + def test_adjacent_raw_ranges_do_not_overlap(self): + """Half-open: touching ranges are not an overlap.""" + analysis = _analysis(sections=[ + _rsrc_section(), _section(".text", 0x8000, 0x100, 0x550, 0x20)]) + assert _run(_tree([_leaf(0x1100, 0x50, 0x500)]), analysis) == [] + + def test_out_of_bounds_data_never_reaches_overlap_checks(self): + """The bounds `continue` short-circuits before overlap detection.""" + analysis = _analysis( + sections=[_rsrc_section(), + _section(".text", 0x1120, 0x100, 0x510, 0x20)], + overlay_offset=0x500) + issues = _run(_tree([_leaf(0x1100, 0, 0x500)]), analysis) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS] + + +# ================================================================= +# String tables +# ================================================================= + +class TestStringTables: + + def test_valid_string_table_is_silent(self): + assert _run(_node(0x1000, 24, []), + string_tables=[{"rva": 0x1100, "size": 0x20}]) == [] + + def test_out_of_bounds_string_table_flagged(self): + issues = _run(_node(0x1000, 24, []), + string_tables=[{"rva": 0x9999, "size": 0x20}]) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_STRING_TABLE_CORRUPT] + assert _details_for( + issues, ReasonCodes.RESOURCE_STRING_TABLE_CORRUPT)[0] == { + "rva": 0x9999, "size": 0x20} + + def test_string_table_overflowing_rsrc_end_flagged(self): + issues = _run(_node(0x1000, 24, []), + string_tables=[{"rva": RSRC_VA + RSRC_VS - 8, + "size": 0x20}]) + assert ReasonCodes.RESOURCE_STRING_TABLE_CORRUPT in make_issue_list(issues) + + def test_only_the_first_corrupt_table_is_reported(self): + """The loop `break`s, so a second corrupt table is not reported.""" + issues = _run(_node(0x1000, 24, []), + string_tables=[{"rva": 0x9999, "size": 0x20}, + {"rva": 0x8888, "size": 0x20}]) + assert len(issues) == 1 + assert _details_for( + issues, ReasonCodes.RESOURCE_STRING_TABLE_CORRUPT)[0]["rva"] == 0x9999 + + def test_valid_table_before_corrupt_one_still_reports(self): + issues = _run(_node(0x1000, 24, []), + string_tables=[{"rva": 0x1100, "size": 0x20}, + {"rva": 0x9999, "size": 0x20}]) + assert _details_for( + issues, ReasonCodes.RESOURCE_STRING_TABLE_CORRUPT)[0]["rva"] == 0x9999 + + def test_missing_string_tables_key_tolerated(self): + """`resources.get("string_tables", [])` - unlike the analysis keys.""" + assert _run(_node(0x1000, 24, [])) == [] + + def test_empty_string_tables_list(self): + assert _run(_node(0x1000, 24, []), string_tables=[]) == [] + + def test_string_tables_checked_even_when_tree_is_clean(self): + issues = _run(_tree([_leaf(0x1100, 0x50, 0x500)]), + string_tables=[{"rva": 0x9999, "size": 0x20}]) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_STRING_TABLE_CORRUPT] + + +# ================================================================= +# Output contract +# ================================================================= + +class TestOutputContract: + + def test_dependency_contract(self): + assert getattr(validate_resources, "_depends_on") == ("internal", + "analysis") + + def test_returns_list(self): + assert isinstance(_run(_tree([_leaf()])), list) + + def test_each_issue_has_issue_and_details(self): + issues = _run(_node(0x1000, 24, [_leaf(0x9999, 0x50)])) + assert issues + for issue in issues: + assert set(issue) == {"issue", "details"} + assert isinstance(issue["issue"], str) + assert isinstance(issue["details"], dict) + + def test_no_details_payload_uses_reserved_reason_key(self): + """ + "reason" is reserved by the heuristics emission layer: _det builds + metadata as {"reason": parent, **details}, so a details["reason"] + would overwrite the parent reason code. This validator uses no + sub-reasons; the guard pins that none is introduced. + """ + analysis = _analysis( + sections=[_rsrc_section(), + _section(".text", 0x1120, 0x100, 0x510, 0x20)], + overlay_offset=0x520) + root = _node(0x1000, 24, [ + _leaf(0x1100, 0x50, 0x500), + _subdir(_node(0x9999, 24, [])), + ]) + issues = _run(root, analysis, + string_tables=[{"rva": 0x9999, "size": 0x20}]) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + + def test_json_serialisable(self): + import json + json.dumps(_run(_node(0x1000, 24, [_leaf(0x9999, 0x50)]), + string_tables=[{"rva": 0x9999, "size": 0x20}])) + + def test_inputs_are_not_mutated(self): + import copy + resources = {"root": _tree([_leaf(0x1100, 0x50, 0x500)]), + "string_tables": [{"rva": 0x1100, "size": 0x20}]} + analysis = _analysis() + res_snapshot = copy.deepcopy(resources) + an_snapshot = copy.deepcopy(analysis) + validate_resources({"resources_struct": resources}, analysis) + assert resources == res_snapshot + assert analysis == an_snapshot + + +# ================================================================= +# Determinism +# ================================================================= + +class TestDeterminism: + + def test_repeated_validation_is_identical(self): + import json + analysis = _analysis( + sections=[_rsrc_section(), + _section(".text", 0x1120, 0x100, 0x510, 0x20)], + overlay_offset=0x520) + root = _node(0x1000, 24, [ + _leaf(0x1100, 0x50, 0x500), + _leaf(0x9999, 0x50, 0x500), + _subdir(_node(0x8888, 24, [])), + ]) + resources = {"root": root, "string_tables": [{"rva": 0x9999, "size": 8}]} + first = json.dumps( + validate_resources({"resources_struct": resources}, analysis), + sort_keys=True) + for _ in range(20): + assert json.dumps( + validate_resources({"resources_struct": resources}, analysis), + sort_keys=True) == first + + def test_emission_order_is_tree_then_string_tables(self): + issues = _run(_node(0x1000, 24, [_leaf(0x9999, 0x50)]), + string_tables=[{"rva": 0x8888, "size": 0x20}]) + assert make_issue_list(issues) == [ + ReasonCodes.RESOURCE_DATA_AT_INVALID_DEPTH, + ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS, + ReasonCodes.RESOURCE_STRING_TABLE_CORRUPT, ] - assert len(details) == 1 - assert details[0]["name"] == "EN-US" - assert details[0]["rva"] == 0x1080 # the language directory's RVA + + def test_entries_processed_in_order(self): + issues = _run(_tree([_leaf(0x9991, 0x50), _leaf(0x9992, 0x50)])) + rvas = [d["data_rva"] for d in _details_for( + issues, ReasonCodes.RESOURCE_DATA_OUT_OF_BOUNDS)] + assert rvas == [0x9991, 0x9992] diff --git a/tests/unit/validators/test_validator_rva_graph.py b/tests/unit/validators/test_validator_rva_graph.py index eacbad7..36136b3 100644 --- a/tests/unit/validators/test_validator_rva_graph.py +++ b/tests/unit/validators/test_validator_rva_graph.py @@ -1,315 +1,754 @@ # Copyright (c) 2026 MalX Labs and contributors # SPDX-License-Identifier: MPL-2.0 +""" +Unit tests for iocx.validators.rva_graph.validate_rva_graph. + +Layer note: @depends_on("metadata", "analysis"), TWO positional arguments. +SizeOfImage and SizeOfHeaders come from metadata["optional_header"]; sections +and overlay_offset from analysis. `data_directories` is read from analysis +FIRST, falling back to metadata - a subtlety pinned below. + +Fixture note: this validator's checks do NOT short-circuit each other. A +directory can be in-headers AND unmapped; an overlay hit can be preceded by a +raw-mismatch. Several fixtures here therefore assert the FULL issue list +rather than mere presence, because a presence-only assertion hides how many +codes a fixture really trips. + +Sections in these fixtures carry an explicit `raw_size` wherever overlay or +raw-mapping logic is exercised. Omitting it makes `sec.get("raw_size", 0)` +zero, which collapses the section's raw range to a point and trips +DATA_DIRECTORY_RAW_MISMATCH before the intended check - an easy fixture trap. +""" + +from __future__ import annotations + +from typing import Any, Dict, List + import pytest -from iocx.validators.rva_graph import validate_rva_graph + +from iocx.validators.rva_graph import ( + validate_rva_graph, + _is_security_directory, + _SECURITY_DIRECTORY_INDEX, + _SECURITY_DIRECTORY_NAME, +) from iocx.reason_codes import ReasonCodes -def make_issue_list(result): - return [i["issue"] for i in result] +# ================================================================= +# Helpers +# ================================================================= +def _dir(name: str = "dir", rva: Any = 0x1000, size: Any = 0x100, + **extra) -> Dict[str, Any]: + d = {"name": name, "rva": rva, "size": size} + d.update(extra) + return d -# --------------------------------------------------------- -# 1) size_of_image missing → early return -# --------------------------------------------------------- -def test_rva_graph_missing_size_of_image(): - metadata = {"optional_header": {}} - analysis = {} - issues = validate_rva_graph(metadata, analysis) - assert issues == [] +def _section(name: str = ".text", va: Any = 0x1000, vs: Any = 0x1000, + raw: Any = 0x400, raw_size: Any = 0x1000) -> Dict[str, Any]: + """A section with a raw range wide enough not to trip RAW_MISMATCH.""" + sec = {"name": name, "virtual_address": va, "virtual_size": vs} + if raw is not None: + sec["raw_address"] = raw + if raw_size is not None: + sec["raw_size"] = raw_size + return sec -# --------------------------------------------------------- -# 2) malformed directory entry → first continue -# --------------------------------------------------------- +def _meta(size_of_image: Any = 0x3000, size_of_headers: Any = None, + **extra) -> Dict[str, Any]: + opt: Dict[str, Any] = {"size_of_image": size_of_image} + if size_of_headers is not None: + opt["size_of_headers"] = size_of_headers + md: Dict[str, Any] = {"optional_header": opt} + md.update(extra) + return md -def test_rva_graph_malformed_directory_entry(): - metadata = {"optional_header": {"size_of_image": 1000}} - analysis = { - "data_directories": [ - {"rva": "bad", "size": 10}, # triggers continue - ] - } - issues = validate_rva_graph(metadata, analysis) - assert issues == [] +def _run(metadata: Dict[str, Any], **analysis) -> List[Dict[str, Any]]: + return validate_rva_graph(metadata, analysis) -# --------------------------------------------------------- -# 3) negative rva/size -# --------------------------------------------------------- -def test_rva_graph_negative_values(): - metadata = {"optional_header": {"size_of_image": 1000}} - analysis = { - "data_directories": [ - {"name": "dir", "rva": -1, "size": 10}, - ] - } - issues = validate_rva_graph(metadata, analysis) - assert ReasonCodes.DATA_DIRECTORY_INVALID_RANGE in make_issue_list(issues) +def make_issue_list(result) -> List[str]: + return [i["issue"] for i in result] -# --------------------------------------------------------- -# 4) empty directory (0,0) -# --------------------------------------------------------- +def _details_for(issues, code) -> List[Dict[str, Any]]: + return [i["details"] for i in issues if i["issue"] == code] -def test_rva_graph_empty_directory(): - metadata = {"optional_header": {"size_of_image": 1000}} - analysis = { - "data_directories": [ - {"name": "dir", "rva": 0, "size": 0}, - ] - } - issues = validate_rva_graph(metadata, analysis) - assert issues == [] +# ================================================================= +# Early return / input tolerance +# ================================================================= -def test_rva_graph_empty_directory_unexpected(monkeypatch): - # Patch REQUIRED_NONZERO_DIRS to force the branch - from iocx.validators import rva_graph - monkeypatch.setattr(rva_graph, "REQUIRED_NONZERO_DIRS", {"dir"}) +class TestEarlyReturn: - metadata = {"optional_header": {"size_of_image": 1000}} - analysis = { - "data_directories": [ - {"name": "dir", "rva": 0, "size": 0}, - ] - } + def test_missing_size_of_image_returns_empty(self): + assert _run({"optional_header": {}}, + data_directories=[_dir(rva=-1, size=-1)]) == [] - issues = rva_graph.validate_rva_graph(metadata, analysis) + def test_missing_optional_header_returns_empty(self): + assert _run({}, data_directories=[_dir(rva=-1, size=-1)]) == [] - assert ReasonCodes.DATA_DIRECTORY_ZERO_SIZE_UNEXPECTED in [ - i["issue"] for i in issues - ] + def test_none_optional_header_returns_empty(self): + assert _run({"optional_header": None}, + data_directories=[_dir(rva=-1, size=-1)]) == [] + @pytest.mark.parametrize("value", ["1000", None, 1000.5, []]) + def test_non_int_size_of_image_returns_empty(self, value): + """A float SizeOfImage is rejected, not coerced.""" + assert _run(_meta(size_of_image=value), + data_directories=[_dir(rva=-1, size=-1)]) == [] -# --------------------------------------------------------- -# 5) zero-RVA non-zero size -# --------------------------------------------------------- + def test_no_directories_returns_empty(self): + assert _run(_meta()) == [] -def test_rva_graph_zero_rva_nonzero_size(): - metadata = {"optional_header": {"size_of_image": 1000}} - analysis = { - "data_directories": [ - {"name": "dir", "rva": 0, "size": 50}, - ] - } - issues = validate_rva_graph(metadata, analysis) - assert ReasonCodes.DATA_DIRECTORY_ZERO_RVA_NONZERO_SIZE in make_issue_list(issues) + def test_none_sections_tolerated(self): + assert _run(_meta(), sections=None, + data_directories=[_dir(rva=0, size=0)]) == [] + @pytest.mark.parametrize("rva,size", [ + ("bad", 0x100), (0x1000, "bad"), (None, 0x100), (0x1000, None), + (0x1000, 1.5), (1.5, 0x100), + ]) + def test_non_int_directory_fields_skipped(self, rva, size): + assert _run(_meta(), data_directories=[_dir(rva=rva, size=size)]) == [] -# --------------------------------------------------------- -# 6) directory in headers -# --------------------------------------------------------- + def test_malformed_section_fields_skipped(self): + """A section with non-int VA/VS never enters section_ranges.""" + issues = _run(_meta(), + sections=[_section(va="bad"), _section(".ok", 0x1000)], + data_directories=[_dir(rva=0x1000, size=0x10)]) + assert issues == [] -def test_rva_graph_in_headers(): - metadata = {"optional_header": {"size_of_image": 1000, "size_of_headers": 300}} - analysis = { - "data_directories": [ - {"name": "dir", "rva": 100, "size": 50}, - ] - } - issues = validate_rva_graph(metadata, analysis) - assert ReasonCodes.DATA_DIRECTORY_IN_HEADERS in make_issue_list(issues) +# ================================================================= +# data_directories source resolution +# ================================================================= -# --------------------------------------------------------- -# 7) out-of-range directory -# --------------------------------------------------------- +class TestDirectorySource: + """`analysis.get(...) or metadata.get(...) or []` - order matters.""" -def test_rva_graph_out_of_range(): - metadata = {"optional_header": {"size_of_image": 200}} - analysis = { - "data_directories": [ - {"name": "dir", "rva": 150, "size": 100}, - ] - } - issues = validate_rva_graph(metadata, analysis) - assert ReasonCodes.DATA_DIRECTORY_OUT_OF_RANGE in make_issue_list(issues) - - -# --------------------------------------------------------- -# 8) overlay detection -# --------------------------------------------------------- - -def test_rva_graph_overlay_detection(): - metadata = {"optional_header": {"size_of_image": 2000}} - analysis = { - "overlay_offset": 300, - "sections": [ - { - "name": ".text", - "virtual_address": 100, - "virtual_size": 500, - "raw_address": 200, - } - ], - "data_directories": [ - {"name": "dir", "rva": 250, "size": 10}, - ], - } - issues = validate_rva_graph(metadata, analysis) - assert ReasonCodes.DATA_DIRECTORY_IN_OVERLAY in make_issue_list(issues) - - -# --------------------------------------------------------- -# 9) zero-length section skip -# --------------------------------------------------------- - -def test_rva_graph_zero_length_section_skip(): - metadata = {"optional_header": {"size_of_image": 2000}} - analysis = { - "sections": [ - { - "name": ".empty", - "virtual_address": 1000, - "virtual_size": 0, - "raw_address": 500, - } - ], - "data_directories": [ - {"name": "dir", "rva": 1000, "size": 10}, # lands exactly on zero-length section - ], - } - issues = validate_rva_graph(metadata, analysis) - assert issues == [] - - -# --------------------------------------------------------- -# 10) not mapped to any section -# --------------------------------------------------------- - -def test_rva_graph_not_mapped_to_section(): - metadata = {"optional_header": {"size_of_image": 2000}} - analysis = { - "sections": [ - { - "name": ".text", - "virtual_address": 100, - "virtual_size": 100, - } - ], - "data_directories": [ - {"name": "dir", "rva": 500, "size": 10}, # outside section - ], - } - issues = validate_rva_graph(metadata, analysis) - assert ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION in make_issue_list(issues) - - -# --------------------------------------------------------- -# 11) spans multiple sections -# --------------------------------------------------------- - -def test_rva_graph_spans_multiple_sections(): - metadata = {"optional_header": {"size_of_image": 2000}} - analysis = { - "sections": [ - {"name": "A", "virtual_address": 100, "virtual_size": 100}, - {"name": "B", "virtual_address": 150, "virtual_size": 100}, - ], - "data_directories": [ - {"name": "dir", "rva": 120, "size": 100}, # overlaps A and B - ], - } - issues = validate_rva_graph(metadata, analysis) - assert ReasonCodes.DATA_DIRECTORY_SPANS_MULTIPLE_SECTIONS in make_issue_list(issues) - - -# --------------------------------------------------------- -# 12) directory overlap detection -# --------------------------------------------------------- - -def test_rva_graph_directory_overlap(): - metadata = {"optional_header": {"size_of_image": 2000}} - analysis = { - "data_directories": [ - {"name": "A", "rva": 100, "size": 100}, - {"name": "B", "rva": 150, "size": 100}, # overlaps A - ] - } - issues = validate_rva_graph(metadata, analysis) - assert ReasonCodes.DATA_DIRECTORY_OVERLAP in make_issue_list(issues) - - -def test_rva_graph_directory_overlap_inner_continue(): - metadata = {"optional_header": {"size_of_image": 2000}} - analysis = { - "data_directories": [ - { - "name": "A", - "rva": 100, - "size": 50, # valid → outer loop does NOT continue - }, - { - "name": "B", - "rva": "bad", # invalid → triggers inner continue - "size": 50, - }, - ] - } + def test_analysis_is_preferred(self): + md = _meta(data_directories=[_dir("FROM_METADATA", 0x2000, 0x10)]) + issues = _run(md, data_directories=[_dir("FROM_ANALYSIS", 0x2000, 0x10)]) + assert _details_for( + issues, ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION + )[0]["directory"] == "FROM_ANALYSIS" + + def test_metadata_used_when_analysis_absent(self): + md = _meta(data_directories=[_dir("FROM_METADATA", 0x2000, 0x10)]) + issues = _run(md) + assert _details_for( + issues, ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION + )[0]["directory"] == "FROM_METADATA" + + def test_empty_analysis_list_falls_through_to_metadata(self): + """ + An empty list is falsy, so `or` continues to metadata. Subtle: an + analysis layer that legitimately reports "no directories" does not + suppress the metadata copy. + """ + md = _meta(data_directories=[_dir("FROM_METADATA", 0x2000, 0x10)]) + issues = _run(md, data_directories=[]) + assert _details_for( + issues, ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION + )[0]["directory"] == "FROM_METADATA" - issues = validate_rva_graph(metadata, analysis) + def test_neither_source_yields_empty(self): + assert _run(_meta()) == [] - # No overlap issue should be produced because the inner loop continues - assert ReasonCodes.DATA_DIRECTORY_OVERLAP not in make_issue_list(issues) +# ================================================================= +# Security directory exclusion +# ================================================================= -def test_rva_graph_raw_mapping_safety_breaks_on_missing_raw_address(): +class TestSecurityDirectoryExclusion: """ - Ensures overlay detection is skipped when a section has no raw data. + IMAGE_DIRECTORY_ENTRY_SECURITY carries a FILE OFFSET, not an RVA, so every + RVA-based check here would be a category error. It must be excluded from + BOTH the per-directory loop and the overlap loop. """ - metadata = { - "optional_header": { - "size_of_image": 0x3000, - "size_of_headers": 0x200, - }, - "data_directories": [ - { - "name": "IMAGE_DIRECTORY_ENTRY_IMPORT", - "rva": 0x1000, # inside .text VA range - "size": 0x100, - } - ], - } - - analysis = { - "overlay_offset": 0x180, # would normally trigger overlay - "sections": [ - { - "name": ".text", - "virtual_address": 0x1000, - "virtual_size": 0x1000, - # CRITICAL: raw_address missing → triggers break - # "raw_address": 0x200, - "raw_size": 0x200, - } + def test_helper_matches_by_index(self): + assert _is_security_directory({"index": _SECURITY_DIRECTORY_INDEX}) + + def test_helper_matches_by_name(self): + assert _is_security_directory({"name": _SECURITY_DIRECTORY_NAME}) + + def test_helper_rejects_other_directories(self): + assert not _is_security_directory({"index": 3, "name": "OTHER"}) + + def test_helper_tolerates_missing_keys(self): + assert not _is_security_directory({}) + + @pytest.mark.parametrize("key,value", [ + ("index", _SECURITY_DIRECTORY_INDEX), + ("name", _SECURITY_DIRECTORY_NAME), + ]) + def test_excluded_from_per_directory_checks(self, key, value): + """ + Negative rva/size would normally trip INVALID_RANGE; for security it + must be silent. + """ + d = {key: value, "rva": -1, "size": -1} + assert _run(_meta(), data_directories=[d]) == [] + + def test_non_security_directory_with_same_values_is_flagged(self): + """Control: proves the exclusion, not the values, causes the silence.""" + issues = _run(_meta(), data_directories=[_dir("OTHER", -1, -1)]) + assert make_issue_list(issues) == [ReasonCodes.DATA_DIRECTORY_INVALID_RANGE] + + def test_excluded_from_overlap_as_first_operand(self): + dirs = [{"index": _SECURITY_DIRECTORY_INDEX, "rva": 0x1000, "size": 0x100}, + _dir("B", 0x1050, 0x100)] + assert ReasonCodes.DATA_DIRECTORY_OVERLAP not in make_issue_list( + _run(_meta(), sections=[_section()], data_directories=dirs)) + + def test_excluded_from_overlap_as_second_operand(self): + dirs = [_dir("A", 0x1000, 0x100), + {"index": _SECURITY_DIRECTORY_INDEX, "rva": 0x1050, + "size": 0x100}] + assert ReasonCodes.DATA_DIRECTORY_OVERLAP not in make_issue_list( + _run(_meta(), sections=[_section()], data_directories=dirs)) + + def test_two_non_security_directories_do_overlap(self): + """Control for the two exclusion tests above.""" + dirs = [_dir("A", 0x1000, 0x100), _dir("B", 0x1050, 0x100)] + assert ReasonCodes.DATA_DIRECTORY_OVERLAP in make_issue_list( + _run(_meta(), sections=[_section()], data_directories=dirs)) + + +# ================================================================= +# Per-directory value checks +# ================================================================= + +class TestDirectoryValueChecks: + + @pytest.mark.parametrize("rva,size", [(-1, 0x10), (0x1000, -1), (-1, -1)]) + def test_negative_values_flagged(self, rva, size): + issues = _run(_meta(), data_directories=[_dir(rva=rva, size=size)]) + assert make_issue_list(issues) == [ReasonCodes.DATA_DIRECTORY_INVALID_RANGE] + + def test_negative_values_short_circuit_later_checks(self): + """`continue` means no mapping or overlay codes follow.""" + issues = _run(_meta(size_of_headers=0x400), + sections=[_section()], + data_directories=[_dir(rva=-1, size=-1)]) + assert len(issues) == 1 + + def test_empty_directory_is_silent(self): + assert _run(_meta(), data_directories=[_dir(rva=0, size=0)]) == [] + + def test_empty_directory_flagged_when_required(self, monkeypatch): + from iocx.validators import rva_graph + monkeypatch.setattr(rva_graph, "REQUIRED_NONZERO_DIRS", {"dir"}) + issues = rva_graph.validate_rva_graph( + _meta(), {"data_directories": [_dir(rva=0, size=0)]}) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_ZERO_SIZE_UNEXPECTED] + + def test_required_nonzero_dirs_is_empty_by_default(self): + """ + The default set is empty, so the ZERO_SIZE_UNEXPECTED branch is + unreachable in production. Pinned so a future addition is deliberate. + """ + from iocx.validators import rva_graph + assert rva_graph.REQUIRED_NONZERO_DIRS == set() + + def test_zero_rva_nonzero_size_flagged(self): + issues = _run(_meta(), data_directories=[_dir(rva=0, size=0x50)]) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_ZERO_RVA_NONZERO_SIZE] + + def test_zero_size_nonzero_rva_flagged(self): + """"Absent" and "present" simultaneously - a distinct malformation.""" + issues = _run(_meta(), data_directories=[_dir(rva=0x1000, size=0)]) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_ZERO_SIZE_NONZERO_RVA] + + def test_zero_rva_and_zero_size_variants_are_distinct(self): + """The three zero-combinations map to three different outcomes.""" + both_zero = _run(_meta(), data_directories=[_dir(rva=0, size=0)]) + rva_zero = _run(_meta(), data_directories=[_dir(rva=0, size=0x50)]) + size_zero = _run(_meta(), data_directories=[_dir(rva=0x1000, size=0)]) + assert both_zero == [] + assert make_issue_list(rva_zero) == [ + ReasonCodes.DATA_DIRECTORY_ZERO_RVA_NONZERO_SIZE] + assert make_issue_list(size_zero) == [ + ReasonCodes.DATA_DIRECTORY_ZERO_SIZE_NONZERO_RVA] + + def test_details_payloads(self): + issues = _run(_meta(), data_directories=[_dir("D", -5, 0x10)]) + assert _details_for(issues, ReasonCodes.DATA_DIRECTORY_INVALID_RANGE)[0] == { + "directory": "D", "rva": -5, "size": 0x10} + + def test_directory_named_by_index_when_name_absent(self): + """`d.get("name") or d.get("index")` - index is the fallback label.""" + issues = _run(_meta(), + data_directories=[{"index": 7, "rva": -1, "size": -1}]) + assert _details_for( + issues, ReasonCodes.DATA_DIRECTORY_INVALID_RANGE)[0]["directory"] == 7 + + +# ================================================================= +# Headers and range +# ================================================================= + +class TestHeadersAndRange: + + def test_directory_in_headers_flagged(self): + issues = _run(_meta(size_of_headers=0x400), + sections=[_section(va=0x100, vs=0x1000)], + data_directories=[_dir(rva=0x200, size=0x10)]) + assert ReasonCodes.DATA_DIRECTORY_IN_HEADERS in make_issue_list(issues) + + def test_rva_exactly_at_size_of_headers_not_flagged(self): + """The comparison is `<`, so the boundary itself is legal.""" + issues = _run(_meta(size_of_headers=0x400), + sections=[_section(va=0x400, vs=0x1000)], + data_directories=[_dir(rva=0x400, size=0x10)]) + assert ReasonCodes.DATA_DIRECTORY_IN_HEADERS not in make_issue_list(issues) + + def test_in_headers_does_not_short_circuit(self): + """ + Unlike the value checks, IN_HEADERS falls through - a directory can be + in the headers AND unmapped. + """ + issues = _run(_meta(size_of_headers=0x400), + sections=[_section(va=0x2000, vs=0x100)], + data_directories=[_dir(rva=0x100, size=0x10)]) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_IN_HEADERS, + ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION] + + def test_missing_size_of_headers_skips_the_check(self): + issues = _run(_meta(), + sections=[_section(va=0x100, vs=0x1000)], + data_directories=[_dir(rva=0x200, size=0x10)]) + assert ReasonCodes.DATA_DIRECTORY_IN_HEADERS not in make_issue_list(issues) + + def test_out_of_range_flagged(self): + issues = _run(_meta(size_of_image=0x200), + data_directories=[_dir(rva=0x150, size=0x100)]) + assert make_issue_list(issues) == [ReasonCodes.DATA_DIRECTORY_OUT_OF_RANGE] + + def test_end_exactly_at_size_of_image_not_flagged(self): + """`rva + size > size_of_image` - the boundary is inclusive.""" + issues = _run(_meta(size_of_image=0x2000), + sections=[_section(va=0x1000, vs=0x1000)], + data_directories=[_dir(rva=0x1F00, size=0x100)]) + assert ReasonCodes.DATA_DIRECTORY_OUT_OF_RANGE not in make_issue_list(issues) + + def test_out_of_range_short_circuits_mapping(self): + """`continue` after OUT_OF_RANGE suppresses overlay and mapping.""" + issues = _run(_meta(size_of_image=0x200), + sections=[_section(va=0x2000, vs=0x100)], + overlay_offset=0, + data_directories=[_dir(rva=0x150, size=0x100)]) + assert make_issue_list(issues) == [ReasonCodes.DATA_DIRECTORY_OUT_OF_RANGE] + + def test_in_headers_and_out_of_range_both_fire(self): + issues = _run(_meta(size_of_image=0x200, size_of_headers=0x400), + data_directories=[_dir(rva=0x150, size=0x100)]) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_IN_HEADERS, + ReasonCodes.DATA_DIRECTORY_OUT_OF_RANGE] + + +# ================================================================= +# Raw mapping and overlay +# ================================================================= + +class TestRawMappingAndOverlay: + + def test_overlay_hit_flagged(self): + issues = _run(_meta(), + overlay_offset=0x500, + sections=[_section(va=0x1000, vs=0x1000, + raw=0x400, raw_size=0x1000)], + data_directories=[_dir(rva=0x1200, size=0x10)]) + # raw_offset = 0x400 + 0x200 = 0x600 >= 0x500 + assert ReasonCodes.DATA_DIRECTORY_IN_OVERLAY in make_issue_list(issues) + assert _details_for( + issues, ReasonCodes.DATA_DIRECTORY_IN_OVERLAY)[0]["raw_offset"] == 0x600 + + def test_raw_offset_exactly_at_overlay_flagged(self): + """The comparison is `>=`.""" + issues = _run(_meta(), + overlay_offset=0x600, + sections=[_section(va=0x1000, vs=0x1000, + raw=0x400, raw_size=0x1000)], + data_directories=[_dir(rva=0x1200, size=0x10)]) + assert ReasonCodes.DATA_DIRECTORY_IN_OVERLAY in make_issue_list(issues) + + def test_raw_offset_below_overlay_not_flagged(self): + issues = _run(_meta(), + overlay_offset=0x601, + sections=[_section(va=0x1000, vs=0x1000, + raw=0x400, raw_size=0x1000)], + data_directories=[_dir(rva=0x1200, size=0x10)]) + assert issues == [] + + def test_missing_overlay_offset_skips_the_check(self): + issues = _run(_meta(), + sections=[_section(va=0x1000, vs=0x1000, + raw=0x400, raw_size=0x1000)], + data_directories=[_dir(rva=0x1200, size=0x10)]) + assert issues == [] + + def test_raw_mismatch_flagged(self): + """ + The RVA maps into the section's VIRTUAL range but the derived raw + offset falls outside its raw data - a virtual size larger than the + raw size. + """ + issues = _run(_meta(), + overlay_offset=0x9999, + sections=[_section(va=0x1000, vs=0x1000, + raw=0x400, raw_size=0x10)], + data_directories=[_dir(rva=0x1800, size=0x10)]) + assert ReasonCodes.DATA_DIRECTORY_RAW_MISMATCH in make_issue_list(issues) + d = _details_for(issues, ReasonCodes.DATA_DIRECTORY_RAW_MISMATCH)[0] + assert d["section_raw_start"] == 0x400 + assert d["section_raw_end"] == 0x410 + assert d["raw_offset"] == 0xC00 + + def test_missing_raw_size_defaults_to_zero_and_mismatches(self): + """ + FIXTURE TRAP: `sec.get("raw_size", 0)` means a section without + raw_size has an empty raw range, so ANY directory mapping into it + trips RAW_MISMATCH. Fixtures exercising overlay must set raw_size. + """ + issues = _run(_meta(), + overlay_offset=0x300, + sections=[_section(va=0x100, vs=0x500, + raw=0x200, raw_size=None)], + data_directories=[_dir(rva=0x250, size=0x10)]) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_RAW_MISMATCH, + ReasonCodes.DATA_DIRECTORY_IN_OVERLAY] + + def test_missing_raw_address_breaks_and_skips_overlay(self): + """ + A section with no raw_address cannot be mapped, so the loop breaks, + raw_offset stays None and `continue` skips BOTH the overlay check and + the later section-mapping checks. + """ + issues = _run(_meta(), + overlay_offset=0x100, + sections=[_section(va=0x1000, vs=0x1000, + raw=None, raw_size=0x200)], + data_directories=[_dir(rva=0x1000, size=0x100)]) + assert issues == [] + + def test_unmapped_rva_is_flagged_regardless_of_overlay_offset(self): + """ + A directory whose RVA maps to no section is reported as unmapped + whether or not an overlay is present. + + Previously the `raw_offset is None` guard used a bare `continue`, + which skipped the section-mapping checks entirely - so the presence of + an unrelated overlay_offset silently suppressed + NOT_MAPPED_TO_SECTION. The guard is now scoped to the overlay check. + """ + with_overlay = _run(_meta(), + overlay_offset=0x100, + sections=[_section(va=0x1000, vs=0x100)], + data_directories=[_dir(rva=0x2000, size=0x10)]) + without_overlay = _run(_meta(), + sections=[_section(va=0x1000, vs=0x100)], + data_directories=[_dir(rva=0x2000, size=0x10)]) + + assert make_issue_list(with_overlay) == [ + ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION] + assert make_issue_list(without_overlay) == make_issue_list(with_overlay) + + def test_same_directory_without_overlay_offset_is_flagged_unmapped(self): + """Control for the test above: the only difference is overlay_offset.""" + issues = _run(_meta(), + sections=[_section(va=0x1000, vs=0x100)], + data_directories=[_dir(rva=0x2000, size=0x10)]) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION] + + def test_mismatching_section_continues_to_a_later_match(self): + """ + On mismatch the loop `continue`s rather than breaking, so a second + section covering the same VA can still resolve raw_offset. Only one + mismatch is emitted and the resolved offset comes from the later + section. + """ + issues = _run(_meta(), + overlay_offset=0x9999, + sections=[ + _section("A", 0x1000, 0x1000, raw=0x400, raw_size=0x10), + _section("B", 0x1000, 0x1000, raw=0x800, raw_size=0x1000), + ], + data_directories=[_dir(rva=0x1800, size=0x10)]) + codes = make_issue_list(issues) + assert codes.count(ReasonCodes.DATA_DIRECTORY_RAW_MISMATCH) == 1 + assert ReasonCodes.DATA_DIRECTORY_SPANS_MULTIPLE_SECTIONS in codes + + +# ================================================================= +# Section mapping +# ================================================================= + +class TestSectionMapping: + + def test_mapped_directory_is_silent(self): + assert _run(_meta(), + sections=[_section(va=0x1000, vs=0x1000)], + data_directories=[_dir(rva=0x1000, size=0x100)]) == [] + + def test_not_mapped_flagged(self): + issues = _run(_meta(), + sections=[_section(va=0x100, vs=0x100)], + data_directories=[_dir(rva=0x500, size=0x10)]) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION] + + def test_no_sections_means_not_mapped(self): + issues = _run(_meta(), data_directories=[_dir(rva=0x1000, size=0x10)]) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION] + + def test_spans_multiple_sections_flagged(self): + issues = _run(_meta(), + sections=[_section("A", 0x100, 0x100), + _section("B", 0x150, 0x100)], + data_directories=[_dir(rva=0x120, size=0x100)]) + assert ReasonCodes.DATA_DIRECTORY_SPANS_MULTIPLE_SECTIONS in make_issue_list(issues) + assert _details_for( + issues, + ReasonCodes.DATA_DIRECTORY_SPANS_MULTIPLE_SECTIONS)[0]["sections"] == ["A", "B"] + + def test_exactly_one_section_is_silent(self): + issues = _run(_meta(), + sections=[_section("A", 0x1000, 0x1000), + _section("B", 0x2000, 0x1000)], + data_directories=[_dir(rva=0x1000, size=0x100)]) + assert issues == [] + + def test_overlap_uses_half_open_interval(self): + """ + `rva < va_end and rva + size > va_start` - a directory ending exactly + at a section start does not count as spanning it. + """ + issues = _run(_meta(), + sections=[_section("A", 0x1000, 0x100), + _section("B", 0x1100, 0x100)], + data_directories=[_dir(rva=0x1000, size=0x100)]) + assert ReasonCodes.DATA_DIRECTORY_SPANS_MULTIPLE_SECTIONS not in make_issue_list(issues) + + def test_zero_length_section_hit_skips_mapping(self): + """ + A directory whose RVA lands exactly on a zero-length section is + skipped: it cannot be meaningfully mapped. + """ + assert _run(_meta(), + sections=[_section(".empty", 0x1000, 0, raw=0x500)], + data_directories=[_dir(rva=0x1000, size=0x10)]) == [] + + def test_zero_length_section_requires_exact_rva_match(self): + """Landing near - not on - a zero-length section is still unmapped.""" + issues = _run(_meta(), + sections=[_section(".empty", 0x1000, 0, raw=0x500)], + data_directories=[_dir(rva=0x1004, size=0x10)]) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION] + + def test_nonzero_length_section_at_same_rva_is_not_skipped(self): + """ + Control: the skip needs BOTH va_start == rva AND vs == 0. A VA match + alone must not skip. + + The fixture spans two sections so that skipping would be observable - + a directory that simply maps cleanly emits nothing either way, and + could not distinguish the two behaviours. + """ + issues = _run(_meta(), + sections=[_section("A", 0x1000, 0x100), + _section("B", 0x1050, 0x100)], + data_directories=[_dir(rva=0x1000, size=0x100)]) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_SPANS_MULTIPLE_SECTIONS] + + +# ================================================================= +# Directory overlap +# ================================================================= + +class TestDirectoryOverlap: + + def test_overlapping_directories_flagged(self): + issues = _run(_meta(), + sections=[_section(va=0x1000, vs=0x1000)], + data_directories=[_dir("A", 0x1000, 0x100), + _dir("B", 0x1050, 0x100)]) + assert ReasonCodes.DATA_DIRECTORY_OVERLAP in make_issue_list(issues) + assert _details_for(issues, ReasonCodes.DATA_DIRECTORY_OVERLAP)[0] == { + "directory_a": "A", "directory_b": "B"} + + def test_adjacent_directories_not_flagged(self): + """`max(start) < min(end)` - touching ranges do not overlap.""" + issues = _run(_meta(), + sections=[_section(va=0x1000, vs=0x1000)], + data_directories=[_dir("A", 0x1000, 0x100), + _dir("B", 0x1100, 0x100)]) + assert ReasonCodes.DATA_DIRECTORY_OVERLAP not in make_issue_list(issues) + + def test_malformed_second_directory_skipped(self): + issues = _run(_meta(), + sections=[_section(va=0x1000, vs=0x1000)], + data_directories=[_dir("A", 0x1000, 0x100), + _dir("B", "bad", 0x100)]) + assert ReasonCodes.DATA_DIRECTORY_OVERLAP not in make_issue_list(issues) + + def test_malformed_first_directory_skipped(self): + issues = _run(_meta(), + sections=[_section(va=0x1000, vs=0x1000)], + data_directories=[_dir("A", "bad", 0x100), + _dir("B", 0x1000, 0x100)]) + assert ReasonCodes.DATA_DIRECTORY_OVERLAP not in make_issue_list(issues) + + def test_each_overlapping_pair_reported_once(self): + """Three mutually overlapping directories give C(3,2) = 3 pairs.""" + issues = _run(_meta(), + sections=[_section(va=0x1000, vs=0x1000)], + data_directories=[_dir("A", 0x1000, 0x100), + _dir("B", 0x1010, 0x100), + _dir("C", 0x1020, 0x100)]) + pairs = [(d["directory_a"], d["directory_b"]) + for d in _details_for(issues, ReasonCodes.DATA_DIRECTORY_OVERLAP)] + assert pairs == [("A", "B"), ("A", "C"), ("B", "C")] + + def test_overlap_runs_even_for_directories_skipped_earlier(self): + """ + The overlap pass is independent of the per-directory loop: a pair that + was `continue`d there (zero size) is still compared here - and a + zero-size range can never overlap, so it stays silent. + """ + issues = _run(_meta(), + sections=[_section(va=0x1000, vs=0x1000)], + data_directories=[_dir("A", 0x1000, 0), + _dir("B", 0x1000, 0x100)]) + codes = make_issue_list(issues) + assert ReasonCodes.DATA_DIRECTORY_ZERO_SIZE_NONZERO_RVA in codes + assert ReasonCodes.DATA_DIRECTORY_OVERLAP not in codes + + def test_overlap_labelled_by_index_when_name_absent(self): + issues = _run(_meta(), + sections=[_section(va=0x1000, vs=0x1000)], + data_directories=[{"index": 1, "rva": 0x1000, "size": 0x100}, + {"index": 2, "rva": 0x1050, "size": 0x100}]) + assert _details_for(issues, ReasonCodes.DATA_DIRECTORY_OVERLAP)[0] == { + "directory_a": 1, "directory_b": 2} + + +# ================================================================= +# Output contract +# ================================================================= + +class TestOutputContract: + + def test_dependency_contract(self): + assert getattr(validate_rva_graph, "_depends_on") == ("metadata", "analysis") + + def test_returns_list(self): + assert isinstance(_run(_meta(), data_directories=[_dir()]), list) + + def test_each_issue_has_issue_and_details(self): + issues = _run(_meta(size_of_headers=0x400), + sections=[_section(va=0x2000, vs=0x100)], + data_directories=[_dir(rva=0x100, size=0x10)]) + assert issues + for issue in issues: + assert set(issue) == {"issue", "details"} + assert isinstance(issue["issue"], str) + assert isinstance(issue["details"], dict) + + def test_no_details_payload_uses_reserved_reason_key(self): + """ + "reason" is reserved by the heuristics emission layer: _det builds + metadata as {"reason": parent, **details}, so a details["reason"] + would overwrite the parent reason code. This validator uses no + sub-reasons; the guard pins that none is introduced. + """ + issues = _run(_meta(size_of_image=0x2000, size_of_headers=0x400), + overlay_offset=0x500, + sections=[_section("A", 0x1000, 0x1000, + raw=0x400, raw_size=0x10)], + data_directories=[_dir("A", 0x100, 0x10), + _dir("B", 0x1800, 0x10), + _dir("C", 0x1800, 0x10), + _dir("D", 0, 0x10), + _dir("E", 0x1000, 0)]) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + + def test_json_serialisable(self): + import json + issues = _run(_meta(size_of_headers=0x400), + sections=[_section(va=0x2000, vs=0x100)], + data_directories=[_dir("A", 0x100, 0x10), + _dir("B", 0x100, 0x10)]) + json.dumps(issues) # must not raise + + def test_inputs_are_not_mutated(self): + import copy + metadata = _meta(size_of_headers=0x400) + analysis = { + "overlay_offset": 0x500, + "sections": [_section("A", 0x1000, 0x1000, raw=0x400, raw_size=0x10)], + "data_directories": [_dir("A", 0x1800, 0x10), _dir("B", 0x1800, 0x10)], + } + md_snapshot = copy.deepcopy(metadata) + an_snapshot = copy.deepcopy(analysis) + validate_rva_graph(metadata, analysis) + assert metadata == md_snapshot + assert analysis == an_snapshot + + +# ================================================================= +# Determinism +# ================================================================= + +class TestDeterminism: + + def test_repeated_validation_is_identical(self): + import json + metadata = _meta(size_of_image=0x2000, size_of_headers=0x400) + analysis = { + "overlay_offset": 0x500, + "sections": [_section("A", 0x1000, 0x1000, raw=0x400, raw_size=0x10), + _section("B", 0x1800, 0x800, raw=0x800, raw_size=0x800)], + "data_directories": [_dir("A", 0x100, 0x10), _dir("B", 0x1800, 0x10), + _dir("C", 0x1800, 0x10), _dir("D", 0, 0x10)], + } + first = json.dumps(validate_rva_graph(metadata, analysis), sort_keys=True) + for _ in range(20): + assert json.dumps(validate_rva_graph(metadata, analysis), + sort_keys=True) == first + + def test_emission_order_is_directories_then_overlaps(self): + """ + All per-directory issues are emitted first, in directory order; the + overlap pass runs afterwards. + """ + issues = _run(_meta(size_of_image=0x2000), + sections=[_section(va=0x1800, vs=0x800)], + data_directories=[_dir("A", 0x100, 0x10), + _dir("B", 0x1800, 0x10), + _dir("C", 0x1800, 0x10)]) + assert make_issue_list(issues) == [ + ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION, # A + ReasonCodes.DATA_DIRECTORY_OVERLAP, # B vs C ] - } - - issues = validate_rva_graph(metadata, analysis) - - # --- Assertions --- - # No overlay anomaly should fire because raw_offset is never computed. - assert not any( - i.issue == ReasonCodes.DATA_DIRECTORY_IN_OVERLAY - for i in issues - ) - - # No raw mismatch either (same reason) - assert not any( - i.issue == ReasonCodes.DATA_DIRECTORY_RAW_MISMATCH - for i in issues - ) - - # Directory *is* mapped to a section, so no NOT_MAPPED_TO_SECTION - assert not any( - i.issue == ReasonCodes.DATA_DIRECTORY_NOT_MAPPED_TO_SECTION - for i in issues - ) - - # Should be completely clean - assert issues == [] diff --git a/tests/unit/validators/test_validator_sections.py b/tests/unit/validators/test_validator_sections.py index 6a8c609..035a82e 100644 --- a/tests/unit/validators/test_validator_sections.py +++ b/tests/unit/validators/test_validator_sections.py @@ -9,6 +9,14 @@ def make_issue_list(result): return [i["issue"] for i in result] +def _has(issues, code, sub_reason=None): + """True if an issue with `code` (and optionally `sub_reason`) was emitted.""" + return any( + i["issue"] == code + and (sub_reason is None or i["details"].get("sub_reason") == sub_reason) + for i in issues + ) + # --------------------------------------------------------- # 1) RWX section @@ -20,7 +28,8 @@ def test_section_rwx(): "sections": [ { "name": ".text", - "characteristics": 0x20000000 | 0x80000000, # EXEC + WRITE + # READ is set so the characteristics are inert scaffolding: EXECUTE without READ would additionally trip SECTION_FLAGS_INCONSISTENT/exec_without_read. + "characteristics": 0x20000000 | 0x80000000 | 0x40000000, } ] } @@ -33,17 +42,15 @@ def test_section_rwx(): # --------------------------------------------------------- def test_section_non_executable_code_like(): - metadata = {} - analysis = { - "sections": [ - { - "name": ".text", - "characteristics": 0x00000020, # CODE flag only - } - ] - } - issues = validate_sections(metadata, analysis) - assert ReasonCodes.SECTION_NON_EXECUTABLE_CODE_LIKE in make_issue_list(issues) + analysis = {"sections": [{ + # ".foo" avoids also tripping SECTION_CODELIKE_NAME_NOT_EXECUTABLE; + # READ avoids SECTION_FLAGS_INCONSISTENT/code_without_read. + "name": ".foo", + "characteristics": 0x00000020 | 0x40000000, + }]} + issues = validate_sections({}, analysis) + assert len(issues) == 1 + assert issues[0]["issue"] == ReasonCodes.SECTION_NON_EXECUTABLE_CODE_LIKE # --------------------------------------------------------- @@ -74,7 +81,7 @@ def test_section_name_non_ascii(): "sections": [ { "name": "têxt", # non-ASCII - "characteristics": 0x20000000, + "characteristics": 0x20000000 | 0x40000000, } ] } @@ -119,7 +126,7 @@ def test_section_name_padding(): "sections": [ { "name": " ", - "characteristics": 0x20000000, + "characteristics": 0x20000000 | 0x40000000, } ] } @@ -132,21 +139,14 @@ def test_section_name_padding(): # --------------------------------------------------------- def test_section_impossible_flags(): - metadata = {} - analysis = { - "sections": [ - { - "name": ".x", - "characteristics": ( - 0x02000000 | # discardable - 0x20000000 | # exec - 0x80000000 # write - ), - } - ] - } - issues = validate_sections(metadata, analysis) - assert ReasonCodes.SECTION_IMPOSSIBLE_FLAGS in make_issue_list(issues) + # D+E+W necessarily also satisfies SECTION_RWX (E+W) and + # SECTION_DISCARDABLE_CODE (D+E); the overlap is inherent to the condition, + # so assert presence rather than an exact issue count. READ is set to + # suppress the unrelated flags_inconsistent noise. + analysis = {"sections": [{"name": ".x", + "characteristics": 0x02000000 | 0x20000000 | 0x80000000 | 0x40000000}]} + issues = validate_sections({}, analysis) + assert _has(issues, ReasonCodes.SECTION_IMPOSSIBLE_FLAGS) # --------------------------------------------------------- @@ -159,7 +159,7 @@ def test_section_raw_misaligned(): "sections": [ { "name": ".data", - "characteristics": 0x20000000, + "characteristics": 0x20000000 | 0x40000000, "raw_address": 123, # not aligned "raw_size": 100, } @@ -179,7 +179,7 @@ def test_section_overlaps_headers(): "sections": [ { "name": ".data", - "characteristics": 0x20000000, + "characteristics": 0x20000000 | 0x40000000, "raw_address": 100, # inside headers "raw_size": 100, } @@ -199,7 +199,7 @@ def test_section_zero_length(): "sections": [ { "name": ".empty", - "characteristics": 0x20000000, + "characteristics": 0x20000000 | 0x40000000, "virtual_address": 1000, "virtual_size": 0, "raw_address": 2000, @@ -221,7 +221,10 @@ def test_section_discardable_code(): "sections": [ { "name": ".text", - "characteristics": 0x02000000 | 0x20000000, # discardable + exec + # DISCARDABLE | EXECUTE is the anomaly; READ is added so the + # fixture doesn't also trip SECTION_FLAGS_INCONSISTENT/ + # exec_without_read. + "characteristics": 0x02000000 | 0x20000000 | 0x40000000, } ] } @@ -234,17 +237,13 @@ def test_section_discardable_code(): # --------------------------------------------------------- def test_section_flags_inconsistent_code_without_read(): - metadata = {} - analysis = { - "sections": [ - { - "name": ".text", - "characteristics": 0x00000020, # CODE but no READ - } - ] - } - issues = validate_sections(metadata, analysis) - assert ReasonCodes.SECTION_FLAGS_INCONSISTENT in make_issue_list(issues) + # ".foo" avoids the code-like-name check; CNT_CODE without READ is the + # anomaly. SECTION_NON_EXECUTABLE_CODE_LIKE also fires - inherent, since + # CNT_CODE without EXECUTE is exactly that condition. + analysis = {"sections": [{"name": ".foo", "characteristics": 0x00000020}]} + issues = validate_sections({}, analysis) + assert _has(issues, ReasonCodes.SECTION_FLAGS_INCONSISTENT, + "code_without_read") def test_section_flags_inconsistent_write_without_read(): @@ -258,7 +257,8 @@ def test_section_flags_inconsistent_write_without_read(): ] } issues = validate_sections(metadata, analysis) - assert ReasonCodes.SECTION_FLAGS_INCONSISTENT in make_issue_list(issues) + assert _has(issues, ReasonCodes.SECTION_FLAGS_INCONSISTENT, + "write_without_read") def test_section_flags_inconsistent_exec_without_read(): @@ -272,7 +272,8 @@ def test_section_flags_inconsistent_exec_without_read(): ] } issues = validate_sections(metadata, analysis) - assert ReasonCodes.SECTION_FLAGS_INCONSISTENT in make_issue_list(issues) + assert _has(issues, ReasonCodes.SECTION_FLAGS_INCONSISTENT, + "exec_without_read") # --------------------------------------------------------- @@ -428,3 +429,30 @@ def test_section_valid_no_issues(): } issues = validate_sections(metadata, analysis) assert issues == [] + + +# -------------------------------------------------------------------- +# 17) Contract and reason key tests +# -------------------------------------------------------------------- + +def test_dependency_contract(): + assert getattr(validate_sections, "_depends_on") == ("metadata", "analysis") + + +def test_no_details_payload_uses_reserved_reason_key(): + """ + "reason" is reserved by the heuristics emission layer: _det builds metadata + as {"reason": parent, **details}, so a details["reason"] would overwrite + the parent reason code. Validators must use "sub_reason". + + CNT_CODE|WRITE|EXECUTE without READ trips all three sub_reason sites. + """ + analysis = {"sections": [{"name": ".foo", + "characteristics": 0x00000020 | 0x80000000 | 0x20000000}]} + issues = validate_sections({}, analysis) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + diff --git a/tests/unit/validators/test_validator_signatures_ext.py b/tests/unit/validators/test_validator_signatures_ext.py index 3c4661a..6e9d6f2 100644 --- a/tests/unit/validators/test_validator_signatures_ext.py +++ b/tests/unit/validators/test_validator_signatures_ext.py @@ -7,12 +7,19 @@ for tag in cert_struct.get("truncations", []) or []: issues.append(StructuralIssue( issue=ReasonCodes.CERTIFICATE_TABLE_MALFORMED, - details={"reason": "truncation", "region": tag})) + details={"sub_reason": "truncation", "region": tag})) Reaching this block requires a certificate_struct that: - has an EMPTY top-level `errors` list (a non-empty one short-circuits at - step 0 with reason "top_level_decode" and returns before 1a), and + step 0 with sub_reason "top_level_decode" and returns before 1a), and - has one or more parser `truncations` tags. + +Details note: CERTIFICATE_TABLE_MALFORMED has TWO emission sites, and both +carried a "reason" key before the migration. Because the heuristics emission +layer merges details over its own reason field, the parent code never appeared +in output at all - it surfaced as the bare strings "top_level_decode" and +"truncation". Both sites now use "sub_reason", so the parent code survives and +the two variants remain distinguishable. These tests pin both. """ from __future__ import annotations @@ -64,7 +71,7 @@ def test_single_truncation_tag_emits_malformed(self): issues = _run(cs, analysis={"file_size": 0x1000}) assert ReasonCodes.CERTIFICATE_TABLE_MALFORMED in _codes(issues) details = _details_for(issues, ReasonCodes.CERTIFICATE_TABLE_MALFORMED) - assert details == [{"reason": "truncation", + assert details == [{"sub_reason": "truncation", "region": "certificate_blob_truncated"}] def test_truncation_with_no_certificates(self): @@ -74,7 +81,7 @@ def test_truncation_with_no_certificates(self): issues = _run(cs, has_signature=False) assert _codes(issues) == [ReasonCodes.CERTIFICATE_TABLE_MALFORMED] assert _details_for(issues, ReasonCodes.CERTIFICATE_TABLE_MALFORMED) == [ - {"reason": "truncation", "region": "certificate_table_truncated"}] + {"sub_reason": "truncation", "region": "certificate_table_truncated"}] def test_multiple_truncation_tags_one_issue_each_in_order(self): cs = _cert_struct( @@ -83,11 +90,13 @@ def test_multiple_truncation_tags_one_issue_each_in_order(self): "certificate_blob_truncated"]) issues = _run(cs, has_signature=False) malformed = _details_for(issues, ReasonCodes.CERTIFICATE_TABLE_MALFORMED) + # "region" is a distinct key and was never subject to the reason + # collision, so it is unchanged by the sub_reason migration. assert [d["region"] for d in malformed] == [ "certificate_table_truncated", "certificate_header_truncated", "certificate_blob_truncated"] - assert all(d["reason"] == "truncation" for d in malformed) + assert all(d["sub_reason"] == "truncation" for d in malformed) def test_no_truncations_does_not_emit_malformed(self): cs = _cert_struct([], truncations=[]) @@ -104,4 +113,61 @@ def test_top_level_error_short_circuits_before_1a(self): assert _codes(issues) == [ReasonCodes.CERTIFICATE_TABLE_MALFORMED] # single issue, and it is the decode variant (not the truncation one) assert _details_for(issues, ReasonCodes.CERTIFICATE_TABLE_MALFORMED) == [ - {"reason": "top_level_decode", "errors": ["raw_file_unavailable"]}] + {"sub_reason": "top_level_decode", + "errors": ["raw_file_unavailable"]}] + + def test_both_variants_share_parent_and_are_distinguishable(self): + """ + CERTIFICATE_TABLE_MALFORMED is emitted from two sites. Both must report + the same parent code while remaining separable by sub_reason. + + This is the property the clobber destroyed: previously the parent was + overwritten, so the two sites surfaced as unrelated top-level strings + ("top_level_decode" and "truncation") and the documented code never + appeared at all. + """ + decode = _run(_cert_struct([], errors=["raw_file_unavailable"])) + trunc = _run(_cert_struct([], truncations=["certificate_blob_truncated"]), + has_signature=False) + + assert _codes(decode) == [ReasonCodes.CERTIFICATE_TABLE_MALFORMED] + assert _codes(trunc) == [ReasonCodes.CERTIFICATE_TABLE_MALFORMED] + assert decode[0]["details"]["sub_reason"] == "top_level_decode" + assert trunc[0]["details"]["sub_reason"] == "truncation" + + +class TestOutputContract: + def test_dependency_contract(self): + assert getattr(validate_signature, "_depends_on") == ( + "internal", "metadata", "analysis") + + def test_no_details_payload_uses_reserved_reason_key(self): + """ + "reason" is reserved by the heuristics emission layer: _det builds + metadata as {"reason": parent, **details}, so a details["reason"] would + overwrite the parent reason code. Validators must use "sub_reason". + + Exercises the truncation, offset-inside-image, multiplicity and + per-certificate field paths together. + """ + certs = [ + {"offset": 0x800, "length": 0x40, "revision": 0x9999, + "cert_type": 0x1234, "errors": []}, + {"offset": 0x900, "length": 0x40, "revision": 0x0200, + "cert_type": 0x0002, "errors": []}, + ] + cs = _cert_struct(certs, truncations=["certificate_blob_truncated"], + overlaps_image=True, image_raw_end=0x5000) + issues = _run(cs, analysis={"file_size": 0x1000}) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + + def test_top_level_decode_avoids_reserved_reason_key(self): + """The short-circuit path is unreachable above; pin it separately.""" + cs = _cert_struct([], errors=["raw_file_unavailable"]) + issues = _run(cs) + assert issues + assert "reason" not in issues[0]["details"] diff --git a/tests/unit/validators/test_validator_tls_ext.py b/tests/unit/validators/test_validator_tls_ext.py index 51d7393..8a6f8c7 100644 --- a/tests/unit/validators/test_validator_tls_ext.py +++ b/tests/unit/validators/test_validator_tls_ext.py @@ -14,6 +14,17 @@ Each fixture is shaped to reach exactly one target while keeping the rest of the validator quiet, so the assertions stay unambiguous. + +Details note: sub-reasons are carried in a "sub_reason" key. The key "reason" +is reserved by the heuristics emission layer, which merges details over its own +reason field - a details["reason"] would overwrite the parent reason code. + +NESTED-KEY CAUTION: two of this validator's sub-reasons are written into dicts +appended to a list (`invalid.append({...})`) and only later promoted to the +top level of `details` via `{**item, ...}`. A key nested that way is NOT safe +from the collision - the splat lifts it into the merged payload just like a +directly-written key. Both were renamed; test_nested_sub_reason_is_promoted +pins that behaviour so the distinction is not lost. """ from __future__ import annotations @@ -77,12 +88,15 @@ def test_truncation_tag_emits_directory_truncated(self): issues = _run(tls) assert ReasonCodes.TLS_DIRECTORY_TRUNCATED in _codes(issues) assert _details_for(issues, ReasonCodes.TLS_DIRECTORY_TRUNCATED) == [ - {"reason": "callback_array", "region": "tls_callbacks_truncated"}] + {"sub_reason": "callback_array", + "region": "tls_callbacks_truncated"}] def test_multiple_truncation_tags_in_order(self): tls = _tls(truncations=["tls_callbacks_read_failed", "tls_callbacks_max_exceeded"]) issues = _run(tls) + # "region" is a distinct key and was never subject to the reason + # collision, so it is unchanged by the sub_reason migration. regions = [d["region"] for d in _details_for(issues, ReasonCodes.TLS_DIRECTORY_TRUNCATED)] assert regions == ["tls_callbacks_read_failed", @@ -124,7 +138,7 @@ def test_va_below_image_base_surfaced(self): issues = _run(tls) assert _codes(issues) == [ReasonCodes.TLS_CALLBACK_RVA_INVALID] assert _details_for(issues, ReasonCodes.TLS_CALLBACK_RVA_INVALID) == [ - {"reason": "tls_callbacks_va_below_image_base"}] + {"sub_reason": "tls_callbacks_va_below_image_base"}] def test_both_resolution_tags_sorted_and_deduped(self): tls = _tls(errors=["tls_image_base_unavailable", @@ -132,7 +146,7 @@ def test_both_resolution_tags_sorted_and_deduped(self): "tls_image_base_unavailable"], # dup callbacks=[]) issues = _run(tls) - reasons = [d["reason"] for d in + reasons = [d["sub_reason"] for d in _details_for(issues, ReasonCodes.TLS_CALLBACK_RVA_INVALID)] # set() dedupes, sorted() orders deterministically assert reasons == ["tls_callbacks_va_below_image_base", @@ -158,7 +172,7 @@ def test_image_base_unavailable_with_callbacks(self): issues = _run(tls) assert _codes(issues).count(ReasonCodes.TLS_CALLBACK_RVA_INVALID) == 1 assert _details_for(issues, ReasonCodes.TLS_CALLBACK_RVA_INVALID) == [ - {"reason": "image_base_unavailable", "callback_count": 2}] + {"sub_reason": "image_base_unavailable", "callback_count": 2}] # ================================================================= @@ -195,7 +209,7 @@ def test_unmapped_target_flagged(self): d = _details_for(issues, ReasonCodes.TLS_CALLBACK_RVA_INVALID) assert d == [{"callback_va": image_base + 0x8000, "callback_rva": 0x8000, - "reason": "not_mapped", + "sub_reason": "not_mapped", "invalid_callback_count": 1}] def test_below_image_base_target_flagged(self): @@ -205,5 +219,94 @@ def test_below_image_base_target_flagged(self): issues = _run(tls, analysis={"extended": [], "sections": []}) d = _details_for(issues, ReasonCodes.TLS_CALLBACK_RVA_INVALID) assert d == [{"callback_va": image_base - 0x10, - "reason": "below_image_base", + "sub_reason": "below_image_base", "invalid_callback_count": 1}] + + +# ================================================================= +# Output contract +# ================================================================= + +class TestOutputContract: + def test_dependency_contract(self): + assert getattr(validate_tls, "_depends_on") == ( + "internal", "metadata", "analysis") + + def test_nested_sub_reason_is_promoted_to_top_level(self): + """ + The per-target sub-reasons are written NESTED, into dicts appended to + `invalid`, then promoted by `details={**item, ...}`. + + This pins that the splat really does lift the key to the top level of + details - which is why nesting offered no protection from the reason + collision and both sites had to be renamed. If the emission is ever + refactored to keep the payload nested (e.g. details={"invalid": item}), + this test will catch the shape change. + """ + image_base = 0x400000 + tls = _tls(image_base=image_base, callbacks=[image_base + 0x8000]) + issues = _run(tls, analysis={"extended": [], + "sections": [{"virtual_address": 0x1000, + "virtual_size": 0x10}]}) + assert issues + details = issues[0]["details"] + # promoted to the top level, not left nested under a sub-object + assert details.get("sub_reason") == "not_mapped" + assert "reason" not in details + + def test_no_details_payload_uses_reserved_reason_key(self): + """ + "reason" is reserved by the heuristics emission layer: _det builds + metadata as {"reason": parent, **details}, so a details["reason"] would + overwrite the parent reason code. Validators must use "sub_reason". + + Exercises the truncation loop, the resolution-tombstone loop and the + per-target (nested/splatted) path together. + """ + image_base = 0x400000 + tls = _tls(image_base=image_base, + truncations=["tls_callbacks_truncated"], + errors=["tls_callbacks_va_below_image_base"], + callbacks=[image_base + 0x8000, image_base - 0x10]) + issues = _run(tls, analysis={"extended": [], + "sections": [{"virtual_address": 0x1000, + "virtual_size": 0x10}]}) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + + def test_header_decode_path_avoids_reserved_reason_key(self): + """The short-circuit path is unreachable above; pin it separately.""" + tls = _tls(errors=["tls_directory_truncated"]) + issues = _run(tls) + assert issues + assert "reason" not in issues[0]["details"] + + def test_image_base_unavailable_path_avoids_reserved_reason_key(self): + """This branch returns early and is mutually exclusive with the rest.""" + tls = _tls(callbacks=[0x401000], image_base=None) + issues = _run(tls) + assert issues + assert all("reason" not in i["details"] for i in issues) + + +# ================================================================= +# Determinism +# ================================================================= + +class TestDeterminism: + def test_repeated_calls_identical(self): + import json + image_base = 0x400000 + tls = _tls(image_base=image_base, + truncations=["tls_callbacks_truncated"], + errors=["tls_callbacks_va_below_image_base"], + callbacks=[image_base + 0x8000, image_base - 0x10]) + analysis = {"extended": [], + "sections": [{"virtual_address": 0x1000, + "virtual_size": 0x10}]} + a = _run(tls, analysis=analysis) + b = _run(tls, analysis=analysis) + assert json.dumps(a, sort_keys=True) == json.dumps(b, sort_keys=True) diff --git a/tests/unit/validators/test_validator_version_info.py b/tests/unit/validators/test_validator_version_info.py index 8c1c940..3751416 100644 --- a/tests/unit/validators/test_validator_version_info.py +++ b/tests/unit/validators/test_validator_version_info.py @@ -12,6 +12,20 @@ cover everything the validator reads. - Each test asserts on the set of REASONCODES emitted and the details payload, since both are part of the contract. + +Layer note: this validator is @depends_on("internal", "analysis") and takes +TWO positional arguments. It reads only analysis["sections"] and does not use +SizeOfImage, so it was unaffected by the metadata-layer migration. + +Details note: sub-reasons are carried in a "sub_reason" key. The key "reason" +is reserved by the heuristics emission layer, which merges details over its own +reason field - a details["reason"] would overwrite the parent reason code. + +CAUTION when editing: several tests below narrow the details list with a +predicate such as `d.get("sub_reason") == "placement"` and then assert the +result is empty. If that key name ever drifts out of sync with the validator, +the predicate silently matches nothing and the test passes vacuously. Each such +test therefore ALSO asserts on the whole issue list, which cannot go vacuous. """ from __future__ import annotations @@ -117,6 +131,14 @@ def _details_for(issues, code) -> List[Dict[str, Any]]: return [issue["details"] for issue in issues if issue["issue"] == code] +def _placement_details(issues) -> List[Dict[str, Any]]: + """Header-issue details whose sub_reason is the placement check.""" + return [ + d for d in _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER) + if d.get("sub_reason") == "placement" + ] + + # ================================================================= # Absence / clean cases # ================================================================= @@ -183,15 +205,13 @@ def test_placement_rva_before_rsrc_flagged(self): vi = _make_vi(rva=0x500, size=100) analysis = _make_analysis(rsrc_va=0x1000, rsrc_vs=0x2000) issues = validate_version_info({"version_info_struct": vi}, analysis) - details = _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER) - assert any(d.get("reason") == "placement" for d in details) + assert len(_placement_details(issues)) == 1 def test_placement_rva_extends_past_rsrc_flagged(self): vi = _make_vi(rva=0x2F00, size=0x200) analysis = _make_analysis(rsrc_va=0x1000, rsrc_vs=0x2000) issues = validate_version_info({"version_info_struct": vi}, analysis) - details = _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER) - placement_details = [d for d in details if d.get("reason") == "placement"] + placement_details = _placement_details(issues) assert len(placement_details) == 1 assert placement_details[0]["rva"] == 0x2F00 assert placement_details[0]["size"] == 0x200 @@ -201,20 +221,17 @@ def test_placement_no_rsrc_section_skipped(self): vi = _make_vi(rva=0x5000, size=100) analysis = _make_analysis(include_rsrc=False) issues = validate_version_info({"version_info_struct": vi}, analysis) - placement_details = [d for d in _details_for( - issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER - ) if d.get("reason") == "placement"] - assert placement_details == [] + assert _placement_details(issues) == [] + # Whole-list assertion: cannot pass vacuously if the predicate drifts. + assert issues == [] def test_placement_rva_none_skipped(self): """If the parser couldn't determine an RVA, placement isn't checked.""" vi = _make_vi(rva=None, size=None) analysis = _make_analysis() issues = validate_version_info({"version_info_struct": vi}, analysis) - placement_details = [d for d in _details_for( - issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER - ) if d.get("reason") == "placement"] - assert placement_details == [] + assert _placement_details(issues) == [] + assert issues == [] def test_placement_size_none_treated_as_zero(self): """size=None is coerced to 0 by `vi['size'] or 0`.""" @@ -222,20 +239,27 @@ def test_placement_size_none_treated_as_zero(self): analysis = _make_analysis(rsrc_va=0x1000, rsrc_vs=0x2000) issues = validate_version_info({"version_info_struct": vi}, analysis) # 0x1100 + 0 is within .rsrc, so no placement issue - placement_details = [d for d in _details_for( - issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER - ) if d.get("reason") == "placement"] - assert placement_details == [] + assert _placement_details(issues) == [] + assert issues == [] def test_placement_exact_boundary_no_issue(self): """RVA at rsrc_va and rva+size == rsrc_va + rsrc_vs is in-bounds.""" vi = _make_vi(rva=0x1000, size=0x2000) analysis = _make_analysis(rsrc_va=0x1000, rsrc_vs=0x2000) issues = validate_version_info({"version_info_struct": vi}, analysis) - placement_details = [d for d in _details_for( - issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER - ) if d.get("reason") == "placement"] - assert placement_details == [] + assert _placement_details(issues) == [] + assert issues == [] + + def test_placement_predicate_is_not_vacuous(self): + """ + Guard for the helper itself: prove _placement_details CAN return a + match. Without this, a drifted key name would make every + "placement isn't checked" assertion above pass for the wrong reason. + """ + vi = _make_vi(rva=0x5000, size=100) + issues = validate_version_info({"version_info_struct": vi}, + _make_analysis()) + assert len(_placement_details(issues)) == 1 # ================================================================= @@ -262,9 +286,9 @@ def test_undecoded_emits_invalid_header_and_returns_early(self): assert ReasonCodes.RESOURCE_VERSIONINFO_INVALID_STRINGFILEINFO not in codes assert ReasonCodes.RESOURCE_VERSIONINFO_INVALID_VARFILEINFO not in codes - # Details should carry the "undecoded" reason and the parser errors + # Details should carry the "undecoded" sub_reason and the parser errors header_details = _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER) - undecoded = [d for d in header_details if d.get("reason") == "undecoded"] + undecoded = [d for d in header_details if d.get("sub_reason") == "undecoded"] assert len(undecoded) == 1 assert undecoded[0]["errors"] == ["too_short"] @@ -272,21 +296,21 @@ def test_szkey_mismatch_emits_invalid_header(self): vi = _make_vi(header_ok=False) issues = validate_version_info({"version_info_struct": vi}, _make_analysis()) details = _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER) - szkey_details = [d for d in details if d.get("reason") == "szkey_mismatch"] + szkey_details = [d for d in details if d.get("sub_reason") == "szkey_mismatch"] assert len(szkey_details) == 1 def test_length_inconsistent_emits_invalid_header(self): vi = _make_vi(length_consistent=False) issues = validate_version_info({"version_info_struct": vi}, _make_analysis()) details = _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER) - length_details = [d for d in details if d.get("reason") == "length_inconsistent"] + length_details = [d for d in details if d.get("sub_reason") == "length_inconsistent"] assert len(length_details) == 1 def test_both_szkey_and_length_emit_two_separate_issues(self): vi = _make_vi(header_ok=False, length_consistent=False) issues = validate_version_info({"version_info_struct": vi}, _make_analysis()) details = _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER) - reasons = [d.get("reason") for d in details] + reasons = [d.get("sub_reason") for d in details] assert "szkey_mismatch" in reasons assert "length_inconsistent" in reasons @@ -295,7 +319,8 @@ def test_undecoded_with_empty_errors_list(self): vi = _make_vi(decoded=False, errors=[]) issues = validate_version_info({"version_info_struct": vi}, _make_analysis()) details = _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_HEADER) - undecoded = [d for d in details if d.get("reason") == "undecoded"] + undecoded = [d for d in details if d.get("sub_reason") == "undecoded"] + assert len(undecoded) == 1 assert undecoded[0]["errors"] == [] @@ -321,13 +346,10 @@ def test_absent_ffi_with_parse_errors_flagged(self): fixed_file_info=None, errors=["fixed_file_info_truncated"], ) - print("VI errors:", vi.get("errors")) # debug - print("VI ffi:", vi.get("fixed_file_info")) # debug issues = validate_version_info({"version_info_struct": vi}, _make_analysis()) - print("Issues:", issues) # debug details = _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_FIXEDINFO) assert len(details) == 1 - assert details[0]["reason"] == "parse_failed" + assert details[0]["sub_reason"] == "parse_failed" assert "fixed_file_info_truncated" in details[0]["errors"] def test_absent_ffi_with_non_ffi_errors_not_flagged(self): @@ -344,7 +366,7 @@ def test_bad_signature_flagged(self): vi = _make_vi(fixed_file_info=ffi) issues = validate_version_info({"version_info_struct": vi}, _make_analysis()) details = _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_FIXEDINFO) - sig_details = [d for d in details if d.get("reason") == "signature"] + sig_details = [d for d in details if d.get("sub_reason") == "signature"] assert len(sig_details) == 1 assert sig_details[0]["signature"] == 0xDEADBEEF @@ -353,7 +375,7 @@ def test_bad_struct_version_flagged(self): vi = _make_vi(fixed_file_info=ffi) issues = validate_version_info({"version_info_struct": vi}, _make_analysis()) details = _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_FIXEDINFO) - sv_details = [d for d in details if d.get("reason") == "struct_version"] + sv_details = [d for d in details if d.get("sub_reason") == "struct_version"] assert len(sv_details) == 1 assert sv_details[0]["struct_version"] == 0x00020000 @@ -362,7 +384,7 @@ def test_both_signature_and_struct_version_emit_two_issues(self): vi = _make_vi(fixed_file_info=ffi) issues = validate_version_info({"version_info_struct": vi}, _make_analysis()) details = _details_for(issues, ReasonCodes.RESOURCE_VERSIONINFO_INVALID_FIXEDINFO) - reasons = [d.get("reason") for d in details] + reasons = [d.get("sub_reason") for d in details] assert "signature" in reasons assert "struct_version" in reasons @@ -580,6 +602,10 @@ def test_placement_plus_sfi_plus_vfi_all_emit(self): class TestOutputContract: """Pin the shape of returned StructuralIssue objects.""" + def test_dependency_contract(self): + assert getattr(validate_version_info, "_depends_on") == ( + "internal", "analysis") + def test_returns_list(self): result = validate_version_info({"version_info_struct": _make_vi()}, _make_analysis()) assert isinstance(result, list) @@ -600,6 +626,44 @@ def test_each_issue_has_issue_and_details(self): assert "details" in issue assert isinstance(issue["details"], dict) + def test_no_details_payload_uses_reserved_reason_key(self): + """ + "reason" is reserved by the heuristics emission layer: _det builds + metadata as {"reason": parent, **details}, so a details["reason"] would + overwrite the parent reason code. Validators must use "sub_reason". + + Exercises placement, both header branches, both FFI branches, and the + SFI/VFI paths together. + """ + vi = _make_vi( + rva=0x5000, size=100, # placement + header_ok=False, # szkey_mismatch + length_consistent=False, # length_inconsistent + fixed_file_info=_make_ffi(signature_ok=False, struct_version_ok=False), + string_file_info=[{"tables": [], "errors": ["string_table_header"]}], + var_file_info=[{"vars": [], "errors": ["var_header"]}], + ) + issues = validate_version_info({"version_info_struct": vi}, _make_analysis()) + assert issues, "fixture should produce issues" + offenders = [i["issue"] for i in issues if "reason" in i["details"]] + assert not offenders, ( + f"details payload used the reserved key 'reason' for: {offenders}" + ) + + def test_undecoded_path_avoids_reserved_reason_key(self): + """The early-return path is unreachable above; pin it separately.""" + vi = _make_vi(decoded=False, fixed_file_info=None, errors=["too_short"]) + issues = validate_version_info({"version_info_struct": vi}, _make_analysis()) + assert issues + assert all("reason" not in i["details"] for i in issues) + + def test_ffi_parse_failed_path_avoids_reserved_reason_key(self): + """The absent-FFI branch is mutually exclusive with the FFI checks.""" + vi = _make_vi(fixed_file_info=None, errors=["fixed_file_info_truncated"]) + issues = validate_version_info({"version_info_struct": vi}, _make_analysis()) + assert issues + assert all("reason" not in i["details"] for i in issues) + # ================================================================= # Determinism