Feature/0.7.6 into main - #47
Merged
Merged
Conversation
Implement the remaining structural PE directories — relocations, certificate table, debug directory, and TLS — as deterministic, pefile-independent struct decoders, bringing IOCX's static PE engine to full directory completeness. Parsers (raw struct decode; never raise; tombstone into errors/truncations): - pe_relocations: IMAGE_BASE_RELOCATION blocks + typed entries (HIGHLOW, DIR64, ...), dual-bounded walk, non-advancing SizeOfBlock guarded. - pe_certificates: WIN_CERTIFICATE table read from raw file bytes (DATA_DIRECTORY[4].VirtualAddress is a FILE OFFSET, not an RVA); records overlaps_image fact; PKCS#7 blob left opaque. - pe_debug: IMAGE_DEBUG_DIRECTORY entries + CodeView PDB path extraction (RSDS/NB10) with canonical mixed-endian GUID; file-pointer-first reads. - pe_tls: IMAGE_TLS_DIRECTORY (PE32/PE32+); VA->RVA callback resolution via ImageBase; dual-bounded callback walk; zero-length raw data valid. Validators: - Add relocations + debug validators (content-level only; directory placement remains owned by rva_graph to avoid double-counting). - Migrate signature + tls validators to consume certificate_struct / tls_struct from InternalMetadata instead of pefile-derived metadata. All existing reason codes and checks preserved; adds the four new codes. Fixes a latent VA/RVA unit mismatch in TLS pointer->section mapping. - tls: surface previously dropped parser tombstones (tls_image_base_unavailable, tls_callbacks_va_below_image_base) as tls_callback_rva_invalid; narrow tls_zero_length_directory so a zero-length raw-data region with valid callbacks is no longer a false positive. Schema / plumbing: - internal_schema: add RelocationStruct, CertificateStruct, DebugStruct, TlsStruct and the four InternalMetadata keys. - reason_codes: add certificate_table_malformed, certificate_offset_inside_image, tls_directory_truncated, tls_callback_rva_invalid (+ relocation_* / debug_* families). All lowercase snake_case, non-overlapping, snapshot-stable. - engine: populate relocation_struct, certificate_struct, debug_struct, tls_struct via the build_*_structure decoders. - dispatcher: register relocations + debug (ascending directory-index order, after the placement backbone, before entropy). signature/tls keep their existing slots.
…itive Three surgical edits to the TLS validator, no changes elsewhere. - Add _CALLBACK_RESOLUTION_ERROR_TAGS for the two parser tombstones that signalled an unresolvable callback array (tls_image_base_unavailable, tls_callbacks_va_below_image_base). - Surface those tags as tls_callback_rva_invalid at the top of _validate_callback_targets, before the guard that was silently swallowing them (parser sets callbacks=[] in both cases). Emitted via sorted(set(...)) for deterministic, de-duplicated output. - Narrow tls_zero_length_directory: only flag a zero-length raw-data region when there are NO resolved callbacks. A zero-length template alongside a valid callback array is legitimate and no longer a false positive. The early return is retained so the degenerate range never reaches the pointer cascade. Both tombstone tags map to the already-registered TLS_CALLBACK_RVA_INVALID; no reason-code or parser changes required. Verified: both findings fixed, all preserved TLS codes unchanged, and deterministic double-run output.
IMAGE_DIRECTORY_ENTRY_SECURITY (index 4) is the one data directory whose VirtualAddress field is a FILE OFFSET, not an RVA; the attribute certificate table is appended to the file and never mapped into the image. The rva_graph validator interpreted it as an RVA, producing a spurious data_directory_out_of_range finding (and potential overlap findings) that double-counted the same corruption already owned, with correct semantics, by the certificate parser / signature validator (certificate_offset_past_eof -> certificate_table_malformed, validated against file_size). Exclude the SECURITY directory from all RVA-based checks: - add _is_security_directory() (matches by index 4 or name) - skip it in the per-directory loop (out-of-range, headers, mapping, overlay) - skip it on both sides of the overlap-detection loop Placement/validity of the security directory is now single-owned by signature / pe_certificates. Pre-existing latent bug surfaced by the v0.7.6 certificate decoder providing a correct second opinion. Updated one affected snapshot golden accordingly.
…ce the v0.7.6 migration
1. TestLocator — valid / zero-rva / zero-size / missing-optional-header / missing-entry. 2. TestReadEntries — the four array-level paths: size_not_entry_aligned, entry_count_exceeds_max, entry_read_failed (via an always-raising PE), entry_truncated (short read). 3. TestDecodeEntry — field decode, unknown type-name → None, and the struct.error → entry_unpack_failed tombstone. 4. CodeView: RSDS via file pointer and the RVA fallback; NB10; unknown signature; too-short; RSDS/NB10 truncated; PDB path unterminated, non-ASCII, and empty→None; size_of_data==0 → max-len fallback. 5. TestReadCodeViewBlob — the three fallback branches, including the one genuine gap I found and closed: __data__ that raises on bytes() (line 252's except…: pass), proven to fall through to the RVA read. 6. _extract_asciiz_path and _format_guid exercised directly (canonical mixed-endian GUID; short→None). test_validator_debug.py: 1. Absence, top-level-decode short-circuit (proves entries/truncations are skipped), truncations (one issue per tag). 2. Entry malformation — priority-first-match, unknown-tag-ignored, clean-entry-clean. 3. Entry RVA validation — unmapped flagged, mapped clean, addr==0 skipped, missing size_of_data→0, no-sections→size_of_image fallback. 4. Combined — truncation+malformed, and malformed and RVA-invalid on the same entry (two issues, one entry). 5. Contract — _depends_on == (internal,analysis), issue shape, JSON-safety, determinism.
Complete the v0.7.6 test sweep, bringing statement coverage to 100% across the entire IOCX codebase with 1620 passing tests. Parsers (byte-level fixtures via struct.pack; fake PE stubs steering get_data / __data__ / sections, including short and raising reads): - pe_relocations: block-header truncation, unpack failure, MAX_BLOCKS exhaustion, entries read-failure and short-read paths. - pe_certificates: locator, __data__ absent/unconvertible, image-raw-end section extents (raising/zero-field sections), full WIN_CERTIFICATE walk (past-EOF, table/header/blob truncation, length-too-small, max-exceeded), revision/type maps, align_up. - pe_debug: entry-array truncations, RSDS/NB10 CodeView decode, PDB path unterminated/non-ASCII/empty, file-pointer->RVA fallback, GUID formatter. - pe_tls: PE32/PE32+ widths, VA->RVA callback resolution, directory and callback-array read/short/unpack paths, image-base-unavailable, looping array cap. Validators (dispatcher-faithful arg binding; assert on reason codes + details): - relocations, debug: block/entry integrity, truncations, per-entry RVA mapping, priority-resolved sub-reasons, placement-owned-by-rva_graph. - signature, tls: post-migration struct sourcing, the new v0.7.6 codes (certificate_table_malformed / certificate_offset_inside_image / tls_directory_truncated / tls_callback_rva_invalid), and the guarded short-circuit boundaries between them. - _directory_invariants: tri-state (True/False/None) contract, key-alias tolerance, int() coercion + except paths, section vs SizeOfImage fallback, and the zero-length-region boundary semantics. All tests are deterministic and JSON-safe;
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.