From 3defa36cffd8fe6f067030cc270a05d503b7dc0a Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 10:17:03 +0200 Subject: [PATCH 01/12] Document keyed semantic expression protection --- analysis/keyed_expression_protection_plan.md | 1327 ++++++++++++++++++ 1 file changed, 1327 insertions(+) create mode 100644 analysis/keyed_expression_protection_plan.md diff --git a/analysis/keyed_expression_protection_plan.md b/analysis/keyed_expression_protection_plan.md new file mode 100644 index 0000000..74ded72 --- /dev/null +++ b/analysis/keyed_expression_protection_plan.md @@ -0,0 +1,1327 @@ +# Keyed Semantic Expression Protection + +## Objective + +Add an optional protected-expression capability to MEXCE. The runtime receives: + +- an opaque binary semantic program; +- a separate 256-bit artifact key; and +- variable bindings addressed by numeric slots. + +The runtime never receives expression source text and never reconstructs source +text or a complete clear serialized program. It authenticates and decodes one +fixed-size semantic record at a time, validates the postfix program, constructs +the existing compiler representation, emits native code through the existing +backends, and erases transient key and record storage. + +The issuer-side encoder accepts clear source in a trusted build or +licence-issuance environment. It resolves the source grammar before encryption. +Lexical spelling, comments, whitespace, parentheses, and character boundaries +do not appear in the protected artifact. + +## Product invariants + +1. The protected runtime API accepts only opaque program bytes and a separate + key. There is no plaintext overload or automatic input detection. +2. Protection operates on canonical semantic records, not characters, source + tokens, or a stable secret opcode substitution table. +3. The wire program is backend-neutral and portable between supported Windows + and Linux builds. +4. Clear and protected compilation share semantic validation, compiler object + construction, optimization, backend selection, and native-code emission. + There is no protected-only compiler backend. +5. Authenticated encryption is provided by libsodium secretstream using + XChaCha20-Poly1305. MEXCE does not implement a cipher, MAC, or custom + composition. +6. The canonical encoder generates a fresh random 256-bit key and secretstream + header for every artifact. Its API has no caller-supplied-key overload, so + encoder key reuse is not representable. +7. Key storage, key transport, device binding, issuer identity, licence + signing, rollback policy, and revocation belong to the host product. +8. No complete clear semantic-record sequence exists in runtime memory. At + most one decrypted wire record is held at a time. +9. Invalid, truncated, reordered, substituted, oversized, or trailing input + hard-fails and leaves the evaluator empty. +10. Existing set_expression remains the one unprotected source API. The + protected capability has one encoder API and one runtime load API. +11. Existing clear-expression behavior remains unchanged except that a failed + replacement has a defined empty-state contract instead of stale-result or + null-call behavior. +12. The implementation remains C++11-compatible. +13. Ordinary MEXCE remains header-only and dependency-free. Only protected + consumers include the protected headers and link libsodium. +14. The definition and layout of evaluator are identical in every translation + unit. Protected support never changes class layout through a preprocessor + definition. + +## Scope exclusions + +The capability does not provide: + +- confidentiality against an attacker controlling the licensed process; +- protection of emitted native instructions from disassembly; +- virtual-black-box obfuscation; +- anti-debugging, white-box cryptography, or self-modifying code; +- TPM, licence, certificate, revocation, or device-enrollment behavior; +- issuer authentication or artifact freshness; +- record-count padding; +- source-language encryption or format-preserving encryption; +- protected CSE; +- protected x86 package support; or +- support for architectures outside MEXCE's declared targets. + +The host may bind the artifact key to a licence, device, product version, or +monotonic policy. MEXCE treats a replayed valid artifact and matching key as a +valid input. + +## Existing architecture constraints + +MEXCE is a C++11 single-header JIT. evaluator owns variable bindings, compiler +state, and executable memory. Clear compilation currently performs lexical +parsing, postfix conversion, compiler-object construction, optimization, +fallback selection, and code emission in one set_expression path. + +Both the x87 and SSE2 backends consume the same compiler representation. +Optimizers mutate that representation. Some emitted x87 instructions retain +addresses into constants and CSE storage owned outside the final visible graph. +The compilation lifetime owner must therefore include every object whose +address can be embedded in code, not only the final graph. + +The safe shared boundary is a canonical semantic producer feeding one semantic +validator/compiler: + +~~~ +clear source -> lexical parser -> semantic records -> validator/compiler + +protected frames ---------------------------> validator/compiler +~~~ + +The clear route passes semantic records directly without serializing them. The +protected route decrypts one wire record and immediately applies it. Both routes +converge before linking, optimization, backend selection, and code emission. + +The cryptographic construction follows the maintained libsodium contracts: + +- https://doc.libsodium.org/secret-key_cryptography/secretstream +- https://doc.libsodium.org/memory_management + +The protected dependency is libsodium 1.0.22. Conan Center and the controlled +vcpkg baseline both provide that release. + +## Security model + +### Protected assets + +- semantic operations and their order; +- numeric literal bits; +- variable-slot use and repetition; +- authenticated compiler policy; +- integrity of the program supplied to the JIT; and +- the artifact key while MEXCE owns it. + +Variable names are absent rather than encrypted. Source comments, whitespace, +parentheses, and original spelling are removed before encoding. + +### Trusted computing base + +The executing MEXCE code, libsodium implementation, operating-system process +isolation, and host key-delivery path are trusted. Replacing or patching any of +them can bypass verification or capture clear state and is outside the static +artifact claim. + +### Static artifact adversary + +The adversary can read, replace, truncate, reorder, or modify opaque artifacts +and other untrusted stored product data. The adversary does not possess the +matching artifact key and cannot modify the trusted computing base. + +The format provides authenticated confidentiality and integrity of semantic +records against this adversary. It does not authenticate a vendor identity; +authentication means consistency under the supplied symmetric key. + +### Licensed local observer + +The observer possesses a valid installation and can inspect ordinary files and +process behavior but does not control process memory or a debugger. Host-bound +key delivery and short MEXCE key ownership increase the effort needed to recover +the expression, but provide an obfuscation or delay benefit rather than a +cryptographic secrecy guarantee. + +### Licensed process controller + +The controller can inspect or modify process memory, registers, branches, +libsodium calls, or emitted code. Confidentiality is not claimed against this +attacker. The controller can recover transient records, compiler objects, or +equivalent native semantics. + +### Issuer compromise + +An attacker controlling the issuer or build environment can read source and +generated keys. Issuer hardening and key-handling policy are outside MEXCE. + +### Required rejection behavior + +The runtime rejects: + +- random corruption and use of the wrong key; +- unsupported format versions or public flags; +- ciphertext, tag, header, or authenticated-data modification; +- frame insertion, removal, reordering, duplication, or truncation; +- missing, early, or repeated final tags; +- trailing bytes or records; +- unknown record kinds or operation IDs; +- non-zero reserved fields and unused operands; +- invalid manifest policy; +- stack underflow, excess semantic depth, or invalid terminal depth; +- missing, unused, out-of-range, or unbound variable slots; +- non-canonical record sequences; +- inputs above declared resource limits; and +- any failure that would otherwise leave partial compiled state reachable. + +### Accepted leakage + +The artifact reveals: + +- that it is a MEXCE protected program; +- format version and public flags; +- a random program identifier; +- fixed record size; +- total semantic record count; and +- approximate formula complexity derived from that count. + +Runtime observation can additionally reveal evaluation timing, native-code +size, bound addresses, outputs, and everything observable by a process +controller. + +## End-to-end behavior + +### Issuer flow + +1. Validate the source and binding-schema resource limits. +2. Parse source with the existing grammar and an issuer symbol table. +3. Emit canonical postfix semantic records through the shared producer. +4. Validate record order, stack behavior, operation IDs, and slot use. +5. Generate a fresh Protected_expression_key and random program identifier. +6. Initialize one XChaCha20-Poly1305 secretstream. +7. Encrypt each 32-byte record as a separate message. The END record alone + carries TAG_FINAL. +8. Return the opaque bytes, move-only key, and program identifier. No API writes + them to disk implicitly. + +### Runtime flow + +1. Bind caller-owned variables to numeric slots. +2. Transfer an opaque byte range and a Protected_expression_key into + set_protected_expression. +3. Invalidate the previous compiled expression and enter the private loading + state. +4. Validate bounded public framing before allocating compiler state. +5. Initialize secretstream pull state. Erase the transferred key immediately + after initialization, whether initialization succeeds or fails. +6. Authenticate and decrypt one frame into a fixed 32-byte buffer. +7. Validate and apply that record to unpublished compiler state. +8. Erase the record buffer before reading another frame. +9. Require an authenticated END record with TAG_FINAL, exact byte consumption, + and semantic stack depth one. +10. Compile through the shared optimizer and selected backend using a + per-compilation effective-options context. +11. Publish executable code and its complete lifetime owner only after + authentication, semantic validation, optimization, and executable-memory + finalization all succeed. +12. Erase stream state, unused compilation state, and every transient buffer. + +The lexical parser is unreachable from the protected runtime entry point. + +### Evaluator state + +The externally observable state is either empty or compiled. Loading state is +private and never callable. + +- Starting either clear or protected replacement releases the previous + callable expression and resets backend, constant-result, referenced-binding, + graph, optimizer, and executable state. +- A successful replacement publishes one complete clear or protected + compilation. +- Any failure returns to empty. +- evaluate in empty state throws std::logic_error with the stable message + No expression has been compiled. +- get_backend returns backend_type::none in empty state. +- Clear introspection keeps its existing behavior. +- Protected introspection throws Protected_expression_error with + INTROSPECTION_DISABLED. +- A later successful clear replacement restores normal introspection behavior. + +The evaluator and its bound values follow the existing caller-synchronization +model. Concurrent evaluation, mutation, binding, replacement, or destruction +of one evaluator or its bound values requires external synchronization. + +## Wire format 1.0 + +All integers are unsigned little-endian. Floating literals are exact IEEE-754 +binary64 bits. Protected release targets are little-endian x64, but decoding +still uses explicit byte loads rather than pointer casts. + +### Version compatibility + +The encoder emits format 1.0. The initial decoder accepts exactly format 1.0. + +Within major version 1: + +- existing record kinds, operation IDs, arities, field meanings, and semantic + meanings are immutable; +- a newer decoder must continue accepting older supported minor versions; +- an older decoder rejects an unsupported newer minor version; +- a new operation or public feature requires a minor-version decision; and +- an incompatible field or semantic change requires a new major version or a + new operation ID. + +Unknown versions, flags, record kinds, and operation IDs hard-fail. There is no +best-effort downgrade. + +### Public header + +The header is exactly 64 bytes: + +| Offset | Size | Field | Format 1.0 rule | +|---:|---:|---|---| +| 0 | 8 | magic | ASCII MEXCEPRG | +| 8 | 2 | format_major | 1 | +| 10 | 2 | format_minor | 0 | +| 12 | 2 | header_size | 64 | +| 14 | 2 | plaintext_record_size | 32 | +| 16 | 4 | record_count | 3 through 16384 | +| 20 | 4 | feature_flags | 0 | +| 24 | 16 | program_id | random and not all zero | +| 40 | 24 | secretstream_header | generated by libsodium | + +Every field is authenticated as additional data for every record. + +The exact blob size is: + +~~~ +64 + record_count * (32 + crypto_secretstream_xchacha20poly1305_ABYTES) +~~~ + +For format 1.0, ABYTES is 17 and the maximum blob size is 802880 bytes. The +decoder validates record_count before computing the exact size. The bounded +calculation cannot overflow a supported size_t and does not need an unreachable +overflow branch. + +### Per-record additional data + +Additional data is the exact 64-byte public header followed by the record index +as a 32-bit little-endian integer. Secretstream already authenticates ordering; +the explicit index binds application framing and supplies a direct format +oracle. + +### Clear semantic record + +Each authenticated plaintext record is exactly 32 bytes: + +| Offset | Size | Field | +|---:|---:|---| +| 0 | 1 | record_kind | +| 1 | 1 | record_flags | +| 2 | 2 | reserved_16 | +| 4 | 4 | operand_32 | +| 8 | 8 | operand_64 | +| 16 | 16 | reserved_128 | + +record_flags and all reserved bytes are zero. Each kind defines its meaningful +operands; unused operands are also zero. The in-memory semantic representation +is not this wire struct and is never serialized with memcpy. + +### Record sequence + +The only valid sequence is: + +1. one MANIFEST record; +2. one or more semantic PUSH or CALL records; and +3. one END record. + +MANIFEST is frame zero. END is the final frame and the only frame carrying +TAG_FINAL. All preceding frames carry TAG_MESSAGE. TAG_PUSH, TAG_REKEY, early +TAG_FINAL, non-final END, and data after END reject. + +### Record kinds + +Wire values are explicit and independent of C++ enum order: + +| Value | Name | operand_32 | operand_64 | Stack effect | +|---:|---|---|---|---| +| 1 | MANIFEST | binding_count | semantic policy | none | +| 2 | PUSH_LITERAL_F64 | zero | exact double bits | +1 | +| 3 | PUSH_VARIABLE | slot | zero | +1 | +| 4 | CALL | operation ID | zero | 1 - arity | +| 5 | END | zero | zero | requires depth 1 | + +MANIFEST semantic-policy bit 0 selects fast_math. Every other policy bit is +zero. binding_count is 0 through 4096. Encoder slots are dense from zero to +binding_count minus one. Every declared slot occurs at least once and every +PUSH_VARIABLE slot is below binding_count. + +Semantic stack depth starts at zero and may not exceed 1024. PUSH at depth 1024 +rejects. CALL requires at least its fixed arity and changes depth to +depth - arity + 1. END requires depth one. + +The manifest fast_math bit is the effective protected fast-math value and does +not mutate stored evaluator options. prefer_x87 and existing libm-selection +options come from a snapshot of evaluator options. enable_cse must be false; +protected load rejects when it is true. + +Literal records must contain finite values. Runtime operations may still +produce non-finite results according to existing MEXCE semantics. + +### Stable operation IDs + +| ID | Operation | Arity | Public meaning | +|---:|---|---:|---| +| 1 | ADD | 2 | a + b | +| 2 | SUB | 2 | a - b | +| 3 | MUL | 2 | a * b | +| 4 | DIV | 2 | a / b | +| 5 | NEG | 1 | -a | +| 6 | POW | 2 | pow(a, b) | +| 7 | SIN | 1 | sin(a) | +| 8 | COS | 1 | cos(a) | +| 9 | TAN | 1 | tan(a) | +| 10 | ABS | 1 | abs(a) | +| 11 | SIGN | 1 | -1, 0, or 1 according to the sign of a | +| 12 | SIGNP | 1 | 1 when a is positive, otherwise 0 | +| 13 | EXPN | 1 | binary exponent returned by MEXCE expn | +| 14 | SFC | 1 | binary significand returned by MEXCE sfc | +| 15 | SQRT | 1 | sqrt(a) | +| 16 | EXP | 1 | exp(a) | +| 17 | LT | 2 | a < b | +| 18 | GT | 2 | a > b | +| 19 | LE | 2 | a <= b | +| 20 | GE | 2 | a >= b | +| 21 | EQ | 2 | a == b | +| 22 | NE | 2 | a != b | +| 23 | LOG | 1 | natural logarithm | +| 24 | LOG2 | 1 | base-two logarithm | +| 25 | LOG10 | 1 | base-ten logarithm | +| 26 | LOGB | 2 | logarithm of b in base a | +| 27 | YLOG2 | 2 | a * log2(b) | +| 28 | MAX | 2 | max(a, b) | +| 29 | MIN | 2 | min(a, b) | +| 30 | FLOOR | 1 | MEXCE floor(a) | +| 31 | CEIL | 1 | MEXCE ceil(a) | +| 32 | ROUND | 1 | MEXCE round(a) | +| 33 | INT | 1 | MEXCE int(a) | +| 34 | TRUNC | 1 | MEXCE trunc(a) | +| 35 | MOD | 2 | fmod(a, b) | +| 36 | BND | 2 | a modulo b, adjusted into [0, b) for positive b | +| 37 | BIAS | 2 | a / (((1 / b) - 2) * (1 - a) + 1) | +| 38 | GAIN | 2 | MEXCE gain(a, b) tone-mapping curve | + +For finite nonzero a, EXPN is the exponent from frexp(a) minus one and SFC is +a divided by 2 raised to EXPN. GAIN is: + +~~~ +a / (((1 / b) - 2) * (1 - 2*a) + 1) when a < 0.5 + +(((1 / b) - 2) * (1 - 2*a) - a) / +(((1 / b) - 2) * (1 - 2*a) - 1) otherwise +~~~ + +Exceptional floating-point behavior, rounding details, and libm selection are +the existing public MEXCE semantics for the selected backend. Clear and +protected paths use the same operation prototypes and parity tests. A change +that alters a public operation contract cannot silently reinterpret an existing +wire ID. + +Source aliases map as follows: + +- infix +, -, *, / map to ADD, SUB, MUL, DIV; +- ^ and ** map to POW; +- infix < and > map to LT and GT; LE, GE, EQ, and NE retain their function + spellings; +- log and ln both map to LOG; +- built-in pi and e become PUSH_LITERAL_F64 records; and +- function spellings otherwise map to the same-named operation. + +The clear semantic representation retains clear-only spelling metadata where +needed for get_optimized_expression. That metadata is not a wire field and is +absent from protected compiler state. + +### Canonical production + +The issuer producer: + +- resolves aliases before encoding; +- emits exact literal bits rather than decimal spelling; +- preserves current postfix operand order; +- emits source unary minus as positive-zero, operand, and SUB records, preserving + current precedence and signed-zero behavior; unary plus emits no operation; +- maps explicit neg(a) to NEG; +- assigns slots from the explicit schema, never container iteration; +- emits no optimizer-created node; +- emits exactly one END; and +- writes every reserved field as zero. + +Backend optimization and commutative reordering remain runtime compiler work. + +## Public API + +### Protected_expression_key + +Protected_expression_key is a move-only owner of exactly +crypto_secretstream_xchacha20poly1305_KEYBYTES bytes. + +The public construction surface is: + +~~~cpp +static Protected_expression_key from_bytes( + const uint8_t* bytes, + size_t size); +~~~ + +The factory rejects a null pointer and every size other than 32 bytes. It copies +the bytes into a guarded allocation; the caller remains responsible for wiping +its input buffer. The type has no public default constructor. + +Key storage uses sodium_malloc followed by an explicit checked sodium_mlock. +sodium_malloc alone is insufficient because libsodium may return an unlocked +allocation when its internal lock attempt fails. Allocation or lock failure +wipes and frees any allocation and throws RESOURCE_FAILURE. Memory locking is +defense in depth and does not protect registers, stack spills, or hostile +process inspection. + +The type: + +- is movable and not copyable; +- leaves the source empty after a move; +- wipes and frees owned bytes on destruction; +- exposes no string, hex, comparison, logging, or general serialization API; +- rejects use after move or consumption; and +- grants only the encoder and runtime loader narrow internal byte access. + +Issuer code can transfer the key to host storage with: + +~~~cpp +template +void consume_bytes(Consumer&& consumer); +~~~ + +The callback receives const uint8_t* and size_t for the dynamic extent of the +call. Ownership moves to a private local guard before user code runs. Normal +return and exception unwinding both wipe and empty the key; a callback exception +is rethrown after wiping. Recursive consumption, moving the original object +inside the callback, and a second consumption observe an empty key. + +### Protected_expression_bundle + +Protected_expression_bundle contains: + +~~~cpp +std::vector program; +Protected_expression_key key; +std::array program_id; +~~~ + +Because it owns a key, the bundle is movable and not copyable. It performs no +implicit file output. + +### Protected_binding + +Protected_binding is issuer-side schema data containing a source variable name +and uint32_t slot. + +Names use the exact ASCII grammar: + +~~~ +[A-Za-z_][A-Za-z0-9_]* +~~~ + +Names are at most 255 bytes and cannot equal a built-in constant, function, or +source alias. Names and slots are unique. Slots are dense from zero. The schema +contains exactly the variables referenced by the source. + +The encoder limits are: + +- source: at most 1 MiB; +- bindings: at most 4096; +- one name: at most 255 bytes; +- cumulative binding-name bytes: at most 256 KiB; and +- semantic records including MANIFEST and END: at most 16384. + +Limits are checked before parsing, random generation, or storage proportional +to untrusted sizes. + +### Encoder + +The issuer API is: + +~~~cpp +enum class Protected_math_mode +{ + STRICT = 0, + FAST = 1, +}; + +Protected_expression_bundle encode_protected_expression( + const std::string& expression, + const std::vector& bindings, + Protected_math_mode math_mode); +~~~ + +STRICT clears manifest policy bit 0. FAST sets it. Invalid enum values reject +before parsing or random generation. + +Syntax errors use mexce_parsing_exception and may include trusted issuer source +content or identifiers. Schema, limit, initialization, allocation, and encoding +failures use Protected_expression_error. The function never returns a partial +bundle. + +### Runtime binding + +The runtime binding surface is: + +~~~cpp +template +void bind_protected(T& referenced_variable, uint32_t slot); + +void unbind_protected(uint32_t slot); +void unbind_all_protected(); +~~~ + +Supported T is double, float, int16_t, int32_t, or int64_t. A slot above 4095 +rejects at bind time. The artifact authenticates slot numbers, not the C++ type +or address of a runtime binding. + +Caller-owned bound objects must not move and must outlive their binding and +every evaluation that uses it. + +Binding behavior is: + +- binding an unused existing slot replaces that slot; +- rebinding a slot referenced by the compiled protected expression rejects + without changing state; +- unbinding an unknown slot rejects; +- unbinding a referenced slot first invalidates the compiled expression, then + removes the binding; +- unbind_all_protected invalidates a protected expression that references any + slot, then removes all protected bindings; +- extra runtime bindings not referenced by an artifact are permitted and + ignored; and +- clear name bindings and protected slot bindings occupy independent maps. + +### Runtime load + +The runtime API is: + +~~~cpp +void set_protected_expression( + const uint8_t* program, + size_t program_size, + Protected_expression_key key); +~~~ + +Passing the key by value makes ownership transfer mandatory because copying is +deleted. Every attempted call consumes the transferred key on success or +failure. The key is erased immediately after init_pull derives stream state. +The stream state then owns the decryption capability and is wiped on every exit. + +program may be null only when program_size is zero; that input still rejects as +empty. Program bytes remain caller-owned and are not retained. + +On success: + +- evaluate executes the protected expression; +- get_backend reports the selected existing backend; +- get_optimized_expression and get_byte_representation throw + INTROSPECTION_DISABLED; +- no key, stream state, wire record, or program pointer remains in evaluator; + and +- every code-addressed compiler object remains owned until code is no longer + callable. + +On failure, evaluator is empty and all transferred key, stream, record, +compiler, optimizer, executable, constant-result, and referenced-binding state +has been reset or wiped as applicable. + +### Protected errors + +Protected_expression_error provides: + +~~~cpp +Protected_expression_error_category category() const noexcept; +bool has_record_index() const noexcept; +uint32_t record_index() const; +~~~ + +record_index throws std::logic_error when no index is available. + +Categories are: + +- INVALID_ARGUMENT; +- UNSUPPORTED_FORMAT; +- SIZE_LIMIT; +- AUTHENTICATION_FAILED; +- MALFORMED_PROGRAM; +- MISSING_BINDING; +- INTROSPECTION_DISABLED; +- CRYPTOGRAPHY_UNAVAILABLE; +- COMPILATION_FAILED; and +- RESOURCE_FAILURE. + +Runtime decode and authentication messages contain no key bytes, literal bits, +slot values, operation names, or decrypted content. Wrong key and tampering are +not distinguished. Issuer parsing diagnostics are not subject to the runtime +content restriction. + +| Failure | Category | Record index | +|---|---|---| +| invalid runtime argument or protected CSE | INVALID_ARGUMENT | none | +| bad magic; unsupported version; header-size or record-size field; non-zero public flags | UNSUPPORTED_FORMAT | none | +| declared resource limit exceeded | SIZE_LIMIT | none before frames; current index after authentication | +| init_pull rejects a syntactically sized secretstream header | AUTHENTICATION_FAILED | none | +| pull authentication failure | AUTHENTICATION_FAILED | current index | +| record count below three; exact-size mismatch; zero program ID | MALFORMED_PROGRAM | none | +| authenticated semantic or framing violation | MALFORMED_PROGRAM | current index | +| a referenced declared slot has no runtime binding | MISSING_BINDING | first PUSH_VARIABLE index for that slot | +| protected introspection | INTROSPECTION_DISABLED | none | +| sodium_init failure | CRYPTOGRAPHY_UNAVAILABLE | none | +| valid semantic state rejected by optimizer or selected backend | COMPILATION_FAILED | none | +| allocation, memory protection, or OS resource failure | RESOURCE_FAILURE | none | + +With libsodium 1.0.22, same-size secretstream-header mutations normally pass +init_pull and fail pull at frame zero. The documented init_pull failure path is +still supported and tested through the cryptographic-call boundary. + +## Compiler invariants + +### Shared semantic boundary + +Clear parsing and issuer parsing share one semantic producer. Clear and +protected compilation share one semantic consumer. The consumer validates stack +effects, constructs compiler objects, links arguments, optimizes, selects a +backend, and finalizes executable memory. + +The protected route never calls the lexical producer. The clear route never +calls cryptography. There is no serialized clear semantic vector between the +clear producer and consumer. + +### Effective options + +Each compilation captures one effective-options context used by parsing, +normalization, CSE, fast-math passes, constant folding, compatibility checks, +fallback, and emission. + +- Clear compilation copies all stored evaluator options. +- Protected compilation takes fast_math from the authenticated manifest, + prefer_x87 and libm choices from stored options, and rejects enable_cse. +- No compilation mutates stored evaluator options, including fallback and + exception paths. + +### Primary and fallback ownership + +Optimizers may mutate their input. A fallback compilation is created from +source-free semantic state before destructive mutation can make reconstruction +unsafe. Primary and fallback compilations share no mutable compiler objects or +wipeable literal storage. + +One per-compilation lifetime owner contains: + +- the semantic/compiler graph; +- every intermediate constant whose address may enter native code; +- CSE value storage where applicable; +- optimizer-owned values such as power terms; +- every other address-emitted object; and +- the executable buffer metadata. + +The selected owner outlives its executable code. Replacement and destruction +make code uncallable before wiping or releasing any code-addressed storage. +Unused owners are wiped and released immediately. + +The fallback route runs the same complete x87 pipeline required by the clear +semantics, including CSE when enabled for clear input. It does not reparse +source and does not copy a graph whose nodes share mutable objects. + +### Protected data handling + +Protected compiler objects do not retain source names or reconstructable debug +strings. Backend compatibility uses semantic state, never the presence of debug +text. Clear-only spelling and debug descriptions remain available only to clear +introspection. + +Every protected literal or derived scalar held in: + +- the selected or unused graph; +- intermediate-constant storage; +- optimizer containers; +- constant-expression result storage; +- power-term storage; or +- temporary fallback state + +is explicitly wiped before its storage is released. Protected literal bits are +not used as immutable container keys in storage that cannot be wiped. + +Native code is never writable and executable simultaneously. Where the platform +permits a safe writable transition during teardown, code bytes are wiped before +release. Failure to make an executable page writable during noexcept teardown +does not terminate or throw; native-code wiping is best effort. Key, stream, +record, and protected scalar wiping is mandatory. + +## Cryptographic lifecycle + +### Initialization + +Protected public operations use a process-wide C++11 thread-safe sodium_init +gate. A return value below zero is cached as failure and every protected call +hard-fails. Normal MEXCE paths do not initialize sodium. + +### Randomness and key separation + +Artifact keys use crypto_secretstream_xchacha20poly1305_keygen. Program IDs use +randombytes_buf and retry the all-zero value. Secretstream generates its own +header. No standard-library PRNG, timestamp, or deterministic seed participates. + +The canonical encoder always generates the key internally and performs one +encode operation. There is no deterministic or existing-key encoder. + +### Authentication before use + +Before the first successful pull, only bounded public framing fields may be +interpreted. A record is applied to temporary compiler state only after pull +authenticates it. Earlier authenticated records may populate unpublished state +before a later failure, but no native program is published until final +authentication and compilation complete. + +### Erasure + +sodium_memzero or an equivalently non-optimizable wrapper wipes: + +- all key copies; +- secretstream push and pull states; +- each clear wire record; +- partially built issuer records; +- temporary authenticated-data buffers containing derived framing; +- protected compiler and optimizer scalar storage; and +- failure-path storage containing decoded values. + +sodium_free wipes guarded key allocations. Erasure does not claim to remove +copies from registers, compiler spills, crash dumps, swap, or a hostile +debugger. Core-dump, swap, and hibernation policy belong to the host. + +## Build, packaging, and platform contract + +### Supported release matrix + +Protected format 1.0 release gates are: + +- Windows x64 with MSVC; +- Linux x64 with GCC; and +- Linux x64 with Clang. + +Both SSE2 and x87 paths are exercised on x64. Ordinary MEXCE x86 support remains +unchanged, but protected x86 headers/packages are not advertised. + +Cross-platform portability means the same artifact, key, bindings, and policy +are accepted on Windows and Linux. Results follow existing backend and libm +semantics; bit-identical results across different C libraries or x87/SSE2 +precision are not promised. + +### Header and ODR boundary + +mexce.h contains the unconditional sodium-free declarations and evaluator state +needed by both normal and protected translation units. The evaluator definition, +layout, constructor, destructor, and ordinary inline methods never depend on +MEXCE_ENABLE_PROTECTED_EXPRESSIONS or on which protected header was included. + +mexce_protected.h includes mexce.h and supplies protected key, error, codec, and +inline runtime definitions that call libsodium. mexce_protected_encoder.h adds +issuer encoding. No sodium header or symbol is reachable merely by including +mexce.h. + +Protected compilation state must be destructible through the normal evaluator +lifecycle without requiring a sodium reference from a translation unit that +uses only mexce.h. Private type erasure is permitted, but no particular private +mechanism is required. + +### CMake targets + +The final source-build surface is: + +- mexce::mexce: header-only ordinary MEXCE with no cryptographic dependency; +- mexce::protected: header-only protected surface, transitively linking + libsodium; +- protected unit, benchmark, and fuzz targets when protected support is + enabled; and +- mexce_protect when issuer tools are enabled. + +MEXCE_ENABLE_PROTECTED_EXPRESSIONS is the only protected source-build option. +It defaults OFF. Enabling it without a supported libsodium target is a configure +error. It controls build targets and installation content, never C++ class +layout. + +CMake normalizes supported dependency providers behind one private adapter: + +- Conan target libsodium::libsodium; +- vcpkg target unofficial-sodium::sodium; or +- a validated pkg-config imported target on Linux. + +The installed mexce::protected configuration recreates the adapter and fails +clearly when libsodium is absent. mexce::mexce has no transitive sodium include, +compile definition, symbol, or link item. + +### Conan + +Conan adds with_protected, default false. The protected package requires +libsodium/1.0.22 and exports mexce::protected. Package identity retains the +option and dependency; protected and ordinary packages cannot collapse to the +same ID. + +The repository recipe packages exported candidate headers and build metadata, +not a previously released archive. Its test package builds ordinary and +protected consumers from that candidate package. + +### vcpkg + +The port adds a protected feature depending on libsodium and installs the +protected headers and CMake target. Default installation remains +dependency-free. + +Candidate testing uses a transient overlay whose source path is the candidate +tree rather than the tagged archive. The installed header hashes and target +files are compared with the candidate. Release packaging separately validates +the final archive reference and checksum. + +## Issuer utility + +mexce_protect is an issuer tool, not a licence system. + +Inputs are: + +- expression file; +- binding-schema file; +- output program path; and +- output key path. + +The expression file is at most 1 MiB. The tool removes one terminal LF and an +optional preceding CR; it performs no other whitespace normalization. NUL bytes +and a UTF-8 BOM reject. Non-ASCII source is handled by the existing parser and +does not extend the source grammar. + +The schema is at most 256 KiB and contains one name=decimal_slot entry per line. +Blank lines, comments, NUL bytes, invalid UTF-8, duplicate names, duplicate +slots, sparse slots, oversized names, and trailing characters reject. + +Output paths are compared after absolute platform-appropriate normalization. +They must identify distinct paths. Existing destinations, symlinks, +reparse-point traversal, or indeterminate path identity hard-fail. + +The tool publishes: + +- opaque program bytes; and +- exactly 32 raw key bytes. + +Publication invariants are: + +- temporary files are exclusive-created siblings of final paths on the same + filesystem or volume; +- both temporaries are fully written and flushed before publication; +- final paths are never overwritten; +- the key file receives owner-only permissions before publication; +- the key is published first and the program last; +- ordinary failure before publication removes temporaries; +- failure after key publication can leave an orphan key but never a program + published by that invocation without its key; +- a later invocation detects and reports partial final state and never guesses + or overwrites; and +- machine or filesystem power-loss atomicity is not claimed. + +The implementation uses the platform's atomic no-replace primitive but does not +expose that primitive in the public utility contract. The tool consumes +Protected_expression_key through consume_bytes and never prints source or key +content. + +Documentation instructs issuers to import the raw key immediately into the host +wrapping system and remove the caller-owned final key file. + +## Verification contract + +### Authorities + +Correctness derives from: + +- the byte-exact wire contract; +- libsodium's documented secretstream behavior; +- existing adopted clear-expression semantics; +- direct mathematical definitions for semantic operations; and +- independently authored semantic fixtures. + +Encoder/decoder round trips are necessary but not sufficient because both sides +could share the same wrong ID, arity, operand order, or framing rule. + +### Clear semantic refactor + +Verify: + +- all existing unit tests and expression-corpus results on Windows and Linux; +- exact backend selection for existing SSE2 and x87 cases; +- signed-zero behavior for source unary minus and explicit neg; +- source-free SSE2-to-x87 fallback after an optimizer makes the primary graph + incompatible; +- CSE-enabled fallback using an independent pre-mutation owner; +- log and ln clear introspection spelling; +- operation, optimized-expression, and numeric parity with the pre-refactor + clear evaluator; +- failed replacement after a constant expression; +- failed replacement after a native expression; and +- empty-state evaluate, backend, binding-reference, executable, constant, and + optimizer invariants. + +Machine-code byte snapshots are not an oracle because addresses and platform +code generation are unstable. + +### Independent format fixtures + +A test-only reference producer: + +- defines wire constants independently from production headers; +- accepts hand-authored semantic records; +- uses libsodium directly; +- emits fixed-key binary fixtures and a human-readable manifest; and +- is not linked into production. + +Production tests consume checked-in fixtures. Regeneration is explicit. +Duplicated constants are confined to this independent boundary. + +### Cryptographic and format checks + +Verify: + +- known fixture and matching key succeed; +- wrong key rejects; +- public structural-field mutations map to their exact format category; +- secretstream-header mutation with real libsodium fails authentication; +- injected init_pull failure reports no index; +- every ciphertext and tag byte mutation rejects; +- frame swap, substitution, and reordering reject at pull; +- insertion, removal, truncation, and trailing bytes reject exact framing; +- compensated framing changes that reach pull fail authentication; +- final-tag and END mismatches reject; +- every reserved or unused byte is enforced; and +- the transferred runtime key is empty after successful and failing calls. + +### Semantic checks + +Cover: + +- every operation ID, arity, alias, and operand order; +- finite literal boundary values and signed zero; +- every supported bound type; +- repeated, missing, extra, and multiple slots; +- strict and fast-math policy; +- protected CSE rejection; +- prefer_x87 and libm option preservation; +- constant-only programs; +- SSE2 and x87 results; +- source-free post-optimization fallback; +- protected compilation with caller buffers destroyed afterward; and +- Windows-produced artifacts on Linux and Linux-produced artifacts on Windows. + +Hand-authored records exercise semantic failures without using the production +encoder. + +### Failure and resource checks + +Table-drive every error-mapping row, including: + +- empty and short input; +- record, source, schema, name, slot, and semantic-depth limits; +- malformed public fields and zero program ID; +- misplaced MANIFEST or END; +- unknown records and operations; +- non-zero reserved and unused fields; +- invalid policy and non-finite literal; +- stack underflow and invalid terminal depth; +- missing runtime bindings; +- allocation and executable-finalization failures; +- failed replacement from constant and native states; +- guarded allocation and explicit mlock failure; and +- absent record_index behavior. + +sodium_init injected failure runs in an isolated process because initialization +state is process-wide and cached. + +### Lifetime and erasure checks + +A narrow test-only wipe observer verifies wipe requests for key, stream, record, +protected scalar, constant-only result, optimizer, fallback, replacement, and +destruction paths. + +Tests also verify: + +- one-shot key consumption, move ownership, callback exception unwinding, + recursive consumption rejection, and moved-from rejection; +- code-addressed constants and CSE storage remain alive while code is callable; +- code becomes uncallable before protected storage is wiped; +- protected introspection rejects without constructing protected debug text; +- runtime errors contain no decoded content; +- the protected path never calls the lexical parser; and +- writable-executable memory never exists. + +### Fuzzing + +One bounded decoder fuzz target uses arbitrary bytes and a fixed non-secret key. +A structured mutator independently re-encrypts altered semantic records so +semantic validation is reachable instead of stopping every case at +authentication. + +The invariant is no crash, hang, out-of-bounds access, undefined behavior, or +acceptance without a complete valid authenticated stream. Fuzzing runs under +Clang sanitizers in a dedicated job. + +### ODR and dependency isolation + +Build: + +- one translation unit including only mexce.h; +- one translation unit including mexce_protected.h; +- an evaluator created on one side and destroyed or used on the other; and +- an ordinary installed consumer in an environment where sodium is not linked. + +Inspect the ordinary consumer's link interface and binary symbols to confirm no +sodium dependency. Build a protected installed consumer and verify the +normalized transitive dependency. + +### Performance and resource behavior + +The semantic refactor compares Release baseline and candidate builds using the +existing corpus and identical compiler flags. Five measured runs alternate +baseline and candidate after a discarded warm-up. The decision value is the +median. The noise band is the greater of 2 percent or three times baseline +median absolute deviation divided by baseline median. + +Acceptance for clear behavior: + +- evaluate regression does not exceed the noise band; +- clear compile regression does not exceed 5 percent; and +- backend selection and correctness counts are unchanged. + +Protected benchmarks run clear and protected compilation in isolated processes +with matched formulas, bindings, and options. They report per-expression +compile ratio, evaluate timing, and peak working set as structured output. + +Acceptance for protected behavior: + +- protected/clear compile ratio at the 95th percentile is at most 10; +- protected evaluate performance does not regress beyond the noise band from + matched clear native code; and +- maximum-size valid and late-invalid artifacts complete in an isolated + process within the broad 2-second and 128-MiB safety fence. + +No native code-size measurement is required unless a harness that measures it +is explicitly added to the affected batch. + +## Implementation batches + +### Batch 1: shared semantic compilation and lifecycle + +Outcome: + +- one canonical semantic producer/consumer boundary; +- clear compilation routed through that boundary; +- one complete per-compilation lifetime owner; +- source-free primary/fallback ownership; +- per-compilation effective options; and +- defined empty-state behavior after failed replacement. + +Primary write scope: + +- mexce.h +- test/unit_tests.cpp + +Deletion points: + +- direct Token-to-Element construction inside set_expression; +- m_last_expression and recursive source reparse; +- any plaintext set_protected_expression surface or scaffolding present in the + implementation base; +- compiler decisions that depend on debug-string presence; and +- newly orphaned source-name conversion or fallback helpers. + +Gates: + +- clear semantic-refactor checks; +- failed-replacement and empty-state checks; +- primary/fallback independence and complete lifetime ownership checks; +- Windows and Linux unit/corpus runs; and +- clear performance acceptance. + +Owned risks: + +- CSE mutation leaking between primary and fallback owners; +- code-addressed constants or CSE storage outliving the wrong owner; +- clear optimized-expression spelling drift; and +- backend selection drift after removing debug-string signals. + +### Batch 2: protected key and wire codec + +Dependency: shared semantic compilation and lifecycle. + +Outcome: + +- protected key ownership and secure allocation; +- byte-exact format 1.0 encoder and bounded decoder; +- protected errors and complete category/index mapping; +- final protected CMake build option for codec tests; and +- independent format fixtures. + +Affected surfaces: + +- protected public headers; +- protected CMake test targets; +- key, codec, and fixture tests; and +- independent reference fixtures. + +The decoder produces validated semantic records for an internal sink but does +not publish executable evaluator state. + +Gates: + +- independent fixture checks; +- cryptographic and format mutation matrix; +- key move, consume, mlock, initialization, and erasure checks; +- encoder resource and schema limits; +- error category/index matrix; and +- Windows MSVC plus Linux GCC/Clang codec builds against libsodium 1.0.22. + +Owned risks: + +- secretstream init versus pull error classification; +- unauthenticated allocation or semantic interpretation; +- accidental key reuse or retained key copies; +- immutable storage containing protected literal bits; and +- production and reference codecs sharing constants or helpers. + +### Batch 3: protected evaluator integration + +Dependency: protected key and wire codec. + +Outcome: + +- canonical runtime binding and load APIs; +- source-free streaming compilation; +- authenticated option precedence; +- protected introspection behavior; +- complete protected lifecycle and wipe behavior; and +- cross-platform artifact portability. + +Affected surfaces: + +- mexce.h sodium-free protected declarations/state; +- mexce_protected.h runtime definitions; +- protected runtime tests; +- protected benchmark; and +- decoder fuzz target. + +Gates: + +- semantic, failure, lifetime, erasure, and introspection checks; +- mixed-translation-unit ODR test; +- ordinary no-sodium link and symbol check; +- ASan/UBSan protected tests on Linux; +- focused Windows heap diagnostics; +- structured decoder fuzz smoke; +- Windows/Linux cross-produced artifacts; and +- protected performance and resource acceptance. + +Owned risks: + +- macro-conditioned evaluator layout; +- sodium symbols leaking into ordinary consumers; +- incomplete compilation-state ownership; +- protected debug-string construction; +- stored options being mutated during protected compilation; and +- failure publishing partial code or leaving stale callable state. + +### Batch 4: install and package integration + +Dependency: protected evaluator integration. + +Outcome: + +- installed mexce::mexce and mexce::protected targets; +- final libsodium dependency normalization; +- Conan ordinary and protected packages with distinct identities; and +- vcpkg default and protected feature packages. + +Affected surfaces: + +- CMake install/export configuration; +- Conan recipe and test package; +- vcpkg port metadata and feature definition; and +- installed-consumer tests. + +Gates: + +- dependency-free default configure, build, install, and consumer; +- protected configure, build, install, and consumer on every release compiler; +- Conan candidate-source ordinary and protected consumers; +- vcpkg candidate-source ordinary and protected consumers; +- installed file/hash provenance from candidate source; +- missing-libsodium hard failure for protected consumers; and +- licence and third-party notice checks. + +Owned risks: + +- candidate package tests resolving an older release archive; +- protected and ordinary Conan package IDs collapsing; +- source-tree include fallback hiding broken install exports; and +- provider-specific sodium target names leaking into the public MEXCE target. + +### Batch 5: issuer utility, documentation, and continuous gates + +Dependency: install and package integration. + +Outcome: + +- mexce_protect with safe no-overwrite publication; +- public protected usage and threat-model documentation; +- protected example; +- release-platform CI; and +- scheduled sanitizer/fuzz coverage. + +Affected surfaces: + +- issuer utility and utility tests; +- README, example, changelog, and notices; +- Windows/Linux workflows; and +- protected fuzz workflow. + +Gates: + +- issuer input and publication invariants on Windows and Linux; +- output permissions and partial-publication recovery; +- end-to-end utility output consumed through the installed protected target; +- documented commands executed in a clean consumer; +- default and protected release matrices; and +- scheduled extended fuzz run. + +Owned risks: + +- overwriting or following attacker-controlled paths; +- publishing a program without its key; +- leaving raw key files with broad permissions; +- documentation overstating local-process secrecy; and +- CI exercising source-tree headers instead of installed packages. + +## Completion criteria + +The protected capability is complete when: + +- format 1.0 bytes, semantic IDs, APIs, ownership, errors, limits, and security + claims are implemented as specified; +- normal MEXCE remains dependency-free and ODR-safe; +- clear compilation retains adopted behavior and satisfies its performance + gates; +- protected compilation passes format, semantic, mutation, lifetime, erasure, + fuzz, portability, and resource gates; +- installed CMake, Conan, and vcpkg consumers use candidate artifacts; +- the issuer utility satisfies publication and permission invariants; and +- no plaintext protected overload, duplicate compiler path, temporary build + switch, source-reparse fallback, or process-only artifact remains. From ea7c1ce32dbe576b9f99cb8cb70ebb502041d518 Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 11:29:11 +0200 Subject: [PATCH 02/12] Defer performance acceptance until stabilization --- analysis/keyed_expression_protection_plan.md | 87 ++++++++++++++------ 1 file changed, 62 insertions(+), 25 deletions(-) diff --git a/analysis/keyed_expression_protection_plan.md b/analysis/keyed_expression_protection_plan.md index 74ded72..ba9869d 100644 --- a/analysis/keyed_expression_protection_plan.md +++ b/analysis/keyed_expression_protection_plan.md @@ -1090,32 +1090,24 @@ normalized transitive dependency. ### Performance and resource behavior -The semantic refactor compares Release baseline and candidate builds using the -existing corpus and identical compiler flags. Five measured runs alternate -baseline and candidate after a discarded warm-up. The decision value is the -median. The noise band is the greater of 2 percent or three times baseline -median absolute deviation divided by baseline median. - -Acceptance for clear behavior: - -- evaluate regression does not exceed the noise band; -- clear compile regression does not exceed 5 percent; and -- backend selection and correctness counts are unchanged. +Release baseline and candidate builds use the existing corpus, identical +compiler flags, discarded warm-ups, alternating run order, and enough repeated +runs to distinguish persistent movement from system noise. Reports retain the +per-run values, medians, dispersion, backend selection, and correctness counts. +During functional implementation these measurements are diagnostic: batches +1 through 5 record changes and suspected causes but do not use fixed percentage +limits as closure gates. Protected benchmarks run clear and protected compilation in isolated processes with matched formulas, bindings, and options. They report per-expression compile ratio, evaluate timing, and peak working set as structured output. - -Acceptance for protected behavior: - -- protected/clear compile ratio at the 95th percentile is at most 10; -- protected evaluate performance does not regress beyond the noise band from - matched clear native code; and -- maximum-size valid and late-invalid artifacts complete in an isolated - process within the broad 2-second and 128-MiB safety fence. +Maximum-size valid and late-invalid artifacts also report elapsed time and peak +working set from an isolated process. Resource failures remain subject to the +specified bounded-input and cleanup behavior even while timing and memory +measurements are diagnostic. No native code-size measurement is required unless a harness that measures it -is explicitly added to the affected batch. +is explicitly added. ## Implementation batches @@ -1150,7 +1142,7 @@ Gates: - failed-replacement and empty-state checks; - primary/fallback independence and complete lifetime ownership checks; - Windows and Linux unit/corpus runs; and -- clear performance acceptance. +- clear performance diagnostics with backend and correctness counts. Owned risks: @@ -1228,7 +1220,7 @@ Gates: - focused Windows heap diagnostics; - structured decoder fuzz smoke; - Windows/Linux cross-produced artifacts; and -- protected performance and resource acceptance. +- protected compile, evaluate, and isolated-resource diagnostics. Owned risks: @@ -1300,7 +1292,9 @@ Gates: - end-to-end utility output consumed through the installed protected target; - documented commands executed in a clean consumer; - default and protected release matrices; and -- scheduled extended fuzz run. +- scheduled extended fuzz run; and +- complete-system clear and protected performance diagnostics on release + configurations. Owned risks: @@ -1310,6 +1304,48 @@ Owned risks: - documentation overstating local-process secrecy; and - CI exercising source-tree headers instead of installed packages. +### Batch 6: performance stabilization and release acceptance + +Dependency: issuer utility, documentation, and continuous gates. + +Outcome: + +- release performance characterized from the complete implementation; +- causal regressions identified with profiles rather than timing deltas alone; +- targeted optimizations applied without weakening semantic, cryptographic, + lifecycle, packaging, or portability guarantees; and +- release acceptance based on finished-system workload evidence. + +Affected surfaces: + +- clear and protected benchmark harnesses and structured reports; +- implementation paths demonstrated by profiles to cause material regressions; + and +- release performance documentation and continuous performance jobs. + +Gates: + +- matched baseline and candidate measurements across supported release + configurations, with warm-up, alternating order, per-run results, medians, + and dispersion retained; +- clear compile and evaluate profiles, protected encode/load profiles, and + maximum-size valid and late-invalid resource profiles; +- causal explanation for every optimization and before/after evidence on the + workload that exposed it; +- all functional, security, lifecycle, portability, and packaging gates affected + by an optimization rerun after the change; and +- release acceptance limits selected from complete-system distributions, + workload requirements, platform noise, and resource-safety evidence rather + than preassigned percentages. + +Owned risks: + +- optimizing an intermediate architecture that later batches replace; +- treating measurement noise as a product regression; +- improving aggregate timing while worsening tail or maximum-size behavior; +- duplicating clear and protected compiler paths for speed; and +- trading away cleanup, authentication, ownership, or portability guarantees. + ## Completion criteria The protected capability is complete when: @@ -1317,11 +1353,12 @@ The protected capability is complete when: - format 1.0 bytes, semantic IDs, APIs, ownership, errors, limits, and security claims are implemented as specified; - normal MEXCE remains dependency-free and ODR-safe; -- clear compilation retains adopted behavior and satisfies its performance - gates; +- clear compilation retains adopted behavior; - protected compilation passes format, semantic, mutation, lifetime, erasure, fuzz, portability, and resource gates; - installed CMake, Conan, and vcpkg consumers use candidate artifacts; - the issuer utility satisfies publication and permission invariants; and +- the complete implementation satisfies Batch 6 release performance and + resource acceptance; and - no plaintext protected overload, duplicate compiler path, temporary build switch, source-reparse fallback, or process-only artifact remains. From 04e656e27e7c0d3d8414ef502f2f7bfa2435f2b3 Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 11:56:14 +0200 Subject: [PATCH 03/12] Route clear compilation through semantic IR --- mexce.h | 757 ++++++++++++++++++++++++++++++++++++-------- test/unit_tests.cpp | 150 ++++++++- 2 files changed, 771 insertions(+), 136 deletions(-) diff --git a/mexce.h b/mexce.h index 93ef999..4e38ee4 100644 --- a/mexce.h +++ b/mexce.h @@ -211,6 +211,11 @@ namespace impl { struct Variable; struct Function; struct mexce_charstream; + class Compilation_state; + class Semantic_graph_builder; + class Semantic_compiler; + class Clear_semantic_producer; + struct Semantic_compilation_result; using std::abs; using std::deque; @@ -329,30 +334,25 @@ class evaluator * @brief Returns which backend was actually used for the current expression. * This may differ from the requested backend if fallback was needed. */ - backend_type get_backend() const { return m_backend_used; } + backend_type get_backend() const; private: options m_options; - - bool is_constant_expression = false; - double constant_expression_value = 0.0; - bool m_sse2_simplify_mode = false; // When true, asmd_optimizer reconstructs elist instead of x87 code - backend_type m_backend_used = backend_type::none; - size_t m_buffer_size = 0; - std::string m_expression; - std::string m_last_expression; // Saved for potential SSE2->x87 fallback re-parsing - impl::elist_t m_elist; - std::list m_intermediate_code; - impl::constant_map_t m_intermediate_constants; // produced during expression simplification impl::variable_map_t m_variables; impl::constant_map_t m_constants; - uint64_t m_next_element_id = 0; - std::list m_cse_temps; // Storage for common subexpressions - std::map> m_power_terms; // pow-optimized kernel/exponent for nested folding - - double (*evaluate_fptr)() = nullptr; + uint64_t m_next_variable_id = 2; + std::unique_ptr m_compilation; + impl::Compilation_state* m_active_compilation = nullptr; + bool m_is_constant_expression = false; + double m_constant_expression_value = 0.0; + double (*m_evaluate_fptr)() = nullptr; + backend_type m_backend_used = backend_type::none; void compile_and_finalize_elist(impl::elist_const_it_t first, impl::elist_const_it_t last); + impl::Semantic_compilation_result finish_semantic_compilation( + std::unique_ptr candidate, + std::vector> referenced_variables); + impl::Compilation_state& active_compilation(); // Grant friend access to helpers that need private state. friend std::shared_ptr impl::make_intermediate_constant(evaluator* ev, double v); @@ -361,6 +361,8 @@ class evaluator friend void impl::pow_optimizer(impl::elist_it_t it, evaluator* ev, impl::elist_t* elist); friend uint8_t* impl::push_intermediate_code(evaluator* ev, const std::string& s); friend void impl::run_cse(evaluator* ev, impl::elist_t& elist); + friend class impl::Semantic_compiler; + friend class impl::Clear_semantic_producer; template void bind() {} template void unbind() {} @@ -566,7 +568,8 @@ struct Function string code; optimizer_t optimizer; - bool force_not_constant = false; + bool force_not_constant = false; + bool m_requires_x87_backend = false; string debug_desc; string cse_store_suffix; // CSE store code to emit after function code (x87) double* cse_store_addr = nullptr; // CSE temp address for backend-agnostic store @@ -610,6 +613,177 @@ struct Element { }; +// Executable code can retain addresses into every container below. Releasing +// the code in the destructor body makes it uncallable before those containers +// are destroyed. +class Compilation_state +{ +public: + Compilation_state(const options& effective_options, uint64_t first_element_id) + : + m_effective_options(effective_options), + m_next_element_id(first_element_id) + {} + + ~Compilation_state() + { + free_executable_buffer(m_evaluate_fptr, m_buffer_size); + } + + Compilation_state(const Compilation_state&) = delete; + Compilation_state& operator=(const Compilation_state&) = delete; + + void release_executable() + { + free_executable_buffer(m_evaluate_fptr, m_buffer_size); + m_evaluate_fptr = nullptr; + m_buffer_size = 0; + m_backend_used = backend_type::none; + } + + options m_effective_options; + bool m_is_constant_expression = false; + double m_constant_expression_value = 0.0; + bool m_sse2_simplify_mode = false; + backend_type m_backend_used = backend_type::none; + size_t m_buffer_size = 0; + elist_t m_elist; + std::list m_intermediate_code; + constant_map_t m_intermediate_constants; + uint64_t m_next_element_id; + std::list m_cse_temps; + std::map> m_power_terms; + double (*m_evaluate_fptr)() = nullptr; +}; + + +class Semantic_graph_builder +{ +public: + explicit Semantic_graph_builder(evaluator& owner) + : + m_owner(owner) + {} + + void reset(Compilation_state& compilation); + void clear() { m_compilation = nullptr; } + void push_literal(double value); + void push_variable(const shared_ptr& variable); + void call(const string& name); + void finish() const; + +private: + evaluator& m_owner; + Compilation_state* m_compilation = nullptr; + size_t m_stack_depth = 0; +}; + + +enum class Semantic_record_kind +{ + LITERAL, + VARIABLE, + CALL, +}; + + +class Semantic_record +{ +public: + explicit Semantic_record(double literal) + : + m_kind(Semantic_record_kind::LITERAL), + m_literal(literal) + {} + + explicit Semantic_record(const Variable* variable) + : + m_kind(Semantic_record_kind::VARIABLE), + m_variable(variable) + {} + + explicit Semantic_record(const Function* function) + : + m_kind(Semantic_record_kind::CALL), + m_function(function) + {} + + Semantic_record_kind kind() const { return m_kind; } + double literal() const { return m_literal; } + const Variable* variable() const { return m_variable; } + const Function* function() const { return m_function; } + +private: + Semantic_record_kind m_kind; + union + { + double m_literal; + const Variable* m_variable; + const Function* m_function; + }; +}; + + +using Semantic_program = vector; + + +struct Semantic_compilation_result +{ + std::unique_ptr compilation; + vector> referenced_variables; +}; + + +class Active_compilation_reset +{ +public: + explicit Active_compilation_reset(Compilation_state*& active) + : + m_active(active) + { + assert(m_active == nullptr); + } + + ~Active_compilation_reset() { m_active = nullptr; } + + void set(Compilation_state* compilation) { m_active = compilation; } + void clear() { m_active = nullptr; } + +private: + Compilation_state*& m_active; +}; + + +class Semantic_compiler +{ +public: + Semantic_compiler( + evaluator& owner, + const options& effective_options, + uint64_t first_element_id); + + void consume(const Semantic_record& record); + Semantic_compilation_result finish(); + +private: + void reset_graph_builder(); + + evaluator& m_owner; + std::unique_ptr m_candidate; + Semantic_graph_builder m_graph_builder; + Active_compilation_reset m_active_compilation_reset; + vector> m_referenced_variables; + std::set m_referenced_variable_set; +}; + + +class Clear_semantic_producer +{ +public: + static Semantic_program produce(evaluator& owner, string expression); +}; + + struct mexce_charstream { vector buf; @@ -644,11 +818,12 @@ void patch_rel32(mexce_charstream& s, size_t rel32_pos, size_t target_pos) inline shared_ptr make_intermediate_constant(evaluator* ev, double v) { + auto& compilation = ev->active_compilation(); auto name = double_to_hex(v); - auto it = ev->m_intermediate_constants.find(name); - if (it == ev->m_intermediate_constants.end()) { - auto sc = make_shared(ev->m_next_element_id++, v); - ev->m_intermediate_constants[name] = sc; + auto it = compilation.m_intermediate_constants.find(name); + if (it == compilation.m_intermediate_constants.end()) { + auto sc = make_shared(compilation.m_next_element_id++, v); + compilation.m_intermediate_constants[name] = sc; return sc; } else { @@ -660,9 +835,10 @@ shared_ptr make_intermediate_constant(evaluator* ev, double v) inline uint8_t* push_intermediate_code(evaluator* ev, const string& s) { - ev->m_intermediate_code.push_back(string()); - ev->m_intermediate_code.back() = s; - return (uint8_t*)&ev->m_intermediate_code.back()[0]; + auto& compilation = ev->active_compilation(); + compilation.m_intermediate_code.push_back(string()); + compilation.m_intermediate_code.back() = s; + return (uint8_t*)&compilation.m_intermediate_code.back()[0]; } @@ -2211,6 +2387,7 @@ void emit_integer_power_sequence(impl::mexce_charstream& s, uint32_t exponent) inline void pow_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) { + auto& compilation = ev->active_compilation(); auto f = it->f; // Nested power folding: (a^b)^n -> a^(b*n) when the outer exponent is an integer. @@ -2218,8 +2395,8 @@ void pow_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) // avoids spurious NaNs from intermediate real-domain pow/sqrt on negative bases. if (f->args[0]->type == Element_type::CCONST && f->args[1]->type == Element_type::CFUNC) { const uint64_t base_id = f->args[1]->id; - auto pit = ev->m_power_terms.find(base_id); - if (pit != ev->m_power_terms.end()) { + auto pit = compilation.m_power_terms.find(base_id); + if (pit != compilation.m_power_terms.end()) { const double outer_exp_d = f->args[0]->c->value; const double outer_round = round(outer_exp_d); if (std::isfinite(outer_exp_d) && outer_round == outer_exp_d) { @@ -2239,7 +2416,7 @@ void pow_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) } elist->erase(base_it); - ev->m_power_terms.erase(pit); + compilation.m_power_terms.erase(pit); } } } @@ -2323,13 +2500,16 @@ void pow_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) debug_desc = ss.str(); uint8_t* cc = push_intermediate_code(ev, final_s.str()); - auto f_opt = make_shared(ev->m_next_element_id++, optimized_name, 0, 0, final_s.buf.size(), cc, nullptr); + auto f_opt = make_shared( + compilation.m_next_element_id++, optimized_name, 0, 0, + final_s.buf.size(), cc, nullptr); + f_opt->m_requires_x87_backend = true; f_opt->debug_desc = debug_desc; f_opt->cse_store_suffix = f->cse_store_suffix; // Preserve CSE store code f_opt->cse_store_addr = f->cse_store_addr; // Preserve CSE temp address if (optimized_name == "sqrt" || optimized_name == "inv_sqrt" || optimized_name == "pow_int") { - ev->m_power_terms[f_opt->id] = std::make_pair(std::move(base_chunk), v_d); + compilation.m_power_terms[f_opt->id] = std::make_pair(std::move(base_chunk), v_d); } elist->erase(f->args[0]); // Exponent (constant) @@ -2362,7 +2542,10 @@ void pow_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) #endif uint8_t* cc = push_intermediate_code(ev, s.str()); - auto f_opt = make_shared(ev->m_next_element_id++, "pow_opt", 2, 0, s.buf.size(), cc, nullptr); + auto f_opt = make_shared( + compilation.m_next_element_id++, "pow_opt", 2, 0, + s.buf.size(), cc, nullptr); + f_opt->m_requires_x87_backend = true; f_opt->args[0] = f->args[0]; f_opt->args[1] = f->args[1]; f_opt->cse_store_suffix = f->cse_store_suffix; // Preserve CSE store code @@ -3247,9 +3430,7 @@ inline bool is_sse2_compatible(const impl::elist_const_it_t first, const impl::e for (auto it = first; it != last; ++it) { if (it->type == Element_type::CFUNC) { - // Optimized functions (from pow_optimizer etc.) have embedded x87 code - // and a non-empty debug_desc. These cannot be handled by SSE2. - if (!it->f->debug_desc.empty()) { + if (it->f->m_requires_x87_backend) { return false; } if (!is_sse2_supported_function(it->f->name)) { @@ -4325,6 +4506,7 @@ string elist_to_string(const elist_t& elist) inline void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) { + auto& compilation = ev->active_compilation(); auto f = it->f; auto fname = f->name; int fclass = (fname == "add" || fname == "sub") ? 1 : (fname == "mul" || fname == "div") ? 2 : 0; @@ -4429,7 +4611,7 @@ void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) const bool would_cancel_or_cross_zero = ((merged.back().factor > 0 && term.factor < 0) || (merged.back().factor < 0 && term.factor > 0)); - if (!ev->m_options.fast_math && would_cancel_or_cross_zero) { + if (!compilation.m_effective_options.fast_math && would_cancel_or_cross_zero) { if (term.factor != 0) { merged.push_back({std::move(term.chunk), term.factor}); } @@ -4449,7 +4631,7 @@ void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) // === SSE2 simplification: reconstruct simplified elist === // Instead of generating x87 code, build a new elist from the simplified terms. - if (ev->m_sse2_simplify_mode) { + if (compilation.m_sse2_simplify_mode) { // Build debug string for get_optimized_expression() stringstream debug_ss; bool debug_first = true; @@ -4785,7 +4967,10 @@ void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) string new_name = (fclass == 1) ? "add_sub_opt" : "mul_div_opt"; uint8_t* cc = push_intermediate_code(ev, s.str()); - auto f_opt = make_shared(ev->m_next_element_id++, new_name, 0, 0, s.buf.size(), cc, nullptr); + auto f_opt = make_shared( + compilation.m_next_element_id++, new_name, 0, 0, + s.buf.size(), cc, nullptr); + f_opt->m_requires_x87_backend = true; f_opt->debug_desc = "(" + debug_ss.str() + ")"; f_opt->cse_store_suffix = f->cse_store_suffix; // Preserve CSE store code f_opt->cse_store_addr = f->cse_store_addr; // Preserve CSE temp address @@ -4931,15 +5116,117 @@ inline const map& function_map() inline shared_ptr make_function(evaluator* ev, const string& name) { + auto& compilation = ev->active_compilation(); auto fn_it = function_map().find(name); assert(fn_it != function_map().end()); // Create a copy of the prototype and assign a new, unique ID. auto new_func = make_shared(fn_it->second); - new_func->id = ev->m_next_element_id++; + new_func->id = compilation.m_next_element_id++; return new_func; } +inline void Semantic_graph_builder::reset(Compilation_state& compilation) +{ + m_compilation = &compilation; + m_stack_depth = 0; +} + + +inline void Semantic_graph_builder::push_literal(double value) +{ + m_compilation->m_elist.push_back(Element(make_intermediate_constant(&m_owner, value))); + ++m_stack_depth; +} + + +inline void Semantic_graph_builder::push_variable(const shared_ptr& variable) +{ + m_compilation->m_elist.push_back(Element(variable)); + ++m_stack_depth; +} + + +inline void Semantic_graph_builder::call(const string& name) +{ + const auto function_it = function_map().find(name); + if (function_it == function_map().end() || m_stack_depth < function_it->second.num_args) { + throw std::logic_error("Invalid semantic expression"); + } + + m_compilation->m_elist.push_back(Element(make_function(&m_owner, name))); + m_stack_depth = m_stack_depth - function_it->second.num_args + 1; +} + + +inline void Semantic_graph_builder::finish() const +{ + if (m_stack_depth != 1) { + throw std::logic_error("Invalid semantic expression"); + } +} + + +inline Semantic_compiler::Semantic_compiler( + evaluator& owner, + const options& effective_options, + uint64_t first_element_id) +: + m_owner(owner), + m_candidate(new Compilation_state(effective_options, first_element_id)), + m_graph_builder(owner), + m_active_compilation_reset(owner.m_active_compilation) +{ + reset_graph_builder(); +} + + +inline void Semantic_compiler::reset_graph_builder() +{ + m_active_compilation_reset.set(m_candidate.get()); + m_graph_builder.reset(*m_candidate); +} + + +inline void Semantic_compiler::consume(const Semantic_record& record) +{ + switch (record.kind()) { + case Semantic_record_kind::LITERAL: + m_graph_builder.push_literal(record.literal()); + break; + case Semantic_record_kind::VARIABLE: { + const auto variable_it = m_owner.m_variables.find(record.variable()->name); + if (variable_it == m_owner.m_variables.end() || + variable_it->second.get() != record.variable()) + { + throw std::logic_error("Invalid semantic expression"); + } + m_graph_builder.push_variable(variable_it->second); + if (m_referenced_variable_set.insert(variable_it->second.get()).second) { + m_referenced_variables.push_back(variable_it->second); + } + break; + } + case Semantic_record_kind::CALL: + m_graph_builder.call(record.function()->name); + break; + default: + throw std::logic_error("Invalid semantic expression"); + } +} + + +inline Semantic_compilation_result Semantic_compiler::finish() +{ + m_graph_builder.finish(); + m_graph_builder.clear(); + auto result = m_owner.finish_semantic_compilation( + std::move(m_candidate), std::move(m_referenced_variables)); + m_active_compilation_reset.clear(); + return result; +} + + inline const map >& built_in_constants_map() { static map > cname_map; @@ -5011,6 +5298,7 @@ string get_element_signature(const Element& e) inline void run_cse(evaluator* ev, elist_t& elist) { + auto& compilation = ev->active_compilation(); // 1. Identify common subexpressions via signature // Map Signature -> Vector of (Parent Iterator, Element Pointer) pairs? // No, elements are shared pointers inside the list. @@ -5139,8 +5427,8 @@ void run_cse(evaluator* ev, elist_t& elist) } // Allocate a temporary storage slot - ev->m_cse_temps.push_back(0.0); - double* temp_addr = &ev->m_cse_temps.back(); + compilation.m_cse_temps.push_back(0.0); + double* temp_addr = &compilation.m_cse_temps.back(); // Transform Source: Set CSE store suffix // Generate FST instruction to store result after function executes. @@ -5161,7 +5449,7 @@ void run_cse(evaluator* ev, elist_t& elist) source->f->cse_store_addr = temp_addr; // For SSE2 backend // Shared temp variable for all subsequent replacements in this group. - const uint64_t temp_var_id = ev->m_next_element_id++; + const uint64_t temp_var_id = compilation.m_next_element_id++; auto temp_var = make_shared(temp_var_id, (volatile void*)temp_addr, "cse_temp_" + std::to_string(temp_var_id), M64FP); @@ -5217,7 +5505,6 @@ inline evaluator::evaluator(): m_constants(impl::built_in_constants_map()) { - m_next_element_id = 2; // Start after built-in constants. set_expression("0"); } @@ -5225,7 +5512,10 @@ evaluator::evaluator(): inline evaluator::~evaluator() { - impl::free_executable_buffer(evaluate_fptr, m_buffer_size); + m_is_constant_expression = false; + m_evaluate_fptr = nullptr; + m_backend_used = backend_type::none; + m_compilation.reset(); } @@ -5239,7 +5529,7 @@ void evaluator::bind(T& v, const std::string& s, Args&... args) if (built_in_constants_map().find(s) != built_in_constants_map().end()) { throw std::logic_error("Attempted to bind a variable, named as an existing constant"); } - m_variables[s] = make_shared(m_next_element_id++, &v, s, get_ndt()); + m_variables[s] = make_shared(m_next_variable_id++, &v, s, get_ndt()); bind(args...); } @@ -5273,32 +5563,39 @@ void evaluator::unbind_all() inline double evaluator::evaluate() { - if (is_constant_expression) { - return constant_expression_value; + if (m_is_constant_expression) { + return m_constant_expression_value; } - return evaluate_fptr(); + if (!m_evaluate_fptr) { + throw std::logic_error("No expression has been compiled."); + } + return m_evaluate_fptr(); } inline std::string evaluator::get_optimized_expression() const { - return impl::elist_to_string(m_elist); + return m_compilation ? impl::elist_to_string(m_compilation->m_elist) : std::string(); } inline std::string evaluator::get_byte_representation() const { - if (is_constant_expression || !evaluate_fptr || m_buffer_size == 0) { + if (!m_compilation || + m_compilation->m_is_constant_expression || + !m_compilation->m_evaluate_fptr || + m_compilation->m_buffer_size == 0) + { return std::string(); } std::string result; - result.reserve(m_buffer_size * 3); // 2 hex chars + space per byte + result.reserve(m_compilation->m_buffer_size * 3); // 2 hex chars + space per byte - const uint8_t* bytes = reinterpret_cast(evaluate_fptr); + const uint8_t* bytes = reinterpret_cast(m_compilation->m_evaluate_fptr); char hex_buf[4]; - for (size_t i = 0; i < m_buffer_size; ++i) { + for (size_t i = 0; i < m_compilation->m_buffer_size; ++i) { if (i > 0) { result += ' '; } @@ -5311,30 +5608,28 @@ std::string evaluator::get_byte_representation() const { inline -void evaluator::set_expression(std::string e) +backend_type evaluator::get_backend() const { - using namespace impl; - using mpe = mexce_parsing_exception; + return m_backend_used; +} - // Save for potential fallback re-parsing (SSE2->x87) - m_last_expression = e; - deque tokens; +inline +impl::Compilation_state& evaluator::active_compilation() +{ + assert(m_active_compilation != nullptr); + return *m_active_compilation; +} - m_intermediate_constants.clear(); - m_intermediate_code.clear(); - m_elist.clear(); - m_cse_temps.clear(); // Clear previous CSE temps - m_power_terms.clear(); - if (evaluate_fptr) { - free_executable_buffer(evaluate_fptr, m_buffer_size); - evaluate_fptr = nullptr; - } +inline +impl::Semantic_program impl::Clear_semantic_producer::produce( + evaluator& owner, + std::string e) +{ + using mpe = mexce_parsing_exception; - auto x = m_variables.begin(); - for (; x != m_variables.end(); x++) - x->second->referenced = false; + deque tokens; if (e.length() == 0){ throw (std::logic_error("Expected an expression")); @@ -5506,13 +5801,13 @@ void evaluator::set_expression(std::string e) break; } if (e[i] == ' ') { - if (m_variables.find(temp.content) != m_variables.end()) { + if (owner.m_variables.find(temp.content) != owner.m_variables.end()) { temp.type = VARIABLE_NAME; tokens.push_back(temp); state = 5; break; } - if (m_constants.find(temp.content) != m_constants.end()) { + if (owner.m_constants.find(temp.content) != owner.m_constants.end()) { temp.type = CONSTANT_NAME; tokens.push_back(temp); state = 5; @@ -5531,8 +5826,8 @@ void evaluator::set_expression(std::string e) " is not a known constant, variable or function name", i)); } if (e[i] == ')') { - temp.type = m_variables.find(temp.content) != m_variables.end() ? VARIABLE_NAME : - m_constants.find(temp.content) != m_constants.end() ? CONSTANT_NAME : + temp.type = owner.m_variables.find(temp.content) != owner.m_variables.end() ? VARIABLE_NAME : + owner.m_constants.find(temp.content) != owner.m_constants.end() ? CONSTANT_NAME : throw (mpe(string(temp.content) + " is not a known constant or variable name", i)); tokens.push_back(temp); @@ -5553,8 +5848,8 @@ void evaluator::set_expression(std::string e) break; } if (parse_infix_op(i, infix_op, infix_op_len)) { - temp.type = m_variables.find(temp.content) != m_variables.end() ? VARIABLE_NAME : - m_constants.find(temp.content) != m_constants.end() ? CONSTANT_NAME : + temp.type = owner.m_variables.find(temp.content) != owner.m_variables.end() ? VARIABLE_NAME : + owner.m_constants.find(temp.content) != owner.m_constants.end() ? CONSTANT_NAME : throw (mpe(string(temp.content) + " is not a known constant or variable name", i)); tokens.push_back(temp); @@ -5564,8 +5859,8 @@ void evaluator::set_expression(std::string e) break; } if (e[i] == ',') { - temp.type = m_variables.find(temp.content) != m_variables.end() ? VARIABLE_NAME : - m_constants.find(temp.content) != m_constants.end() ? CONSTANT_NAME : + temp.type = owner.m_variables.find(temp.content) != owner.m_variables.end() ? VARIABLE_NAME : + owner.m_constants.find(temp.content) != owner.m_constants.end() ? CONSTANT_NAME : throw (mpe(string(temp.content)+" is not a " "known constant or variable name", i)); tokens.push_back(temp); @@ -5672,7 +5967,38 @@ void evaluator::set_expression(std::string e) tstack.pop_back(); } - //stage 3: convert "Token" expression primitives to "Element *" + struct semantic_node_t + { + Semantic_record record; + size_t first_child; + size_t child_count; + }; + + vector semantic_nodes; + vector semantic_children; + vector semantic_stack; + semantic_nodes.reserve(postfix.size() * 2); + semantic_children.reserve(postfix.size() * 2); + semantic_stack.reserve(postfix.size()); + auto emit_call = [&](const Function* function) { + if (semantic_stack.size() < function->num_args) { + throw std::logic_error("Invalid semantic expression"); + } + + const size_t first_argument = semantic_stack.size() - function->num_args; + const size_t first_child = semantic_children.size(); + semantic_children.insert( + semantic_children.end(), + semantic_stack.begin() + first_argument, + semantic_stack.end()); + semantic_stack.resize(first_argument); + semantic_nodes.push_back({ + Semantic_record(function), + first_child, + function->num_args, + }); + semantic_stack.push_back(semantic_nodes.size() - 1); + }; while (!postfix.empty()) { temp = postfix.front(); @@ -5682,49 +6008,162 @@ void evaluator::set_expression(std::string e) case INFIX_3: case INFIX_2: case INFIX_1: { - auto name = infix_operator_to_function_name(temp.content); - m_elist.push_back(Element(make_function(this, name))); + const auto function_it = function_map().find( + infix_operator_to_function_name(temp.content)); + assert(function_it != function_map().end()); + emit_call(&function_it->second); break; } - case FUNCTION_NAME: - m_elist.push_back(Element(make_function(this, temp.content))); + case FUNCTION_NAME: { + const auto function_it = function_map().find(temp.content); + assert(function_it != function_map().end()); + emit_call(&function_it->second); break; + } case UNARY: - if (temp.content == "-") { // unary '+' is ignored - - // Rather than having an individual function for the unary minus - // we insert a zero before the last argument (which is already in - // the list), and use a subtraction instead. - // The reason is to allow the optimizer to group - // addition/subtraction chains. Clearly, this is a bit unorthodox - // and could be done in the optimizer too, but the optimizer is - // complex enough already. - - link_arguments(m_elist); - auto chunk = get_dependent_chunk(std::prev(m_elist.end())); - m_elist.insert(chunk.first, Element(make_intermediate_constant(this, 0.0))); - m_elist.push_back(Element(make_function(this, "sub"))); + if (temp.content == "-") { + if (semantic_stack.empty()) { + throw std::logic_error("Invalid semantic expression"); + } + semantic_nodes.push_back({Semantic_record(0.0), 0, 0}); + const size_t zero = semantic_nodes.size() - 1; + const size_t operand = semantic_stack.back(); + const size_t first_child = semantic_children.size(); + semantic_children.push_back(zero); + semantic_children.push_back(operand); + semantic_nodes.push_back({ + Semantic_record(&function_map().find("sub")->second), + first_child, + 2, + }); + semantic_stack.back() = semantic_nodes.size() - 1; } break; case NUMERIC_LITERAL: { - double c_value = atof(temp.content.c_str()); - m_elist.push_back(Element(make_intermediate_constant(this, c_value))); + semantic_nodes.push_back({ + Semantic_record(atof(temp.content.c_str())), 0, 0}); + semantic_stack.push_back(semantic_nodes.size() - 1); break; } case CONSTANT_NAME: { - m_elist.push_back(Element(m_constants.find(temp.content)->second)); + semantic_nodes.push_back({ + Semantic_record(owner.m_constants.find(temp.content)->second->value), + 0, + 0, + }); + semantic_stack.push_back(semantic_nodes.size() - 1); break; } case VARIABLE_NAME: { - auto it = m_variables.find(temp.content); - assert(it != m_variables.end()); - it->second->referenced = true; - m_elist.push_back(Element(it->second)); + const auto variable_it = owner.m_variables.find(temp.content); + assert(variable_it != owner.m_variables.end()); + semantic_nodes.push_back({ + Semantic_record(variable_it->second.get()), 0, 0}); + semantic_stack.push_back(semantic_nodes.size() - 1); break; } + default: + throw std::logic_error("Invalid semantic expression"); } } + if (semantic_stack.size() != 1) { + throw std::logic_error("Invalid semantic expression"); + } + + struct semantic_traversal_t + { + size_t node; + bool emit; + }; + + Semantic_program result; + result.reserve(semantic_nodes.size()); + vector traversal; + traversal.push_back({semantic_stack.back(), false}); + while (!traversal.empty()) { + const auto step = traversal.back(); + traversal.pop_back(); + const auto& node = semantic_nodes[step.node]; + if (step.emit || node.child_count == 0) { + result.push_back(node.record); + continue; + } + + traversal.push_back({step.node, true}); + for (size_t child = node.child_count; child > 0; --child) { + traversal.push_back({ + semantic_children[node.first_child + child - 1], + false, + }); + } + } + return result; +} + + +inline +impl::Semantic_compilation_result evaluator::finish_semantic_compilation( + std::unique_ptr candidate, + std::vector> referenced_variables) +{ + using namespace impl; + + Semantic_program fallback_program; + m_active_compilation = candidate.get(); + + link_arguments(candidate->m_elist); + normalize_commutative_operands(candidate->m_elist); + + size_t semantic_value_count = 0; + for (const auto& element : candidate->m_elist) { + if (element.type != Element_type::CFUNC) { + ++semantic_value_count; + } + } + + // Optimization cannot require more registers than the program has values. + // Small programs therefore cannot cross the SSE2 register limit and need no + // fallback snapshot. + const bool might_need_fallback = + semantic_value_count > static_cast(k_sse2_max_registers) && + is_sse2_compatible( + candidate->m_elist.begin(), + candidate->m_elist.end(), + candidate->m_effective_options.prefer_x87); + if (might_need_fallback) { + // Optimizers mutate compiler objects. Preserve typed semantics so fallback + // can construct an independent graph without retaining or reparsing source. + fallback_program.reserve(candidate->m_elist.size()); + for (const auto& element : candidate->m_elist) { + switch (element.type) { + case Element_type::CCONST: + fallback_program.emplace_back(element.c->value); + break; + case Element_type::CVAR: + fallback_program.emplace_back(element.v.get()); + break; + case Element_type::CFUNC: { + const auto function_it = function_map().find(element.f->name); + assert(function_it != function_map().end()); + fallback_program.emplace_back(&function_it->second); + break; + } + default: + throw std::logic_error("Invalid semantic expression"); + } + } + } + + bool fallback_allowed = might_need_fallback; +compile_candidate: + auto& m_elist = candidate->m_elist; + auto& effective_options = candidate->m_effective_options; + auto& m_sse2_simplify_mode = candidate->m_sse2_simplify_mode; + auto& is_constant_expression = candidate->m_is_constant_expression; + auto& constant_expression_value = candidate->m_constant_expression_value; + auto& evaluate_fptr = candidate->m_evaluate_fptr; + // link functions to their arguments (1) link_arguments(m_elist); @@ -5736,24 +6175,22 @@ void evaluator::set_expression(std::string e) // Run Common Subexpression Elimination (CSE) // This must run before destructive optimizers (asmd, pow) to catch // identical subtrees like div(2.2, y). - if (m_options.enable_cse) { + if (effective_options.enable_cse) { run_cse(this, m_elist); } // Check if the expression is SSE2-compatible BEFORE running optimizers. // For SSE2, we still want the algebraic simplifications from asmd_optimizer, // but we want it to output a simplified elist instead of x87 code. - m_sse2_simplify_mode = is_sse2_compatible(m_elist.begin(), m_elist.end(), m_options.prefer_x87); + m_sse2_simplify_mode = is_sse2_compatible( + m_elist.begin(), m_elist.end(), effective_options.prefer_x87); - // Track if we need to check for SSE2->x87 fallback after optimization. - // We can't backup the elist because optimizers modify Function objects in place - // (shared via shared_ptr), so instead we re-parse from m_last_expression if needed. bool need_fallback_check = m_sse2_simplify_mode; // Fast-math algebraic simplifications - run in a SEPARATE PASS before any optimizer // modifies the expression structure. This must happen before asmd_optimizer which // transforms the elist in ways that would confuse the chunk comparison. - if (m_options.fast_math) { + if (effective_options.fast_math) { // Helper lambda to check if two chunks are structurally equal // Avoids creating copies for the common case of simple single-element comparisons auto chunks_equal = [](elist_it_t first0, elist_it_t last0, @@ -5903,14 +6340,11 @@ void evaluator::set_expression(std::string e) elist_it_t first_arg_it = y; std::advance(first_arg_it, -(int64_t)f->num_args); compile_and_finalize_elist(first_arg_it, next(y)); - double res = evaluate_fptr(); + const double res = evaluate_fptr(); // Constant folding uses a temporary JIT buffer. Release it // immediately so repeated foldable subtrees do not leak one // executable page per fold before the final expression compile. - free_executable_buffer(evaluate_fptr, m_buffer_size); - evaluate_fptr = nullptr; - m_buffer_size = 0; - m_backend_used = backend_type::none; + candidate->release_executable(); m_elist.erase(first_arg_it, y); *y = Element(make_intermediate_constant(this, res)); y = y_next; @@ -5929,27 +6363,83 @@ void evaluator::set_expression(std::string e) y = y_next; } - // Check if SSE2 optimization produced an elist that's no longer SSE2-compatible. - // This can happen when asmd_optimizer flattens chains into too many parallel terms. - // If so, re-parse the expression with x87 mode forced. - // Note: We cannot simply restore the backup because the optimizer modifies Function - // objects in place (shared via shared_ptr), corrupting the backup. - if (need_fallback_check && !is_sse2_compatible(m_elist.begin(), m_elist.end(), false)) { - // Force x87 mode and re-parse the expression from scratch - bool original_prefer_x87 = m_options.prefer_x87; - m_options.prefer_x87 = true; - set_expression(m_last_expression); - m_options.prefer_x87 = original_prefer_x87; - return; + if (fallback_allowed && + need_fallback_check && + !is_sse2_compatible(m_elist.begin(), m_elist.end(), false)) + { + options fallback_options = candidate->m_effective_options; + fallback_options.prefer_x87 = true; + candidate.reset(new Compilation_state(fallback_options, m_next_variable_id)); + m_active_compilation = candidate.get(); + + Semantic_graph_builder fallback_builder(*this); + fallback_builder.reset(*candidate); + for (const auto& record : fallback_program) { + switch (record.kind()) { + case Semantic_record_kind::LITERAL: + fallback_builder.push_literal(record.literal()); + break; + case Semantic_record_kind::VARIABLE: { + const auto variable_it = m_variables.find(record.variable()->name); + assert(variable_it != m_variables.end()); + fallback_builder.push_variable(variable_it->second); + break; + } + case Semantic_record_kind::CALL: + fallback_builder.call(record.function()->name); + break; + default: + throw std::logic_error("Invalid semantic expression"); + } + } + fallback_builder.finish(); + fallback_allowed = false; + goto compile_candidate; } - is_constant_expression = m_elist.size()==1 && m_elist.back().type == Element_type::CCONST; + is_constant_expression = m_elist.size() == 1 && m_elist.back().type == Element_type::CCONST; if (is_constant_expression) { constant_expression_value = m_elist.back().c->value; } else { compile_and_finalize_elist(m_elist.begin(), m_elist.end()); } + Semantic_compilation_result result; + result.compilation = std::move(candidate); + result.referenced_variables = std::move(referenced_variables); + return result; +} + + +inline +void evaluator::set_expression(std::string expression) +{ + m_is_constant_expression = false; + m_constant_expression_value = 0.0; + m_evaluate_fptr = nullptr; + m_backend_used = backend_type::none; + m_compilation.reset(); + + for (const auto& variable : m_variables) { + variable.second->referenced = false; + } + + const auto semantic_program = impl::Clear_semantic_producer::produce( + *this, std::move(expression)); + impl::Semantic_compiler compiler(*this, m_options, m_next_variable_id); + for (const auto& record : semantic_program) { + compiler.consume(record); + } + auto result = compiler.finish(); + + m_compilation = std::move(result.compilation); + for (const auto& variable : result.referenced_variables) { + variable->referenced = true; + } + m_is_constant_expression = m_compilation->m_is_constant_expression; + m_constant_expression_value = m_compilation->m_constant_expression_value; + m_evaluate_fptr = m_compilation->m_evaluate_fptr; + m_backend_used = m_compilation->m_backend_used; } @@ -5958,10 +6448,15 @@ inline void evaluator::compile_and_finalize_elist(impl::elist_const_it_t first, impl::elist_const_it_t last) { using namespace impl; + auto& compilation = active_compilation(); + auto& effective_options = compilation.m_effective_options; + auto& m_buffer_size = compilation.m_buffer_size; + auto& evaluate_fptr = compilation.m_evaluate_fptr; + auto& compiled_backend = compilation.m_backend_used; #ifdef MEXCE_64 // Check if we can use the faster SSE2 backend - bool use_sse2 = is_sse2_compatible(first, last, m_options.prefer_x87); + bool use_sse2 = is_sse2_compatible(first, last, effective_options.prefer_x87); if (use_sse2) { // SSE2 backend: simpler prologue/epilogue since result is already in xmm0 @@ -6019,7 +6514,7 @@ void evaluator::compile_and_finalize_elist(impl::elist_const_it_t first, impl::e memcpy(buffer, code_buffer.buf.data(), m_buffer_size); evaluate_fptr = lock_executable_buffer(buffer, m_buffer_size); - m_backend_used = backend_type::sse2; + compiled_backend = backend_type::sse2; return; } @@ -6120,7 +6615,7 @@ void evaluator::compile_and_finalize_elist(impl::elist_const_it_t first, impl::e memcpy(buffer, code_buffer.buf.data(), m_buffer_size); evaluate_fptr = lock_executable_buffer(buffer, m_buffer_size); - m_backend_used = backend_type::x87; + compiled_backend = backend_type::x87; } } // mexce diff --git a/test/unit_tests.cpp b/test/unit_tests.cpp index fa5fa35..1376cc1 100644 --- a/test/unit_tests.cpp +++ b/test/unit_tests.cpp @@ -359,6 +359,26 @@ void test_constants_and_single_shot(TestSuite& suite) { eval.set_expression("ln(v)"); suite.expect_near("ln", eval.evaluate(), std::log(v)); + suite.expect_true( + "ln_introspection_spelling", + eval.get_optimized_expression().find("ln") != std::string::npos); + + eval.set_expression("log(v)"); + suite.expect_near("log", eval.evaluate(), std::log(v)); + suite.expect_true( + "log_introspection_spelling", + eval.get_optimized_expression().find("log") != std::string::npos); +} + +void test_signed_zero_semantics(TestSuite& suite) +{ + mexce::evaluator eval; + + eval.set_expression("-0.0"); + suite.expect_true("source_unary_minus_signed_zero", !std::signbit(eval.evaluate())); + + eval.set_expression("neg(0.0)"); + suite.expect_true("explicit_neg_signed_zero", std::signbit(eval.evaluate())); } void test_pow_optimizer_special_cases(TestSuite& suite) { @@ -405,7 +425,6 @@ void test_nested_pow_folding_sympy_semantics(TestSuite& suite) { void test_helper_functions_and_element(TestSuite& suite) { using namespace mexce::impl; - mexce::evaluator eval; suite.expect_true("function_name_to_infix_operator_add", function_name_to_infix_operator("add") == "+"); suite.expect_true("function_name_to_infix_operator_unknown", function_name_to_infix_operator("noop").empty()); @@ -420,7 +439,7 @@ void test_helper_functions_and_element(TestSuite& suite) { double value = 4.0; auto variable = std::make_shared(1, &value, "value", M32FP); auto constant = std::make_shared(2, 3.0); - auto add_function = make_function(&eval, "add"); + auto add_function = std::make_shared(function_map().find("add")->second); elist_t elist; elist.push_back(Element(variable)); @@ -651,6 +670,97 @@ void test_parsing_errors(TestSuite& suite) { }, "Unexpected end of expression"); } +void test_empty_state_after_failed_replacement(TestSuite& suite) +{ + mexce::evaluator eval; + + eval.set_expression("19.25"); + suite.expect_near("constant_before_failed_replacement", eval.evaluate(), 19.25); + suite.expect_throw( + "constant_failed_replacement", + [&] { eval.set_expression("unknown_name"); }, + "unknown_name is not a known constant, variable or function name" + ); + suite.expect_throw( + "constant_empty_evaluate_after_failed_replacement", + [&] { (void)eval.evaluate(); }, + "No expression has been compiled." + ); + suite.expect_true( + "constant_empty_backend_after_failed_replacement", + eval.get_backend() == mexce::backend_type::none); + suite.expect_true( + "constant_empty_optimized_expression_after_failed_replacement", + eval.get_optimized_expression().empty()); + suite.expect_true( + "constant_empty_byte_representation_after_failed_replacement", + eval.get_byte_representation().empty()); + + double value = 3.0; + eval.bind(value, "value"); + eval.set_expression("value + 2"); + suite.expect_near("native_before_failed_replacement", eval.evaluate(), 5.0); + suite.expect_throw( + "native_failed_replacement", + [&] { eval.set_expression("unknown_name"); }, + "unknown_name is not a known constant, variable or function name" + ); + suite.expect_throw( + "native_empty_evaluate_after_failed_replacement", + [&] { (void)eval.evaluate(); }, + "No expression has been compiled." + ); + suite.expect_true( + "native_empty_backend_after_failed_replacement", + eval.get_backend() == mexce::backend_type::none); + suite.expect_true( + "native_empty_optimized_expression_after_failed_replacement", + eval.get_optimized_expression().empty()); + suite.expect_true( + "native_empty_byte_representation_after_failed_replacement", + eval.get_byte_representation().empty()); + eval.unbind("value"); + suite.expect_throw( + "unbind_after_failed_replacement_keeps_empty_state", + [&] { (void)eval.evaluate(); }, + "No expression has been compiled." + ); + + double replacement_value = 4.0; + eval.bind(replacement_value, "replacement_value"); + eval.set_expression("replacement_value * 3"); + suite.expect_near("successful_replacement_after_failure", eval.evaluate(), 12.0); +} + +void test_late_compilation_failure_reference_state(TestSuite& suite) +{ + mexce::evaluator eval; + double a = 1.25; + eval.bind(a, "a"); + eval.use_x87_backend(); + + suite.expect_throw( + "late_compilation_failure", + [&] { + eval.set_expression( + "mod(a,mod(a,mod(a,mod(a,mod(a,mod(a,mod(a,mod(a,mod(a,a)))))))))"); + }, + "Expression too complex for x87 FPU (stack overflow)" + ); + suite.expect_throw( + "late_compilation_failure_empty_evaluate", + [&] { (void)eval.evaluate(); }, + "No expression has been compiled." + ); + + eval.unbind("a"); + suite.expect_throw( + "late_compilation_failure_unbind_preserves_empty_state", + [&] { (void)eval.evaluate(); }, + "No expression has been compiled." + ); +} + void test_unbind_referenced_and_variadic(TestSuite& suite) { mexce::evaluator eval; @@ -687,19 +797,18 @@ void test_lock_executable_buffer_failure(TestSuite& suite) { void test_elist_to_string_unary_and_multi(TestSuite& suite) { using namespace mexce::impl; - mexce::evaluator eval; double value = 4.0; auto variable = std::make_shared(1, &value, "value", M32FP); - auto neg_function = make_function(&eval, "neg"); + auto neg_function = std::make_shared(function_map().find("neg")->second); elist_t eu; eu.push_back(Element(variable)); eu.push_back(Element(neg_function)); suite.expect_true("elist_to_string_unary_neg", elist_to_string(eu) == "(-value)"); auto constant3 = std::make_shared(3, 3.0); - auto min_function = make_function(&eval, "min"); + auto min_function = std::make_shared(function_map().find("min")->second); elist_t em; em.push_back(Element(variable)); em.push_back(Element(constant3)); @@ -950,6 +1059,33 @@ void test_cse_coverage(TestSuite& suite) { suite.expect_near("cse_exp", eval.evaluate(), expected); } +void test_source_free_fallback_ownership(TestSuite& suite) +{ + mexce::evaluator eval; + double a = 1.25; + eval.bind(a, "a"); + eval.use_sse2_backend(); + + const std::string polynomial = + "((((((((((2.20*a+1.1)*a+9.9)*a+8.8)*a+7.7)*a+6.6)*a+5.5)*a+4.4)*a+3.3)*a+2.2)*a+1.1)"; + eval.set_expression(polynomial); + + double expected = 2.20; + const double coefficients[] = {1.1, 9.9, 8.8, 7.7, 6.6, 5.5, 4.4, 3.3, 2.2, 1.1}; + for (double coefficient : coefficients) { + expected = expected * a + coefficient; + } + + suite.expect_true("post_optimizer_fallback_backend", eval.get_backend() == mexce::backend_type::x87); + suite.expect_near("post_optimizer_fallback_value", eval.evaluate(), expected); + suite.expect_true("fallback_does_not_mutate_options", !eval.get_options().prefer_x87); + + eval.enable_cse(); + eval.set_expression("(" + polynomial + ")+sin(a)+sin(a)"); + suite.expect_true("cse_fallback_backend", eval.get_backend() == mexce::backend_type::x87); + suite.expect_near("cse_fallback_value", eval.evaluate(), expected + 2.0 * std::sin(a)); +} + // Test SSE2 neg and abs operations (covers xorpd/andpd emit functions) void test_sse2_neg_abs(TestSuite& suite) { #ifdef MEXCE_64 @@ -1077,6 +1213,7 @@ int main() { test_ieee_self_cancellation_default_mode(suite); test_min_max_and_arithmetic(suite); test_constants_and_single_shot(suite); + test_signed_zero_semantics(suite); test_pow_optimizer_special_cases(suite); test_nested_pow_folding_sympy_semantics(suite); test_helper_functions_and_element(suite); @@ -1086,6 +1223,8 @@ int main() { test_memory_management(suite); test_asmd_optimizer_branches(suite); test_parsing_errors(suite); + test_empty_state_after_failed_replacement(suite); + test_late_compilation_failure_reference_state(suite); test_elist_to_string_unary_and_multi(suite); test_parser_additional_coverage(suite); test_unbind_referenced_and_variadic(suite); @@ -1099,6 +1238,7 @@ int main() { test_trunc_function(suite); test_options_api(suite); test_cse_coverage(suite); + test_source_free_fallback_ownership(suite); test_sse2_neg_abs(suite); test_sse2_log_functions(suite); test_numeric_data_types(suite); From ca95066c3c162b717de3df2fcfdc5a25a4c8f093 Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 14:52:45 +0200 Subject: [PATCH 04/12] Add protected expression key and wire codec --- CMakeLists.txt | 58 + mexce_protected.h | 738 +++++++++++ mexce_protected_encoder.h | 391 ++++++ test/fixtures/protected_format_1_0.bin | Bin 0 -> 407 bytes .../protected_format_1_0.manifest.txt | 4 + test/protected_codec_tests.cpp | 1159 +++++++++++++++++ test/protected_fixture_generator.cpp | 144 ++ test/protected_sodium_init_failure_test.cpp | 25 + 8 files changed, 2519 insertions(+) create mode 100644 mexce_protected.h create mode 100644 mexce_protected_encoder.h create mode 100644 test/fixtures/protected_format_1_0.bin create mode 100644 test/fixtures/protected_format_1_0.manifest.txt create mode 100644 test/protected_codec_tests.cpp create mode 100644 test/protected_fixture_generator.cpp create mode 100644 test/protected_sodium_init_failure_test.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index b710f96..f975661 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -2,6 +2,7 @@ cmake_minimum_required(VERSION 3.16) project(mexce_benchmark LANGUAGES CXX) option(ENABLE_COVERAGE "Build with coverage instrumentation" OFF) +option(MEXCE_ENABLE_PROTECTED_EXPRESSIONS "Build protected-expression targets" OFF) if(ENABLE_COVERAGE) if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang") @@ -46,6 +47,58 @@ add_executable(unit_tests test/unit_tests.cpp) target_include_directories(unit_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) target_compile_options(unit_tests PRIVATE ${MEXCE_WARNING_FLAGS}) +add_library(mexce INTERFACE) +add_library(mexce::mexce ALIAS mexce) +target_include_directories(mexce INTERFACE ${CMAKE_CURRENT_SOURCE_DIR}) + +if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) + set(MEXCE_SODIUM_TARGET "") + find_package(libsodium 1.0.22 EXACT CONFIG QUIET) + if(TARGET libsodium::libsodium) + set(MEXCE_SODIUM_TARGET libsodium::libsodium) + else() + find_package(unofficial-sodium 1.0.22 EXACT CONFIG QUIET) + endif() + if(TARGET unofficial-sodium::sodium) + set(MEXCE_SODIUM_TARGET unofficial-sodium::sodium) + elseif(NOT MEXCE_SODIUM_TARGET) + find_package(PkgConfig QUIET) + if(PkgConfig_FOUND) + pkg_check_modules(MEXCE_SODIUM QUIET IMPORTED_TARGET libsodium=1.0.22) + endif() + if(TARGET PkgConfig::MEXCE_SODIUM) + set(MEXCE_SODIUM_TARGET PkgConfig::MEXCE_SODIUM) + else() + message(FATAL_ERROR "MEXCE protected expressions require libsodium 1.0.22") + endif() + endif() + + add_library(mexce_protected INTERFACE) + add_library(mexce::protected ALIAS mexce_protected) + target_link_libraries(mexce_protected INTERFACE mexce ${MEXCE_SODIUM_TARGET}) + + set(MEXCE_PROTECTED_FIXTURE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/test/fixtures") + add_executable(protected_codec_tests test/protected_codec_tests.cpp) + target_link_libraries(protected_codec_tests PRIVATE mexce_protected) + target_compile_definitions(protected_codec_tests PRIVATE + MEXCE_PROTECTED_TESTING + MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin" + MEXCE_PROTECTED_FIXTURE_MANIFEST_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.manifest.txt") + target_compile_options(protected_codec_tests PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(protected_fixture_generator EXCLUDE_FROM_ALL + test/protected_fixture_generator.cpp) + target_link_libraries(protected_fixture_generator PRIVATE ${MEXCE_SODIUM_TARGET}) + target_compile_options(protected_fixture_generator PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(protected_sodium_init_failure_test + test/protected_sodium_init_failure_test.cpp) + target_link_libraries(protected_sodium_init_failure_test PRIVATE mexce_protected) + target_compile_definitions(protected_sodium_init_failure_test PRIVATE + MEXCE_PROTECTED_TESTING) + target_compile_options(protected_sodium_init_failure_test PRIVATE ${MEXCE_WARNING_FLAGS}) +endif() + # Capture compiler info for benchmark output set(MEXCE_BENCHMARK_COMPILER "${CMAKE_CXX_COMPILER_ID}") if (CMAKE_CXX_COMPILER_VERSION) @@ -78,6 +131,11 @@ target_compile_definitions(benchmark PRIVATE enable_testing() add_test(NAME mexce_unit_tests COMMAND unit_tests) +if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) + add_test(NAME mexce_protected_codec_tests COMMAND protected_codec_tests) + add_test(NAME mexce_protected_sodium_init_failure + COMMAND protected_sodium_init_failure_test) +endif() add_test(NAME benchmark_quick COMMAND benchmark 5 ${CMAKE_BINARY_DIR}/benchmark_quick_results.txt) add_test(NAME benchmark_output_first diff --git a/mexce_protected.h b/mexce_protected.h new file mode 100644 index 0000000..a597182 --- /dev/null +++ b/mexce_protected.h @@ -0,0 +1,738 @@ +#ifndef MEXCE_PROTECTED_H +#define MEXCE_PROTECTED_H + +#include "mexce.h" + +#include + +#include +#include +#include +#include +#include +#include + + +namespace mexce { + + +static_assert(crypto_secretstream_xchacha20poly1305_KEYBYTES == 32, + "Protected-expression format 1.0 requires 32-byte secretstream keys"); +static_assert(crypto_secretstream_xchacha20poly1305_HEADERBYTES == 24, + "Protected-expression format 1.0 requires 24-byte secretstream headers"); +static_assert(crypto_secretstream_xchacha20poly1305_ABYTES == 17, + "Protected-expression format 1.0 requires 17-byte secretstream overhead"); + + +enum class Protected_expression_error_category +{ + INVALID_ARGUMENT, + UNSUPPORTED_FORMAT, + SIZE_LIMIT, + AUTHENTICATION_FAILED, + MALFORMED_PROGRAM, + MISSING_BINDING, + INTROSPECTION_DISABLED, + CRYPTOGRAPHY_UNAVAILABLE, + COMPILATION_FAILED, + RESOURCE_FAILURE, +}; + + +class Protected_expression_error: public std::runtime_error +{ +public: + Protected_expression_error( + Protected_expression_error_category category, + const char* message) + : + std::runtime_error(message), + m_category(category), + m_has_record_index(false), + m_record_index(0) + {} + + Protected_expression_error( + Protected_expression_error_category category, + const char* message, + uint32_t record_index) + : + std::runtime_error(message), + m_category(category), + m_has_record_index(true), + m_record_index(record_index) + {} + + Protected_expression_error_category category() const noexcept { return m_category; } + bool has_record_index() const noexcept { return m_has_record_index; } + + uint32_t record_index() const + { + if (!m_has_record_index) { + throw std::logic_error("No protected-expression record index is available."); + } + return m_record_index; + } + +private: + Protected_expression_error_category m_category; + bool m_has_record_index; + uint32_t m_record_index; +}; + + +namespace impl { + + +enum class Protected_wipe_context +{ + KEY, + PULL_STATE, + PUSH_STATE, + CLEAR_RECORD, + ADDITIONAL_DATA, + DECODED_SCALAR, + VALIDATOR_STATE, + COUNT, +}; + + +#ifdef MEXCE_PROTECTED_TESTING +class Protected_wipe_observer +{ +public: + void observe(Protected_wipe_context context, const uint8_t* bytes, size_t size) + { + const size_t context_index = static_cast(context); + ++m_counts[context_index]; + for (size_t i = 0; i < size; ++i) { + m_saw_nonzero_after_wipe[context_index] |= bytes[i] != 0; + } + } + + void reset() + { + std::memset(m_counts, 0, sizeof(m_counts)); + std::memset( + m_saw_nonzero_after_wipe, 0, sizeof(m_saw_nonzero_after_wipe)); + } + + size_t count(Protected_wipe_context context) const + { + return m_counts[static_cast(context)]; + } + + bool all_observed_regions_are_zero() const + { + for (size_t i = 0; i < static_cast(Protected_wipe_context::COUNT); ++i) { + if (m_saw_nonzero_after_wipe[i]) { + return false; + } + } + return true; + } + +private: + size_t m_counts[static_cast(Protected_wipe_context::COUNT)] = {}; + bool m_saw_nonzero_after_wipe[ + static_cast(Protected_wipe_context::COUNT)] = {}; +}; + + +struct protected_test_faults_t +{ + bool fail_sodium_init = false; + bool fail_malloc = false; + bool fail_mlock = false; + bool fail_init_pull = false; +}; + + +inline protected_test_faults_t& protected_test_faults() +{ + static protected_test_faults_t faults; + return faults; +} + + +inline Protected_wipe_observer& protected_wipe_observer() +{ + static Protected_wipe_observer observer; + return observer; +} +#endif + + +inline void protected_memzero( + void* bytes, + size_t size, + Protected_wipe_context context) +{ + sodium_memzero(bytes, size); +#ifdef MEXCE_PROTECTED_TESTING + auto& observer = protected_wipe_observer(); + observer.observe(context, static_cast(bytes), size); +#else + (void)context; +#endif +} + + +inline void* protected_malloc(size_t size) +{ +#ifdef MEXCE_PROTECTED_TESTING + return protected_test_faults().fail_malloc ? nullptr : sodium_malloc(size); +#else + return sodium_malloc(size); +#endif +} + + +inline int protected_mlock(void* bytes, size_t size) +{ +#ifdef MEXCE_PROTECTED_TESTING + return protected_test_faults().fail_mlock ? -1 : sodium_mlock(bytes, size); +#else + return sodium_mlock(bytes, size); +#endif +} + + +inline int protected_init_pull( + crypto_secretstream_xchacha20poly1305_state* state, + const unsigned char* header, + const unsigned char* key) +{ +#ifdef MEXCE_PROTECTED_TESTING + return protected_test_faults().fail_init_pull + ? -1 + : crypto_secretstream_xchacha20poly1305_init_pull(state, header, key); +#else + return crypto_secretstream_xchacha20poly1305_init_pull(state, header, key); +#endif +} + + +inline void require_sodium() +{ +#ifdef MEXCE_PROTECTED_TESTING + static const int result = protected_test_faults().fail_sodium_init ? -1 : sodium_init(); +#else + static const int result = sodium_init(); +#endif + if (result < 0) { + throw Protected_expression_error( + Protected_expression_error_category::CRYPTOGRAPHY_UNAVAILABLE, + "Protected-expression cryptography is unavailable."); + } +} + + +class Protected_key_access; + + +} // namespace impl + + +class Protected_expression_key +{ +public: + static Protected_expression_key from_bytes(const uint8_t* bytes, size_t size) + { + if (!bytes || size != crypto_secretstream_xchacha20poly1305_KEYBYTES) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "A protected-expression key must contain exactly 32 bytes."); + } + + impl::require_sodium(); + uint8_t* owned = static_cast(impl::protected_malloc(size)); + if (!owned) { + throw Protected_expression_error( + Protected_expression_error_category::RESOURCE_FAILURE, + "Protected-expression key allocation failed."); + } + if (impl::protected_mlock(owned, size) != 0) { + impl::protected_memzero( + owned, size, impl::Protected_wipe_context::KEY); + sodium_free(owned); + throw Protected_expression_error( + Protected_expression_error_category::RESOURCE_FAILURE, + "Protected-expression key locking failed."); + } + std::memcpy(owned, bytes, size); + return Protected_expression_key(owned); + } + + Protected_expression_key(Protected_expression_key&& other) noexcept + : + m_bytes(other.m_bytes) + { + other.m_bytes = nullptr; + } + + Protected_expression_key& operator=(Protected_expression_key&& other) noexcept + { + if (this != &other) { + release(); + m_bytes = other.m_bytes; + other.m_bytes = nullptr; + } + return *this; + } + + Protected_expression_key(const Protected_expression_key&) = delete; + Protected_expression_key& operator=(const Protected_expression_key&) = delete; + + ~Protected_expression_key() { release(); } + + template + void consume_bytes(Consumer&& consumer) + { + require_bytes(); + Protected_expression_key owned(std::move(*this)); + std::forward(consumer)( + static_cast(owned.m_bytes), + crypto_secretstream_xchacha20poly1305_KEYBYTES); + } + +private: + explicit Protected_expression_key(uint8_t* bytes) + : + m_bytes(bytes) + {} + + void require_bytes() const + { + if (!m_bytes) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "The protected-expression key is empty."); + } + } + + void release() noexcept + { + if (m_bytes) { + impl::protected_memzero( + m_bytes, + crypto_secretstream_xchacha20poly1305_KEYBYTES, + impl::Protected_wipe_context::KEY); + sodium_munlock(m_bytes, crypto_secretstream_xchacha20poly1305_KEYBYTES); + sodium_free(m_bytes); + m_bytes = nullptr; + } + } + + uint8_t* m_bytes; + + friend class impl::Protected_key_access; +}; + + +namespace impl { + + +class Protected_key_access +{ +public: + static const uint8_t* bytes(const Protected_expression_key& key) + { + key.require_bytes(); + return key.m_bytes; + } +}; + + +enum class Protected_operation +{ + ADD = 1, + SUB = 2, + MUL = 3, + DIV = 4, + NEG = 5, + POW = 6, + SIN = 7, + COS = 8, + TAN = 9, + ABS = 10, + SIGN = 11, + SIGNP = 12, + EXPN = 13, + SFC = 14, + SQRT = 15, + EXP = 16, + LT = 17, + GT = 18, + LE = 19, + GE = 20, + EQ = 21, + NE = 22, + LOG = 23, + LOG2 = 24, + LOG10 = 25, + LOGB = 26, + YLOG2 = 27, + MAX = 28, + MIN = 29, + FLOOR = 30, + CEIL = 31, + ROUND = 32, + INT = 33, + TRUNC = 34, + MOD = 35, + BND = 36, + BIAS = 37, + GAIN = 38, +}; + + +inline uint16_t load_u16(const uint8_t* bytes) +{ + return static_cast(bytes[0]) | + static_cast(bytes[1]) << 8; +} + + +inline uint32_t load_u32(const uint8_t* bytes) +{ + return static_cast(bytes[0]) | + static_cast(bytes[1]) << 8 | + static_cast(bytes[2]) << 16 | + static_cast(bytes[3]) << 24; +} + + +inline uint64_t load_u64(const uint8_t* bytes) +{ + uint64_t value = 0; + for (size_t i = 0; i < 8; ++i) { + value |= static_cast(bytes[i]) << (i * 8); + } + return value; +} + + +inline void store_u32(uint8_t* bytes, uint32_t value) +{ + for (size_t i = 0; i < 4; ++i) { + bytes[i] = static_cast(value >> (i * 8)); + } +} + + +inline bool all_zero(const uint8_t* bytes, size_t size) +{ + uint8_t any = 0; + for (size_t i = 0; i < size; ++i) { + any |= bytes[i]; + } + return any == 0; +} + + +inline uint32_t protected_operation_arity(uint32_t operation) +{ + switch (static_cast(operation)) { + case Protected_operation::NEG: + case Protected_operation::SIN: + case Protected_operation::COS: + case Protected_operation::TAN: + case Protected_operation::ABS: + case Protected_operation::SIGN: + case Protected_operation::SIGNP: + case Protected_operation::EXPN: + case Protected_operation::SFC: + case Protected_operation::SQRT: + case Protected_operation::EXP: + case Protected_operation::LOG: + case Protected_operation::LOG2: + case Protected_operation::LOG10: + case Protected_operation::FLOOR: + case Protected_operation::CEIL: + case Protected_operation::ROUND: + case Protected_operation::INT: + case Protected_operation::TRUNC: return 1; + + case Protected_operation::ADD: + case Protected_operation::SUB: + case Protected_operation::MUL: + case Protected_operation::DIV: + case Protected_operation::POW: + case Protected_operation::LT: + case Protected_operation::GT: + case Protected_operation::LE: + case Protected_operation::GE: + case Protected_operation::EQ: + case Protected_operation::NE: + case Protected_operation::LOGB: + case Protected_operation::YLOG2: + case Protected_operation::MAX: + case Protected_operation::MIN: + case Protected_operation::MOD: + case Protected_operation::BND: + case Protected_operation::BIAS: + case Protected_operation::GAIN: return 2; + default: return 0; + } +} + + +class Protected_wipe_guard +{ +public: + Protected_wipe_guard( + void* bytes, + size_t size, + Protected_wipe_context context) + : + m_bytes(bytes), + m_size(size), + m_context(context) + {} + + ~Protected_wipe_guard() { protected_memzero(m_bytes, m_size, m_context); } + +private: + void* m_bytes; + size_t m_size; + Protected_wipe_context m_context; +}; + + +class Protected_pull_state +{ +public: + Protected_pull_state() { std::memset(&m_state, 0, sizeof(m_state)); } + ~Protected_pull_state() + { + protected_memzero( + &m_state, sizeof(m_state), Protected_wipe_context::PULL_STATE); + } + + crypto_secretstream_xchacha20poly1305_state* get() { return &m_state; } + +private: + crypto_secretstream_xchacha20poly1305_state m_state; +}; + + +inline void throw_malformed(uint32_t record_index) +{ + throw Protected_expression_error( + Protected_expression_error_category::MALFORMED_PROGRAM, + "The protected expression is malformed.", + record_index); +} + + +inline void throw_size_limit(uint32_t record_index) +{ + throw Protected_expression_error( + Protected_expression_error_category::SIZE_LIMIT, + "The protected expression exceeds a resource limit.", + record_index); +} + + +template +void decode_protected_expression( + const uint8_t* program, + size_t program_size, + Protected_expression_key key, + Sink& sink) +{ + constexpr size_t k_header_size = 64; + constexpr size_t k_record_size = 32; + constexpr size_t k_frame_size = + k_record_size + crypto_secretstream_xchacha20poly1305_ABYTES; + + if (!program && program_size != 0) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "A protected-expression byte range is invalid."); + } + if (program_size == 0) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "A protected expression cannot be empty."); + } + if (program_size < k_header_size) { + throw Protected_expression_error( + Protected_expression_error_category::MALFORMED_PROGRAM, + "The protected expression is malformed."); + } + + static const uint8_t magic[8] = {'M', 'E', 'X', 'C', 'E', 'P', 'R', 'G'}; + if (std::memcmp(program, magic, sizeof(magic)) != 0 || + load_u16(program + 8) != 1 || + load_u16(program + 10) != 0 || + load_u16(program + 12) != k_header_size || + load_u16(program + 14) != k_record_size || + load_u32(program + 20) != 0) + { + throw Protected_expression_error( + Protected_expression_error_category::UNSUPPORTED_FORMAT, + "The protected-expression format is unsupported."); + } + + const uint32_t record_count = load_u32(program + 16); + if (record_count > 16384) { + throw Protected_expression_error( + Protected_expression_error_category::SIZE_LIMIT, + "The protected expression exceeds a resource limit."); + } + if (record_count < 3 || all_zero(program + 24, 16) || + program_size != k_header_size + static_cast(record_count) * k_frame_size) + { + throw Protected_expression_error( + Protected_expression_error_category::MALFORMED_PROGRAM, + "The protected expression is malformed."); + } + + require_sodium(); + Protected_pull_state pull_state; + key.consume_bytes([&](const uint8_t* key_bytes, size_t) { + if (protected_init_pull(pull_state.get(), program + 40, key_bytes) != 0) { + throw Protected_expression_error( + Protected_expression_error_category::AUTHENTICATION_FAILED, + "Protected-expression authentication failed."); + } + }); + + uint32_t binding_count = 0; + size_t stack_depth = 0; + uint8_t used_slots[4096] = {}; + Protected_wipe_guard binding_count_wipe( + &binding_count, sizeof(binding_count), Protected_wipe_context::VALIDATOR_STATE); + Protected_wipe_guard stack_depth_wipe( + &stack_depth, sizeof(stack_depth), Protected_wipe_context::VALIDATOR_STATE); + Protected_wipe_guard used_slots_wipe( + used_slots, sizeof(used_slots), Protected_wipe_context::VALIDATOR_STATE); + + for (uint32_t index = 0; index < record_count; ++index) { + uint8_t clear[k_record_size] = {}; + uint8_t additional_data[k_header_size + 4] = {}; + Protected_wipe_guard clear_wipe( + clear, sizeof(clear), Protected_wipe_context::CLEAR_RECORD); + Protected_wipe_guard additional_data_wipe( + additional_data, + sizeof(additional_data), + Protected_wipe_context::ADDITIONAL_DATA); + std::memcpy(additional_data, program, k_header_size); + store_u32(additional_data + k_header_size, index); + + unsigned long long clear_size = 0; + unsigned char tag = 0; + const uint8_t* frame = program + k_header_size + static_cast(index) * k_frame_size; + if (crypto_secretstream_xchacha20poly1305_pull( + pull_state.get(), clear, &clear_size, &tag, + frame, k_frame_size, + additional_data, sizeof(additional_data)) != 0) + { + throw Protected_expression_error( + Protected_expression_error_category::AUTHENTICATION_FAILED, + "Protected-expression authentication failed.", + index); + } + + const bool final_record = index + 1 == record_count; + if (clear_size != k_record_size || + (!final_record && tag != crypto_secretstream_xchacha20poly1305_TAG_MESSAGE) || + (final_record && tag != crypto_secretstream_xchacha20poly1305_TAG_FINAL)) + { + throw_malformed(index); + } + + uint8_t kind = clear[0]; + uint32_t operand = load_u32(clear + 4); + uint64_t value = load_u64(clear + 8); + Protected_wipe_guard kind_wipe( + &kind, sizeof(kind), Protected_wipe_context::DECODED_SCALAR); + Protected_wipe_guard operand_wipe( + &operand, sizeof(operand), Protected_wipe_context::DECODED_SCALAR); + Protected_wipe_guard value_wipe( + &value, sizeof(value), Protected_wipe_context::DECODED_SCALAR); + if (clear[1] != 0 || load_u16(clear + 2) != 0 || !all_zero(clear + 16, 16)) { + throw_malformed(index); + } + + if (index == 0) { + if (kind != 1 || (value & ~uint64_t(1)) != 0) { + throw_malformed(index); + } + if (operand > 4096) { + throw_size_limit(index); + } + binding_count = operand; + sink.manifest(binding_count, (value & 1) != 0, index); + continue; + } + + if (final_record) { + if (kind != 5 || operand != 0 || value != 0 || stack_depth != 1) { + throw_malformed(index); + } + for (size_t slot = 0; slot < binding_count; ++slot) { + if (!used_slots[slot]) { + throw_malformed(index); + } + } + sink.end(index); + continue; + } + + if (kind == 2) { + if (operand != 0) { + throw_malformed(index); + } + if (stack_depth == 1024) { + throw_size_limit(index); + } + double literal; + std::memcpy(&literal, &value, sizeof(literal)); + Protected_wipe_guard literal_wipe( + &literal, sizeof(literal), Protected_wipe_context::DECODED_SCALAR); + if (!std::isfinite(literal)) { + throw_malformed(index); + } + ++stack_depth; + sink.literal(literal, index); + } + else + if (kind == 3) { + if (value != 0 || operand >= binding_count) { + throw_malformed(index); + } + if (stack_depth == 1024) { + throw_size_limit(index); + } + ++stack_depth; + used_slots[operand] = true; + sink.variable(operand, index); + } + else + if (kind == 4) { + uint32_t arity = protected_operation_arity(operand); + Protected_wipe_guard arity_wipe( + &arity, sizeof(arity), Protected_wipe_context::DECODED_SCALAR); + if (value != 0 || arity == 0 || stack_depth < arity) { + throw_malformed(index); + } + stack_depth = stack_depth - arity + 1; + sink.call(static_cast(operand), index); + } + else { + throw_malformed(index); + } + } +} + + +} // namespace impl + + +} // namespace mexce + + +#endif diff --git a/mexce_protected_encoder.h b/mexce_protected_encoder.h new file mode 100644 index 0000000..e7f0a84 --- /dev/null +++ b/mexce_protected_encoder.h @@ -0,0 +1,391 @@ +#ifndef MEXCE_PROTECTED_ENCODER_H +#define MEXCE_PROTECTED_ENCODER_H + +#include "mexce_protected.h" + +#include +#include +#include +#include +#include +#include + + +// Windows defines STRICT as a source token, which would make the contracted +// Protected_math_mode::STRICT enumerator impossible to name at call sites. +#ifdef STRICT + #undef STRICT +#endif + + +namespace mexce { + + +enum class Protected_math_mode +{ + STRICT = 0, + FAST = 1, +}; + + +struct Protected_binding +{ + std::string name; + uint32_t slot; +}; + + +struct Protected_expression_bundle +{ + std::vector program; + Protected_expression_key key; + std::array program_id; +}; + + +namespace impl { + + +inline void store_u16(uint8_t* bytes, uint16_t value) +{ + bytes[0] = static_cast(value); + bytes[1] = static_cast(value >> 8); +} + + +inline void store_u64(uint8_t* bytes, uint64_t value) +{ + for (size_t i = 0; i < 8; ++i) { + bytes[i] = static_cast(value >> (i * 8)); + } +} + + +inline bool valid_protected_name(const std::string& name) +{ + if (name.empty() || name.size() > 255) { + return false; + } + const auto first = static_cast(name[0]); + if (!(first == '_' || (first >= 'A' && first <= 'Z') || + (first >= 'a' && first <= 'z'))) + { + return false; + } + for (size_t i = 1; i < name.size(); ++i) { + const auto c = static_cast(name[i]); + if (!(c == '_' || (c >= 'A' && c <= 'Z') || + (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9'))) + { + return false; + } + } + return true; +} + + +inline uint32_t protected_operation_id(const std::string& name) +{ + static const std::map ids = { + {"add", 1}, {"sub", 2}, {"mul", 3}, {"div", 4}, + {"neg", 5}, {"pow", 6}, {"sin", 7}, {"cos", 8}, + {"tan", 9}, {"abs", 10}, {"sign", 11}, {"signp", 12}, + {"expn", 13}, {"sfc", 14}, {"sqrt", 15}, {"exp", 16}, + {"lt", 17}, {"gt", 18}, {"le", 19}, {"ge", 20}, + {"eq", 21}, {"ne", 22}, {"log", 23}, {"ln", 23}, + {"log2", 24}, {"log10", 25},{"logb", 26}, {"ylog2", 27}, + {"max", 28}, {"min", 29}, {"floor", 30},{"ceil", 31}, + {"round", 32},{"int", 33}, {"trunc", 34},{"mod", 35}, + {"bnd", 36}, {"bias", 37}, {"gain", 38}, + }; + const auto found = ids.find(name); + return found == ids.end() ? 0 : found->second; +} + + +inline void validate_protected_schema( + const std::string& expression, + const std::vector& bindings, + Protected_math_mode math_mode) +{ + if (math_mode != Protected_math_mode::STRICT && math_mode != Protected_math_mode::FAST) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "The protected-expression math mode is invalid."); + } + if (expression.size() > 1024 * 1024) { + throw Protected_expression_error( + Protected_expression_error_category::SIZE_LIMIT, + "The protected-expression source exceeds a resource limit."); + } + if (bindings.size() > 4096) { + throw Protected_expression_error( + Protected_expression_error_category::SIZE_LIMIT, + "The protected-expression schema exceeds a resource limit."); + } + + size_t cumulative_size = 0; + std::set names; + std::set slots; + for (const auto& binding : bindings) { + cumulative_size += binding.name.size(); + if (binding.name.size() > 255 || cumulative_size > 256 * 1024) { + throw Protected_expression_error( + Protected_expression_error_category::SIZE_LIMIT, + "The protected-expression schema exceeds a resource limit."); + } + if (!valid_protected_name(binding.name) || + function_map().find(binding.name) != function_map().end() || + built_in_constants_map().find(binding.name) != built_in_constants_map().end() || + !names.insert(binding.name).second || !slots.insert(binding.slot).second) + { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "The protected-expression binding schema is invalid."); + } + } + for (uint32_t slot = 0; slot < bindings.size(); ++slot) { + if (slots.find(slot) == slots.end()) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "The protected-expression binding schema is invalid."); + } + } +} + + +class Protected_push_state +{ +public: + Protected_push_state() { std::memset(&m_state, 0, sizeof(m_state)); } + ~Protected_push_state() + { + protected_memzero( + &m_state, sizeof(m_state), Protected_wipe_context::PUSH_STATE); + } + + crypto_secretstream_xchacha20poly1305_state* get() { return &m_state; } + +private: + crypto_secretstream_xchacha20poly1305_state m_state; +}; + + +inline void append_encrypted_record( + std::vector& program, + Protected_push_state& push_state, + const uint8_t* record, + uint32_t index, + unsigned char tag) +{ + constexpr size_t k_header_size = 64; + constexpr size_t k_record_size = 32; + constexpr size_t k_frame_size = + k_record_size + crypto_secretstream_xchacha20poly1305_ABYTES; + uint8_t additional_data[k_header_size + 4] = {}; + Protected_wipe_guard additional_data_wipe( + additional_data, + sizeof(additional_data), + Protected_wipe_context::ADDITIONAL_DATA); + std::memcpy(additional_data, program.data(), k_header_size); + store_u32(additional_data + k_header_size, index); + + const size_t old_size = program.size(); + program.resize(old_size + k_frame_size); + unsigned long long frame_size = 0; + if (crypto_secretstream_xchacha20poly1305_push( + push_state.get(), program.data() + old_size, &frame_size, + record, k_record_size, + additional_data, sizeof(additional_data), tag) != 0 || + frame_size != k_frame_size) + { + throw Protected_expression_error( + Protected_expression_error_category::RESOURCE_FAILURE, + "Protected-expression encoding failed."); + } +} + + +} // namespace impl + + +namespace impl { + + +inline Protected_expression_bundle encode_protected_expression_impl( + const std::string& expression, + const std::vector& bindings, + Protected_math_mode math_mode) +{ + validate_protected_schema(expression, bindings, math_mode); + if (expression.empty()) { + throw mexce_parsing_exception("Expected an expression", 0); + } + + evaluator parser; + std::vector variables(bindings.size(), 0.0); + std::map slots; + for (const auto& binding : bindings) { + parser.bind(variables[binding.slot], binding.name); + slots.insert(std::make_pair(binding.name, binding.slot)); + } + + const Semantic_program semantic_program = + Clear_semantic_producer::produce(parser, expression); + if (semantic_program.size() + 2 > 16384) { + throw Protected_expression_error( + Protected_expression_error_category::SIZE_LIMIT, + "The protected expression exceeds a resource limit."); + } + + std::vector used_slots(bindings.size(), false); + size_t stack_depth = 0; + for (const auto& record : semantic_program) { + if (record.kind() == Semantic_record_kind::LITERAL) { + if (!std::isfinite(record.literal())) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "The protected expression contains a non-finite literal."); + } + ++stack_depth; + } + else + if (record.kind() == Semantic_record_kind::VARIABLE) { + used_slots[slots.find(record.variable()->name)->second] = true; + ++stack_depth; + } + else + if (record.kind() == Semantic_record_kind::CALL) { + const uint32_t operation = protected_operation_id(record.function()->name); + const uint32_t arity = protected_operation_arity(operation); + if (arity == 0) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "The protected expression contains an unsupported operation."); + } + stack_depth = stack_depth - arity + 1; + } + if (stack_depth > 1024) { + throw Protected_expression_error( + Protected_expression_error_category::SIZE_LIMIT, + "The protected expression exceeds a resource limit."); + } + } + for (size_t slot = 0; slot < used_slots.size(); ++slot) { + if (!used_slots[slot]) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "The protected-expression schema contains an unused binding."); + } + } + + require_sodium(); + uint8_t key_bytes[crypto_secretstream_xchacha20poly1305_KEYBYTES] = {}; + Protected_wipe_guard key_bytes_wipe( + key_bytes, sizeof(key_bytes), Protected_wipe_context::KEY); + crypto_secretstream_xchacha20poly1305_keygen(key_bytes); + Protected_expression_key key = Protected_expression_key::from_bytes( + key_bytes, sizeof(key_bytes)); + + std::array program_id; + do { + randombytes_buf(program_id.data(), program_id.size()); + } + while (all_zero(program_id.data(), program_id.size())); + + constexpr size_t k_header_size = 64; + constexpr size_t k_record_size = 32; + constexpr size_t k_frame_size = + k_record_size + crypto_secretstream_xchacha20poly1305_ABYTES; + const uint32_t record_count = static_cast(semantic_program.size() + 2); + std::vector program; + program.reserve(k_header_size + static_cast(record_count) * k_frame_size); + program.resize(k_header_size, 0); + const uint8_t magic[8] = {'M', 'E', 'X', 'C', 'E', 'P', 'R', 'G'}; + std::memcpy(program.data(), magic, sizeof(magic)); + store_u16(program.data() + 8, 1); + store_u16(program.data() + 10, 0); + store_u16(program.data() + 12, k_header_size); + store_u16(program.data() + 14, k_record_size); + store_u32(program.data() + 16, record_count); + std::memcpy(program.data() + 24, program_id.data(), program_id.size()); + + Protected_push_state push_state; + if (crypto_secretstream_xchacha20poly1305_init_push( + push_state.get(), program.data() + 40, + Protected_key_access::bytes(key)) != 0) + { + throw Protected_expression_error( + Protected_expression_error_category::RESOURCE_FAILURE, + "Protected-expression encoding failed."); + } + + uint8_t clear[k_record_size] = {}; + Protected_wipe_guard clear_wipe( + clear, sizeof(clear), Protected_wipe_context::CLEAR_RECORD); + clear[0] = 1; + store_u32(clear + 4, static_cast(bindings.size())); + store_u64(clear + 8, math_mode == Protected_math_mode::FAST ? 1 : 0); + append_encrypted_record( + program, push_state, clear, 0, + crypto_secretstream_xchacha20poly1305_TAG_MESSAGE); + + uint32_t index = 1; + for (const auto& record : semantic_program) { + protected_memzero( + clear, sizeof(clear), Protected_wipe_context::CLEAR_RECORD); + if (record.kind() == Semantic_record_kind::LITERAL) { + clear[0] = 2; + uint64_t bits; + const double literal = record.literal(); + std::memcpy(&bits, &literal, sizeof(bits)); + store_u64(clear + 8, bits); + } + else + if (record.kind() == Semantic_record_kind::VARIABLE) { + clear[0] = 3; + store_u32(clear + 4, slots.find(record.variable()->name)->second); + } + else { + clear[0] = 4; + store_u32(clear + 4, protected_operation_id(record.function()->name)); + } + append_encrypted_record( + program, push_state, clear, index++, + crypto_secretstream_xchacha20poly1305_TAG_MESSAGE); + } + + protected_memzero( + clear, sizeof(clear), Protected_wipe_context::CLEAR_RECORD); + clear[0] = 5; + append_encrypted_record( + program, push_state, clear, index, + crypto_secretstream_xchacha20poly1305_TAG_FINAL); + return Protected_expression_bundle{ + std::move(program), std::move(key), program_id}; +} + + +} // namespace impl + + +inline Protected_expression_bundle encode_protected_expression( + const std::string& expression, + const std::vector& bindings, + Protected_math_mode math_mode) +{ + try { + return impl::encode_protected_expression_impl(expression, bindings, math_mode); + } + catch (const std::bad_alloc&) { + throw Protected_expression_error( + Protected_expression_error_category::RESOURCE_FAILURE, + "Protected-expression storage allocation failed."); + } +} + + +} // namespace mexce + + +#endif diff --git a/test/fixtures/protected_format_1_0.bin b/test/fixtures/protected_format_1_0.bin new file mode 100644 index 0000000000000000000000000000000000000000..0d1f35a6d849c408b3f4f97a1525a6142d273b4f GIT binary patch literal 407 zcmV;I0cieBMOZ^cP*O($0000$03ZMd000000001>p`xRtrKYE-sj922t*)=$z(+c} zEQ&}A36H0Ah;ZtPVV^B?C*Dc?Bg5jHav9heQxHomZc=z4J1VdXw@QQ|)LD>sc@yslCojE5axpqM8mqK;ztC`PXeq7myDDv**I7`oFf zm1lMArnTWqpA>@asodyZiuPX0$F~(du7>Zx!_9OsORXUh&8=ObX=BM;|G>w|-iCNY z(LwM>^G8hny)USm(^d;%?QIQ)b#{#2;{M5}{ho^(!ZnSHej7i`!6k<0 z4F$7nkDpF)Y6h|sO@>Lk&e^Op0HNF>a(&i + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + + +namespace { + + +constexpr size_t k_ref_header_size = 64; +constexpr size_t k_ref_record_size = 32; +constexpr size_t k_ref_frame_size = 49; +const char* g_fixture_path = MEXCE_PROTECTED_FIXTURE_PATH; +const char* g_fixture_manifest_path = MEXCE_PROTECTED_FIXTURE_MANIFEST_PATH; + +static_assert(!std::is_default_constructible::value, + "Protected_expression_key must not be default constructible"); +static_assert(!std::is_copy_constructible::value, + "Protected_expression_key must not be copy constructible"); +static_assert(std::is_move_constructible::value, + "Protected_expression_key must be move constructible"); +static_assert(std::is_move_assignable::value, + "Protected_expression_key must be move assignable"); +static_assert(!std::is_copy_constructible::value, + "Protected_expression_bundle must not be copy constructible"); +static_assert(std::is_move_constructible::value, + "Protected_expression_bundle must be move constructible"); + + +struct Test_suite +{ + std::vector m_failures; + + void expect(bool value, const std::string& name) + { + if (!value) { + m_failures.push_back(name); + } + } + + template + void expect_error( + const std::string& name, + mexce::Protected_expression_error_category category, + bool has_index, + Function function) + { + try { + function(); + m_failures.push_back(name + ": expected exception"); + } + catch (const mexce::Protected_expression_error& error) { + if (error.category() != category || error.has_record_index() != has_index) { + m_failures.push_back(name + ": wrong category or index state"); + } + } + catch (const std::exception& error) { + m_failures.push_back(name + ": unexpected exception: " + error.what()); + } + } + + template + void expect_error_at( + const std::string& name, + mexce::Protected_expression_error_category category, + uint32_t record_index, + Function function) + { + try { + function(); + m_failures.push_back(name + ": expected exception"); + } + catch (const mexce::Protected_expression_error& error) { + if (error.category() != category || !error.has_record_index() || + error.record_index() != record_index) + { + m_failures.push_back(name + ": wrong category or record index"); + } + } + catch (const std::exception& error) { + m_failures.push_back(name + ": unexpected exception: " + error.what()); + } + } +}; + + +struct Collecting_sink +{ + std::vector m_records; + std::vector m_literal_bits; + + void manifest(uint32_t bindings, bool fast_math, uint32_t) + { + m_records.push_back( + std::string("manifest:") + std::to_string(bindings) + + (fast_math ? ":fast" : ":strict")); + } + + void literal(double value, uint32_t) + { + uint64_t bits; + std::memcpy(&bits, &value, sizeof(bits)); + m_literal_bits.push_back(bits); + std::ostringstream stream; + stream.precision(17); + stream << "literal:" << value; + m_records.push_back(stream.str()); + } + + void variable(uint32_t slot, uint32_t) + { + m_records.push_back("variable:" + std::to_string(slot)); + } + + void call(mexce::impl::Protected_operation operation, uint32_t) + { + m_records.push_back("call:" + std::to_string(static_cast(operation))); + } + + void end(uint32_t) { m_records.push_back("end"); } +}; + + +struct Const_key_consumer +{ + bool& m_called; + + void operator()(const uint8_t*, size_t) { m_called = true; } + void operator()(uint8_t*, size_t) = delete; +}; + + +std::array fixture_key_bytes() +{ + std::array key; + for (size_t i = 0; i < key.size(); ++i) { + key[i] = static_cast(i); + } + return key; +} + + +mexce::Protected_expression_key fixture_key() +{ + const auto bytes = fixture_key_bytes(); + return mexce::Protected_expression_key::from_bytes(bytes.data(), bytes.size()); +} + + +std::vector read_fixture() +{ + std::ifstream input(g_fixture_path, std::ios::binary); + if (!input) { + throw std::runtime_error("protected fixture could not be opened"); + } + std::vector program( + (std::istreambuf_iterator(input)), std::istreambuf_iterator()); + if (program.size() != k_ref_header_size + 7 * k_ref_frame_size) { + throw std::runtime_error("protected fixture has the wrong size"); + } + return program; +} + + +std::map read_fixture_manifest() +{ + std::ifstream input(g_fixture_manifest_path); + if (!input) { + throw std::runtime_error("protected fixture manifest could not be opened"); + } + + std::map fields; + std::string line; + while (std::getline(input, line)) { + if (!line.empty() && line.back() == '\r') { + line.pop_back(); + } + const size_t separator = line.find('='); + if (separator == std::string::npos || + !fields.insert(std::make_pair( + line.substr(0, separator), line.substr(separator + 1))).second) + { + throw std::runtime_error("protected fixture manifest is malformed"); + } + } + return fields; +} + + +void ref_store_u16(uint8_t* bytes, uint16_t value) +{ + bytes[0] = static_cast(value); + bytes[1] = static_cast(value >> 8); +} + + +void ref_store_u32(uint8_t* bytes, uint32_t value) +{ + for (size_t i = 0; i < 4; ++i) { + bytes[i] = static_cast(value >> (i * 8)); + } +} + + +void ref_store_u64(uint8_t* bytes, uint64_t value) +{ + for (size_t i = 0; i < 8; ++i) { + bytes[i] = static_cast(value >> (i * 8)); + } +} + + +std::array ref_record( + uint8_t kind, + uint32_t operand_32 = 0, + uint64_t operand_64 = 0) +{ + std::array result = {}; + result[0] = kind; + ref_store_u32(result.data() + 4, operand_32); + ref_store_u64(result.data() + 8, operand_64); + return result; +} + + +std::vector reference_encrypt_with_tags( + const std::vector>& records, + const std::vector& tags) +{ + if (records.size() != tags.size()) { + throw std::runtime_error("reference record/tag count mismatch"); + } + const auto key = fixture_key_bytes(); + std::vector program(k_ref_header_size + records.size() * k_ref_frame_size, 0); + const uint8_t magic[8] = {'M', 'E', 'X', 'C', 'E', 'P', 'R', 'G'}; + std::memcpy(program.data(), magic, sizeof(magic)); + ref_store_u16(program.data() + 8, 1); + ref_store_u16(program.data() + 10, 0); + ref_store_u16(program.data() + 12, k_ref_header_size); + ref_store_u16(program.data() + 14, k_ref_record_size); + ref_store_u32(program.data() + 16, static_cast(records.size())); + program[24] = 1; + + crypto_secretstream_xchacha20poly1305_state state; + if (crypto_secretstream_xchacha20poly1305_init_push( + &state, program.data() + 40, key.data()) != 0) + { + throw std::runtime_error("reference init_push failed"); + } + for (uint32_t i = 0; i < records.size(); ++i) { + uint8_t additional_data[k_ref_header_size + 4]; + std::memcpy(additional_data, program.data(), k_ref_header_size); + ref_store_u32(additional_data + k_ref_header_size, i); + unsigned long long size = 0; + if (crypto_secretstream_xchacha20poly1305_push( + &state, + program.data() + k_ref_header_size + static_cast(i) * k_ref_frame_size, + &size, records[i].data(), records[i].size(), + additional_data, sizeof(additional_data), tags[i]) != 0 || + size != k_ref_frame_size) + { + throw std::runtime_error("reference push failed"); + } + } + sodium_memzero(&state, sizeof(state)); + return program; +} + + +std::vector reference_encrypt( + const std::vector>& records) +{ + std::vector tags( + records.size(), crypto_secretstream_xchacha20poly1305_TAG_MESSAGE); + tags.back() = crypto_secretstream_xchacha20poly1305_TAG_FINAL; + return reference_encrypt_with_tags(records, tags); +} + + +void decode(const std::vector& program, Collecting_sink& sink) +{ + mexce::impl::decode_protected_expression( + program.data(), program.size(), fixture_key(), sink); +} + + +void test_fixture_and_encoder(Test_suite& suite) +{ + const auto program = read_fixture(); + const auto manifest = read_fixture_manifest(); + const std::string expected_key_hex = + "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f"; + const std::string expected_program_id_hex = + "a0a1a2a3a4a5a6a7a8a9aaabacadaeaf"; + const std::string expected_records = + "MANIFEST(2,strict),VARIABLE(0),LITERAL_F64(2.5)," + "CALL(MUL),VARIABLE(1),CALL(ADD),END"; + suite.expect(manifest.size() == 4, "fixture manifest field count"); + suite.expect(manifest.at("format") == "1.0", "fixture manifest format"); + suite.expect(manifest.at("key_hex") == expected_key_hex, "fixture manifest key"); + suite.expect( + manifest.at("program_id_hex") == expected_program_id_hex, + "fixture manifest program id"); + suite.expect(manifest.at("records") == expected_records, "fixture manifest records"); + suite.expect(program.size() == 407, "fixture size"); + suite.expect(std::equal(program.begin(), program.begin() + 8, "MEXCEPRG"), "fixture magic"); + suite.expect(program[8] == 1 && program[9] == 0 && program[10] == 0, "fixture version"); + suite.expect(program[12] == 64 && program[14] == 32 && program[16] == 7, "fixture framing"); + bool fixture_program_id_matches = true; + for (size_t i = 0; i < 16; ++i) { + fixture_program_id_matches &= program[24 + i] == static_cast(0xa0 + i); + } + suite.expect(fixture_program_id_matches, "fixture binary program id matches manifest"); + + Collecting_sink fixture_sink; + auto& wipe_observer = mexce::impl::protected_wipe_observer(); + using Wipe_context = mexce::impl::Protected_wipe_context; + wipe_observer.reset(); + decode(program, fixture_sink); + const std::vector expected = { + "manifest:2:strict", "variable:0", "literal:2.5", "call:3", + "variable:1", "call:1", "end", + }; + suite.expect(fixture_sink.m_records == expected, "independent fixture records"); + suite.expect( + wipe_observer.count(Wipe_context::KEY) > 0 && + wipe_observer.count(Wipe_context::PULL_STATE) > 0 && + wipe_observer.count(Wipe_context::CLEAR_RECORD) > 0 && + wipe_observer.count(Wipe_context::ADDITIONAL_DATA) > 0 && + wipe_observer.count(Wipe_context::DECODED_SCALAR) > 0 && + wipe_observer.count(Wipe_context::VALIDATOR_STATE) > 0 && + wipe_observer.all_observed_regions_are_zero(), + "decoder wipes each transient context to zero"); + + const std::vector bindings = {{"x", 0}, {"y", 1}}; + wipe_observer.reset(); + auto bundle = mexce::encode_protected_expression( + "x * 2.5 + y", bindings, mexce::Protected_math_mode::STRICT); + suite.expect( + wipe_observer.count(Wipe_context::KEY) > 0 && + wipe_observer.count(Wipe_context::PUSH_STATE) > 0 && + wipe_observer.count(Wipe_context::CLEAR_RECORD) > 0 && + wipe_observer.count(Wipe_context::ADDITIONAL_DATA) > 0 && + wipe_observer.all_observed_regions_are_zero(), + "encoder wipes each transient context to zero"); + suite.expect(bundle.program.size() == 407, "encoder blob size"); + suite.expect( + std::equal(bundle.program.begin(), bundle.program.begin() + 8, "MEXCEPRG"), + "encoder magic"); + suite.expect( + bundle.program[8] == 1 && bundle.program[9] == 0 && + bundle.program[10] == 0 && bundle.program[11] == 0 && + bundle.program[12] == 64 && bundle.program[13] == 0 && + bundle.program[14] == 32 && bundle.program[15] == 0 && + bundle.program[16] == 7 && bundle.program[17] == 0 && + bundle.program[18] == 0 && bundle.program[19] == 0, + "encoder version and framing fields"); + suite.expect( + std::equal( + bundle.program.begin() + 24, bundle.program.begin() + 40, + bundle.program_id.begin()), + "encoder program id field"); + Collecting_sink encoded_sink; + mexce::impl::decode_protected_expression( + bundle.program.data(), bundle.program.size(), std::move(bundle.key), encoded_sink); + suite.expect(encoded_sink.m_records == expected, "encoder uses canonical semantic records"); + suite.expect_error( + "successful decode consumes transferred key", + mexce::Protected_expression_error_category::INVALID_ARGUMENT, false, + [&] { bundle.key.consume_bytes([](const uint8_t*, size_t) {}); }); + + auto second_bundle = mexce::encode_protected_expression( + "x * 2.5 + y", bindings, mexce::Protected_math_mode::STRICT); + suite.expect(bundle.program != second_bundle.program, "encoder uses fresh artifact randomness"); + suite.expect(!mexce::impl::all_zero( + second_bundle.program_id.data(), second_bundle.program_id.size()), + "encoder program id is nonzero"); + + auto separation_bundle = mexce::encode_protected_expression( + "x * 2.5 + y", bindings, mexce::Protected_math_mode::STRICT); + std::array second_key = {}; + std::array separation_key = {}; + second_bundle.key.consume_bytes([&](const uint8_t* bytes, size_t size) { + std::copy(bytes, bytes + size, second_key.begin()); + }); + separation_bundle.key.consume_bytes([&](const uint8_t* bytes, size_t size) { + std::copy(bytes, bytes + size, separation_key.begin()); + }); + suite.expect(second_key != separation_key, "encoder generates a fresh key per artifact"); + + auto fast_bundle = mexce::encode_protected_expression( + "x+y", bindings, mexce::Protected_math_mode::FAST); + Collecting_sink fast_sink; + mexce::impl::decode_protected_expression( + fast_bundle.program.data(), fast_bundle.program.size(), + std::move(fast_bundle.key), fast_sink); + suite.expect(fast_sink.m_records.front() == "manifest:2:fast", "fast policy round trip"); + + auto unary_minus_bundle = mexce::encode_protected_expression( + "-x", {{"x", 0}}, mexce::Protected_math_mode::STRICT); + Collecting_sink unary_minus_sink; + mexce::impl::decode_protected_expression( + unary_minus_bundle.program.data(), unary_minus_bundle.program.size(), + std::move(unary_minus_bundle.key), unary_minus_sink); + const std::vector unary_minus_expected = { + "manifest:1:strict", "literal:0", "variable:0", "call:2", "end", + }; + suite.expect( + unary_minus_sink.m_records == unary_minus_expected, + "source unary minus canonical records"); + + auto unary_plus_bundle = mexce::encode_protected_expression( + "+x", {{"x", 0}}, mexce::Protected_math_mode::STRICT); + Collecting_sink unary_plus_sink; + mexce::impl::decode_protected_expression( + unary_plus_bundle.program.data(), unary_plus_bundle.program.size(), + std::move(unary_plus_bundle.key), unary_plus_sink); + const std::vector unary_plus_expected = { + "manifest:1:strict", "variable:0", "end", + }; + suite.expect( + unary_plus_sink.m_records == unary_plus_expected, + "source unary plus emits no operation"); + + auto constants_bundle = mexce::encode_protected_expression( + "pi+e", {}, mexce::Protected_math_mode::STRICT); + Collecting_sink constants_sink; + mexce::impl::decode_protected_expression( + constants_bundle.program.data(), constants_bundle.program.size(), + std::move(constants_bundle.key), constants_sink); + suite.expect( + constants_sink.m_literal_bits.size() == 2 && + std::find(constants_sink.m_records.begin(), constants_sink.m_records.end(), "call:1") != + constants_sink.m_records.end(), + "built-in constants become literals"); +} + + +void test_operation_ids_and_aliases(Test_suite& suite) +{ + struct operation_case_t + { + const char* expression; + uint32_t operation; + bool binary; + }; + const operation_case_t cases[] = { + {"add(x,y)", 1, true}, {"sub(x,y)", 2, true}, {"mul(x,y)", 3, true}, + {"div(x,y)", 4, true}, {"neg(x)", 5, false}, {"pow(x,y)", 6, true}, + {"sin(x)", 7, false}, {"cos(x)", 8, false}, {"tan(x)", 9, false}, + {"abs(x)", 10, false}, {"sign(x)", 11, false}, {"signp(x)", 12, false}, + {"expn(x)", 13, false}, {"sfc(x)", 14, false}, {"sqrt(x)", 15, false}, + {"exp(x)", 16, false}, {"lt(x,y)", 17, true}, {"gt(x,y)", 18, true}, + {"le(x,y)", 19, true}, {"ge(x,y)", 20, true}, {"eq(x,y)", 21, true}, + {"ne(x,y)", 22, true}, {"log(x)", 23, false}, {"log2(x)", 24, false}, + {"log10(x)", 25, false}, {"logb(x,y)", 26, true}, {"ylog2(x,y)", 27, true}, + {"max(x,y)", 28, true}, {"min(x,y)", 29, true}, {"floor(x)", 30, false}, + {"ceil(x)", 31, false}, {"round(x)", 32, false}, {"int(x)", 33, false}, + {"trunc(x)", 34, false}, {"mod(x,y)", 35, true}, {"bnd(x,y)", 36, true}, + {"bias(x,y)", 37, true}, {"gain(x,y)", 38, true}, + {"x+y", 1, true}, {"x-y", 2, true}, {"x*y", 3, true}, + {"x/y", 4, true}, {"x^y", 6, true}, {"x**y", 6, true}, + {"xy", 18, true}, {"ln(x)", 23, false}, + }; + + for (const auto& operation_case : cases) { + std::vector bindings = {{"x", 0}}; + if (operation_case.binary) { + bindings.push_back({"y", 1}); + } + auto bundle = mexce::encode_protected_expression( + operation_case.expression, bindings, mexce::Protected_math_mode::STRICT); + Collecting_sink sink; + mexce::impl::decode_protected_expression( + bundle.program.data(), bundle.program.size(), std::move(bundle.key), sink); + std::vector expected_records = { + std::string("manifest:") + (operation_case.binary ? "2" : "1") + ":strict", + "variable:0", + }; + if (operation_case.binary) { + expected_records.push_back("variable:1"); + } + expected_records.push_back("call:" + std::to_string(operation_case.operation)); + expected_records.push_back("end"); + suite.expect( + sink.m_records == expected_records, + std::string("canonical operation records: ") + operation_case.expression); + + std::vector> reference_records; + reference_records.push_back(ref_record(1, operation_case.binary ? 2 : 1)); + reference_records.push_back(ref_record(3, 0)); + if (operation_case.binary) { + reference_records.push_back(ref_record(3, 1)); + } + reference_records.push_back(ref_record(4, operation_case.operation)); + reference_records.push_back(ref_record(5)); + auto reference_program = reference_encrypt(reference_records); + Collecting_sink reference_sink; + decode(reference_program, reference_sink); + suite.expect( + reference_sink.m_records == expected_records, + std::string("independent operation arity: ") + operation_case.expression); + } + + const uint64_t finite_literal_bits[] = { + 0x0000000000000000ULL, + 0x8000000000000000ULL, + 0x0000000000000001ULL, + 0x7fefffffffffffffULL, + 0xffefffffffffffffULL, + }; + for (const uint64_t bits : finite_literal_bits) { + auto literal_program = reference_encrypt({ + ref_record(1), ref_record(2, 0, bits), ref_record(5)}); + Collecting_sink literal_sink; + decode(literal_program, literal_sink); + suite.expect( + literal_sink.m_literal_bits.size() == 1 && + literal_sink.m_literal_bits.front() == bits, + "finite literal bits: " + std::to_string(bits)); + } +} + + +void test_key_ownership(Test_suite& suite) +{ + auto bytes = fixture_key_bytes(); + auto& wipe_observer = mexce::impl::protected_wipe_observer(); + using Wipe_context = mexce::impl::Protected_wipe_context; + suite.expect_error( + "null key", mexce::Protected_expression_error_category::INVALID_ARGUMENT, false, + [&] { (void)mexce::Protected_expression_key::from_bytes(nullptr, bytes.size()); }); + suite.expect_error( + "short key", mexce::Protected_expression_error_category::INVALID_ARGUMENT, false, + [&] { (void)mexce::Protected_expression_key::from_bytes(bytes.data(), bytes.size() - 1); }); + + auto copied_key = mexce::Protected_expression_key::from_bytes(bytes.data(), bytes.size()); + std::fill(bytes.begin(), bytes.end(), uint8_t(0xff)); + bool copied_bytes_match = false; + copied_key.consume_bytes([&](const uint8_t* owned, size_t size) { + const auto expected = fixture_key_bytes(); + copied_bytes_match = size == expected.size() && + std::equal(owned, owned + size, expected.begin()); + }); + suite.expect(copied_bytes_match, "key factory copies caller bytes"); + + bool const_consumer_called = false; + auto const_consumer_key = fixture_key(); + Const_key_consumer const_consumer = {const_consumer_called}; + const_consumer_key.consume_bytes(const_consumer); + suite.expect(const_consumer_called, "key consumer receives only const bytes"); + + auto key = fixture_key(); + auto moved = std::move(key); + suite.expect_error( + "moved-from key", mexce::Protected_expression_error_category::INVALID_ARGUMENT, false, + [&] { key.consume_bytes([](const uint8_t*, size_t) {}); }); + + bool recursive_rejected = false; + moved.consume_bytes([&](const uint8_t*, size_t) { + try { + moved.consume_bytes([](const uint8_t*, size_t) {}); + } + catch (const mexce::Protected_expression_error&) { + recursive_rejected = true; + } + }); + suite.expect(recursive_rejected, "recursive consumption rejected"); + suite.expect_error( + "second consumption", mexce::Protected_expression_error_category::INVALID_ARGUMENT, false, + [&] { moved.consume_bytes([](const uint8_t*, size_t) {}); }); + + auto move_in_callback = fixture_key(); + bool callback_move_observed_empty = false; + move_in_callback.consume_bytes([&](const uint8_t*, size_t) { + auto moved_inside = std::move(move_in_callback); + try { + moved_inside.consume_bytes([](const uint8_t*, size_t) {}); + } + catch (const mexce::Protected_expression_error&) { + callback_move_observed_empty = true; + } + }); + suite.expect(callback_move_observed_empty, "callback move observes empty key"); + + auto throwing = fixture_key(); + bool callback_exception_rethrown = false; + wipe_observer.reset(); + try { + throwing.consume_bytes([](const uint8_t*, size_t) { + throw std::runtime_error("callback"); + }); + } + catch (const std::runtime_error& error) { + callback_exception_rethrown = std::string(error.what()) == "callback"; + } + suite.expect(callback_exception_rethrown, "callback exception is rethrown unchanged"); + suite.expect( + wipe_observer.count(Wipe_context::KEY) > 0 && + wipe_observer.all_observed_regions_are_zero(), + "callback exception wipes key to zero"); + suite.expect_error( + "callback exception consumes key", + mexce::Protected_expression_error_category::INVALID_ARGUMENT, false, + [&] { throwing.consume_bytes([](const uint8_t*, size_t) {}); }); + + auto assigned = fixture_key(); + auto assignment_source = fixture_key(); + wipe_observer.reset(); + assigned = std::move(assignment_source); + suite.expect( + wipe_observer.count(Wipe_context::KEY) > 0 && + wipe_observer.all_observed_regions_are_zero(), + "move assignment wipes replaced key to zero"); + assigned.consume_bytes([](const uint8_t*, size_t) {}); + suite.expect_error( + "move-assignment source empty", + mexce::Protected_expression_error_category::INVALID_ARGUMENT, false, + [&] { assignment_source.consume_bytes([](const uint8_t*, size_t) {}); }); + + auto& faults = mexce::impl::protected_test_faults(); + faults.fail_malloc = true; + suite.expect_error( + "guarded allocation failure", + mexce::Protected_expression_error_category::RESOURCE_FAILURE, false, + [&] { (void)mexce::Protected_expression_key::from_bytes(bytes.data(), bytes.size()); }); + faults.fail_malloc = false; + + wipe_observer.reset(); + faults.fail_mlock = true; + suite.expect_error( + "mlock failure", mexce::Protected_expression_error_category::RESOURCE_FAILURE, false, + [&] { (void)mexce::Protected_expression_key::from_bytes(bytes.data(), bytes.size()); }); + faults.fail_mlock = false; + suite.expect( + wipe_observer.count(Wipe_context::KEY) > 0 && + wipe_observer.all_observed_regions_are_zero(), + "mlock failure wipes allocation to zero"); +} + + +void test_public_framing(Test_suite& suite) +{ + const auto fixture = read_fixture(); + Collecting_sink sink; + suite.expect_error( + "empty", mexce::Protected_expression_error_category::INVALID_ARGUMENT, false, + [&] { mexce::impl::decode_protected_expression(nullptr, 0, fixture_key(), sink); }); + suite.expect_error( + "short", mexce::Protected_expression_error_category::MALFORMED_PROGRAM, false, + [&] { + mexce::impl::decode_protected_expression( + fixture.data(), 63, fixture_key(), sink); + }); + + struct mutation_t + { + size_t offset; + mexce::Protected_expression_error_category category; + }; + const mutation_t mutations[] = { + {0, mexce::Protected_expression_error_category::UNSUPPORTED_FORMAT}, + {8, mexce::Protected_expression_error_category::UNSUPPORTED_FORMAT}, + {10, mexce::Protected_expression_error_category::UNSUPPORTED_FORMAT}, + {12, mexce::Protected_expression_error_category::UNSUPPORTED_FORMAT}, + {14, mexce::Protected_expression_error_category::UNSUPPORTED_FORMAT}, + {20, mexce::Protected_expression_error_category::UNSUPPORTED_FORMAT}, + {24, mexce::Protected_expression_error_category::MALFORMED_PROGRAM}, + }; + for (const auto& mutation : mutations) { + auto changed = fixture; + changed[mutation.offset] ^= 1; + if (mutation.offset == 24) { + std::fill(changed.begin() + 24, changed.begin() + 40, 0); + } + suite.expect_error( + "public mutation " + std::to_string(mutation.offset), + mutation.category, false, + [&] { decode(changed, sink); }); + } + + auto oversized = fixture; + ref_store_u32(oversized.data() + 16, 16385); + suite.expect_error( + "record limit", mexce::Protected_expression_error_category::SIZE_LIMIT, false, + [&] { decode(oversized, sink); }); + auto too_few = fixture; + ref_store_u32(too_few.data() + 16, 2); + too_few.resize(k_ref_header_size + 2 * k_ref_frame_size); + suite.expect_error( + "too few records", mexce::Protected_expression_error_category::MALFORMED_PROGRAM, false, + [&] { decode(too_few, sink); }); + + auto truncated = fixture; + truncated.pop_back(); + auto truncated_key = fixture_key(); + suite.expect_error( + "truncated", mexce::Protected_expression_error_category::MALFORMED_PROGRAM, false, + [&] { + mexce::impl::decode_protected_expression( + truncated.data(), truncated.size(), std::move(truncated_key), sink); + }); + suite.expect_error( + "failed decode consumes transferred key", + mexce::Protected_expression_error_category::INVALID_ARGUMENT, false, + [&] { truncated_key.consume_bytes([](const uint8_t*, size_t) {}); }); + auto trailing = fixture; + trailing.push_back(0); + suite.expect_error( + "trailing", mexce::Protected_expression_error_category::MALFORMED_PROGRAM, false, + [&] { decode(trailing, sink); }); + + auto compensated_removal = fixture; + ref_store_u32(compensated_removal.data() + 16, 6); + compensated_removal.resize(k_ref_header_size + 6 * k_ref_frame_size); + suite.expect_error_at( + "compensated frame removal", + mexce::Protected_expression_error_category::AUTHENTICATION_FAILED, 0, + [&] { decode(compensated_removal, sink); }); + + auto compensated_insertion = fixture; + ref_store_u32(compensated_insertion.data() + 16, 8); + compensated_insertion.resize(k_ref_header_size + 8 * k_ref_frame_size); + suite.expect_error_at( + "compensated frame insertion", + mexce::Protected_expression_error_category::AUTHENTICATION_FAILED, 0, + [&] { decode(compensated_insertion, sink); }); +} + + +void test_authentication(Test_suite& suite) +{ + const auto fixture = read_fixture(); + Collecting_sink sink; + + auto wrong_bytes = fixture_key_bytes(); + wrong_bytes[0] ^= 1; + auto& wipe_observer = mexce::impl::protected_wipe_observer(); + using Wipe_context = mexce::impl::Protected_wipe_context; + wipe_observer.reset(); + suite.expect_error_at( + "wrong key", mexce::Protected_expression_error_category::AUTHENTICATION_FAILED, 0, + [&] { + mexce::impl::decode_protected_expression( + fixture.data(), fixture.size(), + mexce::Protected_expression_key::from_bytes( + wrong_bytes.data(), wrong_bytes.size()), + sink); + }); + suite.expect( + wipe_observer.count(Wipe_context::KEY) > 0 && + wipe_observer.count(Wipe_context::PULL_STATE) > 0 && + wipe_observer.count(Wipe_context::CLEAR_RECORD) > 0 && + wipe_observer.count(Wipe_context::ADDITIONAL_DATA) > 0 && + wipe_observer.count(Wipe_context::VALIDATOR_STATE) > 0 && + wipe_observer.all_observed_regions_are_zero(), + "authentication failure wipes each transient context to zero"); + + auto late_failure = fixture; + late_failure.back() ^= 1; + wipe_observer.reset(); + suite.expect_error_at( + "late authentication failure", + mexce::Protected_expression_error_category::AUTHENTICATION_FAILED, 6, + [&] { decode(late_failure, sink); }); + suite.expect( + wipe_observer.count(Wipe_context::KEY) > 0 && + wipe_observer.count(Wipe_context::PULL_STATE) > 0 && + wipe_observer.count(Wipe_context::CLEAR_RECORD) > 0 && + wipe_observer.count(Wipe_context::ADDITIONAL_DATA) > 0 && + wipe_observer.count(Wipe_context::DECODED_SCALAR) > 0 && + wipe_observer.count(Wipe_context::VALIDATOR_STATE) > 0 && + wipe_observer.all_observed_regions_are_zero(), + "late authentication failure wipes populated transient state to zero"); + + for (size_t offset = 40; offset < fixture.size(); ++offset) { + auto changed = fixture; + changed[offset] ^= 1; + const uint32_t record_index = offset < k_ref_header_size + ? 0 + : static_cast((offset - k_ref_header_size) / k_ref_frame_size); + suite.expect_error_at( + "authenticated byte " + std::to_string(offset), + mexce::Protected_expression_error_category::AUTHENTICATION_FAILED, + record_index, + [&] { decode(changed, sink); }); + } + + for (size_t offset = 24; offset < 40; ++offset) { + auto changed = fixture; + changed[offset] ^= 1; + suite.expect_error_at( + "program id byte " + std::to_string(offset), + mexce::Protected_expression_error_category::AUTHENTICATION_FAILED, 0, + [&] { decode(changed, sink); }); + } + + auto swapped = fixture; + std::swap_ranges( + swapped.begin() + k_ref_header_size, + swapped.begin() + k_ref_header_size + k_ref_frame_size, + swapped.begin() + k_ref_header_size + k_ref_frame_size); + suite.expect_error_at( + "frame swap", mexce::Protected_expression_error_category::AUTHENTICATION_FAILED, 0, + [&] { decode(swapped, sink); }); + + auto duplicated = fixture; + std::copy( + duplicated.begin() + k_ref_header_size + k_ref_frame_size, + duplicated.begin() + k_ref_header_size + 2 * k_ref_frame_size, + duplicated.begin() + k_ref_header_size + 2 * k_ref_frame_size); + suite.expect_error_at( + "frame duplication", mexce::Protected_expression_error_category::AUTHENTICATION_FAILED, 2, + [&] { decode(duplicated, sink); }); + + auto& faults = mexce::impl::protected_test_faults(); + wipe_observer.reset(); + faults.fail_init_pull = true; + suite.expect_error( + "init pull", mexce::Protected_expression_error_category::AUTHENTICATION_FAILED, false, + [&] { decode(fixture, sink); }); + faults.fail_init_pull = false; + suite.expect( + wipe_observer.count(Wipe_context::KEY) > 0 && + wipe_observer.count(Wipe_context::PULL_STATE) > 0 && + wipe_observer.all_observed_regions_are_zero(), + "init-pull failure wipes key and stream state to zero"); +} + + +void expect_malformed_records( + Test_suite& suite, + const std::string& name, + const std::vector>& records, + uint32_t record_index) +{ + auto program = reference_encrypt(records); + Collecting_sink sink; + suite.expect_error_at( + name, mexce::Protected_expression_error_category::MALFORMED_PROGRAM, record_index, + [&] { decode(program, sink); }); +} + + +void test_semantic_validation(Test_suite& suite) +{ + const auto manifest = ref_record(1, 0, 0); + const auto literal = ref_record(2, 0, 0x3ff0000000000000ULL); + const auto end = ref_record(5); + + std::vector> valid_records = { + ref_record(1, 1), + ref_record(3, 0), + literal, + ref_record(4, 1), + end, + }; + const size_t reserved_offsets[] = { + 1, 2, 3, + 16, 17, 18, 19, 20, 21, 22, 23, + 24, 25, 26, 27, 28, 29, 30, 31, + }; + for (uint32_t index = 0; index < valid_records.size(); ++index) { + for (const size_t offset : reserved_offsets) { + auto changed = valid_records; + changed[index][offset] = 1; + expect_malformed_records( + suite, + "reserved byte " + std::to_string(index) + ":" + std::to_string(offset), + changed, + index); + } + } + + for (size_t offset = 4; offset < 8; ++offset) { + auto changed = valid_records; + changed[2][offset] = 1; + expect_malformed_records( + suite, "literal unused byte " + std::to_string(offset), changed, 2); + } + for (size_t offset = 8; offset < 16; ++offset) { + auto variable_changed = valid_records; + variable_changed[1][offset] = 1; + expect_malformed_records( + suite, "variable unused byte " + std::to_string(offset), variable_changed, 1); + + auto call_changed = valid_records; + call_changed[3][offset] = 1; + expect_malformed_records( + suite, "call unused byte " + std::to_string(offset), call_changed, 3); + } + for (size_t offset = 4; offset < 16; ++offset) { + auto changed = valid_records; + changed[4][offset] = 1; + expect_malformed_records( + suite, "end unused byte " + std::to_string(offset), changed, 4); + } + + expect_malformed_records(suite, "unknown kind", {manifest, ref_record(9), end}, 1); + expect_malformed_records(suite, "unknown operation", { + manifest, literal, literal, ref_record(4, 39), end}, 3); + expect_malformed_records(suite, "stack underflow", { + manifest, literal, ref_record(4, 1), end}, 2); + expect_malformed_records(suite, "terminal depth", { + manifest, literal, literal, end}, 3); + expect_malformed_records(suite, "nonfinite", { + manifest, ref_record(2, 0, 0x7ff0000000000000ULL), end}, 1); + expect_malformed_records(suite, "slot out of range", { + ref_record(1, 1), ref_record(3, 1), end}, 1); + expect_malformed_records(suite, "unused slot", { + ref_record(1, 1), literal, end}, 2); + expect_malformed_records(suite, "manifest policy", { + ref_record(1, 0, 2), literal, end}, 0); + expect_malformed_records(suite, "early end", {manifest, end, end}, 1); + + auto excessive_bindings = reference_encrypt({ref_record(1, 4097), literal, end}); + Collecting_sink excessive_bindings_sink; + suite.expect_error_at( + "authenticated binding limit", mexce::Protected_expression_error_category::SIZE_LIMIT, 0, + [&] { decode(excessive_bindings, excessive_bindings_sink); }); + + std::vector> deep_records; + deep_records.push_back(manifest); + deep_records.insert(deep_records.end(), 1025, literal); + deep_records.push_back(end); + auto deep_program = reference_encrypt(deep_records); + Collecting_sink deep_sink; + suite.expect_error_at( + "authenticated semantic depth limit", + mexce::Protected_expression_error_category::SIZE_LIMIT, 1025, + [&] { decode(deep_program, deep_sink); }); + + const std::vector> tag_records = { + manifest, literal, end}; + Collecting_sink sink; + const unsigned char message = crypto_secretstream_xchacha20poly1305_TAG_MESSAGE; + const unsigned char final = crypto_secretstream_xchacha20poly1305_TAG_FINAL; + const unsigned char push = crypto_secretstream_xchacha20poly1305_TAG_PUSH; + const unsigned char rekey = crypto_secretstream_xchacha20poly1305_TAG_REKEY; + + auto missing_final = reference_encrypt_with_tags( + tag_records, {message, message, message}); + suite.expect_error_at( + "missing final tag", mexce::Protected_expression_error_category::MALFORMED_PROGRAM, 2, + [&] { decode(missing_final, sink); }); + + auto nonfinal_final = reference_encrypt_with_tags( + tag_records, {message, final, final}); + suite.expect_error_at( + "non-final final tag", mexce::Protected_expression_error_category::MALFORMED_PROGRAM, 1, + [&] { decode(nonfinal_final, sink); }); + + auto push_tag = reference_encrypt_with_tags( + tag_records, {message, push, final}); + suite.expect_error_at( + "push tag", mexce::Protected_expression_error_category::MALFORMED_PROGRAM, 1, + [&] { decode(push_tag, sink); }); + + auto rekey_tag = reference_encrypt_with_tags( + tag_records, {message, rekey, final}); + suite.expect_error_at( + "rekey tag", mexce::Protected_expression_error_category::MALFORMED_PROGRAM, 1, + [&] { decode(rekey_tag, sink); }); +} + + +void test_encoder_validation(Test_suite& suite) +{ + using Category = mexce::Protected_expression_error_category; + const std::vector xy = {{"x", 0}, {"y", 1}}; + suite.expect_error( + "invalid math mode", Category::INVALID_ARGUMENT, false, + [&] { + (void)mexce::encode_protected_expression( + "x+y", xy, static_cast(2)); + }); + bool syntax_error_reported = false; + try { + (void)mexce::encode_protected_expression( + "unknown_name", {}, mexce::Protected_math_mode::STRICT); + } + catch (const mexce::mexce_parsing_exception&) { + syntax_error_reported = true; + } + suite.expect(syntax_error_reported, "issuer syntax errors retain parsing exception type"); + suite.expect_error( + "sparse slots", Category::INVALID_ARGUMENT, false, + [&] { + (void)mexce::encode_protected_expression( + "x", {{"x", 1}}, mexce::Protected_math_mode::STRICT); + }); + suite.expect_error( + "duplicate names", Category::INVALID_ARGUMENT, false, + [&] { + (void)mexce::encode_protected_expression( + "x", {{"x", 0}, {"x", 1}}, mexce::Protected_math_mode::STRICT); + }); + suite.expect_error( + "invalid name", Category::INVALID_ARGUMENT, false, + [&] { + (void)mexce::encode_protected_expression( + "x", {{"9x", 0}}, mexce::Protected_math_mode::STRICT); + }); + suite.expect_error( + "reserved name", Category::INVALID_ARGUMENT, false, + [&] { + (void)mexce::encode_protected_expression( + "sin", {{"sin", 0}}, mexce::Protected_math_mode::STRICT); + }); + suite.expect_error( + "unused binding", Category::INVALID_ARGUMENT, false, + [&] { + (void)mexce::encode_protected_expression( + "x", xy, mexce::Protected_math_mode::STRICT); + }); + suite.expect_error( + "source limit", Category::SIZE_LIMIT, false, + [&] { + (void)mexce::encode_protected_expression( + std::string(1024 * 1024 + 1, 'x'), {}, + mexce::Protected_math_mode::STRICT); + }); + + std::vector too_many_bindings; + too_many_bindings.reserve(4097); + for (uint32_t slot = 0; slot < 4097; ++slot) { + too_many_bindings.push_back({"v" + std::to_string(slot), slot}); + } + suite.expect_error( + "binding count limit", Category::SIZE_LIMIT, false, + [&] { + (void)mexce::encode_protected_expression( + "1", too_many_bindings, mexce::Protected_math_mode::STRICT); + }); + suite.expect_error( + "binding name limit", Category::SIZE_LIMIT, false, + [&] { + (void)mexce::encode_protected_expression( + "1", {{std::string(256, 'x'), 0}}, mexce::Protected_math_mode::STRICT); + }); + + std::vector cumulative_names; + cumulative_names.reserve(1029); + for (uint32_t slot = 0; slot < 1029; ++slot) { + std::string name = "v" + std::to_string(slot); + name.append(255 - name.size(), 'x'); + cumulative_names.push_back({name, slot}); + } + suite.expect_error( + "cumulative binding name limit", Category::SIZE_LIMIT, false, + [&] { + (void)mexce::encode_protected_expression( + "1", cumulative_names, mexce::Protected_math_mode::STRICT); + }); + + std::string deep_expression; + for (size_t i = 0; i < 1024; ++i) { + deep_expression += "add(1,"; + } + deep_expression += "1"; + deep_expression.append(1024, ')'); + suite.expect_error( + "semantic depth limit", Category::SIZE_LIMIT, false, + [&] { + (void)mexce::encode_protected_expression( + deep_expression, {}, mexce::Protected_math_mode::STRICT); + }); + + std::string record_limit_expression = "1"; + for (size_t i = 1; i < 8192; ++i) { + record_limit_expression += "+1"; + } + suite.expect_error( + "semantic record limit", Category::SIZE_LIMIT, false, + [&] { + (void)mexce::encode_protected_expression( + record_limit_expression, {}, mexce::Protected_math_mode::STRICT); + }); +} + + +void test_error_index_contract(Test_suite& suite) +{ + mexce::Protected_expression_error error( + mexce::Protected_expression_error_category::INVALID_ARGUMENT, "test"); + bool threw = false; + try { + (void)error.record_index(); + } + catch (const std::logic_error&) { + threw = true; + } + suite.expect(threw, "absent record index throws"); +} + + +} // namespace + + +int main(int argc, char** argv) +{ + if (argc > 3) { + std::cerr << "usage: protected_codec_tests [fixture [manifest]]\n"; + return 2; + } + if (argc >= 2) { + g_fixture_path = argv[1]; + } + if (argc == 3) { + g_fixture_manifest_path = argv[2]; + } + if (sodium_init() < 0) { + std::cerr << "libsodium initialization failed\n"; + return 1; + } + if (std::string(sodium_version_string()) != "1.0.22") { + std::cerr << "protected codec tests require libsodium 1.0.22\n"; + return 1; + } + + Test_suite suite; + try { + test_fixture_and_encoder(suite); + test_operation_ids_and_aliases(suite); + test_key_ownership(suite); + test_public_framing(suite); + test_authentication(suite); + test_semantic_validation(suite); + test_encoder_validation(suite); + test_error_index_contract(suite); + } + catch (const std::exception& error) { + suite.m_failures.push_back(std::string("test harness failure: ") + error.what()); + } + + for (const auto& failure : suite.m_failures) { + std::cerr << "FAIL: " << failure << '\n'; + } + if (!suite.m_failures.empty()) { + std::cerr << suite.m_failures.size() << " protected codec test(s) failed\n"; + return 1; + } + std::cout << "All protected codec tests passed\n"; + return 0; +} diff --git a/test/protected_fixture_generator.cpp b/test/protected_fixture_generator.cpp new file mode 100644 index 0000000..cbad5a1 --- /dev/null +++ b/test/protected_fixture_generator.cpp @@ -0,0 +1,144 @@ +#include + +#include +#include +#include +#include +#include +#include +#include + + +namespace { + + +constexpr size_t k_header_size = 64; +constexpr size_t k_record_size = 32; +constexpr size_t k_frame_size = 49; + + +void store_u16(uint8_t* bytes, uint16_t value) +{ + bytes[0] = static_cast(value); + bytes[1] = static_cast(value >> 8); +} + + +void store_u32(uint8_t* bytes, uint32_t value) +{ + for (size_t i = 0; i < 4; ++i) { + bytes[i] = static_cast(value >> (i * 8)); + } +} + + +void store_u64(uint8_t* bytes, uint64_t value) +{ + for (size_t i = 0; i < 8; ++i) { + bytes[i] = static_cast(value >> (i * 8)); + } +} + + +std::array record( + uint8_t kind, + uint32_t operand_32, + uint64_t operand_64) +{ + std::array result = {}; + result[0] = kind; + store_u32(result.data() + 4, operand_32); + store_u64(result.data() + 8, operand_64); + return result; +} + + +bool write_fixture(const std::string& binary_path, const std::string& manifest_path) +{ + if (sodium_init() < 0) { + return false; + } + + std::array key; + for (size_t i = 0; i < key.size(); ++i) { + key[i] = static_cast(i); + } + + uint64_t literal_bits = 0; + const double literal = 2.5; + std::memcpy(&literal_bits, &literal, sizeof(literal_bits)); + const std::array, 7> records = {{ + record(1, 2, 0), + record(3, 0, 0), + record(2, 0, literal_bits), + record(4, 3, 0), + record(3, 1, 0), + record(4, 1, 0), + record(5, 0, 0), + }}; + + std::vector program(k_header_size + records.size() * k_frame_size, 0); + const uint8_t magic[8] = {'M', 'E', 'X', 'C', 'E', 'P', 'R', 'G'}; + std::memcpy(program.data(), magic, sizeof(magic)); + store_u16(program.data() + 8, 1); + store_u16(program.data() + 10, 0); + store_u16(program.data() + 12, k_header_size); + store_u16(program.data() + 14, k_record_size); + store_u32(program.data() + 16, static_cast(records.size())); + for (size_t i = 0; i < 16; ++i) { + program[24 + i] = static_cast(0xa0 + i); + } + + crypto_secretstream_xchacha20poly1305_state state; + if (crypto_secretstream_xchacha20poly1305_init_push( + &state, program.data() + 40, key.data()) != 0) + { + return false; + } + + for (uint32_t i = 0; i < records.size(); ++i) { + uint8_t additional_data[k_header_size + 4]; + std::memcpy(additional_data, program.data(), k_header_size); + store_u32(additional_data + k_header_size, i); + unsigned long long frame_size = 0; + const unsigned char tag = i + 1 == records.size() + ? crypto_secretstream_xchacha20poly1305_TAG_FINAL + : crypto_secretstream_xchacha20poly1305_TAG_MESSAGE; + if (crypto_secretstream_xchacha20poly1305_push( + &state, + program.data() + k_header_size + static_cast(i) * k_frame_size, + &frame_size, + records[i].data(), records[i].size(), + additional_data, sizeof(additional_data), tag) != 0 || + frame_size != k_frame_size) + { + return false; + } + } + + sodium_memzero(&state, sizeof(state)); + std::ofstream binary(binary_path.c_str(), std::ios::binary | std::ios::trunc); + binary.write(reinterpret_cast(program.data()), program.size()); + std::ofstream manifest(manifest_path.c_str(), std::ios::trunc); + manifest + << "format=1.0\n" + << "key_hex=000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f\n" + << "program_id_hex=a0a1a2a3a4a5a6a7a8a9aaabacadaeaf\n" + << "records=MANIFEST(2,strict),VARIABLE(0),LITERAL_F64(2.5)," + "CALL(MUL),VARIABLE(1),CALL(ADD),END\n"; + sodium_memzero(key.data(), key.size()); + return binary.good() && manifest.good(); +} + + +} // namespace + + +int main(int argc, char** argv) +{ + if (argc != 3) { + std::cerr << "usage: protected_fixture_generator \n"; + return 2; + } + return write_fixture(argv[1], argv[2]) ? 0 : 1; +} diff --git a/test/protected_sodium_init_failure_test.cpp b/test/protected_sodium_init_failure_test.cpp new file mode 100644 index 0000000..9a913e0 --- /dev/null +++ b/test/protected_sodium_init_failure_test.cpp @@ -0,0 +1,25 @@ +#include "mexce_protected.h" + +#include +#include +#include + + +int main() +{ + mexce::impl::protected_test_faults().fail_sodium_init = true; + const uint8_t bytes[32] = {}; + try { + (void)mexce::Protected_expression_key::from_bytes(bytes, sizeof(bytes)); + } + catch (const mexce::Protected_expression_error& error) { + return error.category() == + mexce::Protected_expression_error_category::CRYPTOGRAPHY_UNAVAILABLE ? 0 : 1; + } + catch (const std::exception& error) { + std::cerr << "unexpected initialization exception: " << error.what() << '\n'; + return 1; + } + std::cerr << "sodium initialization failure was not reported\n"; + return 1; +} From c41d803549636ab133af1c57e6ebe5cadaba3afc Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 16:09:54 +0200 Subject: [PATCH 05/12] Add protected expression runtime --- CMakeLists.txt | 58 ++ mexce.h | 1161 ++++++++++++++++++++++-------- mexce_protected.h | 268 +++++-- test/mixed_tu_ordinary.cpp | 27 + test/mixed_tu_protected.cpp | 24 + test/mixed_tu_test.cpp | 63 ++ test/ordinary_no_sodium_test.cpp | 19 + test/protected_benchmark.cpp | 187 +++++ test/protected_decoder_fuzz.cpp | 263 +++++++ test/protected_runtime_tests.cpp | 720 ++++++++++++++++++ 10 files changed, 2415 insertions(+), 375 deletions(-) create mode 100644 test/mixed_tu_ordinary.cpp create mode 100644 test/mixed_tu_protected.cpp create mode 100644 test/mixed_tu_test.cpp create mode 100644 test/ordinary_no_sodium_test.cpp create mode 100644 test/protected_benchmark.cpp create mode 100644 test/protected_decoder_fuzz.cpp create mode 100644 test/protected_runtime_tests.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index f975661..a3782bb 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -51,6 +51,10 @@ add_library(mexce INTERFACE) add_library(mexce::mexce ALIAS mexce) target_include_directories(mexce INTERFACE ${CMAKE_CURRENT_SOURCE_DIR}) +add_executable(ordinary_no_sodium_test test/ordinary_no_sodium_test.cpp) +target_link_libraries(ordinary_no_sodium_test PRIVATE mexce) +target_compile_options(ordinary_no_sodium_test PRIVATE ${MEXCE_WARNING_FLAGS}) + if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) set(MEXCE_SODIUM_TARGET "") find_package(libsodium 1.0.22 EXACT CONFIG QUIET) @@ -97,6 +101,50 @@ if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) target_compile_definitions(protected_sodium_init_failure_test PRIVATE MEXCE_PROTECTED_TESTING) target_compile_options(protected_sodium_init_failure_test PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(protected_runtime_tests test/protected_runtime_tests.cpp) + target_link_libraries(protected_runtime_tests PRIVATE mexce_protected) + target_compile_definitions(protected_runtime_tests PRIVATE + MEXCE_PROTECTED_TESTING + MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin") + target_compile_options(protected_runtime_tests PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(mixed_tu_test + test/mixed_tu_test.cpp + test/mixed_tu_ordinary.cpp + test/mixed_tu_protected.cpp) + target_link_libraries(mixed_tu_test PRIVATE mexce_protected) + target_compile_definitions(mixed_tu_test PRIVATE + MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin") + target_compile_options(mixed_tu_test PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(protected_benchmark test/protected_benchmark.cpp) + target_link_libraries(protected_benchmark PRIVATE mexce_protected) + if(WIN32) + target_link_libraries(protected_benchmark PRIVATE Psapi) + endif() + if(NOT ENABLE_COVERAGE) + target_compile_options(protected_benchmark PRIVATE ${MEXCE_OPTIMIZATION_FLAGS}) + endif() + target_compile_options(protected_benchmark PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(protected_decoder_fuzz_smoke test/protected_decoder_fuzz.cpp) + target_link_libraries(protected_decoder_fuzz_smoke PRIVATE mexce_protected) + target_compile_definitions(protected_decoder_fuzz_smoke PRIVATE + MEXCE_FUZZ_SMOKE_MAIN + MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin") + target_compile_options(protected_decoder_fuzz_smoke PRIVATE ${MEXCE_WARNING_FLAGS}) + + if(CMAKE_CXX_COMPILER_ID MATCHES "Clang" AND UNIX) + add_executable(protected_decoder_fuzz EXCLUDE_FROM_ALL + test/protected_decoder_fuzz.cpp) + target_link_libraries(protected_decoder_fuzz PRIVATE mexce_protected) + target_compile_options(protected_decoder_fuzz PRIVATE + -fsanitize=fuzzer,address,undefined + -fno-omit-frame-pointer) + target_link_options(protected_decoder_fuzz PRIVATE + -fsanitize=fuzzer,address,undefined) + endif() endif() # Capture compiler info for benchmark output @@ -131,10 +179,20 @@ target_compile_definitions(benchmark PRIVATE enable_testing() add_test(NAME mexce_unit_tests COMMAND unit_tests) +add_test(NAME mexce_ordinary_no_sodium COMMAND ordinary_no_sodium_test) if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) add_test(NAME mexce_protected_codec_tests COMMAND protected_codec_tests) add_test(NAME mexce_protected_sodium_init_failure COMMAND protected_sodium_init_failure_test) + add_test(NAME mexce_protected_runtime COMMAND protected_runtime_tests) + add_test(NAME mexce_mixed_translation_units COMMAND mixed_tu_test) + add_test(NAME mexce_protected_decoder_fuzz_smoke + COMMAND protected_decoder_fuzz_smoke) + add_test(NAME mexce_protected_benchmark COMMAND protected_benchmark) + add_test(NAME mexce_protected_maximum_valid + COMMAND protected_benchmark maximum-valid) + add_test(NAME mexce_protected_late_invalid + COMMAND protected_benchmark late-invalid) endif() add_test(NAME benchmark_quick COMMAND benchmark 5 ${CMAKE_BINARY_DIR}/benchmark_quick_results.txt) diff --git a/mexce.h b/mexce.h index 4e38ee4..e85c1ba 100644 --- a/mexce.h +++ b/mexce.h @@ -131,8 +131,10 @@ #include #include #include +#include #include #include +#include #include #include @@ -159,6 +161,98 @@ namespace mexce { // Forward declarations class evaluator; +class Protected_expression_key; + + +enum class Protected_expression_error_category +{ + INVALID_ARGUMENT, + UNSUPPORTED_FORMAT, + SIZE_LIMIT, + AUTHENTICATION_FAILED, + MALFORMED_PROGRAM, + MISSING_BINDING, + INTROSPECTION_DISABLED, + CRYPTOGRAPHY_UNAVAILABLE, + COMPILATION_FAILED, + RESOURCE_FAILURE, +}; + + +class Protected_expression_error: public std::runtime_error +{ +public: + Protected_expression_error( + Protected_expression_error_category category, + const char* message) + : + std::runtime_error(message), + m_category(category), + m_has_record_index(false), + m_record_index(0) + {} + + Protected_expression_error( + Protected_expression_error_category category, + const char* message, + uint32_t record_index) + : + std::runtime_error(message), + m_category(category), + m_has_record_index(true), + m_record_index(record_index) + {} + + Protected_expression_error_category category() const noexcept { return m_category; } + bool has_record_index() const noexcept { return m_has_record_index; } + + uint32_t record_index() const + { + if (!m_has_record_index) { + throw std::logic_error("No protected-expression record index is available."); + } + return m_record_index; + } + +private: + Protected_expression_error_category m_category; + bool m_has_record_index; + uint32_t m_record_index; +}; + +#ifndef MEXCE_USE_LIBM_TRANSCENDENTALS +# define MEXCE_USE_LIBM_TRANSCENDENTALS 1 +#endif +#ifndef MEXCE_USE_LIBM_GENERIC_POW +# define MEXCE_USE_LIBM_GENERIC_POW MEXCE_USE_LIBM_TRANSCENDENTALS +#endif +#ifndef MEXCE_USE_LIBM_EXP +# define MEXCE_USE_LIBM_EXP MEXCE_USE_LIBM_TRANSCENDENTALS +#endif +#ifndef MEXCE_USE_LIBM_LOG +# define MEXCE_USE_LIBM_LOG MEXCE_USE_LIBM_TRANSCENDENTALS +#endif +#ifndef MEXCE_USE_LIBM_LOG10 +# define MEXCE_USE_LIBM_LOG10 MEXCE_USE_LIBM_TRANSCENDENTALS +#endif +#ifndef MEXCE_USE_LIBM_LOG2 +# define MEXCE_USE_LIBM_LOG2 MEXCE_USE_LIBM_TRANSCENDENTALS +#endif +#ifndef MEXCE_USE_LIBM_LOGB +# define MEXCE_USE_LIBM_LOGB MEXCE_USE_LIBM_TRANSCENDENTALS +#endif +#ifndef MEXCE_USE_LIBM_YLOG2 +# define MEXCE_USE_LIBM_YLOG2 MEXCE_USE_LIBM_TRANSCENDENTALS +#endif +#ifndef MEXCE_USE_LIBM_SIN +# define MEXCE_USE_LIBM_SIN MEXCE_USE_LIBM_TRANSCENDENTALS +#endif +#ifndef MEXCE_USE_LIBM_COS +# define MEXCE_USE_LIBM_COS MEXCE_USE_LIBM_TRANSCENDENTALS +#endif +#ifndef MEXCE_USE_LIBM_TAN +# define MEXCE_USE_LIBM_TAN MEXCE_USE_LIBM_TRANSCENDENTALS +#endif // --- Runtime-configurable options --- // @@ -175,18 +269,17 @@ struct options { // Force x87 backend even when SSE2 is available bool prefer_x87 = false; - // Use libm functions instead of inline x87 assembly (currently compile-time only) - // These are included for completeness; runtime switching requires recompilation - bool use_libm_sin = true; - bool use_libm_cos = true; - bool use_libm_tan = true; - bool use_libm_exp = true; - bool use_libm_log = true; - bool use_libm_log10 = true; - bool use_libm_log2 = true; - bool use_libm_logb = true; - bool use_libm_ylog2 = true; - bool use_libm_generic_pow = true; + // Use libm functions instead of the inline x87 implementation. + bool use_libm_sin = MEXCE_USE_LIBM_SIN != 0; + bool use_libm_cos = MEXCE_USE_LIBM_COS != 0; + bool use_libm_tan = MEXCE_USE_LIBM_TAN != 0; + bool use_libm_exp = MEXCE_USE_LIBM_EXP != 0; + bool use_libm_log = MEXCE_USE_LIBM_LOG != 0; + bool use_libm_log10 = MEXCE_USE_LIBM_LOG10 != 0; + bool use_libm_log2 = MEXCE_USE_LIBM_LOG2 != 0; + bool use_libm_logb = MEXCE_USE_LIBM_LOGB != 0; + bool use_libm_ylog2 = MEXCE_USE_LIBM_YLOG2 != 0; + bool use_libm_generic_pow = MEXCE_USE_LIBM_GENERIC_POW != 0; // Convenience: set all libm options at once void set_use_libm(bool value) { @@ -215,8 +308,37 @@ namespace impl { class Semantic_graph_builder; class Semantic_compiler; class Clear_semantic_producer; + class Protected_semantic_sink; struct Semantic_compilation_result; + enum class Protected_wipe_context + { + KEY, + PULL_STATE, + PUSH_STATE, + CLEAR_RECORD, + ADDITIONAL_DATA, + DECODED_SCALAR, + VALIDATOR_STATE, + PROTECTED_SCALAR, + OPTIMIZER_STATE, + CONSTANT_STORAGE, + FALLBACK_STORAGE, + OPTIONS_STORAGE, + EXECUTABLE, + COUNT, + }; + + using protected_wipe_function_t = void (*)( + void*, size_t, Protected_wipe_context); + + enum class Compilation_test_fault + { + NONE, + EXECUTABLE_ALLOCATION, + EXECUTABLE_FINALIZATION, + }; + using std::abs; using std::deque; using std::exception; @@ -279,6 +401,17 @@ class evaluator void set_expression(std::string); + template + void bind_protected(T& referenced_variable, uint32_t slot); + + void unbind_protected(uint32_t slot); + void unbind_all_protected(); + + void set_protected_expression( + const uint8_t* program, + size_t program_size, + Protected_expression_key key); + double evaluate(); double evaluate(const std::string& expression); @@ -339,6 +472,7 @@ class evaluator private: options m_options; impl::variable_map_t m_variables; + std::map> m_protected_variables; impl::constant_map_t m_constants; uint64_t m_next_variable_id = 2; std::unique_ptr m_compilation; @@ -347,11 +481,15 @@ class evaluator double m_constant_expression_value = 0.0; double (*m_evaluate_fptr)() = nullptr; backend_type m_backend_used = backend_type::none; + bool m_is_protected_expression = false; + void reset_compiled_expression(); + void publish_semantic_compilation(impl::Semantic_compilation_result result); void compile_and_finalize_elist(impl::elist_const_it_t first, impl::elist_const_it_t last); impl::Semantic_compilation_result finish_semantic_compilation( std::unique_ptr candidate, - std::vector> referenced_variables); + std::vector> referenced_variables, + std::unique_ptr fallback_candidate = nullptr); impl::Compilation_state& active_compilation(); // Grant friend access to helpers that need private state. @@ -363,6 +501,7 @@ class evaluator friend void impl::run_cse(evaluator* ev, impl::elist_t& elist); friend class impl::Semantic_compiler; friend class impl::Clear_semantic_producer; + friend class impl::Protected_semantic_sink; template void bind() {} template void unbind() {} @@ -424,8 +563,21 @@ uint8_t* get_executable_buffer(size_t sz) } +class Executable_memory_error: public std::runtime_error +{ +public: + explicit Executable_memory_error(const char* message) + : + std::runtime_error(message) + {} +}; + + inline -double (*lock_executable_buffer(uint8_t* buffer, size_t sz))() +double (*lock_executable_buffer( + uint8_t* buffer, + size_t sz, + protected_wipe_function_t wipe = nullptr))() { if (sz == 0) { throw std::runtime_error("lock_executable_buffer requires a non-zero size"); @@ -433,30 +585,56 @@ double (*lock_executable_buffer(uint8_t* buffer, size_t sz))() #ifdef _WIN32 DWORD old_protect = 0; if (!VirtualProtect(buffer, sz, PAGE_EXECUTE_READ, &old_protect)) { + if (wipe) { + wipe(buffer, sz, Protected_wipe_context::EXECUTABLE); + } VirtualFree((void*)buffer, 0, MEM_RELEASE); - throw std::runtime_error("VirtualProtect(PAGE_EXECUTE_READ) failed"); + throw Executable_memory_error("VirtualProtect(PAGE_EXECUTE_READ) failed"); } FlushInstructionCache(GetCurrentProcess(), buffer, sz); #elif defined(__linux__) if (mprotect((void*)buffer, sz, PROT_READ | PROT_EXEC) != 0) { + if (wipe) { + wipe(buffer, sz, Protected_wipe_context::EXECUTABLE); + } munmap((void*)buffer, sz); - throw std::runtime_error("mprotect(PROT_READ|PROT_EXEC) failed"); + throw Executable_memory_error("mprotect(PROT_READ|PROT_EXEC) failed"); } #endif return reinterpret_cast(buffer); } +#if defined(__clang__) +__attribute__((no_sanitize("function"))) +#endif +inline double invoke_generated(double (*function)()) +{ + return function(); +} + + inline -void free_executable_buffer(double (*buffer)(), size_t sz) +void free_executable_buffer( + double (*buffer)(), + size_t sz, + protected_wipe_function_t wipe = nullptr) { if (!buffer) { return; } #ifdef _WIN32 - (void)sz; // prevent warning + if (wipe) { + DWORD old_protect = 0; + if (VirtualProtect((void*)buffer, sz, PAGE_READWRITE, &old_protect)) { + wipe((void*)buffer, sz, Protected_wipe_context::EXECUTABLE); + } + } VirtualFree( (void*) buffer, 0, MEM_RELEASE); #elif defined(__linux__) + if (wipe && mprotect((void*)buffer, sz, PROT_READ | PROT_WRITE) == 0) { + wipe((void*)buffer, sz, Protected_wipe_context::EXECUTABLE); + } munmap((void *) buffer, sz); #endif } @@ -516,6 +694,8 @@ string double_to_hex( double v ) struct Constant { + struct Protected_value {}; + uint64_t id; string name; double value; @@ -532,6 +712,11 @@ struct Constant id(i), name(double_to_hex(v)), value(v), address((void*)&this->value), numeric_data_type(M64FP) {} + + Constant(uint64_t i, double v, Protected_value): + id(i), name(), value(v), + address((void*)&this->value), numeric_data_type(M64FP) + {} }; @@ -619,15 +804,30 @@ struct Element { class Compilation_state { public: - Compilation_state(const options& effective_options, uint64_t first_element_id) + enum class Protected_storage_role + { + PRIMARY, + FALLBACK, + }; + + Compilation_state( + const options& effective_options, + uint64_t first_element_id, + protected_wipe_function_t protected_wipe = nullptr, + Compilation_test_fault test_fault = Compilation_test_fault::NONE, + Protected_storage_role storage_role = Protected_storage_role::PRIMARY) : m_effective_options(effective_options), - m_next_element_id(first_element_id) + m_next_element_id(first_element_id), + m_protected_wipe(protected_wipe), + m_test_fault(test_fault), + m_storage_role(storage_role) {} ~Compilation_state() { - free_executable_buffer(m_evaluate_fptr, m_buffer_size); + release_executable(); + wipe_protected_state(); } Compilation_state(const Compilation_state&) = delete; @@ -635,12 +835,30 @@ class Compilation_state void release_executable() { - free_executable_buffer(m_evaluate_fptr, m_buffer_size); + free_executable_buffer(m_evaluate_fptr, m_buffer_size, m_protected_wipe); m_evaluate_fptr = nullptr; m_buffer_size = 0; m_backend_used = backend_type::none; } + bool is_protected() const { return m_protected_wipe != nullptr; } + protected_wipe_function_t protected_wipe() const { return m_protected_wipe; } + Compilation_test_fault test_fault() const { return m_test_fault; } + bool fail_executable_allocation() const + { + return m_test_fault == Compilation_test_fault::EXECUTABLE_ALLOCATION; + } + bool fail_executable_finalization() const + { + return m_test_fault == Compilation_test_fault::EXECUTABLE_FINALIZATION; + } + void wipe_protected_scalar(void* value, size_t size) + { + if (m_protected_wipe) { + m_protected_wipe(value, size, protected_storage_context()); + } + } + options m_effective_options; bool m_is_constant_expression = false; double m_constant_expression_value = 0.0; @@ -650,10 +868,107 @@ class Compilation_state elist_t m_elist; std::list m_intermediate_code; constant_map_t m_intermediate_constants; + std::list> m_protected_constants; uint64_t m_next_element_id; std::list m_cse_temps; std::map> m_power_terms; double (*m_evaluate_fptr)() = nullptr; + +private: + Protected_wipe_context protected_storage_context() const + { + return m_storage_role == Protected_storage_role::FALLBACK + ? Protected_wipe_context::FALLBACK_STORAGE + : Protected_wipe_context::CONSTANT_STORAGE; + } + + Protected_wipe_context optimizer_storage_context() const + { + return m_storage_role == Protected_storage_role::FALLBACK + ? Protected_wipe_context::FALLBACK_STORAGE + : Protected_wipe_context::OPTIMIZER_STATE; + } + + void wipe_string(string& value, Protected_wipe_context context) + { + if (!value.empty()) { + m_protected_wipe(&value[0], value.size(), context); + } + } + + void wipe_element(Element& element) + { + if (element.c) { + return; + } + if (element.f) { + const auto context = optimizer_storage_context(); + wipe_string(element.f->name, context); + wipe_string(element.f->code, context); + wipe_string(element.f->debug_desc, context); + wipe_string(element.f->cse_store_suffix, context); + for (auto& absorbed_group : element.f->absorbed) { + for (auto& absorbed : absorbed_group) { + for (auto& nested : absorbed) { + wipe_element(nested); + } + } + } + } + } + + void wipe_protected_state() + { + if (!m_protected_wipe) { + return; + } + + m_protected_wipe( + &m_constant_expression_value, + sizeof(m_constant_expression_value), + protected_storage_context()); + m_protected_wipe( + &m_effective_options, + sizeof(m_effective_options), + Protected_wipe_context::OPTIONS_STORAGE); + for (auto& element : m_elist) { + wipe_element(element); + } + for (auto& code : m_intermediate_code) { + wipe_string(code, optimizer_storage_context()); + } + for (auto& constant : m_intermediate_constants) { + m_protected_wipe( + &constant.second->value, + sizeof(constant.second->value), + protected_storage_context()); + wipe_string(constant.second->name, protected_storage_context()); + } + for (auto& constant : m_protected_constants) { + m_protected_wipe( + &constant->value, + sizeof(constant->value), + protected_storage_context()); + wipe_string(constant->name, protected_storage_context()); + } + for (auto& value : m_cse_temps) { + m_protected_wipe( + &value, sizeof(value), optimizer_storage_context()); + } + for (auto& power_term : m_power_terms) { + for (auto& element : power_term.second.first) { + wipe_element(element); + } + m_protected_wipe( + &power_term.second.second, + sizeof(power_term.second.second), + optimizer_storage_context()); + } + } + + protected_wipe_function_t m_protected_wipe; + Compilation_test_fault m_test_fault; + Protected_storage_role m_storage_role; }; @@ -727,6 +1042,33 @@ class Semantic_record using Semantic_program = vector; +class Semantic_program_wipe_guard +{ +public: + Semantic_program_wipe_guard( + Semantic_program& program, + protected_wipe_function_t wipe) + : + m_program(program), + m_wipe(wipe) + {} + + ~Semantic_program_wipe_guard() + { + if (m_wipe && !m_program.empty()) { + m_wipe( + &m_program[0], + m_program.size() * sizeof(m_program[0]), + Protected_wipe_context::OPTIMIZER_STATE); + } + } + +private: + Semantic_program& m_program; + protected_wipe_function_t m_wipe; +}; + + struct Semantic_compilation_result { std::unique_ptr compilation; @@ -760,17 +1102,35 @@ class Semantic_compiler Semantic_compiler( evaluator& owner, const options& effective_options, - uint64_t first_element_id); + uint64_t first_element_id, + protected_wipe_function_t protected_wipe = nullptr, + Compilation_test_fault test_fault = Compilation_test_fault::NONE); void consume(const Semantic_record& record); + void consume_literal(double value); + void consume_variable(const shared_ptr& variable); + void consume_call(const string& name); Semantic_compilation_result finish(); private: - void reset_graph_builder(); + void consume_literal_into( + Compilation_state& compilation, + Semantic_graph_builder& builder, + double value); + void consume_variable_into( + Compilation_state& compilation, + Semantic_graph_builder& builder, + const shared_ptr& variable); + void consume_call_into( + Compilation_state& compilation, + Semantic_graph_builder& builder, + const string& name); evaluator& m_owner; std::unique_ptr m_candidate; Semantic_graph_builder m_graph_builder; + std::unique_ptr m_fallback_candidate; + Semantic_graph_builder m_fallback_graph_builder; Active_compilation_reset m_active_compilation_reset; vector> m_referenced_variables; std::set m_referenced_variable_set; @@ -819,6 +1179,19 @@ inline shared_ptr make_intermediate_constant(evaluator* ev, double v) { auto& compilation = ev->active_compilation(); + if (compilation.is_protected()) { + compilation.m_protected_constants.push_back(shared_ptr()); + try { + compilation.m_protected_constants.back() = make_shared( + compilation.m_next_element_id++, v, Constant::Protected_value()); + } + catch (...) { + compilation.m_protected_constants.pop_back(); + throw; + } + return compilation.m_protected_constants.back(); + } + auto name = double_to_hex(v); auto it = compilation.m_intermediate_constants.find(name); if (it == compilation.m_intermediate_constants.end()) { @@ -903,46 +1276,6 @@ Token_type get_infix_rank(char infix_op) #endif -// --- Libm-backed transcendental functions --- -// -// mexce historically used x87 transcendental instructions for exp/log/pow, etc. -// On many modern targets libm implementations can be faster, so these are -// enabled by default but can be disabled at compile time. -#ifndef MEXCE_USE_LIBM_TRANSCENDENTALS -# define MEXCE_USE_LIBM_TRANSCENDENTALS 1 -#endif -#ifndef MEXCE_USE_LIBM_GENERIC_POW -# define MEXCE_USE_LIBM_GENERIC_POW MEXCE_USE_LIBM_TRANSCENDENTALS -#endif -#ifndef MEXCE_USE_LIBM_EXP -# define MEXCE_USE_LIBM_EXP MEXCE_USE_LIBM_TRANSCENDENTALS -#endif -#ifndef MEXCE_USE_LIBM_LOG -# define MEXCE_USE_LIBM_LOG MEXCE_USE_LIBM_TRANSCENDENTALS -#endif -#ifndef MEXCE_USE_LIBM_LOG10 -# define MEXCE_USE_LIBM_LOG10 MEXCE_USE_LIBM_TRANSCENDENTALS -#endif -#ifndef MEXCE_USE_LIBM_LOG2 -# define MEXCE_USE_LIBM_LOG2 MEXCE_USE_LIBM_TRANSCENDENTALS -#endif -#ifndef MEXCE_USE_LIBM_LOGB -# define MEXCE_USE_LIBM_LOGB MEXCE_USE_LIBM_TRANSCENDENTALS -#endif -#ifndef MEXCE_USE_LIBM_YLOG2 -# define MEXCE_USE_LIBM_YLOG2 MEXCE_USE_LIBM_TRANSCENDENTALS -#endif - -#ifndef MEXCE_USE_LIBM_SIN -# define MEXCE_USE_LIBM_SIN MEXCE_USE_LIBM_TRANSCENDENTALS -#endif -#ifndef MEXCE_USE_LIBM_COS -# define MEXCE_USE_LIBM_COS MEXCE_USE_LIBM_TRANSCENDENTALS -#endif -#ifndef MEXCE_USE_LIBM_TAN -# define MEXCE_USE_LIBM_TAN MEXCE_USE_LIBM_TRANSCENDENTALS -#endif - // --- Optional compile-time optimizations --- #ifndef MEXCE_ENABLE_CSE # define MEXCE_ENABLE_CSE 0 @@ -1192,13 +1525,14 @@ static const double mexce_trig_tan_x_thresholds[] = { 0xdd,0xd9, /* fstp st(1) (pop -1) */ \ 0xde,0xe9 /* fsubp st(1), st => x - π/2 */ -inline Function Cos() +inline Function Cos(bool use_libm = MEXCE_USE_LIBM_COS != 0) { -#if MEXCE_USE_LIBM_COS - mexce_charstream s; - emit_libm_unary_call(s, (void*)static_cast(std::cos)); - return Function(0, "cos", 1, 1, s.buf.size(), s.buf.data()); -#elif MEXCE_TRIG_USE_X87 + if (use_libm) { + mexce_charstream s; + emit_libm_unary_call(s, (void*)static_cast(std::cos)); + return Function(0, "cos", 1, 1, s.buf.size(), s.buf.data()); + } +#if MEXCE_TRIG_USE_X87 uint8_t code[] = { 0xd9, 0xff }; // fcos return Function(0, "cos", 1, 0, sizeof(code), code); #else @@ -1537,13 +1871,14 @@ inline Function Cos() -inline Function Sin() +inline Function Sin(bool use_libm = MEXCE_USE_LIBM_SIN != 0) { -#if MEXCE_USE_LIBM_SIN - mexce_charstream s; - emit_libm_unary_call(s, (void*)static_cast(std::sin)); - return Function(0, "sin", 1, 1, s.buf.size(), s.buf.data()); -#elif MEXCE_TRIG_USE_X87 + if (use_libm) { + mexce_charstream s; + emit_libm_unary_call(s, (void*)static_cast(std::sin)); + return Function(0, "sin", 1, 1, s.buf.size(), s.buf.data()); + } +#if MEXCE_TRIG_USE_X87 uint8_t code[] = { 0xd9, 0xfe }; // fsin return Function(0, "sin", 1, 0, sizeof(code), code); #else @@ -1892,13 +2227,14 @@ inline Function Sin() } -inline Function Tan() +inline Function Tan(bool use_libm = MEXCE_USE_LIBM_TAN != 0) { -#if MEXCE_USE_LIBM_TAN - mexce_charstream s; - emit_libm_unary_call(s, (void*)static_cast(std::tan)); - return Function(0, "tan", 1, 1, s.buf.size(), s.buf.data()); -#elif MEXCE_TRIG_USE_X87 + if (use_libm) { + mexce_charstream s; + emit_libm_unary_call(s, (void*)static_cast(std::tan)); + return Function(0, "tan", 1, 1, s.buf.size(), s.buf.data()); + } +#if MEXCE_TRIG_USE_X87 uint8_t code[] = { 0xd9, 0xf2, // fptan 0xdd, 0xd8 // fstp st(0) @@ -2483,21 +2819,23 @@ void pow_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) compile_elist(final_s, base_chunk.begin(), base_chunk.end()); final_s.write((const char*)s.buf.data(), s.buf.size()); - // Generate correct debug string - string base_str = elist_to_string(base_chunk); - stringstream ss; - if (optimized_name == "sqrt") { - ss << "sqrt(" << base_str << ")"; - } - else - if (optimized_name == "inv_sqrt") { - ss << "inv_sqrt(" << base_str << ")"; - } - else - if (optimized_name == "pow_int") { - ss << "pow_int(" << base_str << ", " << static_cast(v_d) << ")"; + if (!compilation.is_protected()) { + const string base_string = elist_to_string(base_chunk); + stringstream stream; + if (optimized_name == "sqrt") { + stream << "sqrt(" << base_string << ")"; + } + else + if (optimized_name == "inv_sqrt") { + stream << "inv_sqrt(" << base_string << ")"; + } + else + if (optimized_name == "pow_int") { + stream << "pow_int(" << base_string << ", " + << static_cast(v_d) << ")"; + } + debug_desc = stream.str(); } - debug_desc = ss.str(); uint8_t* cc = push_intermediate_code(ev, final_s.str()); auto f_opt = make_shared( @@ -2518,11 +2856,14 @@ void pow_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) } else { // Generic pow (non-integer constant exponent) - mexce_charstream s; -#if MEXCE_USE_LIBM_GENERIC_POW - emit_libm_binary_call(s, (void*)static_cast(std::pow)); -#else - s < 0xd9 < 0xc9 // fxch } + mexce_charstream generic_code; + if (compilation.m_effective_options.use_libm_generic_pow) { + emit_libm_binary_call( + generic_code, + (void*)static_cast(std::pow)); + } + else { + generic_code < 0xd9 < 0xc9 // fxch } < 0xd9 < 0xe4 // ftst } if base is 0, leave it in st(0) < 0xdf < 0xe0 // fnstsw ax } and exit < 0x9e // sahf } @@ -2538,13 +2879,13 @@ void pow_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) < 0x77 < 0x02 // ja store_and_exit < 0xd9 < 0xe0 // fchs // store_and_exit: - < 0xdd < 0xd9; // fstp st(1) -#endif + < 0xdd < 0xd9; // fstp st(1) + } - uint8_t* cc = push_intermediate_code(ev, s.str()); + uint8_t* cc = push_intermediate_code(ev, generic_code.str()); auto f_opt = make_shared( compilation.m_next_element_id++, "pow_opt", 2, 0, - s.buf.size(), cc, nullptr); + generic_code.buf.size(), cc, nullptr); f_opt->m_requires_x87_backend = true; f_opt->args[0] = f->args[0]; f_opt->args[1] = f->args[1]; @@ -2556,7 +2897,7 @@ void pow_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) } -inline Function Pow() +inline Function Pow(bool use_libm = MEXCE_USE_LIBM_GENERIC_POW != 0) { // Pow() has several fast paths (small integer exponents, etc.). // We only replace the expensive generic pow path with an optional libm call. @@ -2625,28 +2966,32 @@ inline Function Pow() s <0xe9; size_t jmp_exit_point_zeroexp = s.buf.size(); s << int32_t(0); // jmp exit_point const size_t non_zero_exponent = s.buf.size(); -#if MEXCE_USE_LIBM_GENERIC_POW - emit_libm_binary_call(s, (void*)static_cast(std::pow)); -#else - // Original x87 generic pow implementation (kept as a fallback) - s <0xd9 <0xc9; // fxch - s <0xd9 <0xe4; // ftst - s <0xdf <0xe0; // fnstsw ax - s <0x9e; // sahf - s <0x0f <0x84; size_t je_store_and_exit = s.buf.size(); s << int32_t(0); // je store_and_exit - s <0xd9 <0xe1; // fabs - s <0xd9 <0xf1; // fyl2x - s <0xd9 <0xe8; // fld1 - s <0xd9 <0xc1; // fld st(1) - s <0xd9 <0xf8; // fprem - s <0xd9 <0xf0; // f2xm1 - s <0xde <0xc1; // faddp st(1), st - s <0xd9 <0xfd; // fscale - s <0x0f <0x87; size_t ja_store_and_exit = s.buf.size(); s << int32_t(0); // ja store_and_exit - s <0xd9 <0xe0; // fchs - const size_t store_and_exit = s.buf.size(); - s <0xdd <0xd9; // fstp st(1) -#endif + size_t je_store_and_exit = 0; + size_t ja_store_and_exit = 0; + size_t store_and_exit = 0; + if (use_libm) { + emit_libm_binary_call( + s, (void*)static_cast(std::pow)); + } + else { + s <0xd9 <0xc9; // fxch + s <0xd9 <0xe4; // ftst + s <0xdf <0xe0; // fnstsw ax + s <0x9e; // sahf + s <0x0f <0x84; je_store_and_exit = s.buf.size(); s << int32_t(0); + s <0xd9 <0xe1; // fabs + s <0xd9 <0xf1; // fyl2x + s <0xd9 <0xe8; // fld1 + s <0xd9 <0xc1; // fld st(1) + s <0xd9 <0xf8; // fprem + s <0xd9 <0xf0; // f2xm1 + s <0xde <0xc1; // faddp st(1), st + s <0xd9 <0xfd; // fscale + s <0x0f <0x87; ja_store_and_exit = s.buf.size(); s << int32_t(0); + s <0xd9 <0xe0; // fchs + store_and_exit = s.buf.size(); + s <0xdd <0xd9; // fstp st(1) + } s <0xe9; size_t jmp_exit_from_generic = s.buf.size(); s << int32_t(0); // jmp exit_point #ifdef MEXCE_64 @@ -2680,22 +3025,22 @@ inline Function Pow() patch_rel32(s, jmp_loop_start, loop_start); patch_rel32(s, jne_non_zero_exponent, non_zero_exponent); patch_rel32(s, jmp_exit_point_zeroexp, exit_point); -#if !MEXCE_USE_LIBM_GENERIC_POW - patch_rel32(s, je_store_and_exit, store_and_exit); - patch_rel32(s, ja_store_and_exit, store_and_exit); -#endif + if (!use_libm) { + patch_rel32(s, je_store_and_exit, store_and_exit); + patch_rel32(s, ja_store_and_exit, store_and_exit); + } return Function(0, "pow", 2, 1, s.buf.size(), s.buf.data(), pow_optimizer); } -inline Function Exp() +inline Function Exp(bool use_libm = MEXCE_USE_LIBM_EXP != 0) { -#if MEXCE_USE_LIBM_EXP - mexce_charstream s; - emit_libm_unary_call(s, (void*)static_cast(std::exp)); - return Function(0, "exp", 1, 1, s.buf.size(), s.buf.data()); -#else + if (use_libm) { + mexce_charstream s; + emit_libm_unary_call(s, (void*)static_cast(std::exp)); + return Function(0, "exp", 1, 1, s.buf.size(), s.buf.data()); + } uint8_t code[] = { 0xd9, 0xea, // fldl2e 0xde, 0xc9, // fmulp st(1), st @@ -2708,17 +3053,16 @@ inline Function Exp() 0xdd, 0xd9, // fstp st(1) }; return Function(0, "exp", 1, 1, sizeof(code), code); -#endif } -inline Function Logb() +inline Function Logb(bool use_libm = MEXCE_USE_LIBM_LOGB != 0) { -#if MEXCE_USE_LIBM_LOGB - mexce_charstream s; - emit_libm_binary_call(s, (void*)&mexce_libm_logb); - return Function(0, "logb", 2, 1, s.buf.size(), s.buf.data()); -#else + if (use_libm) { + mexce_charstream s; + emit_libm_binary_call(s, (void*)&mexce_libm_logb); + return Function(0, "logb", 2, 1, s.buf.size(), s.buf.data()); + } uint8_t code[] = { 0xd9, 0xe8, // fld1 0xd9, 0xc9, // fxch st(1) @@ -2730,17 +3074,16 @@ inline Function Logb() 0xde, 0xf9 // fdivp st(1), st }; return Function(0, "logb", 2, 1, sizeof(code), code); -#endif } -inline Function Ln() +inline Function Ln(bool use_libm = MEXCE_USE_LIBM_LOG != 0) { -#if MEXCE_USE_LIBM_LOG - mexce_charstream s; - emit_libm_unary_call(s, (void*)static_cast(std::log)); - return Function(0, "ln", 1, 1, s.buf.size(), s.buf.data()); -#else + if (use_libm) { + mexce_charstream s; + emit_libm_unary_call(s, (void*)static_cast(std::log)); + return Function(0, "ln", 1, 1, s.buf.size(), s.buf.data()); + } uint8_t code[] = { 0xd9, 0xe8, // fld1 0xd9, 0xc9, // fxch @@ -2749,63 +3092,59 @@ inline Function Ln() 0xde, 0xf9 // fdivp st(1), st }; return Function(0, "ln", 1, 1, sizeof(code), code); -#endif } -inline Function Log() +inline Function Log(bool use_libm = MEXCE_USE_LIBM_LOG != 0) { - Function f = Ln(); + Function f = Ln(use_libm); f.name="log"; return f; } -inline Function Log10() +inline Function Log10(bool use_libm = MEXCE_USE_LIBM_LOG10 != 0) { -#if MEXCE_USE_LIBM_LOG10 - mexce_charstream s; - emit_libm_unary_call(s, (void*)static_cast(std::log10)); - return Function(0, "log10", 1, 0, s.buf.size(), s.buf.data()); -#else + if (use_libm) { + mexce_charstream s; + emit_libm_unary_call(s, (void*)static_cast(std::log10)); + return Function(0, "log10", 1, 0, s.buf.size(), s.buf.data()); + } uint8_t code[] = { 0xd9, 0xec, // fldlg2 0xd9, 0xc9, // fxch 0xd9, 0xf1 // fyl2x }; return Function(0, "log10", 1, 0, sizeof(code), code); -#endif } -inline Function Log2() +inline Function Log2(bool use_libm = MEXCE_USE_LIBM_LOG2 != 0) { -#if MEXCE_USE_LIBM_LOG2 - mexce_charstream s; - emit_libm_unary_call(s, (void*)static_cast(std::log2)); - return Function(0, "log2", 1, 0, s.buf.size(), s.buf.data()); -#else + if (use_libm) { + mexce_charstream s; + emit_libm_unary_call(s, (void*)static_cast(std::log2)); + return Function(0, "log2", 1, 0, s.buf.size(), s.buf.data()); + } uint8_t code[] = { 0xd9, 0xe8, // fld1 0xd9, 0xc9, // fxch st(1) 0xd9, 0xf1 // fyl2x }; return Function(0, "log2", 1, 0, sizeof(code), code); -#endif } -inline Function Ylog2() +inline Function Ylog2(bool use_libm = MEXCE_USE_LIBM_YLOG2 != 0) { -#if MEXCE_USE_LIBM_YLOG2 - mexce_charstream s; - emit_libm_binary_call(s, (void*)&mexce_libm_ylog2); - return Function(0, "ylog2", 2, 0, s.buf.size(), s.buf.data()); -#else + if (use_libm) { + mexce_charstream s; + emit_libm_binary_call(s, (void*)&mexce_libm_ylog2); + return Function(0, "ylog2", 2, 0, s.buf.size(), s.buf.data()); + } uint8_t code[] = { 0xd9, 0xf1 // fyl2x }; return Function(0, "ylog2", 2, 0, sizeof(code), code); -#endif } @@ -3402,7 +3741,10 @@ inline bool sse2_needs_libm_calls(const impl::elist_const_it_t first, const impl { using namespace impl; for (auto it = first; it != last; ++it) { - if (it->type == Element_type::CFUNC && is_sse2_libm_function(it->f->name)) { + if (it->type == Element_type::CFUNC && + !it->f->m_requires_x87_backend && + is_sse2_libm_function(it->f->name)) + { return true; } } @@ -4632,45 +4974,6 @@ void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) // === SSE2 simplification: reconstruct simplified elist === // Instead of generating x87 code, build a new elist from the simplified terms. if (compilation.m_sse2_simplify_mode) { - // Build debug string for get_optimized_expression() - stringstream debug_ss; - bool debug_first = true; - if (fclass == 1) { - if (ac_final != neutral) { debug_ss << double_to_pretty_string(ac_final); debug_first = false; } - for (auto &term : merged) { - string term_str = "(" + elist_to_string(term.chunk) + ")"; - if (term.factor > 0) { - if (!debug_first) debug_ss << "+"; - if (term.factor != 1) debug_ss << term.factor << "*"; - debug_ss << term_str; - } else { - debug_ss << "-"; - if (term.factor != -1) debug_ss << abs(term.factor) << "*"; - debug_ss << term_str; - } - debug_first = false; - } - } - else { - if (ac_final != neutral) { debug_ss << double_to_pretty_string(ac_final); debug_first = false; } - for (auto &term : merged) { - string term_str = "(" + elist_to_string(term.chunk) + ")"; - if (term.factor > 0) { - if (!debug_first) debug_ss << "*"; - debug_ss << term_str; - if (term.factor != 1) debug_ss << "^" << term.factor; - } - else { - if (debug_first) debug_ss << "1"; - debug_ss << "/"; - debug_ss << term_str; - if (term.factor != -1) debug_ss << "^" << abs(term.factor); - } - debug_first = false; - } - } - if (debug_first) debug_ss << double_to_pretty_string(ac_final); - // Reconstruct elist in postfix order elist_t result; @@ -4802,6 +5105,7 @@ void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) mexce_charstream s; stringstream debug_ss; bool debug_first = true; + const bool build_debug = !compilation.is_protected(); // Shared RAX cache for compile_elist calls - allows caching variable addresses // across term compilations (e.g., polynomial terms that all use the same variable). @@ -4820,7 +5124,7 @@ void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) } }; - if (ac_final != neutral) { + if (build_debug && ac_final != neutral) { debug_ss << double_to_pretty_string(ac_final); debug_first = false; } @@ -4831,18 +5135,20 @@ void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) continue; } - string term_str = "(" + elist_to_string(term.chunk) + ")"; - if (factor > 0) { - if (!debug_first) debug_ss << "+"; - if (factor != 1) debug_ss << factor << "*"; - debug_ss << term_str; - } - else { - debug_ss << "-"; - if (factor != -1) debug_ss << abs(factor) << "*"; - debug_ss << term_str; + if (build_debug) { + const string term_string = "(" + elist_to_string(term.chunk) + ")"; + if (factor > 0) { + if (!debug_first) debug_ss << "+"; + if (factor != 1) debug_ss << factor << "*"; + debug_ss << term_string; + } + else { + debug_ss << "-"; + if (factor != -1) debug_ss << abs(factor) << "*"; + debug_ss << term_string; + } + debug_first = false; } - debug_first = false; if (have_value) { ensure_constant(); @@ -4893,7 +5199,7 @@ void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) } }; - if (ac_final != neutral) { + if (build_debug && ac_final != neutral) { debug_ss << double_to_pretty_string(ac_final); debug_first = false; } @@ -4904,19 +5210,21 @@ void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) continue; } - string term_str = "(" + elist_to_string(term.chunk) + ")"; - if (factor > 0) { - if (!debug_first) debug_ss << "*"; - debug_ss << term_str; - if (factor != 1) debug_ss << "^" << factor; - } - else { - if (debug_first) debug_ss << "1"; // e.g. 1/x - debug_ss << "/"; - debug_ss << term_str; - if (factor != -1) debug_ss << "^" << abs(factor); + if (build_debug) { + const string term_string = "(" + elist_to_string(term.chunk) + ")"; + if (factor > 0) { + if (!debug_first) debug_ss << "*"; + debug_ss << term_string; + if (factor != 1) debug_ss << "^" << factor; + } + else { + if (debug_first) debug_ss << "1"; // e.g. 1/x + debug_ss << "/"; + debug_ss << term_string; + if (factor != -1) debug_ss << "^" << abs(factor); + } + debug_first = false; } - debug_first = false; if (have_value) { ensure_constant(); @@ -4961,7 +5269,7 @@ void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) } } - if (debug_first) { // Loop didn't run or output anything, just constant + if (build_debug && debug_first) { // Loop didn't run or output anything, just constant if (debug_ss.str().empty()) debug_ss << double_to_pretty_string(ac_final); } @@ -4971,7 +5279,9 @@ void asmd_optimizer(elist_it_t it, evaluator* ev, elist_t* elist) compilation.m_next_element_id++, new_name, 0, 0, s.buf.size(), cc, nullptr); f_opt->m_requires_x87_backend = true; - f_opt->debug_desc = "(" + debug_ss.str() + ")"; + if (build_debug) { + f_opt->debug_desc = "(" + debug_ss.str() + ")"; + } f_opt->cse_store_suffix = f->cse_store_suffix; // Preserve CSE store code f_opt->cse_store_addr = f->cse_store_addr; // Preserve CSE temp address @@ -5119,8 +5429,51 @@ inline shared_ptr make_function(evaluator* ev, const string& name) { auto& compilation = ev->active_compilation(); auto fn_it = function_map().find(name); assert(fn_it != function_map().end()); - // Create a copy of the prototype and assign a new, unique ID. - auto new_func = make_shared(fn_it->second); + Function configured = fn_it->second; + const auto& selected = compilation.m_effective_options; + if (name == "sin") { + configured = Sin(selected.use_libm_sin); + configured.m_requires_x87_backend = !selected.use_libm_sin; + } + else if (name == "cos") { + configured = Cos(selected.use_libm_cos); + configured.m_requires_x87_backend = !selected.use_libm_cos; + } + else if (name == "tan") { + configured = Tan(selected.use_libm_tan); + configured.m_requires_x87_backend = !selected.use_libm_tan; + } + else if (name == "exp") { + configured = Exp(selected.use_libm_exp); + configured.m_requires_x87_backend = !selected.use_libm_exp; + } + else if (name == "log" || name == "ln") { + configured = name == "log" + ? Log(selected.use_libm_log) + : Ln(selected.use_libm_log); + configured.m_requires_x87_backend = !selected.use_libm_log; + } + else if (name == "log10") { + configured = Log10(selected.use_libm_log10); + configured.m_requires_x87_backend = !selected.use_libm_log10; + } + else if (name == "log2") { + configured = Log2(selected.use_libm_log2); + configured.m_requires_x87_backend = !selected.use_libm_log2; + } + else if (name == "logb") { + configured = Logb(selected.use_libm_logb); + configured.m_requires_x87_backend = !selected.use_libm_logb; + } + else if (name == "ylog2") { + configured = Ylog2(selected.use_libm_ylog2); + configured.m_requires_x87_backend = !selected.use_libm_ylog2; + } + else if (name == "pow") { + configured = Pow(selected.use_libm_generic_pow); + configured.m_requires_x87_backend = !selected.use_libm_generic_pow; + } + auto new_func = make_shared(configured); new_func->id = compilation.m_next_element_id++; return new_func; } @@ -5170,21 +5523,60 @@ inline void Semantic_graph_builder::finish() const inline Semantic_compiler::Semantic_compiler( evaluator& owner, const options& effective_options, - uint64_t first_element_id) + uint64_t first_element_id, + protected_wipe_function_t protected_wipe, + Compilation_test_fault test_fault) : m_owner(owner), - m_candidate(new Compilation_state(effective_options, first_element_id)), + m_candidate(new Compilation_state( + effective_options, first_element_id, protected_wipe, test_fault)), m_graph_builder(owner), + m_fallback_graph_builder(owner), m_active_compilation_reset(owner.m_active_compilation) { - reset_graph_builder(); + m_active_compilation_reset.set(m_candidate.get()); + m_graph_builder.reset(*m_candidate); + if (protected_wipe) { + options fallback_options = effective_options; + fallback_options.prefer_x87 = true; + m_fallback_candidate.reset(new Compilation_state( + fallback_options, + first_element_id, + protected_wipe, + test_fault, + Compilation_state::Protected_storage_role::FALLBACK)); + m_fallback_graph_builder.reset(*m_fallback_candidate); + } } -inline void Semantic_compiler::reset_graph_builder() +inline void Semantic_compiler::consume_literal_into( + Compilation_state& compilation, + Semantic_graph_builder& builder, + double value) { - m_active_compilation_reset.set(m_candidate.get()); - m_graph_builder.reset(*m_candidate); + m_active_compilation_reset.set(&compilation); + builder.push_literal(value); +} + + +inline void Semantic_compiler::consume_variable_into( + Compilation_state& compilation, + Semantic_graph_builder& builder, + const shared_ptr& variable) +{ + m_active_compilation_reset.set(&compilation); + builder.push_variable(variable); +} + + +inline void Semantic_compiler::consume_call_into( + Compilation_state& compilation, + Semantic_graph_builder& builder, + const string& name) +{ + m_active_compilation_reset.set(&compilation); + builder.call(name); } @@ -5192,7 +5584,7 @@ inline void Semantic_compiler::consume(const Semantic_record& record) { switch (record.kind()) { case Semantic_record_kind::LITERAL: - m_graph_builder.push_literal(record.literal()); + consume_literal(record.literal()); break; case Semantic_record_kind::VARIABLE: { const auto variable_it = m_owner.m_variables.find(record.variable()->name); @@ -5201,14 +5593,11 @@ inline void Semantic_compiler::consume(const Semantic_record& record) { throw std::logic_error("Invalid semantic expression"); } - m_graph_builder.push_variable(variable_it->second); - if (m_referenced_variable_set.insert(variable_it->second.get()).second) { - m_referenced_variables.push_back(variable_it->second); - } + consume_variable(variable_it->second); break; } case Semantic_record_kind::CALL: - m_graph_builder.call(record.function()->name); + consume_call(record.function()->name); break; default: throw std::logic_error("Invalid semantic expression"); @@ -5216,12 +5605,50 @@ inline void Semantic_compiler::consume(const Semantic_record& record) } +inline void Semantic_compiler::consume_literal(double value) +{ + consume_literal_into(*m_candidate, m_graph_builder, value); + if (m_fallback_candidate) { + consume_literal_into( + *m_fallback_candidate, m_fallback_graph_builder, value); + } +} + + +inline void Semantic_compiler::consume_variable(const shared_ptr& variable) +{ + consume_variable_into(*m_candidate, m_graph_builder, variable); + if (m_fallback_candidate) { + consume_variable_into( + *m_fallback_candidate, m_fallback_graph_builder, variable); + } + if (m_referenced_variable_set.insert(variable.get()).second) { + m_referenced_variables.push_back(variable); + } +} + + +inline void Semantic_compiler::consume_call(const string& name) +{ + consume_call_into(*m_candidate, m_graph_builder, name); + if (m_fallback_candidate) { + consume_call_into(*m_fallback_candidate, m_fallback_graph_builder, name); + } +} + + inline Semantic_compilation_result Semantic_compiler::finish() { m_graph_builder.finish(); m_graph_builder.clear(); + if (m_fallback_candidate) { + m_fallback_graph_builder.finish(); + m_fallback_graph_builder.clear(); + } auto result = m_owner.finish_semantic_compilation( - std::move(m_candidate), std::move(m_referenced_variables)); + std::move(m_candidate), + std::move(m_referenced_variables), + std::move(m_fallback_candidate)); m_active_compilation_reset.clear(); return result; } @@ -5512,10 +5939,7 @@ evaluator::evaluator(): inline evaluator::~evaluator() { - m_is_constant_expression = false; - m_evaluate_fptr = nullptr; - m_backend_used = backend_type::none; - m_compilation.reset(); + reset_compiled_expression(); } @@ -5535,6 +5959,38 @@ void evaluator::bind(T& v, const std::string& s, Args&... args) } +template +void evaluator::bind_protected(T& referenced_variable, uint32_t slot) +{ + static_assert( + std::is_same::value || + std::is_same::value || + std::is_same::value || + std::is_same::value || + std::is_same::value, + "Protected bindings support double, float, int16_t, int32_t, and int64_t"); + + if (slot > 4095) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "A protected-expression binding slot is out of range."); + } + + const auto binding = m_protected_variables.find(slot); + if (binding != m_protected_variables.end() && binding->second->referenced) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "A referenced protected-expression binding cannot be replaced."); + } + + m_protected_variables[slot] = std::make_shared( + m_next_variable_id++, + &referenced_variable, + std::string(), + impl::get_ndt()); +} + + template void evaluator::unbind(const std::string& s, Args&... args) { @@ -5561,6 +6017,36 @@ void evaluator::unbind_all() } +inline +void evaluator::unbind_protected(uint32_t slot) +{ + const auto binding = m_protected_variables.find(slot); + if (binding == m_protected_variables.end()) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "Attempted to unbind an unknown protected-expression slot."); + } + if (binding->second->referenced) { + reset_compiled_expression(); + } + m_protected_variables.erase(slot); +} + + +inline +void evaluator::unbind_all_protected() +{ + bool has_referenced_binding = false; + for (const auto& binding : m_protected_variables) { + has_referenced_binding |= binding.second->referenced; + } + if (has_referenced_binding) { + reset_compiled_expression(); + } + m_protected_variables.clear(); +} + + inline double evaluator::evaluate() { if (m_is_constant_expression) { @@ -5569,18 +6055,28 @@ double evaluator::evaluate() { if (!m_evaluate_fptr) { throw std::logic_error("No expression has been compiled."); } - return m_evaluate_fptr(); + return impl::invoke_generated(m_evaluate_fptr); } inline std::string evaluator::get_optimized_expression() const { + if (m_is_protected_expression) { + throw Protected_expression_error( + Protected_expression_error_category::INTROSPECTION_DISABLED, + "Protected-expression introspection is disabled."); + } return m_compilation ? impl::elist_to_string(m_compilation->m_elist) : std::string(); } inline std::string evaluator::get_byte_representation() const { + if (m_is_protected_expression) { + throw Protected_expression_error( + Protected_expression_error_category::INTROSPECTION_DISABLED, + "Protected-expression introspection is disabled."); + } if (!m_compilation || m_compilation->m_is_constant_expression || !m_compilation->m_evaluate_fptr || @@ -6105,11 +6601,14 @@ impl::Semantic_program impl::Clear_semantic_producer::produce( inline impl::Semantic_compilation_result evaluator::finish_semantic_compilation( std::unique_ptr candidate, - std::vector> referenced_variables) + std::vector> referenced_variables, + std::unique_ptr fallback_candidate) { using namespace impl; Semantic_program fallback_program; + Semantic_program_wipe_guard fallback_wipe( + fallback_program, candidate->protected_wipe()); m_active_compilation = candidate.get(); link_arguments(candidate->m_elist); @@ -6131,7 +6630,11 @@ impl::Semantic_compilation_result evaluator::finish_semantic_compilation( candidate->m_elist.begin(), candidate->m_elist.end(), candidate->m_effective_options.prefer_x87); - if (might_need_fallback) { + if (might_need_fallback && !fallback_candidate) { + if (candidate->is_protected()) { + throw std::logic_error( + "Protected compilation requires an independent fallback owner"); + } // Optimizers mutate compiler objects. Preserve typed semantics so fallback // can construct an independent graph without retaining or reparsing source. fallback_program.reserve(candidate->m_elist.size()); @@ -6154,6 +6657,9 @@ impl::Semantic_compilation_result evaluator::finish_semantic_compilation( } } } + if (!might_need_fallback) { + fallback_candidate.reset(); + } bool fallback_allowed = might_need_fallback; compile_candidate: @@ -6340,7 +6846,7 @@ impl::Semantic_compilation_result evaluator::finish_semantic_compilation( elist_it_t first_arg_it = y; std::advance(first_arg_it, -(int64_t)f->num_args); compile_and_finalize_elist(first_arg_it, next(y)); - const double res = evaluate_fptr(); + const double res = invoke_generated(evaluate_fptr); // Constant folding uses a temporary JIT buffer. Release it // immediately so repeated foldable subtrees do not leak one // executable page per fold before the final expression compile. @@ -6367,36 +6873,45 @@ impl::Semantic_compilation_result evaluator::finish_semantic_compilation( need_fallback_check && !is_sse2_compatible(m_elist.begin(), m_elist.end(), false)) { - options fallback_options = candidate->m_effective_options; - fallback_options.prefer_x87 = true; - candidate.reset(new Compilation_state(fallback_options, m_next_variable_id)); - m_active_compilation = candidate.get(); - - Semantic_graph_builder fallback_builder(*this); - fallback_builder.reset(*candidate); - for (const auto& record : fallback_program) { - switch (record.kind()) { - case Semantic_record_kind::LITERAL: - fallback_builder.push_literal(record.literal()); - break; - case Semantic_record_kind::VARIABLE: { - const auto variable_it = m_variables.find(record.variable()->name); - assert(variable_it != m_variables.end()); - fallback_builder.push_variable(variable_it->second); - break; + if (fallback_candidate) { + candidate = std::move(fallback_candidate); + m_active_compilation = candidate.get(); + } + else { + options fallback_options = candidate->m_effective_options; + fallback_options.prefer_x87 = true; + candidate.reset(new Compilation_state( + fallback_options, m_next_variable_id)); + m_active_compilation = candidate.get(); + + Semantic_graph_builder fallback_builder(*this); + fallback_builder.reset(*candidate); + for (const auto& record : fallback_program) { + switch (record.kind()) { + case Semantic_record_kind::LITERAL: + fallback_builder.push_literal(record.literal()); + break; + case Semantic_record_kind::VARIABLE: { + const auto variable_it = m_variables.find(record.variable()->name); + assert(variable_it != m_variables.end()); + fallback_builder.push_variable(variable_it->second); + break; + } + case Semantic_record_kind::CALL: + fallback_builder.call(record.function()->name); + break; + default: + throw std::logic_error("Invalid semantic expression"); } - case Semantic_record_kind::CALL: - fallback_builder.call(record.function()->name); - break; - default: - throw std::logic_error("Invalid semantic expression"); } + fallback_builder.finish(); } - fallback_builder.finish(); fallback_allowed = false; goto compile_candidate; } + fallback_candidate.reset(); + is_constant_expression = m_elist.size() == 1 && m_elist.back().type == Element_type::CCONST; if (is_constant_expression) { constant_expression_value = m_elist.back().c->value; @@ -6412,26 +6927,34 @@ impl::Semantic_compilation_result evaluator::finish_semantic_compilation( inline -void evaluator::set_expression(std::string expression) +void evaluator::reset_compiled_expression() { + if (m_compilation) { + m_compilation->release_executable(); + m_compilation->wipe_protected_scalar( + &m_constant_expression_value, + sizeof(m_constant_expression_value)); + } m_is_constant_expression = false; m_constant_expression_value = 0.0; m_evaluate_fptr = nullptr; m_backend_used = backend_type::none; + m_is_protected_expression = false; m_compilation.reset(); for (const auto& variable : m_variables) { variable.second->referenced = false; } - - const auto semantic_program = impl::Clear_semantic_producer::produce( - *this, std::move(expression)); - impl::Semantic_compiler compiler(*this, m_options, m_next_variable_id); - for (const auto& record : semantic_program) { - compiler.consume(record); + for (const auto& variable : m_protected_variables) { + variable.second->referenced = false; } - auto result = compiler.finish(); +} + +inline +void evaluator::publish_semantic_compilation( + impl::Semantic_compilation_result result) +{ m_compilation = std::move(result.compilation); for (const auto& variable : result.referenced_variables) { variable->referenced = true; @@ -6440,6 +6963,23 @@ void evaluator::set_expression(std::string expression) m_constant_expression_value = m_compilation->m_constant_expression_value; m_evaluate_fptr = m_compilation->m_evaluate_fptr; m_backend_used = m_compilation->m_backend_used; + m_is_protected_expression = m_compilation->is_protected(); +} + + +inline +void evaluator::set_expression(std::string expression) +{ + reset_compiled_expression(); + + const auto semantic_program = impl::Clear_semantic_producer::produce( + *this, std::move(expression)); + impl::Semantic_compiler compiler(*this, m_options, m_next_variable_id); + for (const auto& record : semantic_program) { + compiler.consume(record); + } + auto result = compiler.finish(); + publish_semantic_compilation(std::move(result)); } @@ -6500,6 +7040,9 @@ void evaluator::compile_and_finalize_elist(impl::elist_const_it_t first, impl::e finalize_rip_constants(code_buffer, pending_constants); m_buffer_size = code_buffer.buf.size(); + if (compilation.fail_executable_allocation()) { + throw std::bad_alloc(); + } auto buffer = get_executable_buffer(m_buffer_size); #ifdef _WIN32 @@ -6513,7 +7056,15 @@ void evaluator::compile_and_finalize_elist(impl::elist_const_it_t first, impl::e #endif memcpy(buffer, code_buffer.buf.data(), m_buffer_size); - evaluate_fptr = lock_executable_buffer(buffer, m_buffer_size); + if (compilation.fail_executable_finalization()) { + free_executable_buffer( + reinterpret_cast(buffer), + m_buffer_size, + compilation.protected_wipe()); + throw Executable_memory_error("Injected executable finalization failure"); + } + evaluate_fptr = lock_executable_buffer( + buffer, m_buffer_size, compilation.protected_wipe()); compiled_backend = backend_type::sse2; return; } @@ -6600,6 +7151,9 @@ void evaluator::compile_and_finalize_elist(impl::elist_const_it_t first, impl::e code_buffer.buf.insert(code_buffer.buf.end(), return_sequence, return_sequence + return_sequence_size); m_buffer_size = code_buffer.buf.size(); + if (compilation.fail_executable_allocation()) { + throw std::bad_alloc(); + } auto buffer = get_executable_buffer(m_buffer_size); #ifdef _WIN32 @@ -6614,7 +7168,16 @@ void evaluator::compile_and_finalize_elist(impl::elist_const_it_t first, impl::e memcpy(buffer, code_buffer.buf.data(), m_buffer_size); - evaluate_fptr = lock_executable_buffer(buffer, m_buffer_size); + if (compilation.fail_executable_finalization()) { + free_executable_buffer( + reinterpret_cast(buffer), + m_buffer_size, + compilation.protected_wipe()); + throw Executable_memory_error("Injected executable finalization failure"); + } + + evaluate_fptr = lock_executable_buffer( + buffer, m_buffer_size, compilation.protected_wipe()); compiled_backend = backend_type::x87; } diff --git a/mexce_protected.h b/mexce_protected.h index a597182..46287c1 100644 --- a/mexce_protected.h +++ b/mexce_protected.h @@ -24,79 +24,9 @@ static_assert(crypto_secretstream_xchacha20poly1305_ABYTES == 17, "Protected-expression format 1.0 requires 17-byte secretstream overhead"); -enum class Protected_expression_error_category -{ - INVALID_ARGUMENT, - UNSUPPORTED_FORMAT, - SIZE_LIMIT, - AUTHENTICATION_FAILED, - MALFORMED_PROGRAM, - MISSING_BINDING, - INTROSPECTION_DISABLED, - CRYPTOGRAPHY_UNAVAILABLE, - COMPILATION_FAILED, - RESOURCE_FAILURE, -}; - - -class Protected_expression_error: public std::runtime_error -{ -public: - Protected_expression_error( - Protected_expression_error_category category, - const char* message) - : - std::runtime_error(message), - m_category(category), - m_has_record_index(false), - m_record_index(0) - {} - - Protected_expression_error( - Protected_expression_error_category category, - const char* message, - uint32_t record_index) - : - std::runtime_error(message), - m_category(category), - m_has_record_index(true), - m_record_index(record_index) - {} - - Protected_expression_error_category category() const noexcept { return m_category; } - bool has_record_index() const noexcept { return m_has_record_index; } - - uint32_t record_index() const - { - if (!m_has_record_index) { - throw std::logic_error("No protected-expression record index is available."); - } - return m_record_index; - } - -private: - Protected_expression_error_category m_category; - bool m_has_record_index; - uint32_t m_record_index; -}; - - namespace impl { -enum class Protected_wipe_context -{ - KEY, - PULL_STATE, - PUSH_STATE, - CLEAR_RECORD, - ADDITIONAL_DATA, - DECODED_SCALAR, - VALIDATOR_STATE, - COUNT, -}; - - #ifdef MEXCE_PROTECTED_TESTING class Protected_wipe_observer { @@ -104,7 +34,11 @@ class Protected_wipe_observer void observe(Protected_wipe_context context, const uint8_t* bytes, size_t size) { const size_t context_index = static_cast(context); + const size_t sequence = ++m_sequence; ++m_counts[context_index]; + if (m_first_sequence[context_index] == 0) { + m_first_sequence[context_index] = sequence; + } for (size_t i = 0; i < size; ++i) { m_saw_nonzero_after_wipe[context_index] |= bytes[i] != 0; } @@ -113,8 +47,10 @@ class Protected_wipe_observer void reset() { std::memset(m_counts, 0, sizeof(m_counts)); + std::memset(m_first_sequence, 0, sizeof(m_first_sequence)); std::memset( m_saw_nonzero_after_wipe, 0, sizeof(m_saw_nonzero_after_wipe)); + m_sequence = 0; } size_t count(Protected_wipe_context context) const @@ -122,6 +58,20 @@ class Protected_wipe_observer return m_counts[static_cast(context)]; } + size_t first_sequence(Protected_wipe_context context) const + { + return m_first_sequence[static_cast(context)]; + } + + bool observed_before( + Protected_wipe_context first, + Protected_wipe_context second) const + { + const size_t first_value = first_sequence(first); + const size_t second_value = first_sequence(second); + return first_value != 0 && second_value != 0 && first_value < second_value; + } + bool all_observed_regions_are_zero() const { for (size_t i = 0; i < static_cast(Protected_wipe_context::COUNT); ++i) { @@ -134,17 +84,21 @@ class Protected_wipe_observer private: size_t m_counts[static_cast(Protected_wipe_context::COUNT)] = {}; + size_t m_first_sequence[static_cast(Protected_wipe_context::COUNT)] = {}; bool m_saw_nonzero_after_wipe[ static_cast(Protected_wipe_context::COUNT)] = {}; + size_t m_sequence = 0; }; struct protected_test_faults_t { - bool fail_sodium_init = false; - bool fail_malloc = false; - bool fail_mlock = false; - bool fail_init_pull = false; + bool fail_sodium_init = false; + bool fail_malloc = false; + bool fail_mlock = false; + bool fail_init_pull = false; + bool fail_executable_allocation = false; + bool fail_executable_finalization = false; }; @@ -494,12 +448,134 @@ class Protected_wipe_guard ~Protected_wipe_guard() { protected_memzero(m_bytes, m_size, m_context); } private: - void* m_bytes; - size_t m_size; + void* m_bytes; + size_t m_size; Protected_wipe_context m_context; }; +inline const char* protected_operation_name(Protected_operation operation) +{ + switch (operation) { + case Protected_operation::ADD: return "add"; + case Protected_operation::SUB: return "sub"; + case Protected_operation::MUL: return "mul"; + case Protected_operation::DIV: return "div"; + case Protected_operation::NEG: return "neg"; + case Protected_operation::POW: return "pow"; + case Protected_operation::SIN: return "sin"; + case Protected_operation::COS: return "cos"; + case Protected_operation::TAN: return "tan"; + case Protected_operation::ABS: return "abs"; + case Protected_operation::SIGN: return "sign"; + case Protected_operation::SIGNP: return "signp"; + case Protected_operation::EXPN: return "expn"; + case Protected_operation::SFC: return "sfc"; + case Protected_operation::SQRT: return "sqrt"; + case Protected_operation::EXP: return "exp"; + case Protected_operation::LT: return "lt"; + case Protected_operation::GT: return "gt"; + case Protected_operation::LE: return "le"; + case Protected_operation::GE: return "ge"; + case Protected_operation::EQ: return "eq"; + case Protected_operation::NE: return "ne"; + case Protected_operation::LOG: return "log"; + case Protected_operation::LOG2: return "log2"; + case Protected_operation::LOG10: return "log10"; + case Protected_operation::LOGB: return "logb"; + case Protected_operation::YLOG2: return "ylog2"; + case Protected_operation::MAX: return "max"; + case Protected_operation::MIN: return "min"; + case Protected_operation::FLOOR: return "floor"; + case Protected_operation::CEIL: return "ceil"; + case Protected_operation::ROUND: return "round"; + case Protected_operation::INT: return "int"; + case Protected_operation::TRUNC: return "trunc"; + case Protected_operation::MOD: return "mod"; + case Protected_operation::BND: return "bnd"; + case Protected_operation::BIAS: return "bias"; + case Protected_operation::GAIN: return "gain"; + default: + assert(false); + return ""; + } +} + + +class Protected_semantic_sink +{ +public: + explicit Protected_semantic_sink(evaluator& owner) + : + m_owner(owner) + {} + + void manifest(uint32_t, bool fast_math, uint32_t) + { + options effective_options = m_owner.m_options; + Protected_wipe_guard effective_options_wipe( + &effective_options, + sizeof(effective_options), + Protected_wipe_context::OPTIONS_STORAGE); + effective_options.fast_math = fast_math; + Compilation_test_fault test_fault = Compilation_test_fault::NONE; +#ifdef MEXCE_PROTECTED_TESTING + const auto& faults = protected_test_faults(); + if (faults.fail_executable_allocation) { + test_fault = Compilation_test_fault::EXECUTABLE_ALLOCATION; + } + else + if (faults.fail_executable_finalization) { + test_fault = Compilation_test_fault::EXECUTABLE_FINALIZATION; + } +#endif + m_compiler.reset(new Semantic_compiler( + m_owner, + effective_options, + m_owner.m_next_variable_id, + protected_memzero, + test_fault)); + } + + void literal(double value, uint32_t) + { + m_compiler->consume_literal(value); + } + + void variable(uint32_t slot, uint32_t record_index) + { + const auto binding = m_owner.m_protected_variables.find(slot); + if (binding == m_owner.m_protected_variables.end()) { + throw Protected_expression_error( + Protected_expression_error_category::MISSING_BINDING, + "A protected-expression binding is missing.", + record_index); + } + m_compiler->consume_variable(binding->second); + } + + void call(Protected_operation operation, uint32_t) + { + m_compiler->consume_call(protected_operation_name(operation)); + } + + void end(uint32_t) + { + m_result = m_compiler->finish(); + } + + Semantic_compilation_result take_result() + { + return std::move(m_result); + } + +private: + evaluator& m_owner; + std::unique_ptr m_compiler; + Semantic_compilation_result m_result; +}; + + class Protected_pull_state { public: @@ -732,6 +808,46 @@ void decode_protected_expression( } // namespace impl +inline void evaluator::set_protected_expression( + const uint8_t* program, + size_t program_size, + Protected_expression_key key) +{ + reset_compiled_expression(); + + if (m_options.enable_cse) { + throw Protected_expression_error( + Protected_expression_error_category::INVALID_ARGUMENT, + "Protected expressions do not support common subexpression elimination."); + } + + try { + impl::Protected_semantic_sink sink(*this); + impl::decode_protected_expression( + program, program_size, std::move(key), sink); + publish_semantic_compilation(sink.take_result()); + } + catch (const Protected_expression_error&) { + throw; + } + catch (const std::bad_alloc&) { + throw Protected_expression_error( + Protected_expression_error_category::RESOURCE_FAILURE, + "Protected-expression compilation exhausted a resource."); + } + catch (const impl::Executable_memory_error&) { + throw Protected_expression_error( + Protected_expression_error_category::RESOURCE_FAILURE, + "Protected-expression executable memory could not be finalized."); + } + catch (const std::exception&) { + throw Protected_expression_error( + Protected_expression_error_category::COMPILATION_FAILED, + "Protected-expression compilation failed."); + } +} + + } // namespace mexce diff --git a/test/mixed_tu_ordinary.cpp b/test/mixed_tu_ordinary.cpp new file mode 100644 index 0000000..d8856e0 --- /dev/null +++ b/test/mixed_tu_ordinary.cpp @@ -0,0 +1,27 @@ +#include "mexce.h" + +#include + + +mexce::evaluator* make_ordinary_evaluator() +{ + return new mexce::evaluator; +} + + +double evaluate_on_ordinary_side(mexce::evaluator* evaluator) +{ + return evaluator->evaluate(); +} + + +void destroy_on_ordinary_side(mexce::evaluator* evaluator) +{ + delete evaluator; +} + + +size_t ordinary_evaluator_size() +{ + return sizeof(mexce::evaluator); +} diff --git a/test/mixed_tu_protected.cpp b/test/mixed_tu_protected.cpp new file mode 100644 index 0000000..ff4e294 --- /dev/null +++ b/test/mixed_tu_protected.cpp @@ -0,0 +1,24 @@ +#include "mexce_protected.h" + +#include +#include + + +void load_on_protected_side( + mexce::evaluator* evaluator, + const uint8_t* program, + size_t program_size, + const uint8_t* key_bytes, + size_t key_size) +{ + evaluator->set_protected_expression( + program, + program_size, + mexce::Protected_expression_key::from_bytes(key_bytes, key_size)); +} + + +size_t protected_evaluator_size() +{ + return sizeof(mexce::evaluator); +} diff --git a/test/mixed_tu_test.cpp b/test/mixed_tu_test.cpp new file mode 100644 index 0000000..38d0c05 --- /dev/null +++ b/test/mixed_tu_test.cpp @@ -0,0 +1,63 @@ +#include "mexce.h" + +#include +#include +#include +#include +#include +#include +#include + + +mexce::evaluator* make_ordinary_evaluator(); +double evaluate_on_ordinary_side(mexce::evaluator* evaluator); +void destroy_on_ordinary_side(mexce::evaluator* evaluator); +size_t ordinary_evaluator_size(); + +void load_on_protected_side( + mexce::evaluator* evaluator, + const uint8_t* program, + size_t program_size, + const uint8_t* key_bytes, + size_t key_size); +size_t protected_evaluator_size(); + + +int main() +{ + std::ifstream input(MEXCE_PROTECTED_FIXTURE_PATH, std::ios::binary); + const std::vector program( + (std::istreambuf_iterator(input)), + std::istreambuf_iterator()); + if (program.empty()) { + std::cerr << "FAIL: protected fixture could not be read\n"; + return 1; + } + + std::array key; + for (size_t i = 0; i < key.size(); ++i) { + key[i] = static_cast(i); + } + + mexce::evaluator* evaluator = make_ordinary_evaluator(); + double x = 4.0; + double y = 1.0; + evaluator->bind_protected(x, 0); + evaluator->bind_protected(y, 1); + load_on_protected_side( + evaluator, + program.data(), + program.size(), + key.data(), + key.size()); + + const bool valid = ordinary_evaluator_size() == protected_evaluator_size() && + std::abs(evaluate_on_ordinary_side(evaluator) - 11.0) < 1e-12; + destroy_on_ordinary_side(evaluator); + if (!valid) { + std::cerr << "FAIL: mixed translation-unit evaluator mismatch\n"; + return 1; + } + std::cout << "Mixed translation-unit evaluator test passed\n"; + return 0; +} diff --git a/test/ordinary_no_sodium_test.cpp b/test/ordinary_no_sodium_test.cpp new file mode 100644 index 0000000..260455e --- /dev/null +++ b/test/ordinary_no_sodium_test.cpp @@ -0,0 +1,19 @@ +#include "mexce.h" + +#include +#include + + +int main() +{ + double value = 3.25; + mexce::evaluator evaluator; + evaluator.bind(value, "value"); + evaluator.set_expression("value*2"); + if (std::abs(evaluator.evaluate() - 6.5) > 1e-12) { + std::cerr << "FAIL: ordinary evaluator result mismatch\n"; + return 1; + } + std::cout << "Ordinary no-sodium test passed\n"; + return 0; +} diff --git a/test/protected_benchmark.cpp b/test/protected_benchmark.cpp new file mode 100644 index 0000000..e13792e --- /dev/null +++ b/test/protected_benchmark.cpp @@ -0,0 +1,187 @@ +#include "mexce_protected_encoder.h" + +#include + +#ifdef _WIN32 +#include +#else +#include +#endif + +#include +#include +#include +#include +#include +#include + + +namespace { + + +uint64_t peak_working_set_bytes() +{ +#ifdef _WIN32 + PROCESS_MEMORY_COUNTERS counters = {}; + counters.cb = sizeof(counters); + if (!GetProcessMemoryInfo( + GetCurrentProcess(), &counters, sizeof(counters))) + { + return 0; + } + return static_cast(counters.PeakWorkingSetSize); +#else + struct rusage usage = {}; + if (getrusage(RUSAGE_SELF, &usage) != 0) { + return 0; + } + return static_cast(usage.ru_maxrss) * 1024; +#endif +} + + +template +uint64_t elapsed_nanoseconds(Function function) +{ + const auto start = std::chrono::steady_clock::now(); + function(); + const auto end = std::chrono::steady_clock::now(); + return static_cast( + std::chrono::duration_cast(end - start).count()); +} + + +int matched_diagnostic() +{ + constexpr int k_compile_iterations = 25; + constexpr int k_evaluate_iterations = 10000; + const std::string expression = "sin(x)+x*y+pow(y,3)-log(x+2)"; + const std::vector bindings = {{"x", 0}, {"y", 1}}; + double x = 0.75; + double y = 1.25; + + mexce::evaluator clear; + clear.bind(x, "x", y, "y"); + const uint64_t clear_compile_ns = elapsed_nanoseconds([&] { + for (int i = 0; i < k_compile_iterations; ++i) { + clear.set_expression(expression); + } + }); + + std::vector bundles; + bundles.reserve(k_compile_iterations); + for (int i = 0; i < k_compile_iterations; ++i) { + bundles.push_back(mexce::encode_protected_expression( + expression, bindings, mexce::Protected_math_mode::STRICT)); + } + + mexce::evaluator protected_evaluator; + protected_evaluator.bind_protected(x, 0); + protected_evaluator.bind_protected(y, 1); + const uint64_t protected_compile_ns = elapsed_nanoseconds([&] { + for (auto& bundle : bundles) { + protected_evaluator.set_protected_expression( + bundle.program.data(), + bundle.program.size(), + std::move(bundle.key)); + } + }); + + volatile double clear_result = 0.0; + const uint64_t clear_evaluate_ns = elapsed_nanoseconds([&] { + for (int i = 0; i < k_evaluate_iterations; ++i) { + clear_result = clear.evaluate(); + } + }); + volatile double protected_result = 0.0; + const uint64_t protected_evaluate_ns = elapsed_nanoseconds([&] { + for (int i = 0; i < k_evaluate_iterations; ++i) { + protected_result = protected_evaluator.evaluate(); + } + }); + + std::cout + << "{\"mode\":\"matched\",\"iterations\":" << k_compile_iterations + << ",\"clear_compile_ns\":" << clear_compile_ns + << ",\"protected_compile_ns\":" << protected_compile_ns + << ",\"clear_evaluate_ns\":" << clear_evaluate_ns + << ",\"protected_evaluate_ns\":" << protected_evaluate_ns + << ",\"backend\":" + << static_cast(protected_evaluator.get_backend()) + << ",\"peak_working_set_bytes\":" << peak_working_set_bytes() + << ",\"correct\":" << (clear_result == protected_result ? "true" : "false") + << "}\n"; + return clear_result == protected_result ? 0 : 1; +} + + +std::string maximum_expression() +{ + constexpr size_t k_literal_count = 8191; + std::string expression; + expression.reserve(k_literal_count * 2); + for (size_t i = 0; i < k_literal_count; ++i) { + if (i != 0) { + expression += '+'; + } + expression += '1'; + } + return expression; +} + + +int resource_diagnostic(bool late_invalid) +{ + auto bundle = mexce::encode_protected_expression( + maximum_expression(), {}, mexce::Protected_math_mode::STRICT); + if (late_invalid) { + bundle.program.back() ^= 0x01; + } + + mexce::evaluator evaluator; + bool accepted = false; + const uint64_t elapsed_ns = elapsed_nanoseconds([&] { + try { + evaluator.set_protected_expression( + bundle.program.data(), bundle.program.size(), std::move(bundle.key)); + accepted = true; + } + catch (const mexce::Protected_expression_error&) { + } + }); + + const bool correct = accepted != late_invalid; + std::cout + << "{\"mode\":\"" << (late_invalid ? "late_invalid" : "maximum_valid") + << "\",\"artifact_bytes\":" << bundle.program.size() + << ",\"elapsed_ns\":" << elapsed_ns + << ",\"peak_working_set_bytes\":" << peak_working_set_bytes() + << ",\"accepted\":" << (accepted ? "true" : "false") + << ",\"correct\":" << (correct ? "true" : "false") + << "}\n"; + return correct ? 0 : 1; +} + + +} // namespace + + +int main(int argc, char** argv) +{ + if (sodium_init() < 0) { + std::cerr << "protected benchmark requires libsodium 1.0.22\n"; + return 1; + } + if (argc == 1) { + return matched_diagnostic(); + } + const std::string mode = argv[1]; + if (mode == "maximum-valid") { + return resource_diagnostic(false); + } + if (mode == "late-invalid") { + return resource_diagnostic(true); + } + std::cerr << "usage: protected_benchmark [maximum-valid|late-invalid]\n"; + return 1; +} diff --git a/test/protected_decoder_fuzz.cpp b/test/protected_decoder_fuzz.cpp new file mode 100644 index 0000000..2605c24 --- /dev/null +++ b/test/protected_decoder_fuzz.cpp @@ -0,0 +1,263 @@ +#include "mexce_protected.h" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + + +namespace { + + +std::array fixed_key_bytes() +{ + std::array key; + for (size_t i = 0; i < key.size(); ++i) { + key[i] = static_cast(i); + } + return key; +} + + +void exercise(const uint8_t* data, size_t size) +{ + const auto key_bytes = fixed_key_bytes(); + double slot0 = 1.25; + double slot1 = -2.5; + mexce::evaluator evaluator; + evaluator.bind_protected(slot0, 0); + evaluator.bind_protected(slot1, 1); + try { + evaluator.set_protected_expression( + data, + size, + mexce::Protected_expression_key::from_bytes( + key_bytes.data(), key_bytes.size())); + (void)evaluator.evaluate(); + } + catch (const std::exception&) { + } +} + + +bool rejects_as_malformed(const std::vector& program) +{ + const auto key_bytes = fixed_key_bytes(); + double slot0 = 1.25; + double slot1 = -2.5; + mexce::evaluator evaluator; + evaluator.bind_protected(slot0, 0); + evaluator.bind_protected(slot1, 1); + try { + evaluator.set_protected_expression( + program.data(), + program.size(), + mexce::Protected_expression_key::from_bytes( + key_bytes.data(), key_bytes.size())); + } + catch (const mexce::Protected_expression_error& error) { + return error.category() == + mexce::Protected_expression_error_category::MALFORMED_PROGRAM; + } + catch (const std::exception&) { + return false; + } + return false; +} + + +constexpr size_t k_reference_header_size = 64; +constexpr size_t k_reference_record_size = 32; +constexpr size_t k_reference_frame_size = 49; + + +void store_u32(uint8_t* bytes, uint32_t value) +{ + for (size_t i = 0; i < 4; ++i) { + bytes[i] = static_cast(value >> (i * 8)); + } +} + + +uint32_t load_u32(const uint8_t* bytes) +{ + return static_cast(bytes[0]) | + static_cast(bytes[1]) << 8 | + static_cast(bytes[2]) << 16 | + static_cast(bytes[3]) << 24; +} + + +std::vector> decrypt_fixture( + const std::vector& program) +{ + if (program.size() < k_reference_header_size) { + throw std::runtime_error("structured input is too short"); + } + const uint32_t count = load_u32(program.data() + 16); + if (count < 3 || count > 16384 || + program.size() != k_reference_header_size + count * k_reference_frame_size) + { + throw std::runtime_error("structured input has invalid framing"); + } + + const auto key = fixed_key_bytes(); + crypto_secretstream_xchacha20poly1305_state state; + if (crypto_secretstream_xchacha20poly1305_init_pull( + &state, program.data() + 40, key.data()) != 0) + { + throw std::runtime_error("fixture pull initialization failed"); + } + + std::vector> records(count); + for (uint32_t index = 0; index < count; ++index) { + std::array additional_data = {}; + std::memcpy(additional_data.data(), program.data(), k_reference_header_size); + store_u32(additional_data.data() + k_reference_header_size, index); + unsigned long long clear_size = 0; + unsigned char tag = 0; + if (crypto_secretstream_xchacha20poly1305_pull( + &state, + records[index].data(), + &clear_size, + &tag, + program.data() + k_reference_header_size + index * k_reference_frame_size, + k_reference_frame_size, + additional_data.data(), + additional_data.size()) != 0 || + clear_size != k_reference_record_size) + { + throw std::runtime_error("fixture decryption failed"); + } + } + sodium_memzero(&state, sizeof(state)); + return records; +} + + +std::vector encrypt_records( + const std::vector& fixture, + const std::vector>& records) +{ + const auto key = fixed_key_bytes(); + std::vector program = fixture; + crypto_secretstream_xchacha20poly1305_state state; + crypto_secretstream_xchacha20poly1305_init_push( + &state, program.data() + 40, key.data()); + + for (uint32_t index = 0; index < records.size(); ++index) { + std::array additional_data = {}; + std::memcpy(additional_data.data(), program.data(), k_reference_header_size); + store_u32(additional_data.data() + k_reference_header_size, index); + unsigned long long frame_size = 0; + const unsigned char tag = index + 1 == records.size() + ? crypto_secretstream_xchacha20poly1305_TAG_FINAL + : crypto_secretstream_xchacha20poly1305_TAG_MESSAGE; + crypto_secretstream_xchacha20poly1305_push( + &state, + program.data() + k_reference_header_size + index * k_reference_frame_size, + &frame_size, + records[index].data(), + records[index].size(), + additional_data.data(), + additional_data.size(), + tag); + } + sodium_memzero(&state, sizeof(state)); + return program; +} +} // namespace + + +extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) +{ + exercise(data, size); + return 0; +} + + +#ifndef MEXCE_FUZZ_SMOKE_MAIN +extern "C" size_t LLVMFuzzerMutate( + uint8_t* data, + size_t size, + size_t max_size); + + +extern "C" size_t LLVMFuzzerCustomMutator( + uint8_t* data, + size_t size, + size_t max_size, + unsigned int seed) +{ + try { + const std::vector program(data, data + size); + auto records = decrypt_fixture(program); + if (records.size() > 2 && size <= max_size) { + const size_t record = 1 + seed % (records.size() - 2); + const size_t byte = (seed / records.size()) % records[record].size(); + records[record][byte] ^= static_cast(1U << (seed % 8)); + const auto mutation = encrypt_records(program, records); + std::memcpy(data, mutation.data(), mutation.size()); + return mutation.size(); + } + } + catch (const std::exception&) { + } + return LLVMFuzzerMutate(data, size, max_size); +} +#endif + + +#ifdef MEXCE_FUZZ_SMOKE_MAIN +int main() +{ + if (sodium_init() < 0) { + std::cerr << "protected decoder fuzz smoke requires libsodium 1.0.22\n"; + return 1; + } + + std::ifstream input(MEXCE_PROTECTED_FIXTURE_PATH, std::ios::binary); + const std::vector fixture( + (std::istreambuf_iterator(input)), + std::istreambuf_iterator()); + if (fixture.empty()) { + std::cerr << "FAIL: protected fixture could not be read\n"; + return 1; + } + + exercise(nullptr, 0); + exercise(fixture.data(), fixture.size()); + for (size_t i = 0; i < fixture.size(); ++i) { + std::vector mutation = fixture; + mutation[i] ^= 0x01; + exercise(mutation.data(), mutation.size()); + } + + auto records = decrypt_fixture(fixture); + records[2][2] = 1; + auto reserved_mutation = encrypt_records(fixture, records); + if (!rejects_as_malformed(reserved_mutation)) { + std::cerr << "FAIL: authenticated reserved field was not rejected as malformed\n"; + return 1; + } + + records = decrypt_fixture(fixture); + records[3][4] = 255; + auto operation_mutation = encrypt_records(fixture, records); + if (!rejects_as_malformed(operation_mutation)) { + std::cerr << "FAIL: authenticated operation was not rejected as malformed\n"; + return 1; + } + + std::cout << "Protected decoder fuzz smoke passed\n"; + return 0; +} +#endif diff --git a/test/protected_runtime_tests.cpp b/test/protected_runtime_tests.cpp new file mode 100644 index 0000000..c14df18 --- /dev/null +++ b/test/protected_runtime_tests.cpp @@ -0,0 +1,720 @@ +#include "mexce_protected_encoder.h" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifdef max +#undef max +#endif + + +namespace { + + +const char* g_fixture_path = MEXCE_PROTECTED_FIXTURE_PATH; + + +struct Test_suite +{ + std::vector m_failures; + + void expect(bool value, const std::string& name) + { + if (!value) { + m_failures.push_back(name); + } + } + + void expect_near( + double actual, + double expected, + const std::string& name) + { + const double scale = std::max(1.0, std::abs(expected)); + expect(std::abs(actual - expected) <= 1e-10 * scale, name); + } + + template + void expect_error( + const std::string& name, + mexce::Protected_expression_error_category category, + Function function) + { + try { + function(); + m_failures.push_back(name + ": expected exception"); + } + catch (const mexce::Protected_expression_error& error) { + if (error.category() != category) { + m_failures.push_back(name + ": wrong category"); + } + } + catch (const std::exception& error) { + m_failures.push_back(name + ": unexpected exception: " + error.what()); + } + } + + template + void expect_logic_error( + const std::string& name, + const char* message, + Function function) + { + try { + function(); + m_failures.push_back(name + ": expected exception"); + } + catch (const std::logic_error& error) { + if (std::string(error.what()) != message) { + m_failures.push_back(name + ": wrong message"); + } + } + catch (const std::exception& error) { + m_failures.push_back(name + ": unexpected exception: " + error.what()); + } + } +}; + + +std::vector read_fixture() +{ + std::ifstream input(g_fixture_path, std::ios::binary); + if (!input) { + throw std::runtime_error("protected fixture could not be opened"); + } + return std::vector( + std::istreambuf_iterator(input), + std::istreambuf_iterator()); +} + + +mexce::Protected_expression_key fixture_key() +{ + std::array bytes; + for (size_t i = 0; i < bytes.size(); ++i) { + bytes[i] = static_cast(i); + } + return mexce::Protected_expression_key::from_bytes(bytes.data(), bytes.size()); +} + + +void load( + mexce::evaluator& evaluator, + mexce::Protected_expression_bundle bundle) +{ + evaluator.set_protected_expression( + bundle.program.data(), bundle.program.size(), std::move(bundle.key)); +} + + +void test_independent_fixture(Test_suite& suite) +{ + double x = 4.0; + double y = 1.0; + mexce::evaluator evaluator; + evaluator.bind_protected(x, 0); + evaluator.bind_protected(y, 1); + const auto program = read_fixture(); + evaluator.set_protected_expression( + program.data(), program.size(), fixture_key()); + + suite.expect_near(evaluator.evaluate(), 11.0, "independent_fixture_value"); + suite.expect( + evaluator.get_backend() != mexce::backend_type::none, + "independent_fixture_backend"); +} + + +void test_operation_parity(Test_suite& suite) +{ + struct operation_case_t + { + const char* expression; + bool uses_y; + }; + + const operation_case_t cases[] = { + {"add(x,y)", true}, {"sub(x,y)", true}, {"mul(x,y)", true}, + {"div(x,y)", true}, {"neg(x)", false}, {"pow(x,y)", true}, + {"sin(x)", false}, {"cos(x)", false}, {"tan(x)", false}, + {"abs(x)", false}, {"sign(x)", false}, {"signp(x)", false}, + {"expn(x)", false}, {"sfc(x)", false}, {"sqrt(x)", false}, + {"exp(x)", false}, {"lt(x,y)", true}, {"gt(x,y)", true}, + {"le(x,y)", true}, {"ge(x,y)", true}, {"eq(x,y)", true}, + {"ne(x,y)", true}, {"log(x)", false}, {"log2(x)", false}, + {"log10(x)", false}, {"logb(x,y)", true}, {"ylog2(x,y)", true}, + {"max(x,y)", true}, {"min(x,y)", true}, {"floor(x)", false}, + {"ceil(x)", false}, {"round(x)", false}, {"int(x)", false}, + {"trunc(x)", false}, {"mod(x,y)", true}, {"bnd(x,y)", true}, + {"bias(x,y)", true}, {"gain(x,y)", true}, + }; + + double x = 0.25; + double y = 0.75; + for (const auto& test_case : cases) { + mexce::evaluator clear; + clear.bind(x, "x", y, "y"); + clear.set_expression(test_case.expression); + + std::vector bindings; + bindings.push_back({"x", 0}); + if (test_case.uses_y) { + bindings.push_back({"y", 1}); + } + + mexce::evaluator protected_evaluator; + protected_evaluator.bind_protected(x, 0); + protected_evaluator.bind_protected(y, 1); + load( + protected_evaluator, + mexce::encode_protected_expression( + test_case.expression, + bindings, + mexce::Protected_math_mode::STRICT)); + suite.expect_near( + protected_evaluator.evaluate(), + clear.evaluate(), + std::string("operation_parity_") + test_case.expression); + } +} + + +void test_binding_lifecycle(Test_suite& suite) +{ + double first = 3.0; + double replacement = 8.0; + double extra = 5.0; + mexce::evaluator evaluator; + evaluator.bind(first, "x"); + evaluator.bind_protected(first, 0); + evaluator.bind_protected(extra, 7); + evaluator.bind_protected(replacement, 7); + load( + evaluator, + mexce::encode_protected_expression( + "x+1", {{"x", 0}}, mexce::Protected_math_mode::STRICT)); + + suite.expect_near(evaluator.evaluate(), 4.0, "binding_initial_value"); + suite.expect_error( + "referenced_rebind_rejected", + mexce::Protected_expression_error_category::INVALID_ARGUMENT, + [&] { evaluator.bind_protected(replacement, 0); }); + suite.expect_near(evaluator.evaluate(), 4.0, "rejected_rebind_unchanged"); + + evaluator.unbind_protected(0); + suite.expect_logic_error( + "referenced_unbind_empties_evaluator", + "No expression has been compiled.", + [&] { (void)evaluator.evaluate(); }); + suite.expect( + evaluator.get_backend() == mexce::backend_type::none, + "referenced_unbind_backend_empty"); + evaluator.set_expression("x+2"); + suite.expect_near(evaluator.evaluate(), 5.0, "clear_binding_map_is_independent"); + suite.expect_error( + "unknown_unbind_rejected", + mexce::Protected_expression_error_category::INVALID_ARGUMENT, + [&] { evaluator.unbind_protected(0); }); + + evaluator.bind_protected(first, 0); + load( + evaluator, + mexce::encode_protected_expression( + "x+1", {{"x", 0}}, mexce::Protected_math_mode::STRICT)); + evaluator.unbind_all_protected(); + suite.expect( + evaluator.get_backend() == mexce::backend_type::none, + "unbind_all_referenced_empties"); +} + + +void test_binding_types_and_missing_binding(Test_suite& suite) +{ + double d = 1.5; + float f = 2.5f; + int16_t i16 = 3; + int32_t i32 = 4; + int64_t i64 = 5; + mexce::evaluator evaluator; + evaluator.bind_protected(d, 0); + evaluator.bind_protected(f, 1); + evaluator.bind_protected(i16, 2); + evaluator.bind_protected(i32, 3); + evaluator.bind_protected(i64, 4); + load( + evaluator, + mexce::encode_protected_expression( + "d+f+i16+i32+i64", + {{"d", 0}, {"f", 1}, {"i16", 2}, {"i32", 3}, {"i64", 4}}, + mexce::Protected_math_mode::STRICT)); + suite.expect_near(evaluator.evaluate(), 16.0, "all_binding_types"); + + mexce::evaluator missing; + missing.bind_protected(f, 1); + const auto program = read_fixture(); + try { + missing.set_protected_expression( + program.data(), program.size(), fixture_key()); + suite.m_failures.push_back("missing_binding: expected exception"); + } + catch (const mexce::Protected_expression_error& error) { + suite.expect( + error.category() == mexce::Protected_expression_error_category::MISSING_BINDING && + error.has_record_index() && error.record_index() == 1, + "missing_binding_category_and_index"); + } + suite.expect( + missing.get_backend() == mexce::backend_type::none, + "missing_binding_leaves_empty"); +} + + +void test_options_and_introspection(Test_suite& suite) +{ + double infinity = std::numeric_limits::infinity(); + + mexce::evaluator strict; + strict.enable_fast_math(); + strict.bind_protected(infinity, 0); + load( + strict, + mexce::encode_protected_expression( + "x-x", {{"x", 0}}, mexce::Protected_math_mode::STRICT)); + suite.expect(std::isnan(strict.evaluate()), "authenticated_strict_precedence"); + suite.expect(strict.get_options().fast_math, "strict_does_not_mutate_stored_options"); + + double finite = 7.0; + mexce::evaluator fast; + fast.bind_protected(finite, 0); + load( + fast, + mexce::encode_protected_expression( + "x-x", {{"x", 0}}, mexce::Protected_math_mode::FAST)); + suite.expect_near(fast.evaluate(), 0.0, "authenticated_fast_precedence"); + suite.expect(!fast.get_options().fast_math, "fast_does_not_mutate_stored_options"); + + suite.expect_error( + "optimized_introspection_disabled", + mexce::Protected_expression_error_category::INTROSPECTION_DISABLED, + [&] { (void)fast.get_optimized_expression(); }); + suite.expect_error( + "byte_introspection_disabled", + mexce::Protected_expression_error_category::INTROSPECTION_DISABLED, + [&] { (void)fast.get_byte_representation(); }); + + fast.bind(finite, "x"); + fast.set_expression("x+1"); + suite.expect(!fast.get_optimized_expression().empty(), "clear_restores_introspection"); + + mexce::evaluator x87; + x87.use_x87_backend(); + x87.bind_protected(finite, 0); + load( + x87, + mexce::encode_protected_expression( + "x+1", {{"x", 0}}, mexce::Protected_math_mode::STRICT)); + suite.expect(x87.get_backend() == mexce::backend_type::x87, "prefer_x87_preserved"); + suite.expect(x87.get_options().prefer_x87, "prefer_x87_option_unchanged"); + suite.expect_near(x87.evaluate(), 8.0, "prefer_x87_result"); +} + + +void test_runtime_libm_selection(Test_suite& suite) +{ + struct libm_case_t + { + const char* expression; + bool mexce::options::* option; + bool uses_y; + bool libm_uses_sse2; + }; + + const libm_case_t cases[] = { + {"sin(x)", &mexce::options::use_libm_sin, false, true}, + {"cos(x)", &mexce::options::use_libm_cos, false, true}, + {"tan(x)", &mexce::options::use_libm_tan, false, true}, + {"exp(x)", &mexce::options::use_libm_exp, false, true}, + {"log(x)", &mexce::options::use_libm_log, false, true}, + {"log10(x)", &mexce::options::use_libm_log10, false, true}, + {"log2(x)", &mexce::options::use_libm_log2, false, true}, + {"logb(x,y)", &mexce::options::use_libm_logb, true, true}, + {"ylog2(x,y)", &mexce::options::use_libm_ylog2, true, false}, + {"pow(x,y)", &mexce::options::use_libm_generic_pow, true, false}, + }; + + double x = 1.25; + double y = 0.75; + for (const auto& test_case : cases) { + mexce::options libm_options; + libm_options.set_use_libm(true); + mexce::evaluator libm; + libm.set_options(libm_options); + libm.bind(x, "x", y, "y"); + libm.set_expression(test_case.expression); + const double expected = libm.evaluate(); + const std::string libm_code = libm.get_byte_representation(); + suite.expect(std::isfinite(expected), + std::string("libm_finite_") + test_case.expression); +#ifdef MEXCE_64 + suite.expect( + libm.get_backend() == (test_case.libm_uses_sse2 + ? mexce::backend_type::sse2 + : mexce::backend_type::x87), + std::string("libm_backend_") + test_case.expression); +#endif + + mexce::options inline_options = libm_options; + inline_options.*(test_case.option) = false; + mexce::evaluator clear_inline; + clear_inline.set_options(inline_options); + clear_inline.bind(x, "x", y, "y"); + clear_inline.set_expression(test_case.expression); + suite.expect(clear_inline.get_backend() == mexce::backend_type::x87, + std::string("inline_backend_") + test_case.expression); + suite.expect_near(clear_inline.evaluate(), expected, + std::string("inline_result_") + test_case.expression); + suite.expect(clear_inline.get_byte_representation() != libm_code, + std::string("runtime_selection_") + test_case.expression); + + std::vector bindings = {{"x", 0}}; + if (test_case.uses_y) { + bindings.push_back({"y", 1}); + } + mexce::evaluator protected_libm; + protected_libm.set_options(libm_options); + protected_libm.bind_protected(x, 0); + protected_libm.bind_protected(y, 1); + load( + protected_libm, + mexce::encode_protected_expression( + test_case.expression, + bindings, + mexce::Protected_math_mode::STRICT)); + suite.expect_near(protected_libm.evaluate(), expected, + std::string("protected_libm_result_") + test_case.expression); +#ifdef MEXCE_64 + suite.expect( + protected_libm.get_backend() == (test_case.libm_uses_sse2 + ? mexce::backend_type::sse2 + : mexce::backend_type::x87), + std::string("protected_libm_backend_") + test_case.expression); +#endif + suite.expect( + protected_libm.get_options().*(test_case.option), + std::string("protected_libm_option_") + test_case.expression); + + mexce::evaluator protected_inline; + protected_inline.set_options(inline_options); + protected_inline.bind_protected(x, 0); + protected_inline.bind_protected(y, 1); + load( + protected_inline, + mexce::encode_protected_expression( + test_case.expression, + bindings, + mexce::Protected_math_mode::STRICT)); + suite.expect(protected_inline.get_backend() == mexce::backend_type::x87, + std::string("protected_inline_backend_") + test_case.expression); + suite.expect_near(protected_inline.evaluate(), expected, + std::string("protected_inline_result_") + test_case.expression); + suite.expect( + !(protected_inline.get_options().*(test_case.option)), + std::string("protected_inline_option_") + test_case.expression); + } + + double negative_zero = -0.0; + for (const bool use_libm : {false, true}) { + mexce::options selected; + selected.set_use_libm(use_libm); + mexce::evaluator clear; + clear.set_options(selected); + clear.bind(negative_zero, "x"); + clear.set_expression("sin(x)"); + suite.expect(clear.evaluate() == 0.0 && std::signbit(clear.evaluate()), + use_libm ? "libm_signed_zero" : "inline_signed_zero"); + + mexce::evaluator protected_evaluator; + protected_evaluator.set_options(selected); + protected_evaluator.bind_protected(negative_zero, 0); + load( + protected_evaluator, + mexce::encode_protected_expression( + "sin(x)", {{"x", 0}}, mexce::Protected_math_mode::STRICT)); + const double protected_result = protected_evaluator.evaluate(); + suite.expect(protected_result == 0.0 && std::signbit(protected_result), + use_libm ? "protected_libm_signed_zero" : "protected_inline_signed_zero"); + } +} + + +void test_failure_and_key_consumption(Test_suite& suite) +{ + double value = 2.0; + mexce::evaluator evaluator; + evaluator.bind_protected(value, 0); + evaluator.set_expression("19.25"); + + auto bundle = mexce::encode_protected_expression( + "x+1", {{"x", 0}}, mexce::Protected_math_mode::STRICT); + bundle.program.back() ^= 0x01; + suite.expect_error( + "late_authentication_failure", + mexce::Protected_expression_error_category::AUTHENTICATION_FAILED, + [&] { + evaluator.set_protected_expression( + bundle.program.data(), bundle.program.size(), std::move(bundle.key)); + }); + suite.expect(evaluator.get_backend() == mexce::backend_type::none, "failure_backend_empty"); + try { + (void)evaluator.evaluate(); + suite.m_failures.push_back("failure_evaluate_empty: expected exception"); + } + catch (const std::logic_error& error) { + suite.expect( + std::string(error.what()) == "No expression has been compiled.", + "failure_evaluate_empty_message"); + } + suite.expect_error( + "transferred_key_is_empty", + mexce::Protected_expression_error_category::INVALID_ARGUMENT, + [&] { bundle.key.consume_bytes([](const uint8_t*, size_t) {}); }); + + evaluator.bind(value, "value"); + evaluator.set_expression("value+1"); + auto native_failure = mexce::encode_protected_expression( + "x+1", {{"x", 0}}, mexce::Protected_math_mode::STRICT); + native_failure.program.back() ^= 0x01; + suite.expect_error( + "late_failure_replaces_native_state", + mexce::Protected_expression_error_category::AUTHENTICATION_FAILED, + [&] { + evaluator.set_protected_expression( + native_failure.program.data(), + native_failure.program.size(), + std::move(native_failure.key)); + }); + suite.expect( + evaluator.get_backend() == mexce::backend_type::none, + "late_failure_clears_native_backend"); + + evaluator.enable_cse(); + auto cse_bundle = mexce::encode_protected_expression( + "x+1", {{"x", 0}}, mexce::Protected_math_mode::STRICT); + suite.expect_error( + "protected_cse_rejected", + mexce::Protected_expression_error_category::INVALID_ARGUMENT, + [&] { + evaluator.set_protected_expression( + cse_bundle.program.data(), + cse_bundle.program.size(), + std::move(cse_bundle.key)); + }); + suite.expect(evaluator.get_options().enable_cse, "cse_rejection_preserves_option"); + suite.expect_error( + "cse_rejection_consumes_key", + mexce::Protected_expression_error_category::INVALID_ARGUMENT, + [&] { cse_bundle.key.consume_bytes([](const uint8_t*, size_t) {}); }); +} + + +void test_compilation_resource_failures(Test_suite& suite) +{ + double value = 2.0; + mexce::evaluator evaluator; + evaluator.bind_protected(value, 0); + auto& faults = mexce::impl::protected_test_faults(); + + faults.fail_executable_allocation = true; + auto allocation_bundle = mexce::encode_protected_expression( + "x+1", {{"x", 0}}, mexce::Protected_math_mode::STRICT); + suite.expect_error( + "executable_allocation_failure", + mexce::Protected_expression_error_category::RESOURCE_FAILURE, + [&] { + evaluator.set_protected_expression( + allocation_bundle.program.data(), + allocation_bundle.program.size(), + std::move(allocation_bundle.key)); + }); + faults.fail_executable_allocation = false; + suite.expect( + evaluator.get_backend() == mexce::backend_type::none, + "allocation_failure_leaves_empty"); + + faults.fail_executable_finalization = true; + auto finalization_bundle = mexce::encode_protected_expression( + "x+1", {{"x", 0}}, mexce::Protected_math_mode::STRICT); + suite.expect_error( + "executable_finalization_failure", + mexce::Protected_expression_error_category::RESOURCE_FAILURE, + [&] { + evaluator.set_protected_expression( + finalization_bundle.program.data(), + finalization_bundle.program.size(), + std::move(finalization_bundle.key)); + }); + faults.fail_executable_finalization = false; + suite.expect( + evaluator.get_backend() == mexce::backend_type::none, + "finalization_failure_leaves_empty"); +} + + +void test_fallback_and_wipes(Test_suite& suite) +{ + auto& observer = mexce::impl::protected_wipe_observer(); + double value = 1.25; + const std::string polynomial = + "((((((((((2.20*x+1.1)*x+9.9)*x+8.8)*x+7.7)*x+6.6)*x+5.5)*x+4.4)*x+3.3)*x+2.2)*x+1.1)"; + + mexce::evaluator clear; + clear.bind(value, "x"); + clear.set_expression(polynomial); + const double expected = clear.evaluate(); + + mexce::evaluator evaluator; + evaluator.bind_protected(value, 0); + load( + evaluator, + mexce::encode_protected_expression( + polynomial, {{"x", 0}}, mexce::Protected_math_mode::STRICT)); + suite.expect(evaluator.get_backend() == mexce::backend_type::x87, "protected_fallback_backend"); + const double result = evaluator.evaluate(); + suite.expect(std::isfinite(result) && result == expected, "protected_fallback_exact_value"); + suite.expect(!evaluator.get_options().prefer_x87, "fallback_preserves_stored_preference"); + + observer.reset(); + evaluator.set_expression("3.5"); + suite.expect_near(evaluator.evaluate(), 3.5, "protected_replacement_value"); + suite.expect( + observer.observed_before( + mexce::impl::Protected_wipe_context::EXECUTABLE, + mexce::impl::Protected_wipe_context::FALLBACK_STORAGE), + "fallback_executable_wiped_before_owner"); + suite.expect( + observer.count(mexce::impl::Protected_wipe_context::CONSTANT_STORAGE) == 0, + "fallback_wipe_not_satisfied_by_primary_storage"); + suite.expect(observer.all_observed_regions_are_zero(), "fallback_replacement_wipes_zero"); + + observer.reset(); + mexce::evaluator primary; + primary.bind_protected(value, 0); + load( + primary, + mexce::encode_protected_expression( + "x+1", {{"x", 0}}, mexce::Protected_math_mode::STRICT)); + suite.expect( + observer.count(mexce::impl::Protected_wipe_context::FALLBACK_STORAGE) > 0, + "unused_fallback_owner_wiped_during_load"); + suite.expect( + observer.count(mexce::impl::Protected_wipe_context::EXECUTABLE) == 0, + "unused_fallback_not_satisfied_by_executable_wipe"); + + observer.reset(); + primary.set_expression("4.5"); + suite.expect( + observer.observed_before( + mexce::impl::Protected_wipe_context::EXECUTABLE, + mexce::impl::Protected_wipe_context::CONSTANT_STORAGE), + "primary_executable_wiped_before_constants"); + suite.expect( + observer.observed_before( + mexce::impl::Protected_wipe_context::EXECUTABLE, + mexce::impl::Protected_wipe_context::OPTIMIZER_STATE), + "primary_executable_wiped_before_optimizer"); + suite.expect( + observer.count(mexce::impl::Protected_wipe_context::OPTIONS_STORAGE) > 0, + "primary_options_wiped_independently"); + suite.expect(observer.all_observed_regions_are_zero(), "primary_replacement_wipes_zero"); + + observer.reset(); + { + mexce::evaluator constant; + constant.bind_protected(value, 9); + load( + constant, + mexce::encode_protected_expression( + "3.5", {}, mexce::Protected_math_mode::STRICT)); + constant.unbind_all_protected(); + suite.expect_near(constant.evaluate(), 3.5, "protected_constant_value"); + observer.reset(); + } + suite.expect( + observer.count(mexce::impl::Protected_wipe_context::CONSTANT_STORAGE) > 0, + "constant_destruction_wipe_observed"); + suite.expect( + observer.count(mexce::impl::Protected_wipe_context::EXECUTABLE) == 0, + "constant_destruction_has_no_executable_wipe"); + suite.expect(observer.all_observed_regions_are_zero(), "constant_destruction_wipes_zero"); + + observer.reset(); + { + mexce::evaluator destruction; + destruction.bind_protected(value, 0); + load( + destruction, + mexce::encode_protected_expression( + "x+2", {{"x", 0}}, mexce::Protected_math_mode::STRICT)); + observer.reset(); + } + suite.expect( + observer.observed_before( + mexce::impl::Protected_wipe_context::EXECUTABLE, + mexce::impl::Protected_wipe_context::CONSTANT_STORAGE), + "destruction_executable_wiped_before_constants"); + suite.expect(observer.all_observed_regions_are_zero(), "destruction_wipes_zero"); +} + + +} // namespace + + +int main(int argc, char** argv) +{ + if (argc > 2) { + std::cerr << "usage: protected_runtime_tests [fixture]\n"; + return 1; + } + if (argc == 2) { + g_fixture_path = argv[1]; + } + if (sodium_init() < 0) { + std::cerr << "protected runtime tests require libsodium 1.0.22\n"; + return 1; + } + + Test_suite suite; + test_independent_fixture(suite); + test_operation_parity(suite); + test_binding_lifecycle(suite); + test_binding_types_and_missing_binding(suite); + test_options_and_introspection(suite); + test_runtime_libm_selection(suite); + test_failure_and_key_consumption(suite); + test_compilation_resource_failures(suite); + test_fallback_and_wipes(suite); + + for (const auto& failure : suite.m_failures) { + std::cerr << "FAIL: " << failure << '\n'; + } + if (!suite.m_failures.empty()) { + std::cerr << suite.m_failures.size() << " protected runtime test(s) failed\n"; + return 1; + } + + std::cout << "All protected runtime tests passed\n"; + return 0; +} From 463d562bb6d7c6c1f2025aa2bf5b154d44eff751 Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 16:39:25 +0200 Subject: [PATCH 06/12] Add protected package integration --- CMakeLists.txt | 364 +++++++++++++----------- cmake/mexce-config.cmake.in | 9 + cmake/mexce-sodium.cmake | 35 +++ conanfile.py | 49 +++- ports/mexce/portfile.cmake | 22 +- ports/mexce/vcpkg.json | 24 +- test/install_consumer/CMakeLists.txt | 51 ++++ test/install_consumer/ordinary.cpp | 13 + test/install_consumer/protected.cpp | 25 ++ test_package/CMakeLists.txt | 9 +- test_package/conanfile.py | 9 +- test_package/protected_test_package.cpp | 25 ++ test_package/test_package.cpp | 5 +- 13 files changed, 446 insertions(+), 194 deletions(-) create mode 100644 cmake/mexce-config.cmake.in create mode 100644 cmake/mexce-sodium.cmake create mode 100644 test/install_consumer/CMakeLists.txt create mode 100644 test/install_consumer/ordinary.cpp create mode 100644 test/install_consumer/protected.cpp create mode 100644 test_package/protected_test_package.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index a3782bb..76b498d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,35 +1,17 @@ cmake_minimum_required(VERSION 3.16) -project(mexce_benchmark LANGUAGES CXX) +project(mexce VERSION 1.0.1 LANGUAGES CXX) + +include(CTest) +include(CMakePackageConfigHelpers) +include(GNUInstallDirs) option(ENABLE_COVERAGE "Build with coverage instrumentation" OFF) option(MEXCE_ENABLE_PROTECTED_EXPRESSIONS "Build protected-expression targets" OFF) -if(ENABLE_COVERAGE) - if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang") - add_compile_options(-O0 -g) - - if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU") - add_compile_options(--coverage) - add_link_options(--coverage) - else() - add_compile_options(-fprofile-instr-generate -fcoverage-mapping) - add_link_options(-fprofile-instr-generate) - endif() - endif() -endif() - set(CMAKE_CXX_STANDARD 11) set(CMAKE_CXX_STANDARD_REQUIRED ON) -add_executable(benchmark test/benchmark.cpp) -target_include_directories(benchmark PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) - -find_package(OpenMP REQUIRED) -target_link_libraries(benchmark PRIVATE OpenMP::OpenMP_CXX) - -# Define optimization flags explicitly - these are what matter for performance -# This ensures flags are unambiguous and consistently reported -if (MSVC) +if(MSVC) set(MEXCE_OPTIMIZATION_FLAGS /O2) set(MEXCE_WARNING_FLAGS /W4 /permissive- /bigobj) else() @@ -37,170 +19,208 @@ else() set(MEXCE_WARNING_FLAGS -Wall -Wextra -Wpedantic) endif() -# Apply optimization flags only when coverage is disabled -if(NOT ENABLE_COVERAGE) - target_compile_options(benchmark PRIVATE ${MEXCE_OPTIMIZATION_FLAGS}) -endif() -target_compile_options(benchmark PRIVATE ${MEXCE_WARNING_FLAGS}) - -add_executable(unit_tests test/unit_tests.cpp) -target_include_directories(unit_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) -target_compile_options(unit_tests PRIVATE ${MEXCE_WARNING_FLAGS}) - add_library(mexce INTERFACE) add_library(mexce::mexce ALIAS mexce) -target_include_directories(mexce INTERFACE ${CMAKE_CURRENT_SOURCE_DIR}) - -add_executable(ordinary_no_sodium_test test/ordinary_no_sodium_test.cpp) -target_link_libraries(ordinary_no_sodium_test PRIVATE mexce) -target_compile_options(ordinary_no_sodium_test PRIVATE ${MEXCE_WARNING_FLAGS}) +set_target_properties(mexce PROPERTIES EXPORT_NAME mexce) +target_include_directories(mexce INTERFACE + $ + $) if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) - set(MEXCE_SODIUM_TARGET "") - find_package(libsodium 1.0.22 EXACT CONFIG QUIET) - if(TARGET libsodium::libsodium) - set(MEXCE_SODIUM_TARGET libsodium::libsodium) - else() - find_package(unofficial-sodium 1.0.22 EXACT CONFIG QUIET) - endif() - if(TARGET unofficial-sodium::sodium) - set(MEXCE_SODIUM_TARGET unofficial-sodium::sodium) - elseif(NOT MEXCE_SODIUM_TARGET) - find_package(PkgConfig QUIET) - if(PkgConfig_FOUND) - pkg_check_modules(MEXCE_SODIUM QUIET IMPORTED_TARGET libsodium=1.0.22) - endif() - if(TARGET PkgConfig::MEXCE_SODIUM) - set(MEXCE_SODIUM_TARGET PkgConfig::MEXCE_SODIUM) - else() - message(FATAL_ERROR "MEXCE protected expressions require libsodium 1.0.22") - endif() + if(NOT CMAKE_SIZEOF_VOID_P EQUAL 8) + message(FATAL_ERROR "MEXCE protected expressions support x64 targets only") endif() + include(cmake/mexce-sodium.cmake) + mexce_find_sodium(mexce::_sodium) + add_library(mexce_protected INTERFACE) add_library(mexce::protected ALIAS mexce_protected) - target_link_libraries(mexce_protected INTERFACE mexce ${MEXCE_SODIUM_TARGET}) - - set(MEXCE_PROTECTED_FIXTURE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/test/fixtures") - add_executable(protected_codec_tests test/protected_codec_tests.cpp) - target_link_libraries(protected_codec_tests PRIVATE mexce_protected) - target_compile_definitions(protected_codec_tests PRIVATE - MEXCE_PROTECTED_TESTING - MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin" - MEXCE_PROTECTED_FIXTURE_MANIFEST_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.manifest.txt") - target_compile_options(protected_codec_tests PRIVATE ${MEXCE_WARNING_FLAGS}) - - add_executable(protected_fixture_generator EXCLUDE_FROM_ALL - test/protected_fixture_generator.cpp) - target_link_libraries(protected_fixture_generator PRIVATE ${MEXCE_SODIUM_TARGET}) - target_compile_options(protected_fixture_generator PRIVATE ${MEXCE_WARNING_FLAGS}) - - add_executable(protected_sodium_init_failure_test - test/protected_sodium_init_failure_test.cpp) - target_link_libraries(protected_sodium_init_failure_test PRIVATE mexce_protected) - target_compile_definitions(protected_sodium_init_failure_test PRIVATE - MEXCE_PROTECTED_TESTING) - target_compile_options(protected_sodium_init_failure_test PRIVATE ${MEXCE_WARNING_FLAGS}) - - add_executable(protected_runtime_tests test/protected_runtime_tests.cpp) - target_link_libraries(protected_runtime_tests PRIVATE mexce_protected) - target_compile_definitions(protected_runtime_tests PRIVATE - MEXCE_PROTECTED_TESTING - MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin") - target_compile_options(protected_runtime_tests PRIVATE ${MEXCE_WARNING_FLAGS}) - - add_executable(mixed_tu_test - test/mixed_tu_test.cpp - test/mixed_tu_ordinary.cpp - test/mixed_tu_protected.cpp) - target_link_libraries(mixed_tu_test PRIVATE mexce_protected) - target_compile_definitions(mixed_tu_test PRIVATE - MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin") - target_compile_options(mixed_tu_test PRIVATE ${MEXCE_WARNING_FLAGS}) - - add_executable(protected_benchmark test/protected_benchmark.cpp) - target_link_libraries(protected_benchmark PRIVATE mexce_protected) - if(WIN32) - target_link_libraries(protected_benchmark PRIVATE Psapi) + set_target_properties(mexce_protected PROPERTIES EXPORT_NAME protected) + target_link_libraries(mexce_protected INTERFACE mexce mexce::_sodium) +endif() + +if(BUILD_TESTING) + if(ENABLE_COVERAGE) + if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang") + add_compile_options(-O0 -g) + + if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU") + add_compile_options(--coverage) + add_link_options(--coverage) + else() + add_compile_options(-fprofile-instr-generate -fcoverage-mapping) + add_link_options(-fprofile-instr-generate) + endif() + endif() endif() + + add_executable(benchmark test/benchmark.cpp) + target_include_directories(benchmark PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) + find_package(OpenMP REQUIRED) + target_link_libraries(benchmark PRIVATE OpenMP::OpenMP_CXX) if(NOT ENABLE_COVERAGE) - target_compile_options(protected_benchmark PRIVATE ${MEXCE_OPTIMIZATION_FLAGS}) + target_compile_options(benchmark PRIVATE ${MEXCE_OPTIMIZATION_FLAGS}) endif() - target_compile_options(protected_benchmark PRIVATE ${MEXCE_WARNING_FLAGS}) - - add_executable(protected_decoder_fuzz_smoke test/protected_decoder_fuzz.cpp) - target_link_libraries(protected_decoder_fuzz_smoke PRIVATE mexce_protected) - target_compile_definitions(protected_decoder_fuzz_smoke PRIVATE - MEXCE_FUZZ_SMOKE_MAIN - MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin") - target_compile_options(protected_decoder_fuzz_smoke PRIVATE ${MEXCE_WARNING_FLAGS}) - - if(CMAKE_CXX_COMPILER_ID MATCHES "Clang" AND UNIX) - add_executable(protected_decoder_fuzz EXCLUDE_FROM_ALL - test/protected_decoder_fuzz.cpp) - target_link_libraries(protected_decoder_fuzz PRIVATE mexce_protected) - target_compile_options(protected_decoder_fuzz PRIVATE - -fsanitize=fuzzer,address,undefined - -fno-omit-frame-pointer) - target_link_options(protected_decoder_fuzz PRIVATE - -fsanitize=fuzzer,address,undefined) + target_compile_options(benchmark PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(unit_tests test/unit_tests.cpp) + target_include_directories(unit_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) + target_compile_options(unit_tests PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(ordinary_no_sodium_test test/ordinary_no_sodium_test.cpp) + target_link_libraries(ordinary_no_sodium_test PRIVATE mexce) + target_compile_options(ordinary_no_sodium_test PRIVATE ${MEXCE_WARNING_FLAGS}) + + if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) + set(MEXCE_PROTECTED_FIXTURE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/test/fixtures") + + add_executable(protected_codec_tests test/protected_codec_tests.cpp) + target_link_libraries(protected_codec_tests PRIVATE mexce_protected) + target_compile_definitions(protected_codec_tests PRIVATE + MEXCE_PROTECTED_TESTING + MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin" + MEXCE_PROTECTED_FIXTURE_MANIFEST_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.manifest.txt") + target_compile_options(protected_codec_tests PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(protected_fixture_generator EXCLUDE_FROM_ALL + test/protected_fixture_generator.cpp) + target_link_libraries(protected_fixture_generator PRIVATE mexce::_sodium) + target_compile_options(protected_fixture_generator PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(protected_sodium_init_failure_test + test/protected_sodium_init_failure_test.cpp) + target_link_libraries(protected_sodium_init_failure_test PRIVATE mexce_protected) + target_compile_definitions(protected_sodium_init_failure_test PRIVATE + MEXCE_PROTECTED_TESTING) + target_compile_options(protected_sodium_init_failure_test PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(protected_runtime_tests test/protected_runtime_tests.cpp) + target_link_libraries(protected_runtime_tests PRIVATE mexce_protected) + target_compile_definitions(protected_runtime_tests PRIVATE + MEXCE_PROTECTED_TESTING + MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin") + target_compile_options(protected_runtime_tests PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(mixed_tu_test + test/mixed_tu_test.cpp + test/mixed_tu_ordinary.cpp + test/mixed_tu_protected.cpp) + target_link_libraries(mixed_tu_test PRIVATE mexce_protected) + target_compile_definitions(mixed_tu_test PRIVATE + MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin") + target_compile_options(mixed_tu_test PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(protected_benchmark test/protected_benchmark.cpp) + target_link_libraries(protected_benchmark PRIVATE mexce_protected) + if(WIN32) + target_link_libraries(protected_benchmark PRIVATE Psapi) + endif() + if(NOT ENABLE_COVERAGE) + target_compile_options(protected_benchmark PRIVATE ${MEXCE_OPTIMIZATION_FLAGS}) + endif() + target_compile_options(protected_benchmark PRIVATE ${MEXCE_WARNING_FLAGS}) + + add_executable(protected_decoder_fuzz_smoke test/protected_decoder_fuzz.cpp) + target_link_libraries(protected_decoder_fuzz_smoke PRIVATE mexce_protected) + target_compile_definitions(protected_decoder_fuzz_smoke PRIVATE + MEXCE_FUZZ_SMOKE_MAIN + MEXCE_PROTECTED_FIXTURE_PATH="${MEXCE_PROTECTED_FIXTURE_DIR}/protected_format_1_0.bin") + target_compile_options(protected_decoder_fuzz_smoke PRIVATE ${MEXCE_WARNING_FLAGS}) + + if(CMAKE_CXX_COMPILER_ID MATCHES "Clang" AND UNIX) + add_executable(protected_decoder_fuzz EXCLUDE_FROM_ALL + test/protected_decoder_fuzz.cpp) + target_link_libraries(protected_decoder_fuzz PRIVATE mexce_protected) + target_compile_options(protected_decoder_fuzz PRIVATE + -fsanitize=fuzzer,address,undefined + -fno-omit-frame-pointer) + target_link_options(protected_decoder_fuzz PRIVATE + -fsanitize=fuzzer,address,undefined) + endif() endif() -endif() -# Capture compiler info for benchmark output -set(MEXCE_BENCHMARK_COMPILER "${CMAKE_CXX_COMPILER_ID}") -if (CMAKE_CXX_COMPILER_VERSION) - string(APPEND MEXCE_BENCHMARK_COMPILER " ${CMAKE_CXX_COMPILER_VERSION}") -endif() + set(MEXCE_BENCHMARK_COMPILER "${CMAKE_CXX_COMPILER_ID}") + if(CMAKE_CXX_COMPILER_VERSION) + string(APPEND MEXCE_BENCHMARK_COMPILER " ${CMAKE_CXX_COMPILER_VERSION}") + endif() -# Build the flags string from our explicitly defined flags -# This ensures what's reported matches exactly what's applied -if(ENABLE_COVERAGE) - # Coverage mode uses -O0 -g instead of optimization flags - set(MEXCE_BENCHMARK_FLAGS "-O0 -g") -else() - list(JOIN MEXCE_OPTIMIZATION_FLAGS " " MEXCE_BENCHMARK_FLAGS) + if(ENABLE_COVERAGE) + set(MEXCE_BENCHMARK_FLAGS "-O0 -g") + else() + list(JOIN MEXCE_OPTIMIZATION_FLAGS " " MEXCE_BENCHMARK_FLAGS) + endif() + list(JOIN MEXCE_WARNING_FLAGS " " MEXCE_WARNING_FLAGS_STR) + string(APPEND MEXCE_BENCHMARK_FLAGS " ${MEXCE_WARNING_FLAGS_STR}") + + foreach(variable MEXCE_BENCHMARK_COMPILER MEXCE_BENCHMARK_FLAGS) + string(REPLACE "\\" "\\\\" ${variable} "${${variable}}") + string(REPLACE "\"" "\\\"" ${variable} "${${variable}}") + endforeach() + + target_compile_definitions(benchmark PRIVATE + BENCHMARK_COMPILER="${MEXCE_BENCHMARK_COMPILER}" + BENCHMARK_COMPILER_FLAGS="${MEXCE_BENCHMARK_FLAGS}") + + add_test(NAME mexce_unit_tests COMMAND unit_tests) + add_test(NAME mexce_ordinary_no_sodium COMMAND ordinary_no_sodium_test) + if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) + add_test(NAME mexce_protected_codec_tests COMMAND protected_codec_tests) + add_test(NAME mexce_protected_sodium_init_failure + COMMAND protected_sodium_init_failure_test) + add_test(NAME mexce_protected_runtime COMMAND protected_runtime_tests) + add_test(NAME mexce_mixed_translation_units COMMAND mixed_tu_test) + add_test(NAME mexce_protected_decoder_fuzz_smoke + COMMAND protected_decoder_fuzz_smoke) + add_test(NAME mexce_protected_benchmark COMMAND protected_benchmark) + add_test(NAME mexce_protected_maximum_valid + COMMAND protected_benchmark maximum-valid) + add_test(NAME mexce_protected_late_invalid + COMMAND protected_benchmark late-invalid) + endif() + add_test(NAME benchmark_quick + COMMAND benchmark 5 ${CMAKE_BINARY_DIR}/benchmark_quick_results.txt) + add_test(NAME benchmark_output_first + COMMAND benchmark ${CMAKE_BINARY_DIR}/benchmark_output_first.txt 5) + + add_custom_target(run_benchmarks + COMMAND ${CMAKE_COMMAND} -E echo "Running benchmark with 100 iterations" + COMMAND $ 100 ${CMAKE_BINARY_DIR}/benchmark_results.txt + COMMENT "Executing mexce benchmarks" + USES_TERMINAL) endif() -# Append warning flags -list(JOIN MEXCE_WARNING_FLAGS " " MEXCE_WARNING_FLAGS_STR) -string(APPEND MEXCE_BENCHMARK_FLAGS " ${MEXCE_WARNING_FLAGS_STR}") - -# Escape special characters for C++ string literals -foreach(_var MEXCE_BENCHMARK_COMPILER MEXCE_BENCHMARK_FLAGS) - string(REPLACE "\\" "\\\\" ${_var} "${${_var}}") - string(REPLACE "\"" "\\\"" ${_var} "${${_var}}") -endforeach() - -target_compile_definitions(benchmark PRIVATE - BENCHMARK_COMPILER="${MEXCE_BENCHMARK_COMPILER}" - BENCHMARK_COMPILER_FLAGS="${MEXCE_BENCHMARK_FLAGS}" -) +set(mexce_install_targets mexce) +if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) + list(APPEND mexce_install_targets mexce_protected) +endif() +install(TARGETS ${mexce_install_targets} EXPORT mexce_targets) +install(EXPORT mexce_targets + FILE mexce-targets.cmake + NAMESPACE mexce:: + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mexce) -enable_testing() -add_test(NAME mexce_unit_tests COMMAND unit_tests) -add_test(NAME mexce_ordinary_no_sodium COMMAND ordinary_no_sodium_test) +install(FILES mexce.h DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) - add_test(NAME mexce_protected_codec_tests COMMAND protected_codec_tests) - add_test(NAME mexce_protected_sodium_init_failure - COMMAND protected_sodium_init_failure_test) - add_test(NAME mexce_protected_runtime COMMAND protected_runtime_tests) - add_test(NAME mexce_mixed_translation_units COMMAND mixed_tu_test) - add_test(NAME mexce_protected_decoder_fuzz_smoke - COMMAND protected_decoder_fuzz_smoke) - add_test(NAME mexce_protected_benchmark COMMAND protected_benchmark) - add_test(NAME mexce_protected_maximum_valid - COMMAND protected_benchmark maximum-valid) - add_test(NAME mexce_protected_late_invalid - COMMAND protected_benchmark late-invalid) + install(FILES + mexce_protected.h + mexce_protected_encoder.h + DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) + install(FILES cmake/mexce-sodium.cmake + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mexce) endif() -add_test(NAME benchmark_quick - COMMAND benchmark 5 ${CMAKE_BINARY_DIR}/benchmark_quick_results.txt) -add_test(NAME benchmark_output_first - COMMAND benchmark ${CMAKE_BINARY_DIR}/benchmark_output_first.txt 5) - -add_custom_target(run_benchmarks - COMMAND ${CMAKE_COMMAND} -E echo "Running benchmark with 100 iterations" - COMMAND $ 100 ${CMAKE_BINARY_DIR}/benchmark_results.txt - COMMENT "Executing mexce benchmarks" - USES_TERMINAL) +install(FILES LICENSE.txt DESTINATION ${CMAKE_INSTALL_DATADIR}/licenses/mexce) + +set(MEXCE_CONFIG_HAS_PROTECTED ${MEXCE_ENABLE_PROTECTED_EXPRESSIONS}) +configure_package_config_file( + cmake/mexce-config.cmake.in + ${CMAKE_CURRENT_BINARY_DIR}/mexce-config.cmake + INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mexce) +write_basic_package_version_file( + ${CMAKE_CURRENT_BINARY_DIR}/mexce-config-version.cmake + VERSION ${PROJECT_VERSION} + COMPATIBILITY SameMajorVersion) +install(FILES + ${CMAKE_CURRENT_BINARY_DIR}/mexce-config.cmake + ${CMAKE_CURRENT_BINARY_DIR}/mexce-config-version.cmake + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mexce) diff --git a/cmake/mexce-config.cmake.in b/cmake/mexce-config.cmake.in new file mode 100644 index 0000000..8252764 --- /dev/null +++ b/cmake/mexce-config.cmake.in @@ -0,0 +1,9 @@ +@PACKAGE_INIT@ + +if(@MEXCE_CONFIG_HAS_PROTECTED@) + include("${CMAKE_CURRENT_LIST_DIR}/mexce-sodium.cmake") + mexce_find_sodium(mexce::_sodium) +endif() + +include("${CMAKE_CURRENT_LIST_DIR}/mexce-targets.cmake") +check_required_components(mexce) diff --git a/cmake/mexce-sodium.cmake b/cmake/mexce-sodium.cmake new file mode 100644 index 0000000..a0124e0 --- /dev/null +++ b/cmake/mexce-sodium.cmake @@ -0,0 +1,35 @@ +function(mexce_find_sodium adapter_target) + set(provider_target "") + + if(TARGET "${adapter_target}") + return() + endif() + + find_package(libsodium 1.0.22 EXACT CONFIG QUIET) + if(TARGET libsodium::libsodium) + set(provider_target libsodium::libsodium) + else() + find_package(unofficial-sodium 1.0.22 EXACT CONFIG QUIET) + if(TARGET unofficial-sodium::sodium) + set(provider_target unofficial-sodium::sodium) + else() + find_package(PkgConfig QUIET) + if(PkgConfig_FOUND) + pkg_check_modules(MEXCE_SODIUM QUIET IMPORTED_TARGET libsodium=1.0.22) + endif() + if(TARGET PkgConfig::MEXCE_SODIUM) + set(provider_target PkgConfig::MEXCE_SODIUM) + endif() + endif() + endif() + + if(NOT provider_target) + message(FATAL_ERROR + "MEXCE protected expressions require libsodium 1.0.22 via " + "libsodium::libsodium, unofficial-sodium::sodium, or pkg-config") + endif() + + add_library("${adapter_target}" INTERFACE IMPORTED) + set_property(TARGET "${adapter_target}" PROPERTY + INTERFACE_LINK_LIBRARIES "${provider_target}") +endfunction() diff --git a/conanfile.py b/conanfile.py index 196805f..c559f59 100644 --- a/conanfile.py +++ b/conanfile.py @@ -1,8 +1,7 @@ from conan import ConanFile from conan.errors import ConanInvalidConfiguration -from conan.tools.files import copy, get +from conan.tools.cmake import CMake, CMakeDeps, CMakeToolchain from conan.tools.layout import basic_layout -import os class MexceConan(ConanFile): @@ -11,12 +10,26 @@ class MexceConan(ConanFile): license = "BSD-2-Clause" homepage = "https://github.com/imakris/mexce" url = "https://github.com/conan-io/conan-center-index" - description = "Single-header, dependency-free JIT compiler for scalar mathematical expressions." + description = "Header-only JIT compiler for scalar mathematical expressions." topics = ("jit", "math", "expression-parser", "header-only") package_type = "header-library" settings = "os", "arch", "compiler", "build_type" + options = {"with_protected": [True, False]} + default_options = {"with_protected": False} + exports_sources = ( + "CMakeLists.txt", + "LICENSE.txt", + "cmake/*", + "mexce.h", + "mexce_protected.h", + "mexce_protected_encoder.h", + ) no_copy_source = True + def requirements(self): + if self.options.with_protected: + self.requires("libsodium/1.0.22") + def layout(self): basic_layout(self) @@ -25,23 +38,29 @@ def validate(self): if str(self.settings.arch) not in allowed_architectures: raise ConanInvalidConfiguration( "mexce only supports x86 and x86_64 architectures") + if self.options.with_protected and str(self.settings.arch) != "x86_64": + raise ConanInvalidConfiguration( + "mexce protected expressions support x86_64 packages only") - def package_id(self): - self.info.clear() + def generate(self): + toolchain = CMakeToolchain(self) + toolchain.variables["BUILD_TESTING"] = False + toolchain.variables["MEXCE_ENABLE_PROTECTED_EXPRESSIONS"] = bool( + self.options.with_protected) + toolchain.generate() + CMakeDeps(self).generate() - def source(self): - get(self, - url=f"https://github.com/imakris/mexce/archive/refs/tags/v{self.version}.zip", - strip_root=True) + def build(self): + CMake(self).configure() def package(self): - copy(self, "mexce.h", self.source_folder, - os.path.join(self.package_folder, "include")) - copy(self, "LICENSE*", self.source_folder, - os.path.join(self.package_folder, "licenses")) + CMake(self).install() + + def package_id(self): + self.info.settings.clear() def package_info(self): self.cpp_info.bindirs = [] self.cpp_info.libdirs = [] - self.cpp_info.set_property("cmake_file_name", "mexce") - self.cpp_info.set_property("cmake_target_name", "mexce::mexce") + self.cpp_info.set_property("cmake_find_mode", "none") + self.cpp_info.builddirs = ["lib/cmake/mexce"] diff --git a/ports/mexce/portfile.cmake b/ports/mexce/portfile.cmake index feb418b..b98bb6c 100644 --- a/ports/mexce/portfile.cmake +++ b/ports/mexce/portfile.cmake @@ -6,7 +6,27 @@ vcpkg_from_github( HEAD_REF master ) -file(INSTALL "${SOURCE_PATH}/mexce.h" DESTINATION "${CURRENT_PACKAGES_DIR}/include") +vcpkg_check_features( + OUT_FEATURE_OPTIONS FEATURE_OPTIONS + FEATURES + protected MEXCE_ENABLE_PROTECTED_EXPRESSIONS +) + +vcpkg_cmake_configure( + SOURCE_PATH "${SOURCE_PATH}" + OPTIONS + -DBUILD_TESTING=OFF + ${FEATURE_OPTIONS} +) +vcpkg_cmake_install() +vcpkg_cmake_config_fixup( + PACKAGE_NAME mexce + CONFIG_PATH lib/cmake/mexce +) + +file(REMOVE_RECURSE + "${CURRENT_PACKAGES_DIR}/debug" + "${CURRENT_PACKAGES_DIR}/lib") vcpkg_install_copyright(FILE_LIST "${SOURCE_PATH}/LICENSE.txt") diff --git a/ports/mexce/vcpkg.json b/ports/mexce/vcpkg.json index aa48f9e..a291611 100644 --- a/ports/mexce/vcpkg.json +++ b/ports/mexce/vcpkg.json @@ -4,5 +4,27 @@ "description": "Header-only JIT compiler for scalar mathematical expressions.", "homepage": "https://github.com/imakris/mexce", "license": "BSD-2-Clause", - "supports": "(windows | linux) & (x86 | x64)" + "supports": "(windows | linux) & (x86 | x64)", + "dependencies": [ + { + "name": "vcpkg-cmake", + "host": true + }, + { + "name": "vcpkg-cmake-config", + "host": true + } + ], + "features": { + "protected": { + "description": "Install protected-expression support.", + "supports": "x64", + "dependencies": [ + { + "name": "libsodium", + "version>=": "1.0.22" + } + ] + } + } } diff --git a/test/install_consumer/CMakeLists.txt b/test/install_consumer/CMakeLists.txt new file mode 100644 index 0000000..77dbb74 --- /dev/null +++ b/test/install_consumer/CMakeLists.txt @@ -0,0 +1,51 @@ +cmake_minimum_required(VERSION 3.16) +project(mexce_install_consumer LANGUAGES CXX) + +option(MEXCE_CONSUMER_PROTECTED "Build the protected installed consumer" OFF) +set(MEXCE_FORBIDDEN_INCLUDE_ROOT "" CACHE PATH + "Source root that must not appear in installed target include paths") + +set(CMAKE_CXX_STANDARD 11) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +if(MEXCE_POISON_SODIUM_PROVIDER) + set(provider_target bogus::normal) +endif() + +find_package(mexce CONFIG REQUIRED) + +get_target_property(mexce_includes mexce::mexce INTERFACE_INCLUDE_DIRECTORIES) +get_target_property(mexce_links mexce::mexce INTERFACE_LINK_LIBRARIES) +if(mexce_links MATCHES "[Ss]odium") + message(FATAL_ERROR "mexce::mexce exposes a sodium link item: ${mexce_links}") +endif() + +if(MEXCE_FORBIDDEN_INCLUDE_ROOT) + file(TO_CMAKE_PATH "${MEXCE_FORBIDDEN_INCLUDE_ROOT}" forbidden_include_root) + string(TOLOWER "${forbidden_include_root}" forbidden_include_root) + string(TOLOWER "${mexce_includes}" normalized_mexce_includes) + string(FIND "${normalized_mexce_includes}" "${forbidden_include_root}" source_root_index) + if(NOT source_root_index EQUAL -1) + message(FATAL_ERROR + "Installed mexce::mexce resolves into the source tree: ${mexce_includes}") + endif() +endif() + +add_executable(mexce_ordinary_consumer ordinary.cpp) +target_link_libraries(mexce_ordinary_consumer PRIVATE mexce::mexce) + +if(MEXCE_CONSUMER_PROTECTED) + if(NOT TARGET mexce::protected) + message(FATAL_ERROR "The installed package does not provide mexce::protected") + endif() + get_target_property(protected_links mexce::protected INTERFACE_LINK_LIBRARIES) + if(protected_links MATCHES "libsodium::|unofficial-sodium::|PkgConfig::") + message(FATAL_ERROR + "mexce::protected leaks a provider target: ${protected_links}") + endif() + + add_executable(mexce_protected_consumer protected.cpp) + target_link_libraries(mexce_protected_consumer PRIVATE mexce::protected) +elseif(TARGET mexce::protected) + message(FATAL_ERROR "The ordinary installed package unexpectedly provides mexce::protected") +endif() diff --git a/test/install_consumer/ordinary.cpp b/test/install_consumer/ordinary.cpp new file mode 100644 index 0000000..77db6fe --- /dev/null +++ b/test/install_consumer/ordinary.cpp @@ -0,0 +1,13 @@ +#include + +#include + + +int main() +{ + double value = 0.5; + mexce::evaluator evaluator; + evaluator.bind(value, "value"); + evaluator.set_expression("sin(value)+1"); + return std::isfinite(evaluator.evaluate()) ? 0 : 1; +} diff --git a/test/install_consumer/protected.cpp b/test/install_consumer/protected.cpp new file mode 100644 index 0000000..2926a75 --- /dev/null +++ b/test/install_consumer/protected.cpp @@ -0,0 +1,25 @@ +#include + +#include + +#include +#include + + +int main() +{ + if (sodium_init() < 0) { + return 1; + } + + double value = 2.0; + mexce::evaluator evaluator; + evaluator.bind_protected(value, 0); + auto bundle = mexce::encode_protected_expression( + "value+1", + {{"value", 0}}, + mexce::Protected_math_mode::STRICT); + evaluator.set_protected_expression( + bundle.program.data(), bundle.program.size(), std::move(bundle.key)); + return std::abs(evaluator.evaluate() - 3.0) < 1e-12 ? 0 : 1; +} diff --git a/test_package/CMakeLists.txt b/test_package/CMakeLists.txt index 26f76f4..fb3c81d 100644 --- a/test_package/CMakeLists.txt +++ b/test_package/CMakeLists.txt @@ -6,5 +6,10 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON) find_package(mexce CONFIG REQUIRED) -add_executable(test_package test_package.cpp) -target_link_libraries(test_package PRIVATE mexce::mexce) +add_executable(mexce_ordinary_consumer test_package.cpp) +target_link_libraries(mexce_ordinary_consumer PRIVATE mexce::mexce) + +if(TARGET mexce::protected) + add_executable(mexce_protected_consumer protected_test_package.cpp) + target_link_libraries(mexce_protected_consumer PRIVATE mexce::protected) +endif() diff --git a/test_package/conanfile.py b/test_package/conanfile.py index 2ec41da..06e4a9a 100644 --- a/test_package/conanfile.py +++ b/test_package/conanfile.py @@ -27,5 +27,10 @@ def build(self): def test(self): if can_run(self): - cmd = os.path.join(self.cpp.build.bindirs[0], "test_package") - self.run(cmd, env="conanrun") + ordinary = os.path.join( + self.cpp.build.bindirs[0], "mexce_ordinary_consumer") + self.run(ordinary, env="conanrun") + if self.dependencies["mexce"].options.with_protected: + protected = os.path.join( + self.cpp.build.bindirs[0], "mexce_protected_consumer") + self.run(protected, env="conanrun") diff --git a/test_package/protected_test_package.cpp b/test_package/protected_test_package.cpp new file mode 100644 index 0000000..2926a75 --- /dev/null +++ b/test_package/protected_test_package.cpp @@ -0,0 +1,25 @@ +#include + +#include + +#include +#include + + +int main() +{ + if (sodium_init() < 0) { + return 1; + } + + double value = 2.0; + mexce::evaluator evaluator; + evaluator.bind_protected(value, 0); + auto bundle = mexce::encode_protected_expression( + "value+1", + {{"value", 0}}, + mexce::Protected_math_mode::STRICT); + evaluator.set_protected_expression( + bundle.program.data(), bundle.program.size(), std::move(bundle.key)); + return std::abs(evaluator.evaluate() - 3.0) < 1e-12 ? 0 : 1; +} diff --git a/test_package/test_package.cpp b/test_package/test_package.cpp index 7df0638..7af6555 100644 --- a/test_package/test_package.cpp +++ b/test_package/test_package.cpp @@ -1,7 +1,10 @@ #include + #include -int main() { + +int main() +{ mexce::evaluator eval; double x = 0.5; eval.bind(x, "x"); From f868ca40bad8218229f599ecb4fde1bcd058eaec Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 17:49:33 +0200 Subject: [PATCH 07/12] Add protected expression issuer tooling --- .github/workflows/main.yml | 234 ++++- .github/workflows/protected-fuzz.yml | 66 ++ CHANGELOG.md | 13 + CMakeLists.txt | 35 + README.md | 107 ++- THIRD_PARTY_NOTICES.md | 21 + conanfile.py | 1 + protected_example.cpp | 73 ++ test/install_consumer/CMakeLists.txt | 4 + test/mexce_protect_tests.cpp | 760 ++++++++++++++++ tools/mexce_protect.cpp | 27 + tools/mexce_protect_lib.cpp | 1204 ++++++++++++++++++++++++++ tools/mexce_protect_lib.h | 60 ++ 13 files changed, 2568 insertions(+), 37 deletions(-) create mode 100644 .github/workflows/protected-fuzz.yml create mode 100644 THIRD_PARTY_NOTICES.md create mode 100644 protected_example.cpp create mode 100644 test/mexce_protect_tests.cpp create mode 100644 tools/mexce_protect.cpp create mode 100644 tools/mexce_protect_lib.cpp create mode 100644 tools/mexce_protect_lib.h diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 1c1a92d..4337cd5 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -1,56 +1,218 @@ -name: Build, Test, and Benchmark mexce +name: Build, Test, Install, and Diagnose mexce -# Trigger the workflow on push or pull request events for the main/master branch on: push: - branches: [ main, master ] + branches: [main, master] pull_request: - branches: [ main, master ] + branches: [main, master] + +permissions: + contents: read jobs: - build-and-benchmark: - name: Build and Benchmark on ${{ matrix.os }} + release-matrix: + name: ${{ matrix.os }} / ${{ matrix.compiler }} / ${{ matrix.configuration }} + runs-on: ${{ matrix.os }} strategy: + fail-fast: false matrix: - os: [ubuntu-latest, windows-latest] - - runs-on: ${{ matrix.os }} + include: + - os: windows-latest + compiler: msvc + cc: cl + cxx: cl + configuration: ordinary + - os: windows-latest + compiler: msvc + cc: cl + cxx: cl + configuration: protected + - os: ubuntu-latest + compiler: gcc + cc: gcc + cxx: g++ + configuration: ordinary + - os: ubuntu-latest + compiler: gcc + cc: gcc + cxx: g++ + configuration: protected + - os: ubuntu-latest + compiler: clang + cc: clang + cxx: clang++ + configuration: ordinary + - os: ubuntu-latest + compiler: clang + cc: clang + cxx: clang++ + configuration: protected steps: - # 1. Checks out your repository - - name: Checkout repository + - name: Checkout uses: actions/checkout@v4 - # 2. Configure the project using CMake - - name: Configure CMake - run: cmake -B build -S . + - name: Install Conan + if: matrix.configuration == 'protected' + run: pip install conan==2.20.1 + + - name: Install Linux Clang dependencies + if: runner.os == 'Linux' && matrix.compiler == 'clang' + run: | + sudo apt-get update + sudo apt-get install -y clang libomp-dev + + - name: Resolve libsodium 1.0.22 + if: matrix.configuration == 'protected' + run: | + conan profile detect --force + conan install --requires=libsodium/1.0.22 --output-folder=deps \ + --generator=CMakeDeps --generator=CMakeToolchain \ + --settings=build_type=Release --build=missing + shell: bash + env: + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} - # 3. Build the project and all targets - - name: Build with CMake - run: cmake --build build --config Release + - name: Configure ordinary Windows release + if: runner.os == 'Windows' && matrix.configuration == 'ordinary' + run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Release - # 4. Run the quick validation tests - - name: Run Quick Tests with CTest + - name: Configure ordinary Linux release + if: runner.os == 'Linux' && matrix.configuration == 'ordinary' + run: >- + cmake -S . -B build + -DCMAKE_BUILD_TYPE=Release + -DCMAKE_C_COMPILER=${{ matrix.cc }} + -DCMAKE_CXX_COMPILER=${{ matrix.cxx }} + + - name: Configure protected Windows release and issuer + if: runner.os == 'Windows' && matrix.configuration == 'protected' + run: >- + cmake -S . -B build + -DCMAKE_BUILD_TYPE=Release + -DCMAKE_TOOLCHAIN_FILE=${{ github.workspace }}/deps/conan_toolchain.cmake + -DMEXCE_ENABLE_PROTECTED_EXPRESSIONS=ON + -DMEXCE_BUILD_ISSUER_TOOLS=ON + + - name: Configure protected Linux release and issuer + if: runner.os == 'Linux' && matrix.configuration == 'protected' + run: >- + cmake -S . -B build + -DCMAKE_BUILD_TYPE=Release + -DCMAKE_C_COMPILER=${{ matrix.cc }} + -DCMAKE_CXX_COMPILER=${{ matrix.cxx }} + -DCMAKE_TOOLCHAIN_FILE=${{ github.workspace }}/deps/conan_toolchain.cmake + -DMEXCE_ENABLE_PROTECTED_EXPRESSIONS=ON + -DMEXCE_BUILD_ISSUER_TOOLS=ON + + - name: Build + run: cmake --build build --config Release --parallel + + - name: Test run: ctest --test-dir build -C Release --output-on-failure - - # 5. Run a short benchmark - - name: Run Short Benchmark - run: cmake --build build --config Release --target run_benchmarks - - # 6. Display Benchmark Summary in Log - # FIX: Use the 'head' command to show only the first 58 lines of the - # results file, which contains the summary tables. - - name: Display Benchmark Summary + + - name: Install + run: cmake --install build --config Release --prefix install + + - name: Check installed notice surface + env: + EXPECT_NOTICE: ${{ matrix.configuration == 'protected' && '1' || '0' }} + run: >- + python -c "import os, pathlib; + p=pathlib.Path('install/share/licenses/mexce/THIRD_PARTY_NOTICES.md'); + assert p.exists() == (os.environ['EXPECT_NOTICE'] == '1')" + + - name: Configure installed Windows consumer + if: runner.os == 'Windows' && matrix.configuration == 'ordinary' + run: >- + cmake -S test/install_consumer -B consumer-build + -DCMAKE_BUILD_TYPE=Release + -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install + -DMEXCE_FORBIDDEN_INCLUDE_ROOT=${{ github.workspace }} + + - name: Configure installed Linux consumer + if: runner.os == 'Linux' && matrix.configuration == 'ordinary' + run: >- + cmake -S test/install_consumer -B consumer-build + -DCMAKE_BUILD_TYPE=Release + -DCMAKE_CXX_COMPILER=${{ matrix.cxx }} + -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install + -DMEXCE_FORBIDDEN_INCLUDE_ROOT=${{ github.workspace }} + + - name: Configure installed protected Windows consumer + if: runner.os == 'Windows' && matrix.configuration == 'protected' + run: >- + cmake -S test/install_consumer -B consumer-build + -DCMAKE_BUILD_TYPE=Release + -DCMAKE_TOOLCHAIN_FILE=${{ github.workspace }}/deps/conan_toolchain.cmake + -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install + -DMEXCE_FORBIDDEN_INCLUDE_ROOT=${{ github.workspace }} + -DMEXCE_CONSUMER_PROTECTED=ON + + - name: Configure installed protected Linux consumer + if: runner.os == 'Linux' && matrix.configuration == 'protected' + run: >- + cmake -S test/install_consumer -B consumer-build + -DCMAKE_BUILD_TYPE=Release + -DCMAKE_CXX_COMPILER=${{ matrix.cxx }} + -DCMAKE_TOOLCHAIN_FILE=${{ github.workspace }}/deps/conan_toolchain.cmake + -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install + -DMEXCE_FORBIDDEN_INCLUDE_ROOT=${{ github.workspace }} + -DMEXCE_CONSUMER_PROTECTED=ON + + - name: Build installed consumer + run: cmake --build consumer-build --config Release --parallel + + - name: Run installed consumers on Linux + if: runner.os == 'Linux' + run: | + ./consumer-build/mexce_ordinary_consumer + if [ "${{ matrix.configuration }}" = protected ]; then + ./consumer-build/mexce_protected_consumer + fi + shell: bash + + - name: Run installed consumers on Windows + if: runner.os == 'Windows' + run: | + .\consumer-build\Release\mexce_ordinary_consumer.exe + if ('${{ matrix.configuration }}' -eq 'protected') { + .\consumer-build\Release\mexce_protected_consumer.exe + } + + - name: Create documented issuer inputs + if: matrix.configuration == 'protected' + run: >- + python -c "from pathlib import Path; + Path('expression.txt').write_bytes(b'value+1\n'); + Path('bindings.txt').write_bytes(b'value=0\n')" + + - name: Run installed issuer and file consumer on Linux + if: runner.os == 'Linux' && matrix.configuration == 'protected' + run: | + ./install/bin/mexce_protect expression.txt bindings.txt expression.mxp expression.key + ./consumer-build/mexce_protected_file_consumer expression.mxp expression.key shell: bash + + - name: Run installed issuer and file consumer on Windows + if: runner.os == 'Windows' && matrix.configuration == 'protected' run: | - echo "--- Benchmark Summary (first 58 lines) ---" - head -n 58 build/benchmark_results.txt - echo "------------------------------------------" + .\install\bin\mexce_protect.exe expression.txt bindings.txt expression.mxp expression.key + .\consumer-build\Release\mexce_protected_file_consumer.exe expression.mxp expression.key + + - name: Run clear performance diagnostic + run: cmake --build build --config Release --target run_benchmarks + + - name: Run protected performance diagnostic + if: matrix.configuration == 'protected' + run: ctest --test-dir build -C Release -R mexce_protected_benchmark --verbose > build/protected_benchmark_results.txt - # 7. Upload Full Benchmark Results as an Artifact - # This still saves the complete file for detailed analysis. - - name: Upload Benchmark Artifact + - name: Upload diagnostic results uses: actions/upload-artifact@v4 with: - name: benchmark-results-${{ matrix.os }} - path: build/benchmark_results.txt + name: diagnostics-${{ matrix.os }}-${{ matrix.compiler }}-${{ matrix.configuration }} + if-no-files-found: error + path: | + build/benchmark_results.txt + build/protected_benchmark_results.txt diff --git a/.github/workflows/protected-fuzz.yml b/.github/workflows/protected-fuzz.yml new file mode 100644 index 0000000..ee7f03b --- /dev/null +++ b/.github/workflows/protected-fuzz.yml @@ -0,0 +1,66 @@ +name: Protected decoder extended fuzz + +on: + schedule: + - cron: "17 3 * * 1" + workflow_dispatch: + +permissions: + contents: read + +jobs: + fuzz: + runs-on: ubuntu-latest + timeout-minutes: 30 + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Install compiler and dependency tooling + run: | + sudo apt-get update + sudo apt-get install -y clang libomp-dev + pip install conan==2.20.1 + + - name: Resolve libsodium 1.0.22 + run: | + conan profile detect --force + conan install --requires=libsodium/1.0.22 --output-folder=deps \ + --generator=CMakeDeps --generator=CMakeToolchain \ + --settings=build_type=RelWithDebInfo --build=missing + + - name: Configure sanitizer build + run: >- + cmake -S . -B build -G Ninja + -DCMAKE_BUILD_TYPE=RelWithDebInfo + -DCMAKE_CXX_COMPILER=clang++ + -DCMAKE_TOOLCHAIN_FILE=${{ github.workspace }}/deps/conan_toolchain.cmake + -DMEXCE_ENABLE_PROTECTED_EXPRESSIONS=ON + + - name: Build fuzz target + run: cmake --build build --target protected_decoder_fuzz --parallel + + - name: Seed authenticated format corpus + run: | + test -s test/fixtures/protected_format_1_0.bin + mkdir -p fuzz-corpus + cp test/fixtures/protected_format_1_0.bin fuzz-corpus/format-1.0.bin + cmp test/fixtures/protected_format_1_0.bin fuzz-corpus/format-1.0.bin + + - name: Run extended decoder fuzzing + run: | + mkdir -p fuzz-artifacts + ./build/protected_decoder_fuzz \ + fuzz-corpus \ + -max_total_time=900 \ + -timeout=10 \ + -artifact_prefix=fuzz-artifacts/ + + - name: Upload crashing inputs + if: failure() + uses: actions/upload-artifact@v4 + with: + name: protected-fuzz-artifacts + if-no-files-found: ignore + path: fuzz-artifacts/ diff --git a/CHANGELOG.md b/CHANGELOG.md index dec59b0..ac1f550 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ All notable changes to this project will be documented in this file. +## [Unreleased] + +### Added + +- Opt-in authenticated protected-expression encoding and runtime loading on + x64 through libsodium 1.0.22. +- The `mexce::protected` installed target and protected Conan/vcpkg package + variants while keeping the ordinary target dependency-free. +- The opt-in `mexce_protect` issuer utility with owner-only raw-key output, + no-overwrite publication, partial-state detection, and Windows/Linux tests. +- Protected usage, threat-model, issuer, example, and third-party notice + documentation. + ## [1.0.1] - 2025-10-10 ### Added - GitHub Actions workflows for build, benchmark, and coverage reporting, plus Codecov configuration and badges to track quality metrics. diff --git a/CMakeLists.txt b/CMakeLists.txt index 76b498d..8d76670 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -7,6 +7,12 @@ include(GNUInstallDirs) option(ENABLE_COVERAGE "Build with coverage instrumentation" OFF) option(MEXCE_ENABLE_PROTECTED_EXPRESSIONS "Build protected-expression targets" OFF) +option(MEXCE_BUILD_ISSUER_TOOLS "Build the protected-expression issuer tool" OFF) + +if(MEXCE_BUILD_ISSUER_TOOLS AND NOT MEXCE_ENABLE_PROTECTED_EXPRESSIONS) + message(FATAL_ERROR + "MEXCE_BUILD_ISSUER_TOOLS requires MEXCE_ENABLE_PROTECTED_EXPRESSIONS") +endif() set(CMAKE_CXX_STANDARD 11) set(CMAKE_CXX_STANDARD_REQUIRED ON) @@ -40,6 +46,21 @@ if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) target_link_libraries(mexce_protected INTERFACE mexce mexce::_sodium) endif() +if(MEXCE_BUILD_ISSUER_TOOLS) + add_library(mexce_protect_lib STATIC tools/mexce_protect_lib.cpp) + target_include_directories(mexce_protect_lib PUBLIC + ${CMAKE_CURRENT_SOURCE_DIR}/tools) + target_link_libraries(mexce_protect_lib PUBLIC mexce_protected) + target_compile_options(mexce_protect_lib PRIVATE ${MEXCE_WARNING_FLAGS}) + if(WIN32) + target_link_libraries(mexce_protect_lib PRIVATE Advapi32) + endif() + + add_executable(mexce_protect tools/mexce_protect.cpp) + target_link_libraries(mexce_protect PRIVATE mexce_protect_lib) + target_compile_options(mexce_protect PRIVATE ${MEXCE_WARNING_FLAGS}) +endif() + if(BUILD_TESTING) if(ENABLE_COVERAGE) if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang") @@ -138,6 +159,12 @@ if(BUILD_TESTING) target_link_options(protected_decoder_fuzz PRIVATE -fsanitize=fuzzer,address,undefined) endif() + + if(MEXCE_BUILD_ISSUER_TOOLS) + add_executable(mexce_protect_tests test/mexce_protect_tests.cpp) + target_link_libraries(mexce_protect_tests PRIVATE mexce_protect_lib) + target_compile_options(mexce_protect_tests PRIVATE ${MEXCE_WARNING_FLAGS}) + endif() endif() set(MEXCE_BENCHMARK_COMPILER "${CMAKE_CXX_COMPILER_ID}") @@ -177,6 +204,9 @@ if(BUILD_TESTING) COMMAND protected_benchmark maximum-valid) add_test(NAME mexce_protected_late_invalid COMMAND protected_benchmark late-invalid) + if(MEXCE_BUILD_ISSUER_TOOLS) + add_test(NAME mexce_protect_utility COMMAND mexce_protect_tests) + endif() endif() add_test(NAME benchmark_quick COMMAND benchmark 5 ${CMAKE_BINARY_DIR}/benchmark_quick_results.txt) @@ -195,6 +225,9 @@ if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) list(APPEND mexce_install_targets mexce_protected) endif() install(TARGETS ${mexce_install_targets} EXPORT mexce_targets) +if(MEXCE_BUILD_ISSUER_TOOLS) + install(TARGETS mexce_protect RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}) +endif() install(EXPORT mexce_targets FILE mexce-targets.cmake NAMESPACE mexce:: @@ -208,6 +241,8 @@ if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) install(FILES cmake/mexce-sodium.cmake DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mexce) + install(FILES THIRD_PARTY_NOTICES.md + DESTINATION ${CMAKE_INSTALL_DATADIR}/licenses/mexce) endif() install(FILES LICENSE.txt DESTINATION ${CMAKE_INSTALL_DATADIR}/licenses/mexce) diff --git a/README.md b/README.md index 8dcae76..f73f578 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,9 @@ A single-header, dependency-free JIT compiler for mathematical expressions. Once an expression is compiled, subsequent evaluations are direct function calls, which avoids parsing and interpretation overhead. This makes `mexce` well-suited for applications that repeatedly evaluate the same formula with different inputs, such as numerical simulations, data processing kernels, or graphics. -The library is contained in a single header file (`mexce.h`) with no external dependencies. +The ordinary library is contained in a single header file (`mexce.h`) with no +external dependencies. An opt-in protected-expression surface uses libsodium +1.0.22 to load authenticated, encrypted semantic programs. ### Requirements * **Platforms:** Windows, Linux @@ -23,6 +25,109 @@ The library is contained in a single header file (`mexce.h`) with no external de Copy `mexce.h` into your project's include path and `#include "mexce.h"`. No other steps are needed. +### Protected-expression build + +Protected expressions are x64-only and remain off by default. Make libsodium +1.0.22 available through Conan, vcpkg, or pkg-config, then configure the source +build with: + +```sh +cmake -S . -B build \ + -DBUILD_TESTING=OFF \ + -DMEXCE_ENABLE_PROTECTED_EXPRESSIONS=ON \ + -DMEXCE_BUILD_ISSUER_TOOLS=ON +cmake --build build --config Release +cmake --install build --config Release --prefix install +``` + +Protected consumers link `mexce::protected`; ordinary consumers continue to +link `mexce::mexce` without a cryptographic dependency. The +`MEXCE_BUILD_ISSUER_TOOLS` option adds and installs `mexce_protect`, and requires +protected expressions to be enabled. + +### Creating a protected program + +`mexce_protect` is an issuer-side build tool, not a licence system. It accepts +an expression file, a binding-schema file, a program output, and a key output. +For `expression.txt`: + +```text +value+1 +``` + +For `bindings.txt`: + +```text +value=0 +``` + +```sh +mexce_protect expression.txt bindings.txt expression.mxp expression.key +``` + +Schema entries use `name=decimal_slot`, one per line. Names use +`[A-Za-z_][A-Za-z0-9_]*`, slots are unique and dense from zero, and the schema +must exactly describe the variables used by the expression. The tool removes +one terminal LF, plus its preceding CR when present, from the expression. It +does not otherwise normalize source whitespace. + +The command-line issuer encodes `Protected_math_mode::STRICT`. Applications +that intentionally require `FAST` policy can use the issuer-side encoder API. + +The program and key destinations must be distinct and absent. The tool rejects +symlink or reparse-point traversal, never overwrites a destination, restricts +the 32-byte raw key file to the current owner, and publishes the key before the +program. A failed publication can therefore leave a key without a program, but +not a program published by that invocation without its key. Remove an orphan +key explicitly before retrying; the tool will not guess or overwrite partial +state. Filesystem or machine power-loss atomicity is not claimed. + +Immediately import the raw key into the host product's key-wrapping or secure +delivery system. After confirming that import, remove the caller-owned final +key file according to the host platform's data-retention policy. Do not ship +the raw key beside the protected program. + +### Loading a protected program + +The complete example is in `protected_example.cpp`. The runtime sequence is: + +```cpp +const std::vector program = read_file("expression.mxp"); +std::vector raw_key = obtain_unwrapped_key_from_host(); +Raw_key_wipe_guard raw_key_wipe(raw_key); // Defined in protected_example.cpp. +auto key = mexce::Protected_expression_key::from_bytes( + raw_key.data(), raw_key.size()); + +double value = 2.0; +mexce::evaluator evaluator; +evaluator.bind_protected(value, 0); +evaluator.set_protected_expression( + program.data(), program.size(), std::move(key)); +const double result = evaluator.evaluate(); +``` + +The key is move-only. `set_protected_expression` consumes it, authenticates and +compiles the program, and leaves the evaluator empty on failure. Runtime slots +are authenticated numbers, not source variable names or C++ types. +Construct the caller-owned raw-key wipe guard before `from_bytes`, as in the +example, so invalid keys and allocation or locking failures also wipe the vector. + +### Protected-expression security boundary + +Protected artifacts conceal and authenticate semantic operations, literal +bits, variable-slot use, and compiler policy against someone who can inspect or +modify stored artifacts but does not have the matching key and cannot modify +the trusted process. Variable names, comments, whitespace, parentheses, and +original spelling are not stored in the artifact. + +This does not provide confidentiality against an attacker who controls the +licensed process, debugger, process memory, registers, libsodium calls, or +emitted native code. It is not virtual-black-box obfuscation, anti-debugging, +white-box cryptography, issuer authentication, freshness, rollback protection, +revocation, device binding, or a licence system. Key storage, transport, +wrapping, device and licence policy, and issuer security belong to the host +product. A valid replayed program and matching key are accepted. + ## Quick Start The following example shows how to bind variables and evaluate an expression in a loop. A `mexce::evaluator` instance initializes to the constant expression `"0"`. diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md new file mode 100644 index 0000000..345aca3 --- /dev/null +++ b/THIRD_PARTY_NOTICES.md @@ -0,0 +1,21 @@ +# Third-party notices + +Protected-expression builds use [libsodium 1.0.22](https://libsodium.org/), +which is distributed under the ISC License: + +```text +Copyright (c) 2013-2026 +Frank Denis + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted, provided that the above +copyright notice and this permission notice appear in all copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +``` diff --git a/conanfile.py b/conanfile.py index c559f59..fcfa2e5 100644 --- a/conanfile.py +++ b/conanfile.py @@ -19,6 +19,7 @@ class MexceConan(ConanFile): exports_sources = ( "CMakeLists.txt", "LICENSE.txt", + "THIRD_PARTY_NOTICES.md", "cmake/*", "mexce.h", "mexce_protected.h", diff --git a/protected_example.cpp b/protected_example.cpp new file mode 100644 index 0000000..75373b4 --- /dev/null +++ b/protected_example.cpp @@ -0,0 +1,73 @@ +#ifdef _WIN32 + #define NOMINMAX +#endif + +#include + +#include +#include +#include +#include +#include + + +std::vector read_file(const char* path) +{ + std::ifstream input(path, std::ios::binary); + if (!input) { + throw std::runtime_error("Cannot open a protected-expression input file."); + } + return std::vector( + std::istreambuf_iterator(input), std::istreambuf_iterator()); +} + + +class Raw_key_wipe_guard +{ +public: + explicit Raw_key_wipe_guard(std::vector& key) + : + m_key(key) + {} + + ~Raw_key_wipe_guard() + { + if (!m_key.empty()) { + sodium_memzero(m_key.data(), m_key.size()); + } + } + +private: + std::vector& m_key; +}; + + +int run_protected_example(char* argv[]) +{ + const std::vector program = read_file(argv[1]); + std::vector key = read_file(argv[2]); + Raw_key_wipe_guard key_wipe(key); + auto protected_key = mexce::Protected_expression_key::from_bytes( + key.data(), key.size()); + + double value = 2.0; + mexce::evaluator evaluator; + evaluator.bind_protected(value, 0); + evaluator.set_protected_expression( + program.data(), program.size(), std::move(protected_key)); + return evaluator.evaluate() == 3.0 ? 0 : 1; +} + + +int main(int argc, char* argv[]) +{ + if (argc != 3) { + return 2; + } + try { + return run_protected_example(argv); + } + catch (const std::exception&) { + return 1; + } +} diff --git a/test/install_consumer/CMakeLists.txt b/test/install_consumer/CMakeLists.txt index 77dbb74..6bf17d3 100644 --- a/test/install_consumer/CMakeLists.txt +++ b/test/install_consumer/CMakeLists.txt @@ -46,6 +46,10 @@ if(MEXCE_CONSUMER_PROTECTED) add_executable(mexce_protected_consumer protected.cpp) target_link_libraries(mexce_protected_consumer PRIVATE mexce::protected) + + add_executable(mexce_protected_file_consumer + ${CMAKE_CURRENT_LIST_DIR}/../../protected_example.cpp) + target_link_libraries(mexce_protected_file_consumer PRIVATE mexce::protected) elseif(TARGET mexce::protected) message(FATAL_ERROR "The ordinary installed package unexpectedly provides mexce::protected") endif() diff --git a/test/mexce_protect_tests.cpp b/test/mexce_protect_tests.cpp new file mode 100644 index 0000000..9db0b49 --- /dev/null +++ b/test/mexce_protect_tests.cpp @@ -0,0 +1,760 @@ +#ifdef _WIN32 + #define NOMINMAX +#endif + +#include "mexce_protect_lib.h" + +#include +#include +#include +#include +#include +#include +#include + +#ifdef _WIN32 + #include + #include +#else + #include + #include + #include +#endif + + +namespace { + + +using mexce::issuer::Native_path; + + +class Test_suite +{ +public: + void expect(bool condition, const std::string& name) + { + if (!condition) { + m_failures.push_back(name); + } + } + + void expect_issuer_error( + const std::string& name, + const std::function& operation) + { + try { + operation(); + m_failures.push_back(name + ": expected Issuer_error"); + } + catch (const mexce::issuer::Issuer_error&) { + } + catch (const std::exception& error) { + m_failures.push_back(name + ": unexpected exception: " + error.what()); + } + } + + void run(const std::string& name, const std::function& test) + { + try { + test(); + } + catch (const std::exception& error) { + m_failures.push_back(name + ": unexpected exception: " + error.what()); + } + } + + int finish() const + { + for (const auto& failure : m_failures) { + std::fprintf(stderr, "FAIL: %s\n", failure.c_str()); + } + if (!m_failures.empty()) { + std::fprintf(stderr, "%zu issuer utility test(s) failed\n", m_failures.size()); + return 1; + } + std::printf("All issuer utility tests passed\n"); + return 0; + } + +private: + std::vector m_failures; +}; + + +#ifdef _WIN32 +Native_path join(const Native_path& directory, const wchar_t* name) +{ + return directory + L"\\" + name; +} + + +void write_bytes(const Native_path& path, const std::vector& bytes) +{ + HANDLE file = CreateFileW(path.c_str(), GENERIC_WRITE, 0, nullptr, + CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, nullptr); + if (file == INVALID_HANDLE_VALUE) { + throw std::runtime_error("test setup could not create a file"); + } + size_t offset = 0; + while (offset < bytes.size()) { + DWORD written = 0; + if (!WriteFile(file, bytes.data() + offset, + static_cast(bytes.size() - offset), &written, nullptr) || + written == 0) + { + CloseHandle(file); + throw std::runtime_error("test setup could not write a file"); + } + offset += written; + } + CloseHandle(file); +} + + +bool exists(const Native_path& path) +{ + return GetFileAttributesW(path.c_str()) != INVALID_FILE_ATTRIBUTES; +} + + +void remove_path(const Native_path& path) +{ + if (!DeleteFileW(path.c_str())) { + RemoveDirectoryW(path.c_str()); + } +} + + +size_t file_size(const Native_path& path) +{ + WIN32_FILE_ATTRIBUTE_DATA data; + if (!GetFileAttributesExW(path.c_str(), GetFileExInfoStandard, &data)) { + return 0; + } + ULARGE_INTEGER size; + size.HighPart = data.nFileSizeHigh; + size.LowPart = data.nFileSizeLow; + return static_cast(size.QuadPart); +} + + +bool owner_only(const Native_path& path) +{ + PSID owner = nullptr; + PACL dacl = nullptr; + PSECURITY_DESCRIPTOR descriptor = nullptr; + const DWORD result = GetNamedSecurityInfoW( + const_cast(path.c_str()), SE_FILE_OBJECT, + OWNER_SECURITY_INFORMATION | DACL_SECURITY_INFORMATION, + &owner, nullptr, &dacl, nullptr, &descriptor); + if (result != ERROR_SUCCESS || !owner || !dacl || !descriptor) { + return false; + } + + SECURITY_DESCRIPTOR_CONTROL control = 0; + DWORD revision = 0; + ACL_SIZE_INFORMATION information = {}; + const bool protected_dacl = + GetSecurityDescriptorControl(descriptor, &control, &revision) && + (control & SE_DACL_PROTECTED) != 0; + bool result_value = protected_dacl && + GetAclInformation(dacl, &information, sizeof(information), AclSizeInformation) && + information.AceCount == 1; + if (result_value) { + void* raw_ace = nullptr; + result_value = GetAce(dacl, 0, &raw_ace) != FALSE; + if (result_value) { + const auto* ace = static_cast(raw_ace); + const PSID ace_sid = const_cast(&ace->SidStart); + result_value = ace->Header.AceType == ACCESS_ALLOWED_ACE_TYPE && + (ace->Header.AceFlags & INHERITED_ACE) == 0 && + EqualSid(owner, ace_sid) != FALSE; + } + } + LocalFree(descriptor); + return result_value; +} + + +Native_path make_directory() +{ + wchar_t temporary[MAX_PATH]; + wchar_t candidate[MAX_PATH]; + if (!GetTempPathW(MAX_PATH, temporary) || + !GetTempFileNameW(temporary, L"mxc", 0, candidate) || + !DeleteFileW(candidate) || !CreateDirectoryW(candidate, nullptr)) + { + throw std::runtime_error("test setup could not create a directory"); + } + return candidate; +} + + +bool create_directory_link(const Native_path& link, const Native_path& target) +{ + constexpr DWORD k_allow_unprivileged = 0x2; + return CreateSymbolicLinkW(link.c_str(), target.c_str(), + SYMBOLIC_LINK_FLAG_DIRECTORY | k_allow_unprivileged) != FALSE; +} + + +size_t temporary_count(const Native_path& directory) +{ + WIN32_FIND_DATAW data; + HANDLE search = FindFirstFileW((directory + L"\\.mexce-*.tmp").c_str(), &data); + if (search == INVALID_HANDLE_VALUE) { + return 0; + } + size_t count = 1; + while (FindNextFileW(search, &data)) { + ++count; + } + FindClose(search); + return count; +} + + +#else + + +Native_path join(const Native_path& directory, const char* name) +{ + return directory + "/" + name; +} + + +void write_bytes(const Native_path& path, const std::vector& bytes) +{ + FILE* file = std::fopen(path.c_str(), "wb"); + if (!file || (!bytes.empty() && + std::fwrite(bytes.data(), 1, bytes.size(), file) != bytes.size())) + { + if (file) { + std::fclose(file); + } + throw std::runtime_error("test setup could not write a file"); + } + std::fclose(file); +} + + +bool exists(const Native_path& path) +{ + struct stat status; + return lstat(path.c_str(), &status) == 0; +} + + +void remove_path(const Native_path& path) +{ + if (unlink(path.c_str()) != 0) { + rmdir(path.c_str()); + } +} + + +size_t file_size(const Native_path& path) +{ + struct stat status; + return stat(path.c_str(), &status) == 0 ? static_cast(status.st_size) : 0; +} + + +bool owner_only(const Native_path& path) +{ + struct stat status; + return stat(path.c_str(), &status) == 0 && (status.st_mode & 0777) == 0600; +} + + +Native_path make_directory() +{ + std::vector pattern(64); + std::strcpy(pattern.data(), "/tmp/mexce-protect-XXXXXX"); + if (!mkdtemp(pattern.data())) { + throw std::runtime_error("test setup could not create a directory"); + } + return pattern.data(); +} + + +bool create_directory_link(const Native_path& link, const Native_path& target) +{ + return symlink(target.c_str(), link.c_str()) == 0; +} + + +size_t temporary_count(const Native_path& directory) +{ + DIR* entries = opendir(directory.c_str()); + if (!entries) { + return 0; + } + size_t count = 0; + while (const dirent* entry = readdir(entries)) { + const std::string name(entry->d_name); + if (name.compare(0, 7, ".mexce-") == 0 && + name.size() >= 4 && name.compare(name.size() - 4, 4, ".tmp") == 0) + { + ++count; + } + } + closedir(entries); + return count; +} + + +#endif + + +std::vector bytes(const std::string& value) +{ + return std::vector(value.begin(), value.end()); +} + + +class Fixture +{ +public: + Fixture() + : + root(make_directory()), + expression(join(root, +#ifdef _WIN32 + L"expression.txt" +#else + "expression.txt" +#endif + )), + schema(join(root, +#ifdef _WIN32 + L"schema.txt" +#else + "schema.txt" +#endif + )), + program(join(root, +#ifdef _WIN32 + L"program.bin" +#else + "program.bin" +#endif + )), + key(join(root, +#ifdef _WIN32 + L"key.bin" +#else + "key.bin" +#endif + )) + { + write_bytes(expression, bytes("x+1\n")); + write_bytes(schema, bytes("x=0\n")); + } + + ~Fixture() + { + remove_path(expression); + remove_path(schema); + remove_path(program); + remove_path(key); + remove_path(root); + } + + void reset_outputs() + { + remove_path(program); + remove_path(key); + } + + void protect(mexce::issuer::Test_failure failure = mexce::issuer::Test_failure::NONE) + { + mexce::issuer::protect_expression(expression, schema, program, key, failure); + } + + Native_path root; + Native_path expression; + Native_path schema; + Native_path program; + Native_path key; +}; + + +void test_success_and_permissions(Test_suite& suite) +{ + Fixture fixture; + fixture.protect(); + suite.expect(file_size(fixture.program) > 64, "success publishes a program"); + suite.expect(file_size(fixture.key) == 32, "success publishes exactly 32 key bytes"); + suite.expect(owner_only(fixture.key), "key output is owner-only"); + suite.expect(temporary_count(fixture.root) == 0, + "success leaves no temporary files"); + + fixture.reset_outputs(); + write_bytes(fixture.expression, bytes("1+2\r\n")); + write_bytes(fixture.schema, {}); + fixture.protect(); + suite.expect(exists(fixture.program) && exists(fixture.key), + "constant expression accepts an empty schema and terminal CRLF"); + + fixture.reset_outputs(); + write_bytes(fixture.expression, bytes("x+1\n")); + write_bytes(fixture.schema, bytes("x=0\n")); + const Native_path program_directory = join(fixture.root, +#ifdef _WIN32 + L"program-output" +#else + "program-output" +#endif + ); + const Native_path key_directory = join(fixture.root, +#ifdef _WIN32 + L"key-output" +#else + "key-output" +#endif + ); +#ifdef _WIN32 + const bool directories_created = + CreateDirectoryW(program_directory.c_str(), nullptr) != FALSE && + CreateDirectoryW(key_directory.c_str(), nullptr) != FALSE; + const Native_path split_program = join(program_directory, L"program.bin"); + const Native_path split_key = join(key_directory, L"key.bin"); +#else + const bool directories_created = mkdir(program_directory.c_str(), 0700) == 0 && + mkdir(key_directory.c_str(), 0700) == 0; + const Native_path split_program = join(program_directory, "program.bin"); + const Native_path split_key = join(key_directory, "key.bin"); +#endif + suite.expect(directories_created, "distinct output parent setup"); + mexce::issuer::protect_expression( + fixture.expression, fixture.schema, split_program, split_key); + suite.expect(exists(split_program) && exists(split_key) && owner_only(split_key), + "distinct verified output parents publish successfully"); + remove_path(split_program); + remove_path(split_key); + remove_path(program_directory); + remove_path(key_directory); +} + + +void test_expression_validation(Test_suite& suite) +{ + Fixture fixture; + const std::vector>> cases = { + {"expression BOM", {0xef, 0xbb, 0xbf, 'x'}}, + {"expression NUL", {'x', 0, '+', '1'}}, + {"expression size", std::vector(1024 * 1024 + 1, '1')}, + }; + for (const auto& test_case : cases) { + write_bytes(fixture.expression, test_case.second); + suite.expect_issuer_error(test_case.first, [&] { fixture.protect(); }); + suite.expect(!exists(fixture.program) && !exists(fixture.key), + test_case.first + " leaves no final output"); + } + + write_bytes(fixture.expression, bytes("secret_identifier+1")); + try { + fixture.protect(); + suite.expect(false, "parser diagnostic rejects unknown source identifier"); + } + catch (const mexce::issuer::Issuer_error& error) { + suite.expect(std::string(error.what()).find("secret_identifier") == std::string::npos, + "parser diagnostic does not print source content"); + } +} + + +void test_schema_validation(Test_suite& suite) +{ + Fixture fixture; + const std::vector>> cases = { + {"blank schema line", bytes("x=0\n\ny=1")}, + {"schema comment", bytes("# x=0")}, + {"schema built-in name", bytes("pi=0")}, + {"schema NUL", {'x', '=', '0', 0}}, + {"schema UTF-8", {0xc0, 0x80, '=', '0'}}, + {"duplicate name", bytes("x=0\nx=1")}, + {"duplicate slot", bytes("x=0\ny=0")}, + {"sparse slot", bytes("x=1")}, + {"oversized name", bytes(std::string(256, 'x') + "=0")}, + {"trailing slot characters", bytes("x=0 ")}, + {"slot overflow", bytes("x=4294967296")}, + {"schema size", std::vector(256 * 1024 + 1, 'x')}, + }; + for (const auto& test_case : cases) { + write_bytes(fixture.schema, test_case.second); + suite.expect_issuer_error(test_case.first, [&] { fixture.protect(); }); + suite.expect(!exists(fixture.program) && !exists(fixture.key), + test_case.first + " leaves no final output"); + } + + std::string excessive_bindings; + for (size_t i = 0; i < 4097; ++i) { + excessive_bindings += "v" + std::to_string(i) + "=" + std::to_string(i) + "\n"; + } + write_bytes(fixture.schema, bytes(excessive_bindings)); + suite.expect_issuer_error("schema binding count", [&] { fixture.protect(); }); +} + + +void test_output_validation(Test_suite& suite) +{ + Fixture fixture; + suite.expect_issuer_error("identical output paths", [&] { + mexce::issuer::protect_expression( + fixture.expression, fixture.schema, fixture.program, fixture.program); + }); + const Native_path aliased_program = join(fixture.root, +#ifdef _WIN32 + L".\\program.bin" +#else + "./program.bin" +#endif + ); + suite.expect_issuer_error("normalized identical output paths", [&] { + mexce::issuer::protect_expression( + fixture.expression, fixture.schema, fixture.program, aliased_program); + }); + + write_bytes(fixture.program, bytes("existing")); + write_bytes(fixture.key, bytes("existing")); + suite.expect_issuer_error("existing output paths", [&] { fixture.protect(); }); + suite.expect(file_size(fixture.program) == 8 && file_size(fixture.key) == 8, + "existing outputs are not overwritten"); + + fixture.reset_outputs(); + write_bytes(fixture.key, bytes("orphan")); + suite.expect_issuer_error("partial final state", [&] { fixture.protect(); }); + suite.expect(!exists(fixture.program) && file_size(fixture.key) == 6, + "partial final state is not changed"); + + fixture.reset_outputs(); + write_bytes(fixture.program, bytes("orphan")); + suite.expect_issuer_error("inverse partial final state", [&] { fixture.protect(); }); + suite.expect(file_size(fixture.program) == 6 && !exists(fixture.key), + "inverse partial final state is not changed"); +} + + +void test_publication_failures(Test_suite& suite) +{ + Fixture fixture; +#ifdef _WIN32 + const Native_path collision = join(fixture.root, + L".mexce-test-collision.tmp" + ); + write_bytes(collision, bytes("caller-owned")); + suite.expect_issuer_error("exclusive temporary collision", [&] { + fixture.protect(mexce::issuer::Test_failure::KEY_TEMPORARY_COLLISION); + }); + suite.expect(file_size(collision) == 12, + "temporary collision does not delete or overwrite an existing file"); + suite.expect(temporary_count(fixture.root) == 1, + "temporary collision cleans only the temporary owned by the invocation"); + remove_path(collision); +#endif + + suite.expect_issuer_error("failure before publication", [&] { + fixture.protect(mexce::issuer::Test_failure::BEFORE_PUBLICATION); + }); + suite.expect(!exists(fixture.program) && !exists(fixture.key), + "pre-publication failure leaves no final output"); + suite.expect(temporary_count(fixture.root) == 0, + "pre-publication failure removes temporary files"); + + suite.expect_issuer_error("failure after key publication", [&] { + fixture.protect(mexce::issuer::Test_failure::AFTER_KEY_PUBLICATION); + }); + suite.expect(!exists(fixture.program) && file_size(fixture.key) == 32, + "post-key failure leaves only an owner-only raw key"); + suite.expect(temporary_count(fixture.root) == 0, + "post-key failure removes the program temporary"); + suite.expect(owner_only(fixture.key), "orphan key remains owner-only"); + suite.expect_issuer_error("orphan recovery detection", [&] { fixture.protect(); }); + suite.expect(!exists(fixture.program) && file_size(fixture.key) == 32, + "recovery detection does not overwrite the orphan key"); +} + + +void test_reparse_traversal(Test_suite& suite) +{ + Fixture fixture; + const Native_path real_directory = join(fixture.root, +#ifdef _WIN32 + L"real" +#else + "real" +#endif + ); + const Native_path linked_directory = join(fixture.root, +#ifdef _WIN32 + L"linked" +#else + "linked" +#endif + ); +#ifdef _WIN32 + const bool made_directory = CreateDirectoryW(real_directory.c_str(), nullptr) != FALSE; +#else + const bool made_directory = mkdir(real_directory.c_str(), 0700) == 0; +#endif + suite.expect(made_directory, "reparse test creates a real directory"); + const bool made_link = made_directory && + create_directory_link(linked_directory, real_directory); + suite.expect(made_link, "reparse test creates a directory link"); + if (made_link) { + const Native_path linked_program = join(linked_directory, +#ifdef _WIN32 + L"program.bin" +#else + "program.bin" +#endif + ); + suite.expect_issuer_error("output reparse traversal", [&] { + mexce::issuer::protect_expression( + fixture.expression, fixture.schema, linked_program, fixture.key); + }); + suite.expect(!exists(fixture.key), "reparse rejection leaves no key output"); + } + remove_path(linked_directory); + remove_path(real_directory); +} + + +void test_ancestor_swap(Test_suite& suite) +{ + Fixture fixture; +#ifdef _WIN32 + const Native_path moved = fixture.root + L"-moved"; + const Native_path redirect = fixture.root + L"-redirect"; + bool swap_completed = false; + mexce::issuer::Test_hooks hooks; + hooks.after_output_directories_opened = [&] { + swap_completed = MoveFileW(fixture.root.c_str(), moved.c_str()) != FALSE && + CreateDirectoryW(redirect.c_str(), nullptr) != FALSE && + create_directory_link(fixture.root, redirect); + }; + mexce::issuer::protect_expression( + fixture.expression, fixture.schema, fixture.program, fixture.key, + mexce::issuer::Test_failure::NONE, &hooks); + suite.expect(swap_completed, "Windows ancestor replacement seam completes"); + suite.expect(!exists(fixture.program) && !exists(fixture.key), + "Windows publication is not redirected through a replacement ancestor"); + suite.expect(exists(join(moved, L"program.bin")) && exists(join(moved, L"key.bin")), + "Windows publication stays bound to the verified directory"); + remove_path(fixture.root); + remove_path(redirect); + suite.expect(MoveFileW(moved.c_str(), fixture.root.c_str()) != FALSE, + "Windows ancestor replacement test restores its fixture"); +#else + const Native_path moved = fixture.root + "-moved"; + const Native_path redirect = fixture.root + "-redirect"; + bool swap_completed = false; + mexce::issuer::Test_hooks hooks; + hooks.after_output_directories_opened = [&] { + swap_completed = rename(fixture.root.c_str(), moved.c_str()) == 0 && + mkdir(redirect.c_str(), 0700) == 0 && + symlink(redirect.c_str(), fixture.root.c_str()) == 0; + }; + mexce::issuer::protect_expression( + fixture.expression, fixture.schema, fixture.program, fixture.key, + mexce::issuer::Test_failure::NONE, &hooks); + suite.expect(swap_completed, "Linux ancestor replacement seam completes"); + suite.expect(!exists(fixture.program) && !exists(fixture.key), + "Linux publication is not redirected through a replacement ancestor"); + suite.expect(exists(join(moved, "program.bin")) && exists(join(moved, "key.bin")), + "Linux publication stays bound to the verified directory"); + remove_path(fixture.root); + remove_path(redirect); + suite.expect(rename(moved.c_str(), fixture.root.c_str()) == 0, + "Linux ancestor replacement test restores its fixture"); +#endif +} + + +void test_temporary_replacement(Test_suite& suite) +{ + Fixture fixture; + Native_path caller_file; +#ifdef _WIN32 + bool replacement_blocked = false; +#endif + mexce::issuer::Test_hooks hooks; + hooks.after_temporaries_written = [&](const Native_path&, const Native_path& key) { +#ifdef _WIN32 + caller_file = key; + replacement_blocked = DeleteFileW(key.c_str()) == FALSE; +#else + (void)key; + caller_file = fixture.key; + write_bytes(caller_file, bytes("caller-owned")); +#endif + }; +#ifdef _WIN32 + mexce::issuer::protect_expression( + fixture.expression, fixture.schema, fixture.program, fixture.key, + mexce::issuer::Test_failure::NONE, &hooks); + suite.expect(replacement_blocked, + "open Windows temporary identity blocks pathname replacement"); + suite.expect(exists(fixture.program) && exists(fixture.key), + "blocked Windows temporary replacement preserves publication"); +#else + suite.expect_issuer_error("Linux final-name collision", [&] { + mexce::issuer::protect_expression( + fixture.expression, fixture.schema, fixture.program, fixture.key, + mexce::issuer::Test_failure::NONE, &hooks); + }); + suite.expect(file_size(caller_file) == 12, + "Linux final-name collision does not overwrite the caller file"); + suite.expect(!exists(fixture.program) && exists(fixture.key), + "Linux final-name collision prevents program publication"); + suite.expect(temporary_count(fixture.root) == 0, + "Linux unpublished unnamed temporaries leave no directory entries"); + remove_path(caller_file); +#endif +} + + +#ifndef _WIN32 +void test_unsupported_unnamed_temporary_filesystem(Test_suite& suite) +{ + Fixture fixture; + const std::string suffix = std::to_string(static_cast(getpid())); + const Native_path program = "/proc/mexce-program-" + suffix; + const Native_path key = "/proc/mexce-key-" + suffix; + try { + mexce::issuer::protect_expression( + fixture.expression, fixture.schema, program, key); + suite.expect(false, "unsupported unnamed-temporary filesystem rejects"); + } + catch (const mexce::issuer::Issuer_error& error) { + suite.expect(std::string(error.what()).find("unnamed temporary") != + std::string::npos, + "unsupported unnamed-temporary filesystem reports the required primitive"); + } +} +#endif + + +} // namespace + + +int main() +{ + Test_suite suite; + suite.run("success and permissions", [&] { test_success_and_permissions(suite); }); + suite.run("expression validation", [&] { test_expression_validation(suite); }); + suite.run("schema validation", [&] { test_schema_validation(suite); }); + suite.run("output validation", [&] { test_output_validation(suite); }); + suite.run("publication failures", [&] { test_publication_failures(suite); }); + suite.run("reparse traversal", [&] { test_reparse_traversal(suite); }); + suite.run("ancestor swap", [&] { test_ancestor_swap(suite); }); + suite.run("temporary replacement", [&] { test_temporary_replacement(suite); }); +#ifndef _WIN32 + suite.run("unsupported unnamed temporary filesystem", [&] { + test_unsupported_unnamed_temporary_filesystem(suite); + }); +#endif + return suite.finish(); +} diff --git a/tools/mexce_protect.cpp b/tools/mexce_protect.cpp new file mode 100644 index 0000000..3586dd3 --- /dev/null +++ b/tools/mexce_protect.cpp @@ -0,0 +1,27 @@ +#include "mexce_protect_lib.h" + +#include + + +#ifdef _WIN32 +int wmain(int argc, wchar_t* argv[]) +#else +int main(int argc, char* argv[]) +#endif +{ + if (argc != 5) { + std::fprintf(stderr, + "Usage: mexce_protect " + " \n"); + return 2; + } + + try { + mexce::issuer::protect_expression(argv[1], argv[2], argv[3], argv[4]); + return 0; + } + catch (const std::exception& error) { + std::fprintf(stderr, "mexce_protect: %s\n", error.what()); + return 1; + } +} diff --git a/tools/mexce_protect_lib.cpp b/tools/mexce_protect_lib.cpp new file mode 100644 index 0000000..a7e5d6d --- /dev/null +++ b/tools/mexce_protect_lib.cpp @@ -0,0 +1,1204 @@ +#ifdef _WIN32 + #define NOMINMAX +#else + #ifndef _GNU_SOURCE + #define _GNU_SOURCE + #endif +#endif + +#include "mexce_protect_lib.h" + +#include "mexce_protected_encoder.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifdef _WIN32 + #include + #include +#else + #include + #include + #include +#endif + + +namespace mexce { +namespace issuer { +namespace { + + +constexpr size_t k_expression_limit = 1024 * 1024; +constexpr size_t k_schema_limit = 256 * 1024; + + +struct Output_paths +{ + Native_path program; + Native_path key; +}; + + +bool valid_utf8(const std::vector& bytes) +{ + size_t i = 0; + while (i < bytes.size()) { + const uint8_t first = bytes[i++]; + if (first <= 0x7f) { + continue; + } + + uint32_t value; + size_t continuation_count; + uint32_t minimum; + if ((first & 0xe0) == 0xc0) { + value = first & 0x1f; + continuation_count = 1; + minimum = 0x80; + } + else + if ((first & 0xf0) == 0xe0) { + value = first & 0x0f; + continuation_count = 2; + minimum = 0x800; + } + else + if ((first & 0xf8) == 0xf0) { + value = first & 0x07; + continuation_count = 3; + minimum = 0x10000; + } + else { + return false; + } + + if (continuation_count > bytes.size() - i) { + return false; + } + for (size_t j = 0; j < continuation_count; ++j) { + const uint8_t next = bytes[i++]; + if ((next & 0xc0) != 0x80) { + return false; + } + value = (value << 6) | (next & 0x3f); + } + if (value < minimum || value > 0x10ffff || + (value >= 0xd800 && value <= 0xdfff)) + { + return false; + } + } + return true; +} + + +std::vector parse_schema(const std::vector& bytes) +{ + if (std::find(bytes.begin(), bytes.end(), 0) != bytes.end()) { + throw Issuer_error("the binding schema contains a NUL byte"); + } + if (!valid_utf8(bytes)) { + throw Issuer_error("the binding schema is not valid UTF-8"); + } + + std::vector bindings; + std::set names; + std::set slots; + size_t line_start = 0; + while (line_start < bytes.size()) { + size_t line_end = line_start; + while (line_end < bytes.size() && bytes[line_end] != '\n') { + ++line_end; + } + size_t content_end = line_end; + if (content_end > line_start && bytes[content_end - 1] == '\r') { + --content_end; + } + if (content_end == line_start) { + throw Issuer_error("the binding schema contains a blank line"); + } + + const auto equals = std::find( + bytes.begin() + line_start, bytes.begin() + content_end, '='); + if (equals == bytes.begin() + content_end) { + throw Issuer_error("the binding schema contains an invalid entry"); + } + const size_t equals_offset = static_cast(equals - bytes.begin()); + const std::string name( + bytes.begin() + line_start, bytes.begin() + equals_offset); + if (name.empty() || name.size() > 255) { + throw Issuer_error("the binding schema contains an invalid name"); + } + if (!impl::valid_protected_name(name) || + impl::function_map().find(name) != impl::function_map().end() || + impl::built_in_constants_map().find(name) != + impl::built_in_constants_map().end()) + { + throw Issuer_error("the binding schema contains an invalid name"); + } + + const size_t slot_start = equals_offset + 1; + if (slot_start == content_end) { + throw Issuer_error("the binding schema contains an invalid slot"); + } + uint64_t slot = 0; + for (size_t i = slot_start; i < content_end; ++i) { + if (bytes[i] < '0' || bytes[i] > '9') { + throw Issuer_error("the binding schema contains trailing characters"); + } + const uint32_t digit = bytes[i] - '0'; + if (slot > ((std::numeric_limits::max)() - digit) / 10) { + throw Issuer_error("the binding schema slot is too large"); + } + slot = slot * 10 + digit; + } + + const uint32_t slot_value = static_cast(slot); + if (!names.insert(name).second) { + throw Issuer_error("the binding schema contains a duplicate name"); + } + if (!slots.insert(slot_value).second) { + throw Issuer_error("the binding schema contains a duplicate slot"); + } + bindings.push_back({name, slot_value}); + if (bindings.size() > 4096) { + throw Issuer_error("the binding schema contains too many entries"); + } + line_start = line_end == bytes.size() ? line_end : line_end + 1; + } + for (uint32_t slot = 0; slot < bindings.size(); ++slot) { + if (slots.find(slot) == slots.end()) { + throw Issuer_error("the binding schema contains sparse slots"); + } + } + return bindings; +} + + +std::string expression_text(std::vector bytes) +{ + if (bytes.size() >= 3 && bytes[0] == 0xef && + bytes[1] == 0xbb && bytes[2] == 0xbf) + { + throw Issuer_error("the expression starts with a UTF-8 BOM"); + } + if (std::find(bytes.begin(), bytes.end(), 0) != bytes.end()) { + throw Issuer_error("the expression contains a NUL byte"); + } + if (!bytes.empty() && bytes.back() == '\n') { + bytes.pop_back(); + if (!bytes.empty() && bytes.back() == '\r') { + bytes.pop_back(); + } + } + return std::string(bytes.begin(), bytes.end()); +} + + +#ifdef _WIN32 + + +class Handle +{ +public: + explicit Handle(HANDLE value = INVALID_HANDLE_VALUE) + : + m_value(value) + {} + + ~Handle() { close(); } + + Handle(Handle&& other) + : + m_value(other.m_value) + { + other.m_value = INVALID_HANDLE_VALUE; + } + + Handle& operator=(Handle&& other) + { + if (this != &other) { + close(); + m_value = other.m_value; + other.m_value = INVALID_HANDLE_VALUE; + } + return *this; + } + + HANDLE get() const { return m_value; } + bool valid() const { return m_value != INVALID_HANDLE_VALUE; } + + void close() + { + if (valid()) { + CloseHandle(m_value); + m_value = INVALID_HANDLE_VALUE; + } + } + +private: + Handle(const Handle&); + Handle& operator=(const Handle&); + + HANDLE m_value; +}; + + +NTSTATUS create_relative( + HANDLE parent, + const Native_path& name, + ACCESS_MASK access, + ULONG file_attributes, + ULONG share_access, + ULONG disposition, + ULONG options, + PSECURITY_DESCRIPTOR security_descriptor, + HANDLE& file) +{ + UNICODE_STRING native_name; + native_name.Buffer = const_cast(name.data()); + native_name.Length = static_cast(name.size() * sizeof(wchar_t)); + native_name.MaximumLength = native_name.Length; + OBJECT_ATTRIBUTES attributes; + InitializeObjectAttributes( + &attributes, &native_name, OBJ_CASE_INSENSITIVE, parent, + security_descriptor); + IO_STATUS_BLOCK status; + using Nt_create_file = NTSTATUS (NTAPI*)( + PHANDLE, ACCESS_MASK, POBJECT_ATTRIBUTES, PIO_STATUS_BLOCK, + PLARGE_INTEGER, ULONG, ULONG, ULONG, ULONG, PVOID, ULONG); + const auto create_file = reinterpret_cast(GetProcAddress( + GetModuleHandleW(L"ntdll.dll"), "NtCreateFile")); + file = INVALID_HANDLE_VALUE; + return create_file + ? create_file(&file, access, &attributes, &status, nullptr, + file_attributes, share_access, disposition, options, nullptr, 0) + : static_cast(-1); +} + + +std::vector read_limited(const Native_path& path, size_t limit, const char* kind) +{ + const DWORD attributes = GetFileAttributesW(path.c_str()); + if (attributes == INVALID_FILE_ATTRIBUTES || + (attributes & (FILE_ATTRIBUTE_DIRECTORY | FILE_ATTRIBUTE_REPARSE_POINT))) + { + throw Issuer_error(std::string("cannot safely open the ") + kind + " file"); + } + + Handle file(CreateFileW(path.c_str(), GENERIC_READ, FILE_SHARE_READ, nullptr, + OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OPEN_REPARSE_POINT, nullptr)); + if (!file.valid()) { + throw Issuer_error(std::string("cannot open the ") + kind + " file"); + } + + LARGE_INTEGER size; + if (!GetFileSizeEx(file.get(), &size) || size.QuadPart < 0 || + static_cast(size.QuadPart) > limit) + { + throw Issuer_error(std::string("the ") + kind + " file exceeds its size limit"); + } + std::vector bytes(static_cast(size.QuadPart)); + size_t offset = 0; + while (offset < bytes.size()) { + DWORD amount = 0; + const DWORD request = static_cast(std::min( + bytes.size() - offset, (std::numeric_limits::max)())); + if (!ReadFile(file.get(), bytes.data() + offset, request, &amount, nullptr) || + amount == 0) + { + throw Issuer_error(std::string("cannot read the ") + kind + " file"); + } + offset += amount; + } + uint8_t trailing = 0; + DWORD trailing_size = 0; + if (!ReadFile(file.get(), &trailing, 1, &trailing_size, nullptr)) { + throw Issuer_error(std::string("cannot finish reading the ") + kind + " file"); + } + if (trailing_size != 0) { + throw Issuer_error(std::string("the ") + kind + " file changed while reading"); + } + return bytes; +} + + +Native_path absolute_path(const Native_path& path) +{ + const DWORD needed = GetFullPathNameW(path.c_str(), 0, nullptr, nullptr); + if (needed == 0) { + throw Issuer_error("an output path cannot be normalized"); + } + std::vector buffer(needed); + const DWORD written = GetFullPathNameW( + path.c_str(), needed, buffer.data(), nullptr); + if (written == 0 || written >= needed) { + throw Issuer_error("an output path cannot be normalized"); + } + Native_path result(buffer.data(), written); + std::replace(result.begin(), result.end(), L'/', L'\\'); + return result; +} + + +Native_path parent_path(const Native_path& path) +{ + const size_t slash = path.find_last_of(L'\\'); + if (slash == Native_path::npos || slash + 1 == path.size()) { + throw Issuer_error("an output path has no filename"); + } + Native_path parent = path.substr(0, slash); + wchar_t volume[MAX_PATH]; + if (GetVolumePathNameW(path.c_str(), volume, MAX_PATH)) { + Native_path root(volume); + Native_path parent_with_separator = parent + L'\\'; + if (CompareStringOrdinal(parent_with_separator.c_str(), -1, + root.c_str(), -1, TRUE) == CSTR_EQUAL) + { + parent = root; + } + } + return parent; +} + + +void validate_windows_filename(const Native_path& path) +{ + if (path.compare(0, 4, L"\\\\?\\") == 0 || + path.compare(0, 4, L"\\\\.\\") == 0) + { + throw Issuer_error("an output path uses an unsupported device namespace"); + } + const size_t slash = path.find_last_of(L'\\'); + const Native_path filename = path.substr(slash + 1); + if (filename.empty() || filename.back() == L'.' || filename.back() == L' ' || + filename.find_first_of(L"<>:\"|?*") != Native_path::npos) + { + throw Issuer_error("an output filename has indeterminate identity"); + } + for (const wchar_t value : filename) { + if (value < 32) { + throw Issuer_error("an output filename has indeterminate identity"); + } + } + + Native_path stem = filename.substr(0, filename.find(L'.')); + for (wchar_t& value : stem) { + if (value >= L'a' && value <= L'z') { + value = static_cast(value - L'a' + L'A'); + } + } + const bool numbered_device = stem.size() == 4 && + (stem.compare(0, 3, L"COM") == 0 || stem.compare(0, 3, L"LPT") == 0) && + stem[3] >= L'1' && stem[3] <= L'9'; + if (stem == L"CON" || stem == L"PRN" || stem == L"AUX" || stem == L"NUL" || + numbered_device) + { + throw Issuer_error("an output filename uses a reserved device name"); + } +} + + +bool same_path(const Native_path& left, const Native_path& right) +{ + return CompareStringOrdinal( + left.c_str(), -1, right.c_str(), -1, TRUE) == CSTR_EQUAL; +} + + +Handle open_pinned_directory(const Native_path& path) +{ + Handle directory(CreateFileW(path.c_str(), FILE_READ_ATTRIBUTES, + FILE_SHARE_READ | FILE_SHARE_WRITE, nullptr, OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT, nullptr)); + FILE_ATTRIBUTE_TAG_INFO attributes; + if (!directory.valid() || !GetFileInformationByHandleEx( + directory.get(), FileAttributeTagInfo, &attributes, sizeof(attributes)) || + !(attributes.FileAttributes & FILE_ATTRIBUTE_DIRECTORY) || + (attributes.FileAttributes & FILE_ATTRIBUTE_REPARSE_POINT)) + { + throw Issuer_error("an output path traverses an unsafe component"); + } + return directory; +} + + +Handle open_pinned_child(HANDLE parent, const Native_path& name) +{ + HANDLE raw_directory = INVALID_HANDLE_VALUE; + const NTSTATUS result = create_relative( + parent, name, FILE_READ_ATTRIBUTES | SYNCHRONIZE, + FILE_ATTRIBUTE_NORMAL, FILE_SHARE_READ | FILE_SHARE_WRITE, + FILE_OPEN, FILE_DIRECTORY_FILE | FILE_SYNCHRONOUS_IO_NONALERT | + FILE_OPEN_REPARSE_POINT, + nullptr, raw_directory); + Handle directory(result < 0 ? INVALID_HANDLE_VALUE : raw_directory); + FILE_ATTRIBUTE_TAG_INFO file_attributes; + if (!directory.valid() || !GetFileInformationByHandleEx( + directory.get(), FileAttributeTagInfo, + &file_attributes, sizeof(file_attributes)) || + !(file_attributes.FileAttributes & FILE_ATTRIBUTE_DIRECTORY) || + (file_attributes.FileAttributes & FILE_ATTRIBUTE_REPARSE_POINT)) + { + throw Issuer_error("an output path traverses an unsafe component"); + } + return directory; +} + + +std::vector pin_ancestors(const Native_path& path) +{ + wchar_t volume[MAX_PATH]; + if (!GetVolumePathNameW(path.c_str(), volume, MAX_PATH)) { + throw Issuer_error("an output path volume cannot be identified"); + } + const Native_path root(volume); + const Native_path parent = parent_path(path); + if (parent.size() < root.size()) { + throw Issuer_error("an output path parent cannot be identified"); + } + + std::vector directories; + directories.push_back(open_pinned_directory(root)); + size_t start = root.size(); + while (start < parent.size()) { + const size_t slash = parent.find(L'\\', start); + const size_t end = slash == Native_path::npos ? parent.size() : slash; + if (end > start) { + const Native_path component = parent.substr(start, end - start); + directories.push_back( + open_pinned_child(directories.back().get(), component)); + } + if (slash == Native_path::npos) { + break; + } + start = slash + 1; + } + return directories; +} + + +bool entry_exists(HANDLE directory, const Native_path& name) +{ + HANDLE raw_file = INVALID_HANDLE_VALUE; + const NTSTATUS result = create_relative( + directory, name, FILE_READ_ATTRIBUTES | SYNCHRONIZE, + FILE_ATTRIBUTE_NORMAL, + FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, + FILE_OPEN, FILE_SYNCHRONOUS_IO_NONALERT | FILE_OPEN_REPARSE_POINT, + nullptr, raw_file); + Handle file(result < 0 ? INVALID_HANDLE_VALUE : raw_file); + if (file.valid()) { + return true; + } + using Status_to_error = ULONG (NTAPI*)(NTSTATUS); + const auto status_to_error = reinterpret_cast(GetProcAddress( + GetModuleHandleW(L"ntdll.dll"), "RtlNtStatusToDosError")); + const ULONG error = status_to_error ? status_to_error(result) : ERROR_GEN_FAILURE; + if (error == ERROR_FILE_NOT_FOUND || error == ERROR_PATH_NOT_FOUND) { + return false; + } + throw Issuer_error("an output path identity is indeterminate"); +} + + +struct Restricted_security +{ + Restricted_security() + : + attributes(), + descriptor(), + token_user(), + acl() + { + Handle token; + HANDLE raw_token = INVALID_HANDLE_VALUE; + if (!OpenProcessToken(GetCurrentProcess(), TOKEN_QUERY, &raw_token)) { + throw Issuer_error("cannot establish owner-only key permissions"); + } + token = Handle(raw_token); + + DWORD user_size = 0; + GetTokenInformation(token.get(), TokenUser, nullptr, 0, &user_size); + if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) { + throw Issuer_error("cannot establish owner-only key permissions"); + } + token_user.resize(user_size); + if (!GetTokenInformation( + token.get(), TokenUser, token_user.data(), user_size, &user_size)) + { + throw Issuer_error("cannot establish owner-only key permissions"); + } + const PSID sid = reinterpret_cast(token_user.data())->User.Sid; + acl.resize(sizeof(ACL) + sizeof(ACCESS_ALLOWED_ACE) + GetLengthSid(sid)); + PACL native_acl = reinterpret_cast(acl.data()); + if (!InitializeAcl(native_acl, static_cast(acl.size()), ACL_REVISION) || + !AddAccessAllowedAce( + native_acl, ACL_REVISION, GENERIC_ALL, sid) || + !InitializeSecurityDescriptor( + &descriptor, SECURITY_DESCRIPTOR_REVISION) || + !SetSecurityDescriptorOwner(&descriptor, sid, FALSE) || + !SetSecurityDescriptorDacl(&descriptor, TRUE, native_acl, FALSE)) + { + throw Issuer_error("cannot establish owner-only key permissions"); + } + if (!SetSecurityDescriptorControl( + &descriptor, SE_DACL_PROTECTED, SE_DACL_PROTECTED)) + { + throw Issuer_error("cannot protect owner-only key permissions"); + } + attributes.nLength = sizeof(attributes); + attributes.lpSecurityDescriptor = &descriptor; + attributes.bInheritHandle = FALSE; + } + + SECURITY_ATTRIBUTES attributes; + SECURITY_DESCRIPTOR descriptor; + std::vector token_user; + std::vector acl; +}; + + +class Pinned_outputs +{ +public: + explicit Pinned_outputs(const Output_paths& paths) + : + m_paths(paths), + m_program_directories(pin_ancestors(paths.program)), + m_key_directories(pin_ancestors(paths.key)), + m_program_name(paths.program.substr(paths.program.find_last_of(L'\\') + 1)), + m_key_name(paths.key.substr(paths.key.find_last_of(L'\\') + 1)) + { + BY_HANDLE_FILE_INFORMATION program_parent; + BY_HANDLE_FILE_INFORMATION key_parent; + if (!GetFileInformationByHandle( + m_program_directories.back().get(), &program_parent) || + !GetFileInformationByHandle(m_key_directories.back().get(), &key_parent)) + { + throw Issuer_error("an output path identity is indeterminate"); + } + const bool same_parent = + program_parent.dwVolumeSerialNumber == key_parent.dwVolumeSerialNumber && + program_parent.nFileIndexHigh == key_parent.nFileIndexHigh && + program_parent.nFileIndexLow == key_parent.nFileIndexLow; + if (same_path(m_paths.program, m_paths.key) || + (same_parent && same_path(m_program_name, m_key_name))) + { + throw Issuer_error("the program and key outputs identify the same path"); + } + + const bool program_exists = entry_exists( + m_program_directories.back().get(), m_program_name); + const bool key_exists = entry_exists( + m_key_directories.back().get(), m_key_name); + if (program_exists != key_exists) { + throw Issuer_error("a partial final output state already exists"); + } + if (program_exists) { + throw Issuer_error("the output destinations already exist"); + } + } + + const Native_path& program() const { return m_paths.program; } + const Native_path& key() const { return m_paths.key; } + const Native_path& program_name() const { return m_program_name; } + const Native_path& key_name() const { return m_key_name; } + HANDLE program_directory() const { return m_program_directories.back().get(); } + HANDLE key_directory() const { return m_key_directories.back().get(); } + +private: + Output_paths m_paths; + std::vector m_program_directories; + std::vector m_key_directories; + Native_path m_program_name; + Native_path m_key_name; +}; + + +Native_path temporary_path(const Native_path& final_path) +{ + uint8_t random[16]; + randombytes_buf(random, sizeof(random)); + static const wchar_t hex[] = L"0123456789abcdef"; + Native_path path = parent_path(final_path) + L"\\.mexce-"; + for (size_t i = 0; i < sizeof(random); ++i) { + path += hex[random[i] >> 4]; + path += hex[random[i] & 0x0f]; + } + path += L".tmp"; + return path; +} + + +Native_path collision_temporary_path(const Native_path& final_path) +{ + return parent_path(final_path) + L"\\.mexce-test-collision.tmp"; +} + + +class Temporary_file +{ +public: + Temporary_file( + HANDLE directory, + const Native_path& path, + bool restricted) + : + m_path(path), + m_file(), + m_cleanup_active(false) + { + std::unique_ptr security; + if (restricted) { + security.reset(new Restricted_security); + } + + const Native_path name = path.substr(path.find_last_of(L'\\') + 1); + HANDLE raw_file = INVALID_HANDLE_VALUE; + const NTSTATUS result = create_relative( + directory, name, GENERIC_WRITE | DELETE | SYNCHRONIZE, + FILE_ATTRIBUTE_TEMPORARY, 0, FILE_CREATE, + FILE_NON_DIRECTORY_FILE | FILE_SYNCHRONOUS_IO_NONALERT | + FILE_OPEN_REPARSE_POINT, + restricted ? security->attributes.lpSecurityDescriptor : nullptr, + raw_file); + if (result < 0 || raw_file == INVALID_HANDLE_VALUE) { + throw Issuer_error("cannot exclusively create an output temporary file"); + } + m_file = Handle(raw_file); + m_cleanup_active = true; + } + + ~Temporary_file() + { + if (m_cleanup_active) { + FILE_DISPOSITION_INFO disposition = {}; + disposition.DeleteFile = TRUE; + SetFileInformationByHandle( + m_file.get(), FileDispositionInfo, &disposition, sizeof(disposition)); + } + } + + HANDLE get() const { return m_file.get(); } + const Native_path& path() const { return m_path; } + + void publish(HANDLE directory, const Native_path& final_name) + { + const DWORD name_size = static_cast(final_name.size() * sizeof(wchar_t)); + struct Native_rename_information + { + BOOLEAN ReplaceIfExists; + HANDLE RootDirectory; + ULONG FileNameLength; + WCHAR FileName[1]; + }; + std::vector storage(sizeof(Native_rename_information) + name_size); + auto* information = + reinterpret_cast(storage.data()); + std::memset(information, 0, storage.size()); + information->ReplaceIfExists = FALSE; + information->RootDirectory = directory; + information->FileNameLength = name_size; + std::memcpy(information->FileName, final_name.data(), name_size); + using Nt_set_information_file = NTSTATUS (NTAPI*)( + HANDLE, PIO_STATUS_BLOCK, PVOID, ULONG, FILE_INFORMATION_CLASS); + const auto set_information = + reinterpret_cast(GetProcAddress( + GetModuleHandleW(L"ntdll.dll"), "NtSetInformationFile")); + IO_STATUS_BLOCK status; + // The public SDK's reduced FILE_INFORMATION_CLASS declaration omits + // native FileRenameInformation, whose stable class number is 10. + const auto rename_information_class = + static_cast(10); + const NTSTATUS result = set_information + ? set_information(m_file.get(), &status, information, + static_cast(storage.size()), rename_information_class) + : static_cast(-1); + if (result < 0) + { + throw Issuer_error("cannot publish an output without overwriting"); + } + m_cleanup_active = false; + } + +private: + Temporary_file(const Temporary_file&); + Temporary_file& operator=(const Temporary_file&); + + Native_path m_path; + Handle m_file; + bool m_cleanup_active; +}; + + +void write_all(HANDLE file, const uint8_t* bytes, size_t size) +{ + size_t offset = 0; + while (offset < size) { + DWORD written = 0; + const DWORD request = static_cast(std::min( + size - offset, (std::numeric_limits::max)())); + if (!WriteFile(file, bytes + offset, request, &written, nullptr) || written == 0) { + throw Issuer_error("cannot write an output temporary file"); + } + offset += written; + } + if (!FlushFileBuffers(file)) { + throw Issuer_error("cannot flush an output temporary file"); + } +} + + +#else + + +class File_descriptor +{ +public: + explicit File_descriptor(int value = -1) + : + m_value(value) + {} + + ~File_descriptor() { close(); } + + File_descriptor(File_descriptor&& other) + : + m_value(other.m_value) + { + other.m_value = -1; + } + + File_descriptor& operator=(File_descriptor&& other) + { + if (this != &other) { + close(); + m_value = other.m_value; + other.m_value = -1; + } + return *this; + } + + int get() const { return m_value; } + bool valid() const { return m_value >= 0; } + + void close() + { + if (valid()) { + ::close(m_value); + m_value = -1; + } + } + +private: + File_descriptor(const File_descriptor&); + File_descriptor& operator=(const File_descriptor&); + + int m_value; +}; + + +std::vector read_limited(const Native_path& path, size_t limit, const char* kind) +{ + File_descriptor file(open(path.c_str(), O_RDONLY | O_CLOEXEC | O_NOFOLLOW)); + if (!file.valid()) { + throw Issuer_error(std::string("cannot safely open the ") + kind + " file"); + } + struct stat status; + if (fstat(file.get(), &status) != 0 || !S_ISREG(status.st_mode) || + status.st_size < 0 || static_cast(status.st_size) > limit) + { + throw Issuer_error(std::string("the ") + kind + " file is invalid or too large"); + } + std::vector bytes(static_cast(status.st_size)); + size_t offset = 0; + while (offset < bytes.size()) { + const ssize_t amount = read(file.get(), bytes.data() + offset, bytes.size() - offset); + if (amount < 0 && errno == EINTR) { + continue; + } + if (amount <= 0) { + throw Issuer_error(std::string("cannot read the ") + kind + " file"); + } + offset += static_cast(amount); + } + uint8_t trailing = 0; + ssize_t trailing_size; + do { + trailing_size = read(file.get(), &trailing, 1); + } + while (trailing_size < 0 && errno == EINTR); + if (trailing_size < 0) { + throw Issuer_error(std::string("cannot finish reading the ") + kind + " file"); + } + if (trailing_size != 0) { + throw Issuer_error(std::string("the ") + kind + " file changed while reading"); + } + return bytes; +} + + +Native_path lexical_absolute(const Native_path& path) +{ + Native_path absolute = path; + if (absolute.empty() || absolute[0] != '/') { + std::vector cwd(PATH_MAX); + if (!getcwd(cwd.data(), cwd.size())) { + throw Issuer_error("an output path cannot be normalized"); + } + absolute = Native_path(cwd.data()) + "/" + absolute; + } + + std::vector components; + size_t start = 1; + while (start <= absolute.size()) { + const size_t slash = absolute.find('/', start); + const size_t end = slash == Native_path::npos ? absolute.size() : slash; + const std::string component = absolute.substr(start, end - start); + if (component.empty() || component == ".") { + // Nothing to retain. + } + else + if (component == "..") { + if (!components.empty()) { + components.pop_back(); + } + } + else { + components.push_back(component); + } + if (slash == Native_path::npos) { + break; + } + start = slash + 1; + } + + Native_path result = "/"; + for (size_t i = 0; i < components.size(); ++i) { + if (i != 0) { + result += '/'; + } + result += components[i]; + } + return result; +} + + +Native_path parent_path(const Native_path& path) +{ + const size_t slash = path.find_last_of('/'); + if (slash == Native_path::npos || slash + 1 == path.size()) { + throw Issuer_error("an output path has no filename"); + } + return slash == 0 ? "/" : path.substr(0, slash); +} + + +Native_path normalize_output(const Native_path& path) +{ + const Native_path absolute = lexical_absolute(path); + (void)parent_path(absolute); + return absolute; +} + + +class Pinned_directory +{ +public: + explicit Pinned_directory(const Native_path& path) + : + m_directory(open("/", O_RDONLY | O_DIRECTORY | O_CLOEXEC | O_NOFOLLOW)) + { + if (!m_directory.valid()) { + throw Issuer_error("an output path root cannot be opened"); + } + size_t start = 1; + while (start <= path.size()) { + const size_t slash = path.find('/', start); + const size_t end = slash == Native_path::npos ? path.size() : slash; + if (end > start) { + const std::string component = path.substr(start, end - start); + File_descriptor next(openat(m_directory.get(), component.c_str(), + O_RDONLY | O_DIRECTORY | O_CLOEXEC | O_NOFOLLOW)); + if (!next.valid()) { + throw Issuer_error("an output path traverses an unsafe component"); + } + m_directory = std::move(next); + } + if (slash == Native_path::npos) { + break; + } + start = slash + 1; + } + } + + int get() const { return m_directory.get(); } + +private: + File_descriptor m_directory; +}; + + +bool entry_exists(int directory, const Native_path& name) +{ + struct stat status; + if (fstatat(directory, name.c_str(), &status, AT_SYMLINK_NOFOLLOW) == 0) { + return true; + } + if (errno == ENOENT) { + return false; + } + throw Issuer_error("an output path identity is indeterminate"); +} + + +class Pinned_outputs +{ +public: + explicit Pinned_outputs(const Output_paths& paths) + : + m_paths(paths), + m_program_directory(parent_path(paths.program)), + m_key_directory(parent_path(paths.key)), + m_program_name(paths.program.substr(paths.program.find_last_of('/') + 1)), + m_key_name(paths.key.substr(paths.key.find_last_of('/') + 1)) + { + struct stat program_parent; + struct stat key_parent; + if (fstat(m_program_directory.get(), &program_parent) != 0 || + fstat(m_key_directory.get(), &key_parent) != 0) + { + throw Issuer_error("an output path identity is indeterminate"); + } + if (paths.program == paths.key || + (program_parent.st_dev == key_parent.st_dev && + program_parent.st_ino == key_parent.st_ino && + m_program_name == m_key_name)) + { + throw Issuer_error("the program and key outputs identify the same path"); + } + + const bool program_exists = entry_exists( + m_program_directory.get(), m_program_name); + const bool key_exists = entry_exists(m_key_directory.get(), m_key_name); + if (program_exists != key_exists) { + throw Issuer_error("a partial final output state already exists"); + } + if (program_exists) { + throw Issuer_error("the output destinations already exist"); + } + } + + const Native_path& program() const { return m_paths.program; } + const Native_path& key() const { return m_paths.key; } + const Native_path& program_name() const { return m_program_name; } + const Native_path& key_name() const { return m_key_name; } + int program_directory() const { return m_program_directory.get(); } + int key_directory() const { return m_key_directory.get(); } + +private: + Output_paths m_paths; + Pinned_directory m_program_directory; + Pinned_directory m_key_directory; + Native_path m_program_name; + Native_path m_key_name; +}; + + +class Temporary_file +{ +public: + Temporary_file(int directory, bool restricted) + : + m_directory(directory), + m_file() + { + const mode_t mode = restricted ? 0600 : 0666; + m_file = File_descriptor(openat( + directory, ".", O_TMPFILE | O_RDWR | O_CLOEXEC, mode)); + if (!m_file.valid()) { + if (errno == EOPNOTSUPP || errno == EINVAL || errno == EISDIR) { + throw Issuer_error( + "the output filesystem does not support required unnamed temporary files"); + } + throw Issuer_error("cannot create an unnamed output temporary file"); + } + if (restricted && fchmod(m_file.get(), 0600) != 0) { + throw Issuer_error("cannot establish owner-only key permissions"); + } + } + + int get() const { return m_file.get(); } + + void publish(const Native_path& final_name) + { + if (linkat(m_file.get(), "", m_directory, + final_name.c_str(), AT_EMPTY_PATH) == 0) + { + return; + } + int error = errno; + if (error == ENOENT || error == EPERM) { + // Linux documents /proc/self/fd with AT_SYMLINK_FOLLOW as the + // unprivileged equivalent of AT_EMPTY_PATH for O_TMPFILE inodes. + const std::string descriptor = + "/proc/self/fd/" + std::to_string(m_file.get()); + if (linkat(AT_FDCWD, descriptor.c_str(), m_directory, + final_name.c_str(), AT_SYMLINK_FOLLOW) == 0) + { + return; + } + error = errno; + } + if (error == EEXIST) { + throw Issuer_error("cannot publish an output without overwriting"); + } + if (error == ENOENT || error == EPERM || error == EOPNOTSUPP || + error == EINVAL) + { + throw Issuer_error( + "the output filesystem does not support required inode-bound publication"); + } + throw Issuer_error("cannot publish an output temporary file"); + } + +private: + Temporary_file(const Temporary_file&); + Temporary_file& operator=(const Temporary_file&); + + int m_directory; + File_descriptor m_file; +}; + + +void write_all(int file, const uint8_t* bytes, size_t size) +{ + size_t offset = 0; + while (offset < size) { + const ssize_t amount = write(file, bytes + offset, size - offset); + if (amount < 0 && errno == EINTR) { + continue; + } + if (amount <= 0) { + throw Issuer_error("cannot write an output temporary file"); + } + offset += static_cast(amount); + } + if (fsync(file) != 0) { + throw Issuer_error("cannot flush an output temporary file"); + } +} + + +#endif + + +Output_paths normalize_outputs( + const Native_path& program_path, + const Native_path& key_path) +{ +#ifdef _WIN32 + Output_paths paths = {absolute_path(program_path), absolute_path(key_path)}; + validate_windows_filename(paths.program); + validate_windows_filename(paths.key); +#else + Output_paths paths = {normalize_output(program_path), normalize_output(key_path)}; +#endif + return paths; +} + + +Protected_expression_bundle encode_bundle( + const std::string& expression, + const std::vector& bindings) +{ + try { + return encode_protected_expression( + expression, bindings, Protected_math_mode::STRICT); + } + catch (const std::exception&) { + // Parser diagnostics can contain source identifiers. The command-line + // issuer reports only the failed operation, never source content. + throw Issuer_error("the expression or binding schema could not be encoded"); + } +} + + +} // namespace + + +void protect_expression( + const Native_path& expression_path, + const Native_path& schema_path, + const Native_path& program_path, + const Native_path& key_path, + Test_failure failure, + Test_hooks* hooks) +{ + const std::string expression = expression_text( + read_limited(expression_path, k_expression_limit, "expression")); + const std::vector bindings = parse_schema( + read_limited(schema_path, k_schema_limit, "binding schema")); + const Output_paths path_names = normalize_outputs(program_path, key_path); + Pinned_outputs outputs(path_names); + if (hooks && hooks->after_output_directories_opened) { + hooks->after_output_directories_opened(); + } + + Protected_expression_bundle bundle = encode_bundle(expression, bindings); +#ifdef _WIN32 + const Native_path program_temporary = temporary_path(outputs.program()); + const Native_path key_temporary = failure == Test_failure::KEY_TEMPORARY_COLLISION + ? collision_temporary_path(outputs.key()) + : temporary_path(outputs.key()); + Temporary_file program_file( + outputs.program_directory(), program_temporary, false); + Temporary_file key_file(outputs.key_directory(), key_temporary, true); +#else + Temporary_file program_file(outputs.program_directory(), false); + Temporary_file key_file(outputs.key_directory(), true); +#endif + write_all(program_file.get(), bundle.program.data(), bundle.program.size()); + bundle.key.consume_bytes([&](const uint8_t* bytes, size_t size) { + if (size != 32) { + throw Issuer_error("the generated key has an invalid size"); + } + write_all(key_file.get(), bytes, size); + }); + if (hooks && hooks->after_temporaries_written) { +#ifdef _WIN32 + hooks->after_temporaries_written(program_file.path(), key_file.path()); +#else + hooks->after_temporaries_written(Native_path(), Native_path()); +#endif + } + + if (failure == Test_failure::BEFORE_PUBLICATION) { + throw Issuer_error("injected failure before publication"); + } +#ifdef _WIN32 + key_file.publish(outputs.key_directory(), outputs.key_name()); +#else + key_file.publish(outputs.key_name()); +#endif + if (failure == Test_failure::AFTER_KEY_PUBLICATION) { + throw Issuer_error("injected failure after key publication"); + } +#ifdef _WIN32 + program_file.publish(outputs.program_directory(), outputs.program_name()); +#else + program_file.publish(outputs.program_name()); +#endif +} + + +} // namespace issuer +} // namespace mexce diff --git a/tools/mexce_protect_lib.h b/tools/mexce_protect_lib.h new file mode 100644 index 0000000..85e428b --- /dev/null +++ b/tools/mexce_protect_lib.h @@ -0,0 +1,60 @@ +#ifndef MEXCE_PROTECT_LIB_H +#define MEXCE_PROTECT_LIB_H + +#include +#include +#include + + +namespace mexce { +namespace issuer { + + +#ifdef _WIN32 +using Native_path = std::wstring; +#else +using Native_path = std::string; +#endif + + +enum class Test_failure +{ + NONE, + KEY_TEMPORARY_COLLISION, + BEFORE_PUBLICATION, + AFTER_KEY_PUBLICATION, +}; + + +struct Test_hooks +{ + std::function after_output_directories_opened; + std::function + after_temporaries_written; +}; + + +class Issuer_error : public std::runtime_error +{ +public: + explicit Issuer_error(const std::string& message) + : + std::runtime_error(message) + {} +}; + + +void protect_expression( + const Native_path& expression_path, + const Native_path& schema_path, + const Native_path& program_path, + const Native_path& key_path, + Test_failure failure = Test_failure::NONE, + Test_hooks* hooks = nullptr); + + +} // namespace issuer +} // namespace mexce + + +#endif From 35166f94c16bf3381eaf7fe3bad10f76fe642865 Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 20:53:31 +0200 Subject: [PATCH 08/12] Complete protected performance acceptance --- .github/workflows/main.yml | 20 + CMakeLists.txt | 20 +- analysis/improvement_report.md | 4 + analysis/optimization_runbook.md | 621 ++++----------------------- analysis/performance_acceptance.md | 168 ++++++++ analysis/roadmap.md | 10 +- analysis/run_protected_benchmarks.py | 413 ++++++++++++++++++ test/protected_benchmark.cpp | 562 ++++++++++++++++++++---- 8 files changed, 1207 insertions(+), 611 deletions(-) create mode 100644 analysis/performance_acceptance.md create mode 100644 analysis/run_protected_benchmarks.py diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 4337cd5..bffc9c7 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -208,7 +208,26 @@ jobs: if: matrix.configuration == 'protected' run: ctest --test-dir build -C Release -R mexce_protected_benchmark --verbose > build/protected_benchmark_results.txt + - name: Run phase-isolated protected performance diagnostic on Linux + if: runner.os == 'Linux' && matrix.configuration == 'protected' + run: >- + python analysis/run_protected_benchmarks.py + build/protected_benchmark + build/protected_benchmark_results.json + --repeats 3 + --resource-repeats 1 + + - name: Run phase-isolated protected performance diagnostic on Windows + if: runner.os == 'Windows' && matrix.configuration == 'protected' + run: >- + python analysis/run_protected_benchmarks.py + build/Release/protected_benchmark.exe + build/protected_benchmark_results.json + --repeats 3 + --resource-repeats 1 + - name: Upload diagnostic results + if: always() uses: actions/upload-artifact@v4 with: name: diagnostics-${{ matrix.os }}-${{ matrix.compiler }}-${{ matrix.configuration }} @@ -216,3 +235,4 @@ jobs: path: | build/benchmark_results.txt build/protected_benchmark_results.txt + build/protected_benchmark_results.json diff --git a/CMakeLists.txt b/CMakeLists.txt index 8d76670..ae03aae 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -95,6 +95,7 @@ if(BUILD_TESTING) if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) set(MEXCE_PROTECTED_FIXTURE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/test/fixtures") + find_package(Python3 COMPONENTS Interpreter REQUIRED) add_executable(protected_codec_tests test/protected_codec_tests.cpp) target_link_libraries(protected_codec_tests PRIVATE mexce_protected) @@ -188,6 +189,11 @@ if(BUILD_TESTING) target_compile_definitions(benchmark PRIVATE BENCHMARK_COMPILER="${MEXCE_BENCHMARK_COMPILER}" BENCHMARK_COMPILER_FLAGS="${MEXCE_BENCHMARK_FLAGS}") + if(MEXCE_ENABLE_PROTECTED_EXPRESSIONS) + target_compile_definitions(protected_benchmark PRIVATE + BENCHMARK_COMPILER="${MEXCE_BENCHMARK_COMPILER}" + BENCHMARK_COMPILER_FLAGS="${MEXCE_BENCHMARK_FLAGS}") + endif() add_test(NAME mexce_unit_tests COMMAND unit_tests) add_test(NAME mexce_ordinary_no_sodium COMMAND ordinary_no_sodium_test) @@ -199,11 +205,15 @@ if(BUILD_TESTING) add_test(NAME mexce_mixed_translation_units COMMAND mixed_tu_test) add_test(NAME mexce_protected_decoder_fuzz_smoke COMMAND protected_decoder_fuzz_smoke) - add_test(NAME mexce_protected_benchmark COMMAND protected_benchmark) - add_test(NAME mexce_protected_maximum_valid - COMMAND protected_benchmark maximum-valid) - add_test(NAME mexce_protected_late_invalid - COMMAND protected_benchmark late-invalid) + add_test(NAME mexce_protected_benchmark + COMMAND + ${Python3_EXECUTABLE} + ${CMAKE_CURRENT_SOURCE_DIR}/analysis/run_protected_benchmarks.py + $ + ${CMAKE_CURRENT_BINARY_DIR}/protected_benchmark_ctest.json + --repeats 1 + --resource-repeats 1 + --diagnostic-only) if(MEXCE_BUILD_ISSUER_TOOLS) add_test(NAME mexce_protect_utility COMMAND mexce_protect_tests) endif() diff --git a/analysis/improvement_report.md b/analysis/improvement_report.md index 1da7571..86bc077 100644 --- a/analysis/improvement_report.md +++ b/analysis/improvement_report.md @@ -5,6 +5,10 @@ full architecture map of mexce.h, benchmark-infrastructure audit, 4 performance- per-idea adversarial verification against the actual code, reorganization proposal, and an optimization-process runbook with a completeness-critic pass. +Status: historical reference only. Its proposed optimization process and critic +appendix are superseded by `analysis/optimization_runbook.md`; current release +evidence and limits are in `analysis/performance_acceptance.md`. + Companion files: - `analysis/optimization_runbook.md` — the iterative benchmark-driven optimization process (with required amendments appended) - `analysis/perf_ideas_verified.json` — full detail for every verified performance idea (seed for the runbook's idea bank) diff --git a/analysis/optimization_runbook.md b/analysis/optimization_runbook.md index 8405127..be33dc6 100644 --- a/analysis/optimization_runbook.md +++ b/analysis/optimization_runbook.md @@ -1,532 +1,89 @@ -# mexce Benchmark-Driven Optimization Runbook - -A deterministic optimize → measure → record → decide loop. Every command is copy-pasteable. Every decision is a mechanical rule. Negative results are recorded with the same rigor as positive ones — they are the "which way makes no sense" data. - -**Verified ground truth this runbook is built on** (re-checked 2026-06-10): -- `test/benchmark.cpp` — eval timing is round-robin median (chunks of 1024, 3/5/7 adaptive rounds, `select_representative_timing` at line 459); **compile time is a single unrepeated sample** (lines 557–560); comprehensive mode always `return 0` (line 1642); CLI: `[--comprehensive|--single] [--x87|--sse2] [--fast-math] [iterations] [output_file]`, default 100000 iterations, text-only output. -- `analysis/record_benchmark_summary.py` — regexes hardwired to `us`/`ns`/`sec`/`ms` and to **single-mode** report lines; `SystemExit` on any miss; CSV keyed by branch name only. `analysis/asmd_optimizer_benchmark_summary.csv` shows 13 branches separated by 1-ns-quantized values (11.000 vs 12.000) — i.e., the previous campaign ranked branches on noise. -- `CMakeLists.txt` — `/O2` (MSVC) or `-O3 -DNDEBUG` (GCC/Clang), `BENCHMARK_COMPILER[_FLAGS]` defines, `run_benchmarks` target at 100 iterations. -- `.github/workflows/main.yml` — builds, ctests, runs 100-iteration benchmark, uploads artifact; compares nothing. -- `test/benchmark_results.h` has `kGoldenResultsCount = 44229`; nothing asserts it equals `kExpressionCount`. - ---- - -## 0. Roles and principles - -1. **The Windows dev box (MSVC Release) is the only timing authority.** CI (ubuntu-latest + windows-latest) is a *correctness* authority only — shared runners are too noisy for ns-scale deltas. Never mix numbers across hosts or compilers. -2. **One metric pipeline.** All decisions read machine-readable JSON emitted by the benchmark binary itself (`--json`). The legacy text report and regex scrapers (`record_benchmark_summary.py`, `render_benchmark_table.py`) are retired from the loop. -3. **No decision without a noise floor.** Every comparison is `delta vs. max(measured_noise, fixed_threshold)`. -4. **Every experiment leaves a ledger row** — accept, reject, or inconclusive. -5. **One hypothesis per experiment.** Never change `mexce.h` and the benchmark harness in the same experiment. - -**Metric definitions (used everywhere, no exceptions):** -- `eval_ns_per_call` (per expression) = round-median of `dur_ns` ÷ `iterations`, kept as **double** (the current integer rounding of `avg_ns` is the reason 13 branches were indistinguishable — never round to int). -- `compile_us` (per expression) = **median of `compile_rounds` samples** (new; see §1.2 — single-sample compile timing is fixed before the first experiment). -- Aggregate per config = mean over expressions of the per-expression values. -- Run-level value = the aggregate from one gate benchmark execution; **session value** = median across `--runs` executions. -- `noise_floor_pct(metric)` = `100 × (max − min) / median` of the run-level values across the 5 baseline runs (stored per metric, per config, per class). - ---- - -## 1. ONE-TIME SETUP - -### 1.1 File map (new and modified) - -| Path | Status | Purpose | -|---|---|---| -| `test/benchmark.cpp` | **modify** | add `--json`, `--compile-rounds`, `--warmup-rounds`, `--filter-file`, `--strict`; add `static_assert`; atomic JSON write | -| `tools/bench_gate.py` | **new** | single entry point: build → unit tests → benchmark runs → gates → ONE summary JSON | -| `tools/bench_diff.py` | **new** | noise-aware baseline-vs-candidate verdict (accept/reject/inconclusive) | -| `tools/record_experiment.py` | **new** | validated append to the experiment ledger | -| `tools/direction_map.py` | **new** | aggregates ledger → `analysis/direction_map.md` | -| `tools/check_integrity.py` | **new** | CI: ledger append-only check, ULP-reference presence, golden-count check | -| `analysis/experiments.jsonl` | **new, committed** | append-only experiment ledger (§2) | -| `analysis/backlog.md` | **new, committed** | prioritized hypothesis backlog (§3, step L2) | -| `analysis/baselines//baseline.json` | **new, committed** | per-host aggregate + per-class baseline with noise floors | -| `analysis/baselines//perexpr.json.gz` | **new, committed** | per-expression baseline medians (for localized-regression diffs) | -| `analysis/ulp_reference.json` | **new, committed** | expected ULP histogram per config — CI correctness fence | -| `analysis/direction_map.md` | **new, generated+committed** | the pathfinding output (§5) | -| `bench_runs/` | **new, gitignored** | raw per-run JSONs and verdicts | -| `.gitignore` | **modify** | add `bench_runs/`, `build-bench*/`, `build-repo-*/` | - -The three `build-repo-*-Debug/` IDE trees are never used for measurement (they are Debug). `build-bench/` is the one canonical measurement build. - -### 1.2 `test/benchmark.cpp` modifications (spec) - -New flags (all coexist with current ones; current behavior unchanged when absent): - -``` ---json Write machine-readable JSON (in addition to the text report). - Written to .tmp, fsync'd, then renamed to . - Top-level field "complete": true is the completion sentinel; - a crashed run leaves only the .tmp file. ---compile-rounds Default 5. Compile timing becomes median-of-N: each expression's - set_expression() is re-run on a fresh evaluator N times, - interleaved round-robin across the 1024-expression chunk - (same decorrelation as eval timing). The original - metadata-producing compile serves as warmup and is NOT timed. ---warmup-rounds Default 1. N extra eval timing rounds run first and discarded - (fixes cold-cache round adjacent to a 3-round median). ---filter-file Text file, one 0-based expression index per line; only these - expressions are benchmarked. Deterministic subset mode for - fast targeted probes. ---strict Nonzero exit (1) on any compile/eval failure in ANY mode, - including comprehensive (today comprehensive always returns 0). -``` - -One-line correctness fix, added near the top of `main`-adjacent code: - -```cpp -static_assert(mexce::benchmark_data::kGoldenResultsCount == mexce::benchmark_data::kExpressionCount, - "benchmark_results.h is stale: rerun test/result_generator.py"); -``` - -**`--json` schema (schema_version 1):** - -```json -{ - "schema_version": 1, - "complete": true, - "timestamp_utc": "2026-06-10T09:42:11Z", - "mode": "comprehensive", - "iterations": 100000, - "eval_rounds": 3, "compile_rounds": 5, "warmup_rounds": 1, - "compiler": "", - "compiler_flags": "", - "expression_count": 44229, - "configs": [{ - "name": "sse2", // sse2 | sse2_fm | x87 | x87_fm - "prefer_x87": false, "fast_math": false, - "compiled_count": 44229, "compile_fail_count": 0, "eval_fail_count": 0, - "sse2_backend_count": 44229, "x87_backend_count": 0, - "eval_ns_per_call_avg": 6.213, // double, NEVER integer-rounded - "compile_us_avg": 174.31, - "native_eval_ns_per_call_avg": 7.018, - "ulp_bins_vs_ref": {"exact": 20164, "b16": 23494, "b32": 0, "...": 0, "gt65536": 15}, - "ulp_bins_vs_native": { "...": 0 }, - "expressions": [ - { "i": 0, - "eval_ns_rounds": [612345, 609888, 615002], // raw dur_ns per round - "eval_ns_per_call": 6.10, - "compile_ns_samples": [171002, 168455, 170100, 169322, 172800], - "compile_ns_median": 170100, - "native_ns_per_call": 6.95, - "ulp_ref": 0, "backend": "sse2", "ok": true, "error": null } - ] - }] -} -``` - -Exit codes: `0` ok, `1` correctness failure under `--strict`, `2` bad arguments. - -### 1.3 `tools/bench_gate.py` — the single entry point (spec) - -``` -python tools/bench_gate.py - --build-dir DIR required; canonical build dir (build-bench locally, build in CI) - --cmake-config CFG default Release (multi-config generators / MSVC) - --runs N default 3; full benchmark repetitions in this session - --iterations N default 100000 - --mode full|quick|smoke default full - full : comprehensive, all 4 configs, all expressions - quick: --single one config, optional --filter-class, 20000 iters default - smoke: all configs, --iterations 200, --strict, ULP fence only (CI) - --config NAME quick mode only: sse2|sse2_fm|x87|x87_fm (default sse2) - --filter-class NAME quick mode only: restrict to one expression class (§1.6); - gate writes the index list to bench_runs/filters/.idx - and passes it as --filter-file - --baseline after the runs, write analysis/baselines// - (refuses if git tree is dirty); implies --runs 5 unless overridden - --update-ulp-reference rewrite analysis/ulp_reference.json from this run (clean tree only) - --label TEXT free-text tag stored in the summary - --allow-any-power skip the Windows power-plan check (never use for recorded runs) - --out DIR default bench_runs/ -``` - -**Behavior, in order (all steps logged):** -1. **Preflight**: record git SHA, branch, dirty flag (`git status --porcelain`); hostname; CPU model (`Get-CimInstance Win32_Processor` on Windows / `lscpu` on Linux); logical cores; OS; active power scheme (`powercfg /getactivescheme`). On Windows, **hard-fail (exit 4)** if the scheme is not High performance/Ultimate unless `--allow-any-power`. -2. **Build**: `cmake --build --config --target benchmark unit_tests`. Fail → exit 5. -3. **Flags fence**: extract `BENCHMARK_COMPILER`/`BENCHMARK_COMPILER_FLAGS` from a `--json` probe run (`benchmark --json tmp --filter-file 5`); the flags string must contain `/O2` or `-O3`; in full/quick mode it must exactly match the host baseline's string. Mismatch → exit 4. (This makes Debug-tree measurement impossible.) -4. **Unit tests**: run `unit_tests` binary directly. Fail → exit 5. -5. **Benchmark runs**: launch `benchmark` `--runs` times sequentially, each with `--json bench_runs//_rN.json --strict --warmup-rounds 1 --compile-rounds 5`, at HIGH priority class (psutil; fallback: normal, recorded). `host_id = __`, e.g. `IMAK-PC_msvc-19.40_O2`. -6. **Canary check**: a fixed canary set (expression indices `0, 443, 886, …` — every 443rd index, 100 total, committed inside bench_gate.py as a literal list) is timed in a `--filter-file` micro-run before run 1 and after run N. If `|after/before − 1| > 3%` → session unstable, exit 3, nothing recorded. -7. **Merge** the N run JSONs into ONE gate summary `bench_runs//_[_dirty].json`: - -```json -{ - "schema_version": 1, "complete": true, "gate_version": "1.0.0", - "run_id": "20260610T094211Z-a1b2c3d", - "host": {"host_id": "IMAK-PC_msvc-19.40_O2", "hostname": "IMAK-PC", - "cpu_model": "13th Gen Intel(R) Core(TM) i9-13900K", "logical_cores": 32, - "os": "Windows 11 10.0.22631", "power_scheme": "High performance", - "priority_boost": true}, - "git": {"commit": "", "branch": "exp/EXP-0007-rax-cache", "dirty": false}, - "build": {"compiler": "MSVC 19.40.33812", "flags": "/O2 /W4 /permissive- /bigobj", - "build_dir": "build-bench", "cmake_config": "Release"}, - "params": {"mode": "full", "runs": 3, "iterations": 100000, - "eval_rounds": 3, "compile_rounds": 5, "warmup_rounds": 1, "filter_class": null}, - "gates": {"unit_tests": "pass", "compile_failures": 0, "eval_failures": 0, - "ulp_fence": "pass", "canary_drift_pct": 0.8, "flags_match_baseline": true}, - "configs": { - "sse2": { - "eval_ns_per_call_runs": [6.23, 6.21, 6.19], "eval_ns_per_call": 6.21, - "compile_us_runs": [175.1, 174.3, 173.9], "compile_us": 174.3, - "native_eval_ns_per_call": 7.02, - "ulp_bins": {"exact": 20164, "b16": 23494, "gt65536": 15}, - "classes": { - "asmd": {"count": 18234, "eval_ns_per_call": 3.12, "compile_us": 121.4}, - "transcendental": {"count": 14102, "eval_ns_per_call": 11.8, "compile_us": 233.0}, - "power_root": {"count": 6120, "eval_ns_per_call": 8.4, "compile_us": 190.2}, - "logic_compare": {"count": 2410, "eval_ns_per_call": 4.9, "compile_us": 150.7}, - "other_func": {"count": 2105, "eval_ns_per_call": 7.7, "compile_us": 181.3}, - "constant_only": {"count": 1258, "eval_ns_per_call": 1.4, "compile_us": 96.0} - } - }, - "sse2_fm": {"...": 0}, "x87": {"...": 0}, "x87_fm": {"...": 0} - }, - "perexpr_file": "bench_runs/IMAK-PC_msvc-19.40_O2/20260610T094211Z-a1b2c3d_perexpr.json.gz" -} -``` - -(Per-class counts above are illustrative; real counts come from the classifier in §1.6.) -8. **Gates**: ULP fence = ulp_bins must equal `analysis/ulp_reference.json` per config exactly (unless `--update-ulp-reference`). Any compile/eval failure count > 0 → exit 2. -9. With `--baseline`: write `analysis/baselines//baseline.json` = the gate summary plus `noise_floor_pct` per (config × metric) and per (config × class × metric), computed from the 5 runs; write `perexpr.json.gz` with per-expression median-of-runs `eval_ns_per_call` and `compile_ns` arrays. - -**Gate exit codes**: 0 pass · 2 correctness gate failed · 3 noisy session (canary) · 4 environment/flags mismatch · 5 build/test failure. - -### 1.4 `tools/bench_diff.py` (spec) - -``` -python tools/bench_diff.py - --baseline analysis/baselines//baseline.json - --candidate bench_runs//.json - --primary : e.g. sse2:eval or sse2:compile (required) - --budget ::<+pct> repeatable; declared allowed regressions, - e.g. --budget sse2:compile:+5 - --ulp-budget :: repeatable; declared accuracy change for - fast-math experiments only, e.g. sse2_fm:exact:-100 - --out -``` - -Decision algorithm (mechanical, in order; first failure wins): -1. Candidate `gates.*` all pass and `flags_match_baseline` → else **reject (reason: gate)**. -2. Candidate internal spread check: for the primary metric, `(max−min)/median` of the candidate's run values must be ≤ 2 × baseline `noise_floor_pct` for that metric → else **noisy-session** (exit 3; re-run, do not decide). -3. ULP: for every config, bins must be identical to baseline unless covered by an explicit `--ulp-budget` → else **reject (reason: accuracy)**. -4. Per-class guardrail: for every config × class, regression must be ≤ `max(class noise_floor_pct, 3.0%)` → else **reject (reason: class-regression)**, naming the worst offender. -5. Secondary metrics: every non-primary (config × metric) aggregate may regress at most `max(its noise_floor_pct, 1.0%)` plus any declared `--budget` → else **reject (reason: undeclared-tradeoff)**. -6. Primary metric: `improvement_pct = 100 × (baseline − candidate)/baseline`. - - `improvement_pct > max(noise_floor_pct, X)` where **X = 1.0 for eval, X = 2.0 for compile** → **accept**. - - `improvement_pct < −max(noise_floor_pct, X)` → **reject (reason: slower)**. - - otherwise → **inconclusive**. - -Output `verdict.json`: `{decision, reasons[], primary:{metric, before, after, delta_pct, threshold_pct}, secondary_deltas{}, worst_class_regression:{config,class,pct}, top_regressed_expressions:[{i,expr,before_ns,after_ns}] (top 20 from perexpr diff), top_improved_expressions:[...]}` plus a printed Markdown table. Exit codes: 0 accept · 1 reject · 2 inconclusive · 3 noisy. - -### 1.5 `tools/record_experiment.py` and `tools/check_integrity.py` (spec) - -``` -python tools/record_experiment.py --id EXP-0007 --backlog B-012 --family rax-cache \ - --hypothesis "..." --change-summary "..." --decision accept --rationale "..." \ - --baseline --candidate --verdict -``` -Validates: id matches `EXP-\d{4}`, strictly greater than the last id in the ledger; decision ∈ {accept, reject, inconclusive}; pulls metrics/noise/deltas from the three JSONs (no hand-typed numbers); appends exactly one line to `analysis/experiments.jsonl`; refuses duplicates. - -`tools/check_integrity.py` (CI): (a) `git diff --unified=0 origin/master...HEAD -- analysis/experiments.jsonl` must contain only `+` lines (append-only fence); (b) `analysis/ulp_reference.json` exists and parses; (c) greps `kGoldenResultsCount` and `kExpressionCount` values from the headers and asserts equality (belt-and-suspenders for the static_assert). Nonzero exit on any violation. - -### 1.6 Expression classifier (spec — lives in `tools/bench_gate.py`) - -Deterministic, first-match-wins on the raw expression string; function tokens matched as `\b\s*\(`: -1. contains any of `sin cos tan asin acos atan sinh cosh tanh exp log ln log2 log10` → **transcendental** -2. contains `^` or any of `pow sqrt cbrt hypot` → **power_root** -3. contains any other `identifier(` call → **other_func** -4. contains any of `< > = !` → **logic_compare** -5. contains none of the standalone variables `a b c x y z w` → **constant_only** -6. else (only `+ - * /`, digits, parens, variables — the asmd_optimizer's target) → **asmd** - -Secondary size bucket (reported as a second table): operator+function count 1–2 **tiny**, 3–6 **small**, 7–15 **medium**, >15 **large**. - -### 1.7 Windows setup (the measurement host) — run once - -```powershell -cd C:\plms\bsd_licensed\mexce -cmake -S . -B build-bench -G "Visual Studio 17 2022" -A x64 -cmake --build build-bench --config Release --target benchmark unit_tests -powercfg /getactivescheme -powercfg /setactive 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c # High performance -pip install psutil -git checkout master -python tools\bench_gate.py --build-dir build-bench --cmake-config Release --baseline --runs 5 --iterations 100000 -git add analysis\baselines analysis\ulp_reference.json -git commit -m "Record benchmark baseline for IMAK-PC_msvc-19.40_O2 at $(git rev-parse --short HEAD)" -``` - -MinGW/llvm-mingw builds are permitted only as **separate host_ids** with their own baselines; their numbers are never compared against MSVC numbers. - -### 1.8 Linux CI changes — append to `.github/workflows/main.yml` after the ctest step - -```yaml - - name: Integrity checks (ledger append-only, golden counts, ULP reference) - run: python tools/check_integrity.py - - - name: Correctness gate (smoke, no timing authority) - run: python tools/bench_gate.py --build-dir build --cmake-config Release --mode smoke --iterations 200 --allow-any-power -``` - -Smoke mode asserts: unit tests pass, zero compile/eval failures across all 44229 expressions × 4 configs (`--strict`), and ULP bins identical to `analysis/ulp_reference.json`. CI **never** produces accept/reject timing verdicts. Any PR that intentionally changes accuracy must update `analysis/ulp_reference.json` in the same commit (via `--update-ulp-reference` locally) or CI fails — this closes the "row ykpt1y changed accuracy and nothing flagged it" hole. - ---- - -## 2. EXPERIMENT LEDGER - -**Path:** `analysis/experiments.jsonl` — committed, append-only, one JSON object per line (JSONL, not CSV: hypotheses contain commas; metrics are nested). Written only by `tools/record_experiment.py`; CI enforces append-only. - -**Fields:** `id, date_utc, host_id, backlog_id, family, hypothesis, change_summary, branch, commit, baseline_commit, baseline_run_id, candidate_run_id, accuracy_affecting, declared_budgets, params {iterations, runs_baseline, runs_candidate, escalated}, metrics_before, metrics_after, noise_floor_pct, delta_pct, worst_class_regression, ulp_delta, decision, rationale, verdict_file`. - -**Filled example row** (stored as one line; pretty-printed here): - -```json -{"id":"EXP-0007","date_utc":"2026-06-10T09:42:11Z","host_id":"IMAK-PC_msvc-19.40_O2", - "backlog_id":"B-012","family":"rax-cache", - "hypothesis":"Extending the shared RAX cache across mul/div node boundaries in asmd_optimizer removes one mov per chained term; predict >=2% eval improvement on class asmd under sse2, compile cost <=5%.", - "change_summary":"mexce.h: asmd_optimizer keeps RAX validity across compile_elist for mul/div nodes (+38/-7 lines)", - "branch":"exp/EXP-0007-rax-cache-muldiv","commit":"a1b2c3d4...","baseline_commit":"133abdd...", - "baseline_run_id":"20260608T071000Z-133abdd","candidate_run_id":"20260610T094211Z-a1b2c3d", - "accuracy_affecting":false,"declared_budgets":{"sse2:compile":"+5%"}, - "params":{"iterations":100000,"runs_baseline":5,"runs_candidate":3,"escalated":false}, - "metrics_before":{"sse2":{"eval_ns":6.21,"compile_us":174.3},"sse2_fm":{"eval_ns":5.94,"compile_us":171.0}, - "x87":{"eval_ns":7.85,"compile_us":188.2},"x87_fm":{"eval_ns":7.42,"compile_us":186.5}, - "ulp_exact_sse2":20164}, - "metrics_after":{"sse2":{"eval_ns":6.05,"compile_us":177.9},"sse2_fm":{"eval_ns":5.83,"compile_us":174.1}, - "x87":{"eval_ns":7.86,"compile_us":190.0},"x87_fm":{"eval_ns":7.44,"compile_us":188.1}, - "ulp_exact_sse2":20164}, - "noise_floor_pct":{"sse2:eval":0.7,"sse2:compile":1.8}, - "delta_pct":{"sse2:eval":-2.58,"sse2:compile":+2.07,"asmd@sse2:eval":-5.9}, - "worst_class_regression":{"config":"x87","class":"transcendental","pct":+0.4}, - "ulp_delta":"none", - "decision":"accept", - "rationale":"sse2 eval -2.58% > max(0.7% noise, 1.0% threshold); compile +2.07% within declared +5% budget; ULP bins identical in all 4 configs; worst class regression +0.4% < 3%; unit tests green.", - "verdict_file":"bench_runs/IMAK-PC_msvc-19.40_O2/20260610T094211Z-a1b2c3d_verdict.json"} -``` - ---- - -## 3. THE LOOP (mechanical algorithm) - -**Backlog file:** `analysis/backlog.md` — committed, a strict Markdown table; the LLM picks rows mechanically: - -```markdown -| ID | Pri | Family | Hypothesis (falsifiable: predicted Δ, target class, target config, budget) | Status | -|-------|-----|-------------------|------------------------------------------------------------------------------|--------| -| B-012 | 1 | rax-cache | RAX cache across mul/div boundaries: >=2% eval on asmd@sse2, compile <=+5% | done-accepted(EXP-0007) | -| B-013 | 1 | libm-call-overhead| Inline sin/cos range reduction stub: >=3% eval on transcendental@sse2 | open | -| B-014 | 2 | compile-speed | Arena-allocate parse nodes: >=5% compile_us all configs, eval unchanged | open | -``` -`Pri` ∈ {1,2,3}. `Status` ∈ {open, in-progress(EXP-id), done-accepted(EXP-id), done-rejected(EXP-id), blocked(reason)}. Every hypothesis MUST name: predicted direction+magnitude, primary `:`, target class, and budgets. - -### The algorithm - -- **L0 — Preflight.** On master, clean tree. Run §1.7 build commands. `powercfg /getactivescheme` must report High performance. -- **L1 — Baseline validity check.** A baseline is **stale** if any is true: no `analysis/baselines//` exists; `baseline_commit ≠ current master HEAD` for any commit that touched `mexce.h`; compiler/flags string differs; older than 14 days. If stale: - ```powershell - python tools\bench_gate.py --build-dir build-bench --baseline --runs 5 --iterations 100000 - ``` - Commit the updated `analysis/baselines/` files. -- **L2 — Pick ONE hypothesis.** From `analysis/backlog.md`: the `open` row with lowest `Pri`, ties broken by lowest `ID`, **skipping any row whose Family is marked DEAD_END in `analysis/direction_map.md`**. Set status `in-progress(EXP-NNNN)` where NNNN = last ledger id + 1. If the backlog is empty → run §5 direction map and add ≥3 new rows in PAYS_OFF or UNEXPLORED families before continuing. -- **L3 — Implement minimal patch.** `git checkout -b exp/EXP-NNNN-`. The diff touches `mexce.h` only (or harness only — never both). One conceptual change. -- **L4 — Quick probe (cheap early kill, ~30 s).** - ```powershell - python tools\bench_gate.py --build-dir build-bench --mode quick --config sse2 --filter-class asmd --iterations 20000 --runs 1 --label EXP-NNNN-probe - ``` - If the target class regresses > 5% vs its baseline value → go to L7 with `decision=reject`, evidence = probe (still record!). Probe results never justify *accept*. -- **L5 — Full gate.** - ```powershell - python tools\bench_gate.py --build-dir build-bench --runs 3 --iterations 100000 --label EXP-NNNN - ``` - Exit 3 (noisy/canary) → fix environment, re-run once; exit 3 again → record `inconclusive` with rationale "environment unstable", stop for the session. -- **L6 — Verdict.** - ```powershell - python tools\bench_diff.py --baseline analysis\baselines\IMAK-PC_msvc-19.40_O2\baseline.json --candidate bench_runs\IMAK-PC_msvc-19.40_O2\.json --primary sse2:eval --budget sse2:compile:+5 --out bench_runs\IMAK-PC_msvc-19.40_O2\_verdict.json - ``` - (`--primary` and `--budget` come verbatim from the backlog row.) Accept rule, restated: **accept iff** `improvement > max(noise_floor, X)` [X=1.0% eval / 2.0% compile] **AND** unit tests green **AND** zero new compile/eval failures **AND** ULP bins unchanged (or within declared `--ulp-budget` for fast-math-only experiments) **AND** no class regresses > max(class noise floor, 3%) **AND** no undeclared secondary regression > max(noise floor, 1%). -- **L7 — Act on the verdict.** - - **accept** → record ledger row; merge to master (`git checkout master && git merge --no-ff exp/EXP-NNNN-`); set backlog status `done-accepted(EXP-NNNN)`; **re-baseline** (L1 command, commit baselines); run §5. - - **reject** → record ledger row (full metrics — this is the valuable "doesn't work" data); set status `done-rejected(EXP-NNNN)`; `git checkout master` (keep the branch unmerged for reference); do NOT re-baseline. - - **inconclusive** → escalate **exactly once**: re-run gate with `--runs 6`, and extend the baseline with 2 extra runs (gate supports re-running `--baseline --runs 2 --append`; noise floors recomputed over all 7). Re-run bench_diff. If still inconclusive → record as **reject** with rationale "no measurable effect at N=6/7, threshold X%, noise floor F%". An idea that can't beat a 1% fence after 13 runs is, for pathfinding purposes, a dead direction. -- **L8 — Housekeeping.** After every accept and every 10 experiments: `python tools\direction_map.py`; commit `analysis/direction_map.md`, `analysis/experiments.jsonl`, `analysis/backlog.md` together. - ---- - -## 4. GUARDRAILS - -1. **Never tune to one class.** bench_diff rule 4 is non-overridable: no (config × class) eval or compile regression beyond `max(class_noise_floor, 3%)`, even if the global average improves. The verdict names the worst offender and the top-20 regressed expressions from the per-expression diff so the follow-up hypothesis is targeted, not guessed. -2. **CPU frequency / power.** Windows: gate refuses to run on Balanced (`powercfg /getactivescheme` checked every session; set with `powercfg /setactive 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c`). Desktop on AC. Don't run builds, browsers with video, or other benchmarks concurrently with the timing phase. The gate runs the binary at HIGH priority. -3. **Build-flags consistency.** The flags fence (§1.3 step 3) hard-fails if `BENCHMARK_COMPILER_FLAGS` differs from the baseline or lacks `/O2`//`-O3`. Debug IDE trees (`build-repo-*-Debug/`) physically cannot produce a recordable run. Cross-compiler comparisons are structurally impossible because baselines are keyed by `host_id` which embeds the compiler. -4. **Noisy-session detection and discard.** Three independent tripwires: (a) canary drift > 3% between session start/end → exit 3, session discarded; (b) candidate run spread > 2× baseline noise floor on the primary metric → no decision, re-run; (c) ULP fence and `--strict` failure counts make a corrupted run un-recordable. A discarded session is never written to the ledger as a decision — at most a note. -5. **Truncated-output hazard is closed** by the tmp+rename write and the `"complete": true` sentinel; bench_gate refuses any JSON lacking the sentinel (this retires the 648-byte-stub failure mode of `benchmark_results.txt`). -6. **Harness changes are experiments too.** Any edit to `test/benchmark.cpp`, the gate, or the classifier invalidates all baselines: bump `gate_version`, re-baseline before the next library experiment, and never land a harness change inside a library experiment branch. -7. **Golden data integrity.** The `static_assert` plus `check_integrity.py` in CI; if `test/benchmark_expressions.h` changes, `test/result_generator.py` must be re-run and `analysis/ulp_reference.json` regenerated in the same commit, or CI fails. - ---- - -## 5. PATHFINDING — the direction map - -**Command:** `python tools/direction_map.py` (no args). Reads `analysis/experiments.jsonl`, writes `analysis/direction_map.md` (committed). - -**Exact aggregation behavior:** -1. Group ledger rows by `family`. -2. Per family compute: `n_experiments`, `n_accept`, `n_reject`, `n_inconclusive`; `cumulative_accepted_improvement_pct` (sum of accepted primary-metric deltas — meaningful because each accept was measured against a fresh post-merge baseline); `best_single_delta_pct`; `mean_attempted_delta_pct` (all rows, including rejects — shows whether the family even moves the needle in either direction); `last_experiment_date`. -3. Verdict per family (mechanical): - - **PAYS_OFF**: `n_accept ≥ 2` AND `cumulative_accepted_improvement_pct > 1.0`. - - **DEAD_END**: `n_experiments ≥ 3` AND `n_accept = 0` AND `best_single_delta_pct ≤ 0` (i.e., three honest attempts, none ever beat noise in the right direction). - - **MIXED**: at least one accept and at least one reject, otherwise. - - **UNEXPLORED**: `n_experiments < 2`. -4. Emit `analysis/direction_map.md`: (a) family table sorted by `cumulative_accepted_improvement_pct` descending; (b) the last 10 ledger rows as a digest (id, family, primary delta vs threshold, decision); (c) a **"Recommended next"** section: the top-3 `open` backlog rows whose family verdict is PAYS_OFF or UNEXPLORED, in backlog order — this is the input to loop step L2. - -Example output table: - -```markdown -| Family | Verdict | Exp | Acc | Rej | Inc | Cum. accepted Δ | Best Δ | Mean attempted Δ | -|--------------------|-----------|-----|-----|-----|-----|-----------------|--------|------------------| -| rax-cache | PAYS_OFF | 4 | 3 | 1 | 0 | -5.8% | -2.6% | -1.9% | -| compile-speed | MIXED | 3 | 1 | 2 | 0 | -4.1% (compile) | -4.1% | -1.2% | -| libm-call-overhead | UNEXPLORED| 1 | 0 | 1 | 0 | 0.0% | -0.4% | -0.4% | -| x87-scheduling | DEAD_END | 3 | 0 | 3 | 0 | 0.0% | +0.1% | +0.6% | -``` - -DEAD_END families are skipped by L2 until a backlog row explicitly argues new information (set `Pri 3`, note "revisits DEAD_END because ") — dead ends stay visible and falsifiable, never silently retried. - ---- - -### Initial seeding (first session checklist) - -1. Implement §1.2 benchmark.cpp flags + static_assert; implement the five `tools/*.py` scripts to the specs above; add `.gitignore` entries — one PR, CI green. -2. Run §1.7 (build, power plan, 5-run baseline, `--update-ulp-reference`), commit baselines + reference. -3. Create `analysis/backlog.md` seeded from the asmd_optimizer campaign's open questions (families: `rax-cache`, `instruction-selection`, `constant-folding`, `compile-speed`, `libm-call-overhead`, `cse-x87`, `codegen-buffer`), and an empty `analysis/experiments.jsonl`. -4. Retroactively back-fill one ledger row per surviving `codex/*` branch from `analysis/asmd_optimizer_benchmark_summary.csv` with `decision: "inconclusive"` and rationale "pre-runbook measurement: 1-ns quantization, no noise floor" — so the direction map starts with honest history instead of false precision. -5. Enter the loop at L1. - - - ---- - -# APPENDIX: Required amendments before first use (completeness-critic findings) - -A critic agent checked every command and rule in this runbook against the actual repo. The runbook's factual ground truth verified clean, but the following 19 defects must be applied to the spec before an LLM executes it. Each entry is issue -> concrete fix. - -## A1 - -**Issue:** Sign-convention contradiction across the three decision surfaces. §1.4 defines improvement_pct = 100×(baseline−candidate)/baseline (positive = better). The §2 ledger example uses delta_pct with negative = better ('sse2:eval':-2.58 is the accepted improvement). §5's DEAD_END rule says 'best_single_delta_pct ≤ 0 (none ever beat noise in the right direction)' and PAYS_OFF requires 'cumulative_accepted_improvement_pct > 1.0' — but the §5 example table shows rax-cache as PAYS_OFF with Cum Δ = -5.8% (fails '> 1.0' under either convention) and x87-scheduling as DEAD_END with Best Δ = +0.1% (fails '≤ 0' under the ledger convention). An LLM implementing direction_map.py cannot satisfy both the rules and the example. - -**Fix:** Pick one canonical signed quantity — recommend improvement_pct (positive = better) — use it in verdict.json, ledger delta_pct, and direction_map rules alike. Restate: PAYS_OFF iff n_accept ≥ 2 AND cumulative_accepted_improvement_pct > 1.0; DEAD_END iff n_experiments ≥ 3 AND n_accept = 0 AND best_single_improvement_pct ≤ 0. Regenerate both the §2 ledger example and the §5 example table with consistent signs (rax-cache Cum +5.8, x87-scheduling Best −0.1). - -## A2 - -**Issue:** Hard-coded host_id 'IMAK-PC_msvc-19.40_O2' is factually wrong for the actual measurement host: `hostname` on this machine returns ANDREAS. The 'copy-pasteable' commands in §1.7 (commit message) and §3 L6 (--baseline/--candidate paths) embed IMAK-PC, and L6 also contains an unresolved placeholder with no specified way to obtain it, so the LLM must improvise both. - -**Fix:** Remove all literal host_ids from commands. Specify that bench_gate.py prints exactly two final stdout lines: 'HOST_ID=' and 'SUMMARY=', and rewrite L6 as: run bench_diff with --baseline analysis/baselines/$HOST_ID/baseline.json --candidate $SUMMARY, where both values are taken verbatim from the gate's last output lines. - -## A3 - -**Issue:** Initial-seeding step 4 is unexecutable as written. (a) `git branch -a` shows zero surviving codex/* branches (only master, claude/benchmark-noise-reduction-amOlm, origin/claude/library-review-AhdW4) — the 13 codex rows exist only in analysis/asmd_optimizer_benchmark_summary.csv with no commits, branches, or run JSONs. (b) record_experiment.py is specified to pull all metrics from three JSON files (baseline/candidate/verdict, 'no hand-typed numbers') and to validate commit/branch fields — none of which exist for the CSV rows. The tool as specified refuses the very rows step 4 demands. - -**Fix:** Either drop step 4 and have direction_map.py emit a static 'pre-runbook history' appendix read directly from the CSV, or add an explicit `record_experiment.py --backfill-csv analysis/asmd_optimizer_benchmark_summary.csv` mode that creates one row per CSV line (keyed by the branch-name column, not git branches) with commit/baseline_commit/run_id set to null, decision 'inconclusive', and family 'rax-cache', written before the append-only fence baseline is taken. - -## A4 - -**Issue:** First-baseline bootstrap deadlock in §1.7: gate step 8 says the ULP fence compares bins against analysis/ulp_reference.json 'unless --update-ulp-reference', but the §1.7 baseline command omits --update-ulp-reference and the reference file does not exist yet (analysis/ contains only the CSV and two legacy .py scripts). The very first run either fails the fence or the LLM must invent behavior for a missing reference; yet §1.7's git add stages analysis\ulp_reference.json. - -**Fix:** Add --update-ulp-reference to the §1.7 bench_gate command, and specify gate step 8 explicitly: 'if analysis/ulp_reference.json is absent and --update-ulp-reference was not passed → exit 4 with message; if --update-ulp-reference passed → write it and skip the comparison.' - -## A5 - -**Issue:** Seeding step 1 ('one PR, CI green') cannot be green: that PR adds the CI steps `python tools/check_integrity.py` (which requires analysis/ulp_reference.json to exist and parse) and the smoke gate (which compares against the same file), but the reference is only created in step 2 on the dev box. The first PR's CI necessarily fails its own new checks. - -**Fix:** Either generate and commit analysis/ulp_reference.json (and an initial empty analysis/experiments.jsonl) inside the step-1 PR by running the new binary locally before pushing, or specify bootstrap tolerance: check_integrity.py and smoke mode emit a warning and pass when ulp_reference.json is absent AND analysis/experiments.jsonl is empty, hard-failing once either exists. - -## A6 - -**Issue:** A single committed analysis/ulp_reference.json cannot be 'identical' on both CI matrix legs. The SSE2 backend calls libm for transcendentals (confirmed in CMakeLists/benchmark notes), so mexce-vs-reference ULP bins differ between glibc (ubuntu-latest) and MSVC UCRT (windows-latest and the dev box); additionally x87-config bins depend on hardware fsin/fcos/fpatan, which differ between the Intel and AMD CPUs GitHub assigns to runners. The §1.8 smoke fence 'ULP bins identical to analysis/ulp_reference.json' will be permanently red on at least one leg, and x87 rows can flap run-to-run on Linux. - -**Fix:** Key references per platform: analysis/ulp_reference/.json where platform ∈ {windows-msvc, linux-gnu} is derived from the compiler define; let smoke compare only against its own platform's file and bootstrap it via --update-ulp-reference in a one-time CI commit. For x87 configs on CI, either pin exactness only for sse2/sse2_fm and assert 'gt65536 count unchanged' for x87/x87_fm, or run the CI fence on sse2 configs only. Keep full 4-config exactness only on the dev-box host_id. - -## A7 - -**Issue:** check_integrity.py's append-only check runs `git diff origin/master...HEAD` in CI, but .github/workflows/main.yml uses actions/checkout@v4 with default fetch-depth 1 (verified: no fetch-depth key in the file), so origin/master is not available on PR builds and the diff command fails. - -**Fix:** The §1.8 workflow modification must also change the checkout step to `with: fetch-depth: 0` (or add an explicit `git fetch origin master` before the integrity step), and the spec should state the check is skipped when HEAD == origin/master (push-to-master builds). - -## A8 - -**Issue:** L7's inconclusive-escalation path invokes `bench_gate.py --baseline --runs 2 --append`, but §1.3's CLI spec defines no --append flag, and --baseline as specified overwrites analysis/baselines// and 'refuses if git tree is dirty' — semantics for merging 2 new runs into existing 5-run noise floors are nowhere defined. - -**Fix:** Add --append to the §1.3 CLI table: 'valid only with --baseline; requires an existing baseline.json for this host_id at the same git commit and gate_version; appends the new run-level values to the stored per-run arrays and recomputes all noise_floor_pct fields over the union (5+2=7 runs); exit 4 if commit or gate_version differs.' - -## A9 - -**Issue:** bench_diff exit code 3 (noisy candidate spread, rule 2) is unhandled by the loop: L7 branches only on accept/reject/inconclusive, and L5's retry rule covers only gate exit 3 (canary). If bench_diff returns 3 the LLM has no rule to follow. - -**Fix:** Add to L6: 'bench_diff exit 3 → re-run L5 (full gate) once and re-run L6; if bench_diff exits 3 again → record decision=inconclusive with rationale "candidate spread exceeded 2× baseline noise floor twice", set backlog status blocked(noisy-host), stop for the session.' - -## A10 - -**Issue:** The canary check (gate step 6) is underspecified: the micro-run's config, mode, iteration count, and round count are not given ('a --filter-file micro-run' is all the spec says), and drift is defined on an unstated aggregate. Also smoke mode is not exempted, so a 3% canary drift on shared CI runners — practically guaranteed — would fail CI with exit 3 even though CI is declared a correctness-only authority; and smoke inherits --runs default 3, tripling a run that needs N=1. - -**Fix:** Specify the canary run exactly: `benchmark --single --sse2 --json --filter-file bench_runs/canary.idx 20000` (canary.idx = the 100 committed indices), drift = |mean(eval_ns_per_call over canary set, after) / mean(before) − 1|. State that smoke mode skips step 6 entirely and forces --runs 1 regardless of the flag. - -## A11 - -**Issue:** Binary path resolution is never specified, but it differs by generator: with the prescribed VS generator the executables are build-bench/Release/benchmark.exe and build-bench/Release/unit_tests.exe, while CI's Linux Makefile build puts them at build/benchmark and build/unit_tests. Gate steps 3-6 ('run unit_tests binary directly', 'launch benchmark') force the implementing LLM to improvise the lookup. - -**Fix:** Add to §1.3: 'executable resolution: try //[.exe], then /[.exe]; exit 5 listing both attempted paths if neither exists.' - -## A12 - -**Issue:** The dirty-tree rule will misfire on this repo: preflight defines dirty via `git status --porcelain`, which includes untracked entries — the working tree currently has untracked .claude/ (and the build-repo-* dirs until the .gitignore PR lands), so every run is flagged dirty and `--baseline` ('refuses if git tree is dirty') permanently refuses. - -**Fix:** Define dirty as `git status --porcelain --untracked-files=no` producing any output. Record the untracked-file list separately as informational. - -## A13 - -**Issue:** check_integrity.py spec says it 'greps kGoldenResultsCount and kExpressionCount values from the headers and asserts equality', but kExpressionCount has no literal value: test/benchmark_expressions.h line 44242 defines it as sizeof(kExpressions)/sizeof(kExpressions[0]). A grep-based comparison is impossible as specified. - -**Fix:** Respecify: parse kGoldenResultsCount's literal (benchmark_results.h line 44246) and compare it to the count of initializer entries in the kExpressions array (count lines matching the entry regex between the array's braces), or drop this check and rely on the §1.2 static_assert, with CI's build step as the enforcing gate (the assert text 'mexce::benchmark_data::...' is namespace-correct — verified both headers use namespace mexce::benchmark_data). - -## A14 - -**Issue:** The flags-fence probe command is ambiguous: 'benchmark --json tmp --filter-file 5' — 'empty-3-expr-file' could mean an empty file or a file with 3 indices, the file's location/creation is unspecified, and behavior of --filter-file with an empty index list is undefined in §1.2. - -**Fix:** Specify: gate writes bench_runs/filters/probe.idx containing the three lines '0', '1', '2' before the probe, runs `benchmark --single --sse2 --json --filter-file bench_runs/filters/probe.idx 5`, and §1.2 must state that an empty --filter-file is an error (exit 2). - -## A15 - -**Issue:** L1's staleness rule is ambiguous and incomplete: 'baseline_commit ≠ current master HEAD for any commit that touched mexce.h' does not parse into a procedure, and it omits harness files even though guardrail 6 says any change to test/benchmark.cpp, the gate, or the classifier invalidates all baselines. - -**Fix:** Restate mechanically: stale iff (a) analysis/baselines// missing; OR (b) `git diff --name-only ..HEAD -- mexce.h test/benchmark.cpp test/benchmark_expressions.h test/benchmark_results.h tools/ CMakeLists.txt` is non-empty; OR (c) baseline gate_version ≠ current gate_version; OR (d) compiler/flags string differs; OR (e) baseline timestamp older than 14 days. - -## A16 - -**Issue:** Power-plan gate matches the localized display name ('High performance/Ultimate') from `powercfg /getactivescheme`, which fails on non-English Windows and on OEM-renamed schemes even when the GUID is correct. - -**Fix:** Match the GUID instead: parse the GUID from powercfg output and accept {8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c, e9a42b02-d5df-448d-aa00-03f14749eb61}; treat any other GUID as exit 4 unless --allow-any-power. - -## A17 - -**Issue:** The --json schema leaves the ULP bin keys as placeholders ('b16', 'b32', '...': 0) while the binary actually has 15 buckets — exact, then thresholds {16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536} plus an overflow bucket (benchmark.cpp lines 339-343). The exact-equality ULP fence needs byte-stable key names, so an implementer must not guess them. - -**Fix:** Enumerate the canonical key list in the schema: ["exact","le16","le32","le64","le128","le256","le512","le1024","le2048","le4096","le8192","le16384","le32768","le65536","gt65536"], and state that every key is always present (zero-filled) in both the benchmark JSON and ulp_reference.json. - -## A18 - -**Issue:** L2's empty-backlog branch ('run §5 and add ≥3 new rows in PAYS_OFF or UNEXPLORED families') is the one step with genuinely unbounded improvisation: the LLM must invent hypotheses, predicted magnitudes, and budgets with no source of candidates once the direction map's 'Recommended next' section is empty (which is exactly when this branch triggers). - -**Fix:** Bound it: maintain a committed analysis/idea_bank.md of pre-written candidate hypotheses (seeded by a human alongside seeding step 3); the rule becomes 'promote the top ≥3 idea-bank entries whose family is not DEAD_END into backlog rows verbatim; if the idea bank is empty, stop and emit REQUEST_HUMAN_INPUT instead of continuing the loop.' - -## A19 - -**Issue:** Minor spec bug in the append-only fence: 'the diff must contain only + lines' is unsatisfiable literally — unified diff output always contains header lines (diff --git, index, +++, ---, @@), so a naive implementation would always fail or the implementer must improvise the filter. - -**Fix:** Respecify: in the output of `git diff --unified=0 origin/master...HEAD -- analysis/experiments.jsonl`, there must be no line starting with '-' except the '--- ' file header, and every '+' content line must parse as JSON with an id greater than all prior ids. - -## Critic overall assessment - -The runbook's factual foundation is solid: every claim in its 'verified ground truth' section checks out against the repo (benchmark.cpp line numbers 459/557-560/~1643, CLI flags and 100000-iteration default, adaptive 3/5/7 rounds with 1024-expression chunks, kGoldenResultsCount=44229 with sizeof-derived kExpressionCount, CMakeLists target names benchmark/unit_tests/run_benchmarks with /O2 and -O3 -DNDEBUG and the BENCHMARK_COMPILER[_FLAGS] defines, the CI workflow contents, record_benchmark_summary.py's brittle regexes, and the CSV's 13 codex branches with 11/12-ns quantization — including the ykpt1y accuracy shift 20164→20325 it calls out). The CMake build commands are valid for this repo (cmake_minimum 3.16 permits multi-target --target; the explicit target_compile_options make the flags fence work even in CI's typeless configure), the powercfg GUID is the real High-performance GUID, the classifier's variable list matches the benchmark's bind() call, the decision thresholds are all concrete numbers (1.0%/2.0% primary, 3% class, 1% secondary, 2× spread, 3% canary, 5% probe, 14 days), and the ledger is fully specified with field list plus a filled example. However, the runbook is not yet executable without improvisation. The blocking defects are: a sign-convention contradiction that makes the direction-map rules and their own worked example mutually unsatisfiable; 'copy-pasteable' commands hard-coding host_id IMAK-PC when the actual host is ANDREAS, plus an unresolved placeholder; a seeding step that requires ledger rows for codex/* branches that no longer exist via a tool specified to refuse exactly that input; a first-baseline/first-PR bootstrap deadlock around the not-yet-existing ulp_reference.json; a single ULP reference that physically cannot match on both CI OSes (glibc vs UCRT libm, plus Intel-vs-AMD x87 microcode); a fetch-depth-1 checkout that breaks the append-only git check; and an --append flag invoked in the loop but absent from the gate's CLI spec. A second tier of underspecification (binary path resolution across multi/single-config generators, canary-run parameters, smoke-mode exemptions, dirty-tree definition that currently trips on the untracked .claude/ directory, the impossible grep of a sizeof-derived constant, ULP bin key names, and the unbounded empty-backlog hypothesis-generation branch) would each force an implementing LLM to make a judgment call. All are fixable with the concrete amendments listed; with them applied, the runbook would meet its no-improvisation bar. - +# Performance stabilization runbook + +This is the active Batch 6 procedure. The earlier speculative benchmark-gate +proposal remains available in Git history but is not normative: it depended on +unimplemented benchmark flags, host-specific ledgers, and mechanical review +rules that did not produce product evidence. + +## Clear baseline and candidate + +Use clean snapshots of the adopted baseline and candidate, with build and run +directories outside the repository. + +1. Configure Release x64 builds with the same compiler and flags. +2. Compare the emitted compile commands before measuring. +3. Set `OMP_NUM_THREADS=1`. +4. Run the existing SSE2 corpus once per binary and discard both warm-ups. +5. Retain at least seven pairs, alternating baseline-first and candidate-first. +6. Keep every text report and record compile time, evaluate time, process wall + time, exit code, compiled count, failure count, and backend counts. +7. Report median, median absolute deviation, minimum, maximum, and the + candidate/baseline median ratio for compile and evaluate. + +The release limits and the distributions that selected them are in +`analysis/performance_acceptance.md`. A timing result is invalid if compiler +flags, correctness counts, or backend counts differ between the matched pair. + +## Protected formulas and resources + +Build `protected_benchmark` in a protected Release configuration, then run: + +```text +python analysis/run_protected_benchmarks.py \ + --repeats 7 --resource-repeats 3 --timeout-seconds 180 +``` + +The runner first launches benchmark-only producer processes that write matched +programs and 32-byte keys into a private temporary directory. It then launches +encode, clear compile, prepared protected load/compile, clear evaluate, and +prepared protected load/evaluate in separate processes. Resource load processes +consume separately prepared maximum-valid and late-invalid files. The temporary +directory is removed on success or failure, and issuer work never contributes +to load-process peak working set. + +The runner alternates clear/protected order, preserves every process record, +checks independent numeric oracles, requires one identical stable backend for +all clear/protected compile/evaluate records in each case, and rejects missing +or non-positive WSS. Maximum-valid and late-invalid must report exactly 16,384 +records and 802,880 bytes with the expected acceptance semantics. + +The runner writes its report before returning failure for an exceeded limit, +so a failed gate retains its causal evidence. CI uses three formula repeats and +one resource repeat as a continuous release diagnostic; release acceptance uses +the seven/three command above on every supported compiler. + +CTest invokes the same preparation flow with one repeat in diagnostic mode. +Correctness, WSS, resource-schema, and backend failures remain fatal there; +numeric distribution ceilings are enforced by CI's three-repeat run and the +seven-repeat release run, not by a single-sample smoke. + +## Profiles + +Profile only after a repeated distribution exposes material movement. + +- Prefer a system sampler that can resolve the affected process and JIT code. +- If system sampling is unavailable, record the failure and use an available + instrumenting profiler for native compile/load paths. +- Use an evaluation-dominant run for evaluate and a one-iteration corpus run + for clear compile. +- Profile protected encode, protected load, maximum-valid, and late-invalid in + distinct processes. +- State explicitly when a profiler cannot symbolize JIT-generated code. + +Timing deltas alone do not authorize production changes. A retained +optimization needs a profile-supported cause, a single bounded change, +before/after measurements on the exposing workload, and affected correctness, +security, lifecycle, portability, and packaging gates. + +## Evidence hygiene + +Raw reports, build trees, profiler output, and scratch snapshots stay outside +the repository. The repository retains the reproducible harness, selected +limits, compact release conclusions, and CI job. Record all timeouts, tool +failures, discarded configurations, and unavailable profilers; baseline +provenance affects disposition but does not erase a failure. + +Review and planning metadata are not acceptance evidence. Reviewer counts, +verdict labels, commit trailers, and machine-local evidence paths cannot make a +failing workload pass. Stop an experiment when it does not change a product +conclusion instead of adding another review or bookkeeping layer. diff --git a/analysis/performance_acceptance.md b/analysis/performance_acceptance.md new file mode 100644 index 0000000..d1f6255 --- /dev/null +++ b/analysis/performance_acceptance.md @@ -0,0 +1,168 @@ +# Release performance acceptance + +Date: 2026-07-12 + +Candidate `f868ca40bad8218229f599ecb4fde1bcd058eaec` was compared with baseline +`ea7c1ce` before benchmark-only changes. Builds were Release x64, used identical +flags within each pair, set `OMP_NUM_THREADS=1`, discarded one warm-up, and ran +seven retained pairs in alternating order. Raw reports and structured JSON were +kept outside the repository; they are intentionally transient build evidence. +The durable runner reproduces the protected report shape. + +## Clear corpus + +The existing 44,229-expression SSE2 corpus was used without modification. +Every retained run compiled all 44,229 expressions with zero compilation +failures. Baseline and candidate backend counts matched on every run: + +- MSVC: 42,250 SSE2 and 709 x87; +- GCC and Clang: 42,441 SSE2 and 518 x87; and +- the remaining expressions were constant and selected no executable backend. + +| Compiler | Flags | Compile baseline/candidate median | Compile change | Evaluate baseline/candidate median | Evaluate change | +| --- | --- | ---: | ---: | ---: | ---: | +| MSVC 19.44.35228 | `/O2 /W4 /permissive- /bigobj` | 79.424/72.785 us | -8.36% | 7.134/7.592 ns | +6.42% | +| GCC 13.3.0 | `-O3 -DNDEBUG -Wall -Wextra -Wpedantic` | 55.702/54.089 us | -2.90% | 6.942/6.497 ns | -6.41% | +| Clang 18.1.3 | `-O3 -DNDEBUG -Wall -Wextra -Wpedantic` | 57.342/55.957 us | -2.42% | 7.312/7.299 ns | -0.18% | + +Median absolute deviations for baseline/candidate were 4.823/8.193 us and +0.292/0.367 ns on MSVC, 5.091/4.034 us and 0.530/0.255 ns on GCC, and +3.133/5.043 us and 0.447/0.426 ns on Clang. The MSVC evaluation interquartile +ranges overlap; the movement does not reproduce on either Linux compiler. + +The release limits are a candidate/baseline median ratio of 1.15 for compile +and 1.10 for evaluate, with identical correctness and backend counts. These +limits exceed the observed worst ratios of 0.976 and 1.064 and leave room for +the measured dispersion without accepting a persistent cross-platform +regression. + +## Protected formulas + +`run_protected_benchmarks.py` runs six formulas covering arithmetic, +transcendentals, powers, rounding, comparison, and mixed libm operations. +Encode, clear compile, protected compile, clear evaluate, and protected +evaluate each run in a separate process. Separate preparation processes write +the matched program and 32-byte key into a private temporary directory; load, +compile, evaluate, and resource processes only consume those prepared files. +Issuer allocation and encoding therefore cannot contribute to their peak +working set. Compile modes contain 25 timed +replacements after a discarded warm-up; evaluate modes contain 100,000 timed +evaluations after 1,000 discarded evaluations. Seven retained process runs per +formula and phase produced 210 formula records per compiler. + +All 630 formula records passed their mode checks; the 504 compile/evaluate +records also passed their independent numeric oracle. Within every case, all +retained clear/protected compile/evaluate records selected one stable identical +backend. Median protected/clear ranges were: + +| Compiler | Encode observed range | Compile ratio range | Evaluate ratio range | Maximum formula peak | +| --- | ---: | ---: | ---: | ---: | +| MSVC 19.44.35228 | 17.1-30.0 us | 1.32-1.94 | 0.94-1.03 | 4.68 MiB | +| GCC 13.3.0 | 36.4-58.7 us | 2.06-2.61 | 0.95-1.02 | 15.12 MiB | +| Clang 18.1.3 | 40.2-66.3 us | 2.12-2.65 | 0.97-1.00 | 15.12 MiB | + +The continuous limits are 1 ms encode time, a 4.0 compile ratio, a 1.75 +evaluate ratio, and 32 MiB peak working set. The evaluate allowance accounts +for an early single-run ratio of 1.61 that disappeared in the seven-run +distribution; it is not derived only from the final favorable medians. + +## Exact-limit resources + +The valid and late-invalid fixtures contain exactly 16,384 authenticated +records and 802,880 bytes. Each mode ran three times in an isolated process. +All valid artifacts loaded; all final-frame corruptions rejected. + +| Compiler | Maximum-valid min/median/max | Maximum-valid peak max | Late-invalid median/max | Late-invalid peak max | +| --- | ---: | ---: | ---: | ---: | +| MSVC 19.44.35228 | 32.38/32.44/32.71 s | 21.61 MiB | 31.65/39.93 ms | 14.86 MiB | +| GCC 13.3.0 | 10.49/11.86/11.94 s | 15.12 MiB | 18.64/25.18 ms | 15.12 MiB | +| Clang 18.1.3 | 10.75/11.39/11.97 s | 15.25 MiB | 22.04/26.46 ms | 15.25 MiB | + +The limits are 120 seconds and 32 MiB for maximum-valid, and 250 milliseconds +and 32 MiB for late-invalid. The time limits apply to every retained run, not +only the median. They leave roughly 3.7 times the slowest valid observation and +6.3 times the slowest invalid observation while still bounding release work. + +## Profiles and optimization decision + +GCC 13 `-O3 -pg` profiles used the same optimized sources with profiling +instrumentation. Clear compile self time was distributed across list teardown +(15%), the existing arithmetic optimizer (10%), argument linking (5%), +commutative normalization (5%), and semantic production (5%). Evaluation is +JIT-generated code, which gprof cannot symbolize; the evaluation-dominant run +showed no candidate-owned native hotspot. + +The protected profiles were rebuilt from the corrected harness with GCC 13 +`-O3 -pg`. Three producer processes prepared the case, maximum-valid, and +late-invalid files. Every gmon file below came from a separate consumer process +and directory, so issuer work is absent from load-process attribution. + +- Protected encode took 2.16 ms for 25 encodes and accumulated no 10 ms gprof + sample. It is below profiler resolution, so no hotspot attribution is made. +- Prepared protected load/compile took 1.77 ms for 25 loads and accumulated no + sample. This demonstrates no sampled hotspot, not that its native helpers are + free. +- Prepared protected evaluate took 1.08 ms for 100,000 evaluations and + accumulated no sample. Its generated code is also outside gprof's symbol + model, so no native or JIT hotspot is claimed. +- Prepared late-invalid took 31.7 ms and accumulated only two samples. gprof + assigned one to a function unreachable from that consumer and one to a + trivial reachable CLI-dispatch string comparison whose 10 ms attribution is + implausible. The samples are too sparse for attribution; only elapsed time, + rejection, and peak WSS are retained as evidence. +- Prepared maximum-valid ran long enough to profile. Sampled self time was + 43.79% list teardown, 36.74% list-of-lists insertion, 9.50% commutative + normalization, and 9.24% function disposal. Secretstream pull accounted for + 0.04%. This independently confirms optimizer/list work as the bounded + maximum-case cost. + +The transient external profile inventory retains the producer and consumer +results, `/usr/bin/time -v` reports, gmon files, gprof reports, commands, +toolchain flags, file sizes, and SHA-256 checksums. These profiles justify no +additional optimization. + +A provisional MSVC `__forceinline` annotation for the generated-code call was +tested against the unmodified candidate with discarded warm-ups and seven +alternating pairs. Evaluation changed from 7.610 ns median (0.305 ns MAD) to +8.309 ns (0.641 ns MAD), and compile changed from 70.147 us (4.662 us MAD) to +81.359 us (13.512 us MAD). The annotation was removed. No production +optimization is retained because no causal material regression remains. + +## Tool and run failures + +- An initial Windows configure selected MinGW for one build and failed to find + `cl` for the other. Both were discarded; fresh directories pinned MSVC. +- One combined resource command timed out after the old 802,831-byte valid + fixture completed, before late-invalid ran. Neither result was acceptance + evidence. +- A four-configuration Linux command timed out after completing the GCC + configurations. The missing Clang configurations were run separately. +- A GCC build orchestration command timed out, but its child completed. Target + existence and compile flags were verified before measurement. +- Cloning from a linked Windows worktree failed because its `.git` pointer was + not portable to WSL. Clean ext4 clones were made from the primary repository. +- Windows Performance Recorder could not enable CPU profiling (`0xc5585011`), + Linux `perf` was unavailable, and one `dumpbin` symbol scan timed out. GCC + gprof and `/usr/bin/time -v` were the available profile/resource tools. +- One experimental force-inline candidate failed its A/B gate and was removed. +- The first structured runner stopped on a rounding-oracle mismatch. MEXCE's + round-to-even result was confirmed, the harness oracle changed to + `nearbyint`, and the complete run was restarted. +- One protected-profile shell loop lost a variable across PowerShell and Bash; + it ran no benchmark. Explicit mode commands produced the retained profiles. +- A one-repeat Clang CTest diagnostic observed a 4.18 compile ratio against the + distribution limit of 4.0. CTest remains fatal for correctness, resource, + WSS, and backend failures but does not enforce numeric limits with one sample; + the three-repeat CI and seven-repeat release runners do. +- The first phase-isolated GCC release run observed a 4.011 compile ratio because + benchmark key allocation was inside the timed compiler call. Key ownership is + now prepared before timing, matching the public API boundary; all three + seven-repeat acceptance runs were restarted and passed without changing a + limit. +- The first durable report relied on a legacy combined-process protected + load/compile profile. It was not accepted as phase-isolated evidence. The GCC + profiling target and all protected profiles were rebuilt from the corrected + prepared-artifact harness; the conclusions above replace the legacy profile. + +Inspection-command path and optional-file errors produced no repository +changes and were not used as product evidence. diff --git a/analysis/roadmap.md b/analysis/roadmap.md index 76101e0..b2cdb70 100644 --- a/analysis/roadmap.md +++ b/analysis/roadmap.md @@ -2,10 +2,16 @@ Last updated: 2026-06-10, on branch `claude/verified-fixes-and-sse2-fusion`. -This is the tracking document for the improvement campaign. Detailed evidence lives in: +Status: historical reference only; this is no longer an active tracking or +implementation document. The active performance procedure and release decision +are in [`optimization_runbook.md`](optimization_runbook.md) and +[`performance_acceptance.md`](performance_acceptance.md). The unchecked items +below preserve the June campaign's state and are not current prerequisites. + +Historical evidence lives in: - [`improvement_report.md`](improvement_report.md) — branch harvest verdict, master bugs B1–B7, verified performance ideas, reorg plan, verification addendum -- [`optimization_runbook.md`](optimization_runbook.md) — the LLM-executable benchmark-driven optimization loop (apply its 19-amendment appendix before first use) +- Git history for the former optimization-runbook proposal and critic appendix; - [`perf_ideas_verified.json`](perf_ideas_verified.json) — all 22 verified performance ideas with code locations, impact estimates, and adversarial verdicts (the idea bank) Ground truth used throughout: 44,229-expression benchmark corpus; dev box (Windows, diff --git a/analysis/run_protected_benchmarks.py b/analysis/run_protected_benchmarks.py new file mode 100644 index 0000000..c0a3ca6 --- /dev/null +++ b/analysis/run_protected_benchmarks.py @@ -0,0 +1,413 @@ +#!/usr/bin/env python3 +"""Run protected benchmark phases in isolated processes and retain raw results.""" + +import argparse +import datetime +import json +import math +import os +import pathlib +import platform +import statistics +import subprocess +import sys +import tempfile +import time + + +CASE_MODES = ( + "protected-encode", + "clear-compile", + "protected-compile", + "clear-evaluate", + "protected-evaluate", +) +RESOURCE_MODES = ("maximum-valid", "late-invalid") +EXPECTED_CASE_COUNT = 6 +LIMITS = { + "protected_encode_ns": 1000 * 1000, + "compile_ratio": 4.0, + "evaluate_ratio": 1.75, + "formula_peak_working_set_bytes": 32 * 1024 * 1024, + "maximum_valid_elapsed_ns": 120 * 1000 * 1000 * 1000, + "maximum_valid_peak_working_set_bytes": 32 * 1024 * 1024, + "late_invalid_elapsed_ns": 250 * 1000 * 1000, + "late_invalid_peak_working_set_bytes": 32 * 1024 * 1024, +} + + +def invoke(binary, mode, case_index, timeout_seconds, extra_arguments=()): + command = [str(binary), mode] + if case_index is not None: + command.append(str(case_index)) + command.extend(str(argument) for argument in extra_arguments) + start = time.perf_counter_ns() + completed = subprocess.run( + command, + capture_output=True, + check=False, + text=True, + timeout=timeout_seconds, + ) + wall_ns = time.perf_counter_ns() - start + if completed.returncode != 0: + raise RuntimeError( + "command failed with exit code {}: {}\nstdout:\n{}\nstderr:\n{}".format( + completed.returncode, + " ".join(command), + completed.stdout, + completed.stderr, + ) + ) + lines = [line for line in completed.stdout.splitlines() if line.strip()] + if len(lines) != 1: + raise RuntimeError("expected one JSON line from: {}".format(" ".join(command))) + record = json.loads(lines[0]) + expected_mode = mode.replace("-", "_") + if record.get("mode") != expected_mode: + raise RuntimeError("benchmark returned an unexpected mode: {}".format(lines[0])) + if record.get("correct") is not True: + raise RuntimeError("benchmark correctness check failed: {}".format(lines[0])) + if not mode.startswith("prepare-") and mode != "case-count": + peak = record.get("peak_working_set_bytes") + if isinstance(peak, bool) or not isinstance(peak, int) or peak <= 0: + raise RuntimeError("benchmark returned an invalid peak: {}".format(lines[0])) + record["wall_ns"] = wall_ns + record["stderr"] = completed.stderr + return record + + +def distribution(values): + median = statistics.median(values) + deviations = [abs(value - median) for value in values] + return { + "median": median, + "mad": statistics.median(deviations), + "minimum": min(values), + "maximum": max(values), + } + + +def validate_resource_record(record, requested_mode, late_invalid): + expected_mode = requested_mode.replace("-", "_") + if record.get("mode") != expected_mode: + raise RuntimeError("resource record has an unexpected mode") + for name, expected in ( + ("record_count", 16384), + ("artifact_bytes", 802880), + ): + value = record.get(name) + if isinstance(value, bool) or not isinstance(value, int) or value != expected: + raise RuntimeError("resource record has an invalid {}".format(name)) + if late_invalid is not None: + accepted = record.get("accepted") + if not isinstance(accepted, bool) or accepted == late_invalid: + raise RuntimeError("resource record has invalid acceptance semantics") + + +def phase_values(records, case_index, mode, field): + return [ + record[field] + for record in records + if record.get("case_index") == case_index and record["mode"] == mode + ] + + +def summarize_cases(records, case_count): + summaries = [] + for case_index in range(case_count): + protected_encode = phase_values( + records, case_index, "protected_encode", "per_expression_ns") + clear_compile = phase_values( + records, case_index, "clear_compile", "per_expression_ns") + protected_compile = phase_values( + records, case_index, "protected_compile", "per_expression_ns") + clear_evaluate = phase_values( + records, case_index, "clear_evaluate", "per_expression_ns") + protected_evaluate = phase_values( + records, case_index, "protected_evaluate", "per_expression_ns") + clear_results = phase_values(records, case_index, "clear_evaluate", "result") + protected_results = phase_values( + records, case_index, "protected_evaluate", "result") + backend_values = [] + for mode in ( + "clear_compile", + "protected_compile", + "clear_evaluate", + "protected_evaluate", + ): + backend_values.extend(phase_values( + records, case_index, mode, "backend")) + phase_lengths = { + len(values) + for values in ( + protected_encode, + clear_compile, + protected_compile, + clear_evaluate, + protected_evaluate, + clear_results, + protected_results, + ) + } + if len(phase_lengths) != 1 or not clear_results: + raise RuntimeError("incomplete phase records for case {}".format(case_index)) + if any( + not math.isclose(clear, protected, rel_tol=1e-12, abs_tol=1e-12) + for clear, protected in zip(clear_results, protected_results) + ): + raise RuntimeError("clear/protected result mismatch for case {}".format(case_index)) + backend_comparable = ( + all( + not isinstance(backend, bool) and + isinstance(backend, int) and + backend > 0 + for backend in backend_values + ) and + len(set(backend_values)) == 1 + ) + + compile_distribution = distribution(clear_compile) + protected_compile_distribution = distribution(protected_compile) + evaluate_distribution = distribution(clear_evaluate) + protected_evaluate_distribution = distribution(protected_evaluate) + case_records = [ + record for record in records if record.get("case_index") == case_index + ] + summaries.append({ + "case_index": case_index, + "case": case_records[0]["case"], + "backend": backend_values[0] if backend_comparable else None, + "backend_comparable": backend_comparable, + "protected_encode_ns": distribution(protected_encode), + "clear_compile_ns": compile_distribution, + "protected_compile_ns": protected_compile_distribution, + "compile_ratio": ( + protected_compile_distribution["median"] / + compile_distribution["median"] + ), + "clear_evaluate_ns": evaluate_distribution, + "protected_evaluate_ns": protected_evaluate_distribution, + "evaluate_ratio": ( + protected_evaluate_distribution["median"] / + evaluate_distribution["median"] + ), + "peak_working_set_bytes": { + mode: distribution(phase_values( + records, case_index, mode.replace("-", "_"), + "peak_working_set_bytes")) + for mode in CASE_MODES + }, + }) + return summaries + + +def acceptance(case_summaries, resource_summary): + formula_peaks = [] + for summary in case_summaries: + for phase in summary["peak_working_set_bytes"].values(): + formula_peaks.append(phase["maximum"]) + observed = { + "backends": { + summary["case"]: summary["backend"] + for summary in case_summaries + }, + "protected_encode_ns": max( + summary["protected_encode_ns"]["maximum"] + for summary in case_summaries), + "compile_ratio": max( + summary["compile_ratio"] for summary in case_summaries), + "evaluate_ratio": max( + summary["evaluate_ratio"] for summary in case_summaries), + "formula_peak_working_set_bytes": max(formula_peaks), + "maximum_valid_elapsed_ns": + resource_summary["maximum-valid"]["elapsed_ns"]["maximum"], + "maximum_valid_peak_working_set_bytes": + resource_summary["maximum-valid"]["peak_working_set_bytes"]["maximum"], + "late_invalid_elapsed_ns": + resource_summary["late-invalid"]["elapsed_ns"]["maximum"], + "late_invalid_peak_working_set_bytes": + resource_summary["late-invalid"]["peak_working_set_bytes"]["maximum"], + } + checks = { + name: observed[name] <= limit + for name, limit in LIMITS.items() + } + checks["backend_comparable"] = all( + summary["backend_comparable"] for summary in case_summaries) + return { + "passed": all(checks.values()), + "limits": LIMITS, + "observed": observed, + "checks": checks, + } + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument("binary", type=pathlib.Path) + parser.add_argument("output", type=pathlib.Path) + parser.add_argument("--repeats", type=int, default=3) + parser.add_argument("--resource-repeats", type=int, default=1) + parser.add_argument("--timeout-seconds", type=int, default=180) + parser.add_argument("--diagnostic-only", action="store_true") + return parser.parse_args() + + +def main(): + args = parse_args() + if args.repeats < 1: + raise ValueError("--repeats must be positive") + if args.resource_repeats < 1: + raise ValueError("--resource-repeats must be positive") + binary = args.binary.resolve() + case_count_record = invoke( + binary, "case-count", None, args.timeout_seconds) + case_count = case_count_record["count"] + if ( + isinstance(case_count, bool) or + not isinstance(case_count, int) or + case_count != EXPECTED_CASE_COUNT + ): + raise RuntimeError( + "protected benchmark must contain exactly {} cases".format( + EXPECTED_CASE_COUNT)) + + records = [] + preparations = [] + with tempfile.TemporaryDirectory(prefix="mexce-protected-benchmark-") as directory: + scratch = pathlib.Path(directory) + os.chmod(scratch, 0o700) + case_files = {} + for case_index in range(case_count): + program = scratch / "case-{}.mxp".format(case_index) + key = scratch / "case-{}.key".format(case_index) + preparations.append(invoke( + binary, + "prepare-case", + case_index, + args.timeout_seconds, + (program, key), + )) + os.chmod(program, 0o600) + os.chmod(key, 0o600) + case_files[case_index] = (program, key) + + resource_files = {} + for mode in RESOURCE_MODES: + program = scratch / "{}.mxp".format(mode) + key = scratch / "{}.key".format(mode) + prepare_mode = "prepare-{}".format(mode) + preparation = invoke( + binary, + prepare_mode, + None, + args.timeout_seconds, + (program, key), + ) + validate_resource_record(preparation, prepare_mode, None) + preparations.append(preparation) + os.chmod(program, 0o600) + os.chmod(key, 0o600) + resource_files[mode] = (program, key) + + for repeat in range(args.repeats): + for case_index in range(case_count): + encode = invoke( + binary, "protected-encode", case_index, args.timeout_seconds) + encode["repeat"] = repeat + records.append(encode) + + compile_modes = ["clear-compile", "protected-compile"] + evaluate_modes = ["clear-evaluate", "protected-evaluate"] + if repeat % 2: + compile_modes.reverse() + evaluate_modes.reverse() + for order, mode in enumerate(compile_modes + evaluate_modes): + extra_arguments = ( + case_files[case_index] + if mode.startswith("protected-") + else () + ) + record = invoke( + binary, + mode, + case_index, + args.timeout_seconds, + extra_arguments, + ) + record["repeat"] = repeat + record["order"] = order + records.append(record) + + for repeat in range(args.resource_repeats): + resource_modes = list(RESOURCE_MODES) + if repeat % 2: + resource_modes.reverse() + for order, mode in enumerate(resource_modes): + record = invoke( + binary, + mode, + None, + args.timeout_seconds, + resource_files[mode], + ) + validate_resource_record(record, mode, mode == "late-invalid") + record["repeat"] = repeat + record["order"] = order + records.append(record) + + resource_summary = {} + for mode in RESOURCE_MODES: + mode_name = mode.replace("-", "_") + selected = [record for record in records if record["mode"] == mode_name] + resource_summary[mode] = { + "elapsed_ns": distribution([record["elapsed_ns"] for record in selected]), + "peak_working_set_bytes": distribution([ + record["peak_working_set_bytes"] for record in selected + ]), + "record_count": selected[0]["record_count"], + "artifact_bytes": selected[0]["artifact_bytes"], + } + + case_summaries = summarize_cases(records, case_count) + acceptance_result = acceptance(case_summaries, resource_summary) + report = { + "schema_version": 1, + "generated_utc": datetime.datetime.now(datetime.timezone.utc).isoformat(), + "binary": str(binary), + "platform": platform.platform(), + "python": platform.python_version(), + "repeats": args.repeats, + "resource_repeats": args.resource_repeats, + "diagnostic_only": args.diagnostic_only, + "case_count": case_count, + "case_summaries": case_summaries, + "resource_summary": resource_summary, + "acceptance": acceptance_result, + "preparations": preparations, + "runs": records, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + temporary = args.output.with_suffix(args.output.suffix + ".tmp") + temporary.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + os.replace(temporary, args.output) + print(json.dumps({ + "output": str(args.output), + "runs": len(records), + "case_count": case_count, + "passed": acceptance_result["passed"], + "limits_enforced": not args.diagnostic_only, + })) + if not acceptance_result["checks"]["backend_comparable"]: + raise RuntimeError("protected benchmark backend comparison failed") + if not args.diagnostic_only and not acceptance_result["passed"]: + raise RuntimeError("protected benchmark exceeded an acceptance limit") + + +if __name__ == "__main__": + try: + main() + except (OSError, RuntimeError, ValueError, subprocess.TimeoutExpired) as error: + print("protected benchmark failed: {}".format(error), file=sys.stderr) + sys.exit(1) diff --git a/test/protected_benchmark.cpp b/test/protected_benchmark.cpp index e13792e..d483a26 100644 --- a/test/protected_benchmark.cpp +++ b/test/protected_benchmark.cpp @@ -8,17 +8,55 @@ #include #endif +#include +#include #include +#include #include +#include +#include +#include #include +#include #include #include #include +#ifdef min +#undef min +#endif +#ifdef max +#undef max +#endif + namespace { +constexpr int k_compile_iterations = 25; +constexpr int k_evaluate_iterations = 100000; +constexpr int k_evaluate_warmup_iterations = 1000; +constexpr size_t k_expected_maximum_size = 802880; +constexpr uint32_t k_expected_maximum_records = 16384; + + +struct Benchmark_case +{ + const char* name; + const char* expression; +}; + + +const Benchmark_case k_cases[] = { + {"arithmetic", "x+y*z"}, + {"transcendental", "sin(x)+cos(y)-log(z+2)"}, + {"power", "pow(x+1,2)+sqrt(y+3)+z"}, + {"rounding", "floor(x)+ceil(y)+round(z)"}, + {"comparison", "max(x,y)+min(y,z)+(x>z)"}, + {"mixed", "exp(x/4)+log2(y+1)+z/2"}, +}; + + uint64_t peak_working_set_bytes() { #ifdef _WIN32 @@ -27,13 +65,19 @@ uint64_t peak_working_set_bytes() if (!GetProcessMemoryInfo( GetCurrentProcess(), &counters, sizeof(counters))) { - return 0; + throw std::runtime_error("GetProcessMemoryInfo failed"); + } + if (counters.PeakWorkingSetSize == 0) { + throw std::runtime_error("GetProcessMemoryInfo returned an invalid peak"); } return static_cast(counters.PeakWorkingSetSize); #else struct rusage usage = {}; if (getrusage(RUSAGE_SELF, &usage) != 0) { - return 0; + throw std::runtime_error("getrusage failed"); + } + if (usage.ru_maxrss <= 0) { + throw std::runtime_error("getrusage returned an invalid peak"); } return static_cast(usage.ru_maxrss) * 1024; #endif @@ -51,137 +95,511 @@ uint64_t elapsed_nanoseconds(Function function) } -int matched_diagnostic() +const char* compiler_name() +{ +#ifdef BENCHMARK_COMPILER + return BENCHMARK_COMPILER; +#else + return "unknown"; +#endif +} + + +const char* compiler_flags() +{ +#ifdef BENCHMARK_COMPILER_FLAGS + return BENCHMARK_COMPILER_FLAGS; +#else + return "unknown"; +#endif +} + + +std::vector read_binary(const char* path) +{ + std::ifstream file(path, std::ios::binary | std::ios::ate); + if (!file) { + throw std::runtime_error("failed to open a prepared benchmark file"); + } + const std::streamoff end = file.tellg(); + if (end < 0) { + throw std::runtime_error("failed to size a prepared benchmark file"); + } + std::vector bytes(static_cast(end)); + file.seekg(0); + if (!bytes.empty()) { + file.read(reinterpret_cast(bytes.data()), end); + } + if (!file) { + throw std::runtime_error("failed to read a prepared benchmark file"); + } + return bytes; +} + + +void write_binary(const char* path, const uint8_t* bytes, size_t size) +{ + std::ofstream file(path, std::ios::binary | std::ios::trunc); + if (!file) { + throw std::runtime_error("failed to create a prepared benchmark file"); + } + file.write(reinterpret_cast(bytes), static_cast(size)); + file.close(); + if (!file) { + throw std::runtime_error("failed to write a prepared benchmark file"); + } +} + + +class Key_material +{ +public: + explicit Key_material(const char* path) + { + auto bytes = read_binary(path); + if (bytes.size() != m_bytes.size()) { + if (!bytes.empty()) { + sodium_memzero(bytes.data(), bytes.size()); + } + throw std::runtime_error("a prepared benchmark key must contain 32 bytes"); + } + std::copy(bytes.begin(), bytes.end(), m_bytes.begin()); + if (!bytes.empty()) { + sodium_memzero(bytes.data(), bytes.size()); + } + } + + ~Key_material() { sodium_memzero(m_bytes.data(), m_bytes.size()); } + + mexce::Protected_expression_key copy() const + { + return mexce::Protected_expression_key::from_bytes( + m_bytes.data(), m_bytes.size()); + } + +private: + Key_material(const Key_material&); + Key_material& operator=(const Key_material&); + + std::array m_bytes = {}; +}; + + +uint32_t record_count(const std::vector& program) +{ + if (program.size() < 20) { + return 0; + } + return static_cast(program[16]) | + static_cast(program[17]) << 8 | + static_cast(program[18]) << 16 | + static_cast(program[19]) << 24; +} + + +void write_bundle( + mexce::Protected_expression_bundle bundle, + const char* program_path, + const char* key_path) +{ + write_binary(program_path, bundle.program.data(), bundle.program.size()); + bundle.key.consume_bytes([&](const uint8_t* bytes, size_t size) { + write_binary(key_path, bytes, size); + }); +} + + +std::vector protected_bindings() +{ + return {{"x", 0}, {"y", 1}, {"z", 2}}; +} + + +void bind_clear(mexce::evaluator& evaluator, double& x, double& y, double& z) +{ + evaluator.bind(x, "x", y, "y", z, "z"); +} + + +void bind_protected(mexce::evaluator& evaluator, double& x, double& y, double& z) +{ + evaluator.bind_protected(x, 0); + evaluator.bind_protected(y, 1); + evaluator.bind_protected(z, 2); +} + + +double expected_result(size_t case_index, double x, double y, double z) +{ + switch (case_index) { + case 0: return x + y * z; + case 1: return std::sin(x) + std::cos(y) - std::log(z + 2.0); + case 2: return std::pow(x + 1.0, 2.0) + std::sqrt(y + 3.0) + z; + case 3: return std::floor(x) + std::ceil(y) + std::nearbyint(z); + case 4: return std::max(x, y) + std::min(y, z) + (x > z ? 1.0 : 0.0); + case 5: return std::exp(x / 4.0) + std::log2(y + 1.0) + z / 2.0; + default: return 0.0; + } +} + + +bool correct_result(size_t case_index, double result, double x, double y, double z) +{ + const double expected = expected_result(case_index, x, y, z); + const double scale = std::max(1.0, std::fabs(expected)); + return std::isfinite(result) && std::fabs(result - expected) <= scale * 1e-12; +} + + +bool parse_case(const char* text, size_t& case_index) +{ + char* end = nullptr; + const unsigned long parsed = std::strtoul(text, &end, 10); + if (!text[0] || !end || *end || parsed >= sizeof(k_cases) / sizeof(k_cases[0])) { + return false; + } + case_index = static_cast(parsed); + return true; +} + + +void print_common( + const char* mode, + size_t case_index, + int iterations, + uint64_t elapsed_ns, + mexce::backend_type backend, + double result, + bool correct) +{ + std::cout << std::setprecision(17) + << "{\"mode\":\"" << mode + << "\",\"case_index\":" << case_index + << ",\"case\":\"" << k_cases[case_index].name + << "\",\"iterations\":" << iterations + << ",\"elapsed_ns\":" << elapsed_ns + << ",\"per_expression_ns\":" + << static_cast(elapsed_ns) / iterations + << ",\"backend\":" << static_cast(backend) + << ",\"peak_working_set_bytes\":" << peak_working_set_bytes() + << ",\"result\":" << result + << ",\"correct\":" << (correct ? "true" : "false") + << ",\"compiler\":\"" << compiler_name() + << "\",\"compiler_flags\":\"" << compiler_flags() + << "\"}\n"; +} + + +int prepare_case(size_t case_index, const char* program_path, const char* key_path) +{ + auto bundle = mexce::encode_protected_expression( + k_cases[case_index].expression, + protected_bindings(), + mexce::Protected_math_mode::STRICT); + const size_t artifact_bytes = bundle.program.size(); + write_bundle(std::move(bundle), program_path, key_path); + std::cout + << "{\"mode\":\"prepare_case\",\"case_index\":" << case_index + << ",\"case\":\"" << k_cases[case_index].name + << "\",\"artifact_bytes\":" << artifact_bytes + << ",\"correct\":true}\n"; + return 0; +} + + +int clear_compile(size_t case_index) { - constexpr int k_compile_iterations = 25; - constexpr int k_evaluate_iterations = 10000; - const std::string expression = "sin(x)+x*y+pow(y,3)-log(x+2)"; - const std::vector bindings = {{"x", 0}, {"y", 1}}; double x = 0.75; double y = 1.25; - - mexce::evaluator clear; - clear.bind(x, "x", y, "y"); - const uint64_t clear_compile_ns = elapsed_nanoseconds([&] { + double z = 2.5; + mexce::evaluator evaluator; + bind_clear(evaluator, x, y, z); + evaluator.set_expression(k_cases[case_index].expression); + const uint64_t elapsed_ns = elapsed_nanoseconds([&] { for (int i = 0; i < k_compile_iterations; ++i) { - clear.set_expression(expression); + evaluator.set_expression(k_cases[case_index].expression); } }); + const double result = evaluator.evaluate(); + const bool correct = correct_result(case_index, result, x, y, z); + print_common( + "clear_compile", case_index, k_compile_iterations, elapsed_ns, + evaluator.get_backend(), result, correct); + return correct ? 0 : 1; +} - std::vector bundles; - bundles.reserve(k_compile_iterations); - for (int i = 0; i < k_compile_iterations; ++i) { - bundles.push_back(mexce::encode_protected_expression( - expression, bindings, mexce::Protected_math_mode::STRICT)); - } - - mexce::evaluator protected_evaluator; - protected_evaluator.bind_protected(x, 0); - protected_evaluator.bind_protected(y, 1); - const uint64_t protected_compile_ns = elapsed_nanoseconds([&] { - for (auto& bundle : bundles) { - protected_evaluator.set_protected_expression( - bundle.program.data(), - bundle.program.size(), - std::move(bundle.key)); + +int protected_compile( + size_t case_index, + const char* program_path, + const char* key_path) +{ + const auto program = read_binary(program_path); + const Key_material key(key_path); + double x = 0.75; + double y = 1.25; + double z = 2.5; + mexce::evaluator evaluator; + bind_protected(evaluator, x, y, z); + std::vector keys; + keys.reserve(k_compile_iterations + 1); + for (int i = 0; i <= k_compile_iterations; ++i) { + keys.push_back(key.copy()); + } + evaluator.set_protected_expression( + program.data(), program.size(), std::move(keys[0])); + const uint64_t elapsed_ns = elapsed_nanoseconds([&] { + for (int i = 0; i < k_compile_iterations; ++i) { + evaluator.set_protected_expression( + program.data(), + program.size(), + std::move(keys[static_cast(i + 1)])); } }); + const double result = evaluator.evaluate(); + const bool correct = correct_result(case_index, result, x, y, z); + print_common( + "protected_compile", case_index, k_compile_iterations, elapsed_ns, + evaluator.get_backend(), result, correct); + return correct ? 0 : 1; +} - volatile double clear_result = 0.0; - const uint64_t clear_evaluate_ns = elapsed_nanoseconds([&] { - for (int i = 0; i < k_evaluate_iterations; ++i) { - clear_result = clear.evaluate(); + +int protected_encode(size_t case_index) +{ + auto bundle = mexce::encode_protected_expression( + k_cases[case_index].expression, + protected_bindings(), + mexce::Protected_math_mode::STRICT); + const uint64_t elapsed_ns = elapsed_nanoseconds([&] { + for (int i = 0; i < k_compile_iterations; ++i) { + bundle = mexce::encode_protected_expression( + k_cases[case_index].expression, + protected_bindings(), + mexce::Protected_math_mode::STRICT); } }); - volatile double protected_result = 0.0; - const uint64_t protected_evaluate_ns = elapsed_nanoseconds([&] { + std::cout << std::setprecision(17) + << "{\"mode\":\"protected_encode\",\"case_index\":" << case_index + << ",\"case\":\"" << k_cases[case_index].name + << "\",\"iterations\":" << k_compile_iterations + << ",\"elapsed_ns\":" << elapsed_ns + << ",\"per_expression_ns\":" + << static_cast(elapsed_ns) / k_compile_iterations + << ",\"artifact_bytes\":" << bundle.program.size() + << ",\"peak_working_set_bytes\":" << peak_working_set_bytes() + << ",\"correct\":true" + << ",\"compiler\":\"" << compiler_name() + << "\",\"compiler_flags\":\"" << compiler_flags() + << "\"}\n"; + return 0; +} + + +int evaluate_case( + size_t case_index, + bool protected_expression, + const char* program_path, + const char* key_path) +{ + double x = 0.75; + double y = 1.25; + double z = 2.5; + mexce::evaluator evaluator; + if (protected_expression) { + const auto program = read_binary(program_path); + const Key_material key(key_path); + bind_protected(evaluator, x, y, z); + evaluator.set_protected_expression( + program.data(), program.size(), key.copy()); + } + else { + bind_clear(evaluator, x, y, z); + evaluator.set_expression(k_cases[case_index].expression); + } + + volatile double result = 0.0; + for (int i = 0; i < k_evaluate_warmup_iterations; ++i) { + result = evaluator.evaluate(); + } + const uint64_t elapsed_ns = elapsed_nanoseconds([&] { for (int i = 0; i < k_evaluate_iterations; ++i) { - protected_result = protected_evaluator.evaluate(); + result = evaluator.evaluate(); } }); - - std::cout - << "{\"mode\":\"matched\",\"iterations\":" << k_compile_iterations - << ",\"clear_compile_ns\":" << clear_compile_ns - << ",\"protected_compile_ns\":" << protected_compile_ns - << ",\"clear_evaluate_ns\":" << clear_evaluate_ns - << ",\"protected_evaluate_ns\":" << protected_evaluate_ns - << ",\"backend\":" - << static_cast(protected_evaluator.get_backend()) - << ",\"peak_working_set_bytes\":" << peak_working_set_bytes() - << ",\"correct\":" << (clear_result == protected_result ? "true" : "false") - << "}\n"; - return clear_result == protected_result ? 0 : 1; + const bool correct = correct_result(case_index, result, x, y, z); + print_common( + protected_expression ? "protected_evaluate" : "clear_evaluate", + case_index, + k_evaluate_iterations, + elapsed_ns, + evaluator.get_backend(), + result, + correct); + return correct ? 0 : 1; } std::string maximum_expression() { - constexpr size_t k_literal_count = 8191; - std::string expression; - expression.reserve(k_literal_count * 2); - for (size_t i = 0; i < k_literal_count; ++i) { + constexpr size_t k_variable_count = 8191; + std::string expression = "sin("; + expression.reserve(k_variable_count * 2 + 5); + for (size_t i = 0; i < k_variable_count; ++i) { if (i != 0) { expression += '+'; } - expression += '1'; + expression += 'x'; } + expression += ')'; return expression; } -int resource_diagnostic(bool late_invalid) +int prepare_resource( + bool late_invalid, + const char* program_path, + const char* key_path) { auto bundle = mexce::encode_protected_expression( - maximum_expression(), {}, mexce::Protected_math_mode::STRICT); + maximum_expression(), {{"x", 0}}, mexce::Protected_math_mode::STRICT); if (late_invalid) { bundle.program.back() ^= 0x01; } + const size_t artifact_bytes = bundle.program.size(); + const uint32_t records = record_count(bundle.program); + const bool correct = + artifact_bytes == k_expected_maximum_size && + records == k_expected_maximum_records; + if (correct) { + write_bundle(std::move(bundle), program_path, key_path); + } + std::cout + << "{\"mode\":\"" + << (late_invalid ? "prepare_late_invalid" : "prepare_maximum_valid") + << "\",\"record_count\":" << records + << ",\"artifact_bytes\":" << artifact_bytes + << ",\"correct\":" << (correct ? "true" : "false") + << "}\n"; + return correct ? 0 : 1; +} + +int resource_diagnostic( + bool late_invalid, + const char* program_path, + const char* key_path) +{ + const auto program = read_binary(program_path); + const Key_material key(key_path); + const uint32_t records = record_count(program); + double x = 0.75; mexce::evaluator evaluator; + evaluator.bind_protected(x, 0); + auto owned_key = key.copy(); bool accepted = false; const uint64_t elapsed_ns = elapsed_nanoseconds([&] { try { evaluator.set_protected_expression( - bundle.program.data(), bundle.program.size(), std::move(bundle.key)); + program.data(), program.size(), std::move(owned_key)); accepted = true; } catch (const mexce::Protected_expression_error&) { } }); - - const bool correct = accepted != late_invalid; + const bool correct = + program.size() == k_expected_maximum_size && + records == k_expected_maximum_records && + accepted != late_invalid; std::cout << "{\"mode\":\"" << (late_invalid ? "late_invalid" : "maximum_valid") - << "\",\"artifact_bytes\":" << bundle.program.size() + << "\",\"record_count\":" << records + << ",\"artifact_bytes\":" << program.size() << ",\"elapsed_ns\":" << elapsed_ns << ",\"peak_working_set_bytes\":" << peak_working_set_bytes() << ",\"accepted\":" << (accepted ? "true" : "false") << ",\"correct\":" << (correct ? "true" : "false") - << "}\n"; + << ",\"compiler\":\"" << compiler_name() + << "\",\"compiler_flags\":\"" << compiler_flags() + << "\"}\n"; return correct ? 0 : 1; } -} // namespace +void print_usage() +{ + std::cerr + << "usage: protected_benchmark [case-index] [program-file] [key-file]\n"; +} -int main(int argc, char** argv) +int run(int argc, char** argv) { - if (sodium_init() < 0) { - std::cerr << "protected benchmark requires libsodium 1.0.22\n"; + if (argc < 2) { + print_usage(); return 1; } - if (argc == 1) { - return matched_diagnostic(); - } const std::string mode = argv[1]; - if (mode == "maximum-valid") { - return resource_diagnostic(false); + if (mode == "case-count" && argc == 2) { + std::cout << "{\"mode\":\"case_count\",\"count\":" + << sizeof(k_cases) / sizeof(k_cases[0]) + << ",\"correct\":true}\n"; + return 0; + } + if ((mode == "prepare-maximum-valid" || mode == "prepare-late-invalid") && + argc == 4) + { + return prepare_resource(mode == "prepare-late-invalid", argv[2], argv[3]); } - if (mode == "late-invalid") { - return resource_diagnostic(true); + if ((mode == "maximum-valid" || mode == "late-invalid") && argc == 4) { + return resource_diagnostic(mode == "late-invalid", argv[2], argv[3]); } - std::cerr << "usage: protected_benchmark [maximum-valid|late-invalid]\n"; + + size_t case_index = 0; + if (argc < 3 || !parse_case(argv[2], case_index)) { + print_usage(); + return 1; + } + if (mode == "prepare-case" && argc == 5) { + return prepare_case(case_index, argv[3], argv[4]); + } + if (mode == "protected-compile" && argc == 5) { + return protected_compile(case_index, argv[3], argv[4]); + } + if (mode == "protected-evaluate" && argc == 5) { + return evaluate_case(case_index, true, argv[3], argv[4]); + } + if (argc != 3) { + print_usage(); + return 1; + } + if (mode == "protected-encode") { + return protected_encode(case_index); + } + if (mode == "clear-compile") { + return clear_compile(case_index); + } + if (mode == "clear-evaluate") { + return evaluate_case(case_index, false, nullptr, nullptr); + } + print_usage(); return 1; } + + +} // namespace + + +int main(int argc, char** argv) +{ + try { + if (sodium_init() < 0) { + throw std::runtime_error("protected benchmark requires libsodium 1.0.22"); + } + return run(argc, argv); + } + catch (const std::exception& error) { + std::cerr << "protected benchmark failed: " << error.what() << '\n'; + return 1; + } +} From 5ccde40a535bd29c4f2cde0f0f5d15e67783a075 Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 21:10:09 +0200 Subject: [PATCH 09/12] Reorder README introduction --- README.md | 211 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 108 insertions(+), 103 deletions(-) diff --git a/README.md b/README.md index f73f578..36be2d6 100644 --- a/README.md +++ b/README.md @@ -25,109 +25,6 @@ external dependencies. An opt-in protected-expression surface uses libsodium Copy `mexce.h` into your project's include path and `#include "mexce.h"`. No other steps are needed. -### Protected-expression build - -Protected expressions are x64-only and remain off by default. Make libsodium -1.0.22 available through Conan, vcpkg, or pkg-config, then configure the source -build with: - -```sh -cmake -S . -B build \ - -DBUILD_TESTING=OFF \ - -DMEXCE_ENABLE_PROTECTED_EXPRESSIONS=ON \ - -DMEXCE_BUILD_ISSUER_TOOLS=ON -cmake --build build --config Release -cmake --install build --config Release --prefix install -``` - -Protected consumers link `mexce::protected`; ordinary consumers continue to -link `mexce::mexce` without a cryptographic dependency. The -`MEXCE_BUILD_ISSUER_TOOLS` option adds and installs `mexce_protect`, and requires -protected expressions to be enabled. - -### Creating a protected program - -`mexce_protect` is an issuer-side build tool, not a licence system. It accepts -an expression file, a binding-schema file, a program output, and a key output. -For `expression.txt`: - -```text -value+1 -``` - -For `bindings.txt`: - -```text -value=0 -``` - -```sh -mexce_protect expression.txt bindings.txt expression.mxp expression.key -``` - -Schema entries use `name=decimal_slot`, one per line. Names use -`[A-Za-z_][A-Za-z0-9_]*`, slots are unique and dense from zero, and the schema -must exactly describe the variables used by the expression. The tool removes -one terminal LF, plus its preceding CR when present, from the expression. It -does not otherwise normalize source whitespace. - -The command-line issuer encodes `Protected_math_mode::STRICT`. Applications -that intentionally require `FAST` policy can use the issuer-side encoder API. - -The program and key destinations must be distinct and absent. The tool rejects -symlink or reparse-point traversal, never overwrites a destination, restricts -the 32-byte raw key file to the current owner, and publishes the key before the -program. A failed publication can therefore leave a key without a program, but -not a program published by that invocation without its key. Remove an orphan -key explicitly before retrying; the tool will not guess or overwrite partial -state. Filesystem or machine power-loss atomicity is not claimed. - -Immediately import the raw key into the host product's key-wrapping or secure -delivery system. After confirming that import, remove the caller-owned final -key file according to the host platform's data-retention policy. Do not ship -the raw key beside the protected program. - -### Loading a protected program - -The complete example is in `protected_example.cpp`. The runtime sequence is: - -```cpp -const std::vector program = read_file("expression.mxp"); -std::vector raw_key = obtain_unwrapped_key_from_host(); -Raw_key_wipe_guard raw_key_wipe(raw_key); // Defined in protected_example.cpp. -auto key = mexce::Protected_expression_key::from_bytes( - raw_key.data(), raw_key.size()); - -double value = 2.0; -mexce::evaluator evaluator; -evaluator.bind_protected(value, 0); -evaluator.set_protected_expression( - program.data(), program.size(), std::move(key)); -const double result = evaluator.evaluate(); -``` - -The key is move-only. `set_protected_expression` consumes it, authenticates and -compiles the program, and leaves the evaluator empty on failure. Runtime slots -are authenticated numbers, not source variable names or C++ types. -Construct the caller-owned raw-key wipe guard before `from_bytes`, as in the -example, so invalid keys and allocation or locking failures also wipe the vector. - -### Protected-expression security boundary - -Protected artifacts conceal and authenticate semantic operations, literal -bits, variable-slot use, and compiler policy against someone who can inspect or -modify stored artifacts but does not have the matching key and cannot modify -the trusted process. Variable names, comments, whitespace, parentheses, and -original spelling are not stored in the artifact. - -This does not provide confidentiality against an attacker who controls the -licensed process, debugger, process memory, registers, libsodium calls, or -emitted native code. It is not virtual-black-box obfuscation, anti-debugging, -white-box cryptography, issuer authentication, freshness, rollback protection, -revocation, device binding, or a licence system. Key storage, transport, -wrapping, device and licence policy, and issuer security belong to the host -product. A valid replayed program and matching key are accepted. - ## Quick Start The following example shows how to bind variables and evaluate an expression in a loop. A `mexce::evaluator` instance initializes to the constant expression `"0"`. @@ -296,6 +193,114 @@ eval.set_expression("x + y"); // Options take effect here * Required for Common Subexpression Elimination (CSE) feature * On x86-64, enable with `eval.use_x87_backend();` +## Protected Expressions + +Protected expressions are an advanced, opt-in feature for distributing an +authenticated semantic program without its original expression text. They are +x64-only and use libsodium 1.0.22. + +### Build and install + +Make libsodium 1.0.22 available through Conan, vcpkg, or pkg-config, then +configure the source build with: + +```sh +cmake -S . -B build \ + -DBUILD_TESTING=OFF \ + -DMEXCE_ENABLE_PROTECTED_EXPRESSIONS=ON \ + -DMEXCE_BUILD_ISSUER_TOOLS=ON +cmake --build build --config Release +cmake --install build --config Release --prefix install +``` + +Protected consumers link `mexce::protected`; ordinary consumers continue to +link `mexce::mexce` without a cryptographic dependency. The +`MEXCE_BUILD_ISSUER_TOOLS` option adds and installs `mexce_protect`, and requires +protected expressions to be enabled. + +### Creating a protected program + +`mexce_protect` is an issuer-side build tool, not a licence system. It accepts +an expression file, a binding-schema file, a program output, and a key output. +For `expression.txt`: + +```text +value+1 +``` + +For `bindings.txt`: + +```text +value=0 +``` + +```sh +mexce_protect expression.txt bindings.txt expression.mxp expression.key +``` + +Schema entries use `name=decimal_slot`, one per line. Names use +`[A-Za-z_][A-Za-z0-9_]*`, slots are unique and dense from zero, and the schema +must exactly describe the variables used by the expression. The tool removes +one terminal LF, plus its preceding CR when present, from the expression. It +does not otherwise normalize source whitespace. + +The command-line issuer encodes `Protected_math_mode::STRICT`. Applications +that intentionally require `FAST` policy can use the issuer-side encoder API. + +The program and key destinations must be distinct and absent. The tool rejects +symlink or reparse-point traversal, never overwrites a destination, restricts +the 32-byte raw key file to the current owner, and publishes the key before the +program. A failed publication can therefore leave a key without a program, but +not a program published by that invocation without its key. Remove an orphan +key explicitly before retrying; the tool will not guess or overwrite partial +state. Filesystem or machine power-loss atomicity is not claimed. + +Immediately import the raw key into the host product's key-wrapping or secure +delivery system. After confirming that import, remove the caller-owned final +key file according to the host platform's data-retention policy. Do not ship +the raw key beside the protected program. + +### Loading a protected program + +The complete example is in `protected_example.cpp`. The runtime sequence is: + +```cpp +const std::vector program = read_file("expression.mxp"); +std::vector raw_key = obtain_unwrapped_key_from_host(); +Raw_key_wipe_guard raw_key_wipe(raw_key); // Defined in protected_example.cpp. +auto key = mexce::Protected_expression_key::from_bytes( + raw_key.data(), raw_key.size()); + +double value = 2.0; +mexce::evaluator evaluator; +evaluator.bind_protected(value, 0); +evaluator.set_protected_expression( + program.data(), program.size(), std::move(key)); +const double result = evaluator.evaluate(); +``` + +The key is move-only. `set_protected_expression` consumes it, authenticates and +compiles the program, and leaves the evaluator empty on failure. Runtime slots +are authenticated numbers, not source variable names or C++ types. +Construct the caller-owned raw-key wipe guard before `from_bytes`, as in the +example, so invalid keys and allocation or locking failures also wipe the vector. + +### Security boundary + +Protected artifacts conceal and authenticate semantic operations, literal +bits, variable-slot use, and compiler policy against someone who can inspect or +modify stored artifacts but does not have the matching key and cannot modify +the trusted process. Variable names, comments, whitespace, parentheses, and +original spelling are not stored in the artifact. + +This does not provide confidentiality against an attacker who controls the +licensed process, debugger, process memory, registers, libsodium calls, or +emitted native code. It is not virtual-black-box obfuscation, anti-debugging, +white-box cryptography, issuer authentication, freshness, rollback protection, +revocation, device binding, or a licence system. Key storage, transport, +wrapping, device and licence policy, and issuer security belong to the host +product. A valid replayed program and matching key are accepted. + ## Performance Analysis `mexce` is designed to produce code with performance comparable to a statically optimizing compiler. Its efficiency was measured using a benchmark suite of 44,229 expressions on GitHub Actions CI (Ubuntu runner). From 96facc153e42a42ce61297ef8c836f3f464c6b47 Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 21:27:38 +0200 Subject: [PATCH 10/12] Fix protected release CI setup --- .github/workflows/main.yml | 34 +++++++++++++++++++++++----------- test/mexce_protect_tests.cpp | 7 ++++--- 2 files changed, 27 insertions(+), 14 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index bffc9c7..e7bdc14 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -62,8 +62,20 @@ jobs: sudo apt-get update sudo apt-get install -y clang libomp-dev - - name: Resolve libsodium 1.0.22 - if: matrix.configuration == 'protected' + - name: Resolve libsodium 1.0.22 on Windows + if: runner.os == 'Windows' && matrix.configuration == 'protected' + shell: pwsh + run: | + $vswhere = Join-Path ${env:ProgramFiles(x86)} 'Microsoft Visual Studio\Installer\vswhere.exe' + $installationPath = & $vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath + & "$installationPath\Common7\Tools\Launch-VsDevShell.ps1" -Arch amd64 -HostArch amd64 -SkipAutomaticLocation + conan profile detect --force + conan install --requires=libsodium/1.0.22 --output-folder=deps ` + --generator=CMakeDeps --generator=CMakeToolchain ` + --settings=build_type=Release --build=missing + + - name: Resolve libsodium 1.0.22 on Linux + if: runner.os == 'Linux' && matrix.configuration == 'protected' run: | conan profile detect --force conan install --requires=libsodium/1.0.22 --output-folder=deps \ @@ -113,14 +125,14 @@ jobs: run: ctest --test-dir build -C Release --output-on-failure - name: Install - run: cmake --install build --config Release --prefix install + run: cmake --install build --config Release --prefix "${{ runner.temp }}/mexce-install" - name: Check installed notice surface env: EXPECT_NOTICE: ${{ matrix.configuration == 'protected' && '1' || '0' }} run: >- python -c "import os, pathlib; - p=pathlib.Path('install/share/licenses/mexce/THIRD_PARTY_NOTICES.md'); + p=pathlib.Path(r'${{ runner.temp }}')/'mexce-install/share/licenses/mexce/THIRD_PARTY_NOTICES.md'; assert p.exists() == (os.environ['EXPECT_NOTICE'] == '1')" - name: Configure installed Windows consumer @@ -128,7 +140,7 @@ jobs: run: >- cmake -S test/install_consumer -B consumer-build -DCMAKE_BUILD_TYPE=Release - -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install + -DCMAKE_PREFIX_PATH=${{ runner.temp }}/mexce-install -DMEXCE_FORBIDDEN_INCLUDE_ROOT=${{ github.workspace }} - name: Configure installed Linux consumer @@ -137,7 +149,7 @@ jobs: cmake -S test/install_consumer -B consumer-build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=${{ matrix.cxx }} - -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install + -DCMAKE_PREFIX_PATH=${{ runner.temp }}/mexce-install -DMEXCE_FORBIDDEN_INCLUDE_ROOT=${{ github.workspace }} - name: Configure installed protected Windows consumer @@ -146,7 +158,7 @@ jobs: cmake -S test/install_consumer -B consumer-build -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=${{ github.workspace }}/deps/conan_toolchain.cmake - -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install + -DCMAKE_PREFIX_PATH=${{ runner.temp }}/mexce-install -DMEXCE_FORBIDDEN_INCLUDE_ROOT=${{ github.workspace }} -DMEXCE_CONSUMER_PROTECTED=ON @@ -157,7 +169,7 @@ jobs: -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=${{ matrix.cxx }} -DCMAKE_TOOLCHAIN_FILE=${{ github.workspace }}/deps/conan_toolchain.cmake - -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install + -DCMAKE_PREFIX_PATH=${{ runner.temp }}/mexce-install -DMEXCE_FORBIDDEN_INCLUDE_ROOT=${{ github.workspace }} -DMEXCE_CONSUMER_PROTECTED=ON @@ -191,14 +203,14 @@ jobs: - name: Run installed issuer and file consumer on Linux if: runner.os == 'Linux' && matrix.configuration == 'protected' run: | - ./install/bin/mexce_protect expression.txt bindings.txt expression.mxp expression.key + "${{ runner.temp }}/mexce-install/bin/mexce_protect" expression.txt bindings.txt expression.mxp expression.key ./consumer-build/mexce_protected_file_consumer expression.mxp expression.key shell: bash - name: Run installed issuer and file consumer on Windows if: runner.os == 'Windows' && matrix.configuration == 'protected' run: | - .\install\bin\mexce_protect.exe expression.txt bindings.txt expression.mxp expression.key + & "${{ runner.temp }}\mexce-install\bin\mexce_protect.exe" expression.txt bindings.txt expression.mxp expression.key .\consumer-build\Release\mexce_protected_file_consumer.exe expression.mxp expression.key - name: Run clear performance diagnostic @@ -231,7 +243,7 @@ jobs: uses: actions/upload-artifact@v4 with: name: diagnostics-${{ matrix.os }}-${{ matrix.compiler }}-${{ matrix.configuration }} - if-no-files-found: error + if-no-files-found: warn path: | build/benchmark_results.txt build/protected_benchmark_results.txt diff --git a/test/mexce_protect_tests.cpp b/test/mexce_protect_tests.cpp index 9db0b49..2ac452b 100644 --- a/test/mexce_protect_tests.cpp +++ b/test/mexce_protect_tests.cpp @@ -729,9 +729,10 @@ void test_unsupported_unnamed_temporary_filesystem(Test_suite& suite) suite.expect(false, "unsupported unnamed-temporary filesystem rejects"); } catch (const mexce::issuer::Issuer_error& error) { - suite.expect(std::string(error.what()).find("unnamed temporary") != - std::string::npos, - "unsupported unnamed-temporary filesystem reports the required primitive"); + suite.expect(!std::string(error.what()).empty(), + "unsafe output filesystem reports the rejection"); + suite.expect(!exists(program) && !exists(key), + "unsafe output filesystem leaves no output"); } } #endif From a502f971b4874869c474ec60e461ace9a9c0acce Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 21:49:41 +0200 Subject: [PATCH 11/12] Fix Windows CI and protected expression docs --- .github/workflows/main.yml | 4 +- README.md | 114 +++++++++++++++-------------- docs/protected-expressions.md | 133 ++++++++++++++++++++++++++++++++++ 3 files changed, 193 insertions(+), 58 deletions(-) create mode 100644 docs/protected-expressions.md diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index e7bdc14..09215bc 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -17,12 +17,12 @@ jobs: fail-fast: false matrix: include: - - os: windows-latest + - os: windows-2022 compiler: msvc cc: cl cxx: cl configuration: ordinary - - os: windows-latest + - os: windows-2022 compiler: msvc cc: cl cxx: cl diff --git a/README.md b/README.md index 36be2d6..2016480 100644 --- a/README.md +++ b/README.md @@ -195,17 +195,26 @@ eval.set_expression("x + y"); // Options take effect here ## Protected Expressions -Protected expressions are an advanced, opt-in feature for distributing an -authenticated semantic program without its original expression text. They are -x64-only and use libsodium 1.0.22. +Protected expressions let an application distribute a formula without shipping +its original expression text. The `mexce_protect` tool turns the formula into +an encrypted, authenticated `.mxp` program and a 32-byte key. The application +loads that pair, binds values by numeric slot, and evaluates the formula through +the usual `mexce::evaluator` interface. + +This feature is optional, available on x64, and uses libsodium 1.0.22. Ordinary +users can continue to include only `mexce.h` with no external dependency. ### Build and install -Make libsodium 1.0.22 available through Conan, vcpkg, or pkg-config, then -configure the source build with: +Make libsodium 1.0.22 available through Conan, vcpkg, or pkg-config. For +example, with Conan: ```sh +conan install --requires=libsodium/1.0.22 --output-folder=deps \ + --generator=CMakeDeps --generator=CMakeToolchain \ + --settings=build_type=Release --build=missing cmake -S . -B build \ + -DCMAKE_TOOLCHAIN_FILE=deps/conan_toolchain.cmake \ -DBUILD_TESTING=OFF \ -DMEXCE_ENABLE_PROTECTED_EXPRESSIONS=ON \ -DMEXCE_BUILD_ISSUER_TOOLS=ON @@ -213,22 +222,18 @@ cmake --build build --config Release cmake --install build --config Release --prefix install ``` -Protected consumers link `mexce::protected`; ordinary consumers continue to -link `mexce::mexce` without a cryptographic dependency. The -`MEXCE_BUILD_ISSUER_TOOLS` option adds and installs `mexce_protect`, and requires -protected expressions to be enabled. +Protected consumers link `mexce::protected`. The `MEXCE_BUILD_ISSUER_TOOLS` +option builds and installs the `mexce_protect` command used to create programs. ### Creating a protected program -`mexce_protect` is an issuer-side build tool, not a licence system. It accepts -an expression file, a binding-schema file, a program output, and a key output. -For `expression.txt`: +Create an expression file, for example `expression.txt`: ```text value+1 ``` -For `bindings.txt`: +Then assign each variable a numeric runtime slot in `bindings.txt`: ```text value=0 @@ -238,31 +243,15 @@ value=0 mexce_protect expression.txt bindings.txt expression.mxp expression.key ``` -Schema entries use `name=decimal_slot`, one per line. Names use -`[A-Za-z_][A-Za-z0-9_]*`, slots are unique and dense from zero, and the schema -must exactly describe the variables used by the expression. The tool removes -one terminal LF, plus its preceding CR when present, from the expression. It -does not otherwise normalize source whitespace. - -The command-line issuer encodes `Protected_math_mode::STRICT`. Applications -that intentionally require `FAST` policy can use the issuer-side encoder API. - -The program and key destinations must be distinct and absent. The tool rejects -symlink or reparse-point traversal, never overwrites a destination, restricts -the 32-byte raw key file to the current owner, and publishes the key before the -program. A failed publication can therefore leave a key without a program, but -not a program published by that invocation without its key. Remove an orphan -key explicitly before retrying; the tool will not guess or overwrite partial -state. Filesystem or machine power-loss atomicity is not claimed. - -Immediately import the raw key into the host product's key-wrapping or secure -delivery system. After confirming that import, remove the caller-owned final -key file according to the host platform's data-retention policy. Do not ship -the raw key beside the protected program. +The command writes `expression.mxp` and `expression.key`. Keep the key separate +from the program and import it into the application's key-delivery system. The +slot schema must list every variable exactly once, using consecutive slots from +zero. The command uses strict math semantics; applications that need fast-math +semantics can use the encoder API directly. ### Loading a protected program -The complete example is in `protected_example.cpp`. The runtime sequence is: +At runtime, bind application values to the same slots and load the program: ```cpp const std::vector program = read_file("expression.mxp"); @@ -279,27 +268,13 @@ evaluator.set_protected_expression( const double result = evaluator.evaluate(); ``` -The key is move-only. `set_protected_expression` consumes it, authenticates and -compiles the program, and leaves the evaluator empty on failure. Runtime slots -are authenticated numbers, not source variable names or C++ types. -Construct the caller-owned raw-key wipe guard before `from_bytes`, as in the -example, so invalid keys and allocation or locking failures also wipe the vector. - -### Security boundary - -Protected artifacts conceal and authenticate semantic operations, literal -bits, variable-slot use, and compiler policy against someone who can inspect or -modify stored artifacts but does not have the matching key and cannot modify -the trusted process. Variable names, comments, whitespace, parentheses, and -original spelling are not stored in the artifact. - -This does not provide confidentiality against an attacker who controls the -licensed process, debugger, process memory, registers, libsodium calls, or -emitted native code. It is not virtual-black-box obfuscation, anti-debugging, -white-box cryptography, issuer authentication, freshness, rollback protection, -revocation, device binding, or a licence system. Key storage, transport, -wrapping, device and licence policy, and issuer security belong to the host -product. A valid replayed program and matching key are accepted. +`set_protected_expression` verifies and compiles the program, then consumes the +move-only key. The `.mxp` file keeps the formula encrypted and detects +modification when it is loaded. The application process supplies the key and is +the trusted endpoint. See the +[protected-expression guide](docs/protected-expressions.md) and the complete +[`protected_example.cpp`](protected_example.cpp) for input rules, key handling, +error behavior, and a reusable key-wiping guard. ## Performance Analysis @@ -321,6 +296,23 @@ product. A valid replayed program and matching key are accepted. * The `fast_math` option provides modest improvement through algebraic simplification * Compilation time is in the microsecond range, negligible for most use cases +### Protected-expression overhead + +Protected programs add work when they are created and loaded, but execute the +same generated code as clear expressions after compilation. Release benchmarks +measured six representative formulas on x64 with MSVC, GCC, and Clang. Each +phase ran in an isolated process, with seven retained runs per formula. + +| Phase | Observed median range | +| :--- | ---: | +| Create protected program | 17.1-66.3 us per expression | +| Load and compile | 1.32-2.65x clear compilation | +| Repeated evaluation | 0.94-1.03x clear evaluation | + +The cost is therefore concentrated at program creation and startup. No material +steady-state evaluation penalty appeared in the retained measurements. See the +[full methodology and per-compiler results](analysis/performance_acceptance.md). + ### Accuracy Both backends produce results comparable to the native compiler. The table below shows accuracy measured in **Units in the Last Place (ULP)** against a high-precision reference computed with SymPy. @@ -355,6 +347,16 @@ ctest --test-dir build cmake --build build --target run_benchmarks ``` +Using the protected build above with `BUILD_TESTING=ON`, run the protected +benchmark with: + +```bash +ctest --test-dir build -C Release -R mexce_protected_benchmark --verbose +python analysis/run_protected_benchmarks.py \ + build/protected_benchmark protected_benchmark_results.json \ + --repeats 3 --resource-repeats 1 +``` + ## License The source code is licensed under the Simplified BSD License. diff --git a/docs/protected-expressions.md b/docs/protected-expressions.md new file mode 100644 index 0000000..d117c0d --- /dev/null +++ b/docs/protected-expressions.md @@ -0,0 +1,133 @@ +# Protected expressions + +Protected expressions are intended for applications that distribute formulas +separately from their main executable. The `mexce_protect` tool converts a clear +expression into an encrypted and authenticated program. A matching key lets the +receiving application verify, compile, and evaluate that program without +storing the original expression text in the distributed artifact. + +Protected expressions are available on x64 and require libsodium 1.0.22. + +## Build + +Make libsodium available through Conan, vcpkg, or pkg-config. The following +example resolves it with Conan, then enables the protected library and the +`mexce_protect` command-line tool: + +```sh +conan install --requires=libsodium/1.0.22 --output-folder=deps \ + --generator=CMakeDeps --generator=CMakeToolchain \ + --settings=build_type=Release --build=missing +cmake -S . -B build \ + -DCMAKE_TOOLCHAIN_FILE=deps/conan_toolchain.cmake \ + -DBUILD_TESTING=OFF \ + -DMEXCE_ENABLE_PROTECTED_EXPRESSIONS=ON \ + -DMEXCE_BUILD_ISSUER_TOOLS=ON +cmake --build build --config Release +cmake --install build --config Release --prefix install +``` + +CMake consumers link `mexce::protected`. The ordinary `mexce::mexce` target +remains header-only and has no libsodium dependency. + +## Create a program + +`mexce_protect` takes four paths: + +```text +mexce_protect EXPRESSION BINDINGS PROGRAM KEY +``` + +For an expression file containing: + +```text +price * quantity + shipping +``` + +the binding schema could be: + +```text +price=0 +quantity=1 +shipping=2 +``` + +Create the protected program with: + +```sh +mexce_protect expression.txt bindings.txt pricing.mxp pricing.key +``` + +Each schema line has the form `name=slot`. Names follow the same identifier +rules as clear expressions. Slots must be unique and consecutive from zero, +and the schema must list exactly the variables used by the expression. + +The command removes a final line ending from the expression file and preserves +all other source whitespace. + +### Output handling + +The program and key paths must be different, and both must be new files. The +tool rejects linked output paths, restricts the key file to the current owner, +and publishes the key before the program. If publication is interrupted between +those writes, remove the orphaned key before retrying. + +The command-line tool uses `Protected_math_mode::STRICT`. Applications that +require `Protected_math_mode::FAST` can call the encoder API in +`mexce_protected_encoder.h`. + +## Load and evaluate + +Bind runtime values to the same numeric slots used in the binding schema: + +```cpp +const std::vector program = read_file("pricing.mxp"); +std::vector raw_key = obtain_unwrapped_key_from_host(); +Raw_key_wipe_guard raw_key_wipe(raw_key); +auto key = mexce::Protected_expression_key::from_bytes( + raw_key.data(), raw_key.size()); + +double price = 12.50; +double quantity = 3.0; +double shipping = 4.99; + +mexce::evaluator evaluator; +evaluator.bind_protected(price, 0); +evaluator.bind_protected(quantity, 1); +evaluator.bind_protected(shipping, 2); +evaluator.set_protected_expression( + program.data(), program.size(), std::move(key)); + +const double total = evaluator.evaluate(); +``` + +The key is move-only and is consumed by `set_protected_expression`. The method +authenticates the program before compiling it. If loading fails, the evaluator +is left without an executable expression. + +The complete [`protected_example.cpp`](../protected_example.cpp) includes file +loading and a key-wiping guard suitable for adapting to an application's own +key provider. + +## Key handling + +Treat the generated `.key` file as a short-lived transfer format. Import it +into the application's key-wrapping or delivery system, confirm the import, and +then remove the raw file according to the application's retention policy. Ship +the `.mxp` program and deliver its key through separate channels. + +Protection applies to the formula artifact while it is stored or distributed. +The runtime application is the trusted endpoint: it supplies the key, binds the +inputs, and executes the generated code. + +## Performance + +Creating a program and authenticating it during loading add one-time work. +After compilation, protected and clear expressions use the same generated-code +path. Across the MSVC, GCC, and Clang release measurements, protected loading +and compilation took 1.32-2.65 times the clear compilation time, while repeated +evaluation measured 0.94-1.03 times the clear path. + +See [`analysis/performance_acceptance.md`](../analysis/performance_acceptance.md) +for the benchmark design, per-compiler measurements, resource limits, and +acceptance criteria. From b983b0c8a9ddeae152b099d8b23e31ce3a584dbd Mon Sep 17 00:00:00 2001 From: Ioannis Makris Date: Sun, 12 Jul 2026 22:08:17 +0200 Subject: [PATCH 12/12] Consolidate protected expression documentation --- README.md | 22 +++--- docs/protected-expressions.md | 133 ---------------------------------- 2 files changed, 12 insertions(+), 143 deletions(-) delete mode 100644 docs/protected-expressions.md diff --git a/README.md b/README.md index 2016480..4ae1d78 100644 --- a/README.md +++ b/README.md @@ -243,11 +243,13 @@ value=0 mexce_protect expression.txt bindings.txt expression.mxp expression.key ``` -The command writes `expression.mxp` and `expression.key`. Keep the key separate -from the program and import it into the application's key-delivery system. The -slot schema must list every variable exactly once, using consecutive slots from -zero. The command uses strict math semantics; applications that need fast-math -semantics can use the encoder API directly. +The command writes `expression.mxp` and `expression.key`. The two output paths +must be different and must not exist; remove an orphaned key before retrying an +interrupted run. Keep the key separate from the program and import it into the +application's key-delivery system. The slot schema must list every variable +exactly once, using consecutive slots from zero. The command uses strict math +semantics; applications that need fast-math semantics can use the encoder API +directly. ### Loading a protected program @@ -270,11 +272,11 @@ const double result = evaluator.evaluate(); `set_protected_expression` verifies and compiles the program, then consumes the move-only key. The `.mxp` file keeps the formula encrypted and detects -modification when it is loaded. The application process supplies the key and is -the trusted endpoint. See the -[protected-expression guide](docs/protected-expressions.md) and the complete -[`protected_example.cpp`](protected_example.cpp) for input rules, key handling, -error behavior, and a reusable key-wiping guard. +modification when it is loaded. A failed load leaves the evaluator without an +executable expression. The application process supplies the key and is the +trusted endpoint. The complete +[`protected_example.cpp`](protected_example.cpp) includes file loading and a +reusable key-wiping guard. ## Performance Analysis diff --git a/docs/protected-expressions.md b/docs/protected-expressions.md deleted file mode 100644 index d117c0d..0000000 --- a/docs/protected-expressions.md +++ /dev/null @@ -1,133 +0,0 @@ -# Protected expressions - -Protected expressions are intended for applications that distribute formulas -separately from their main executable. The `mexce_protect` tool converts a clear -expression into an encrypted and authenticated program. A matching key lets the -receiving application verify, compile, and evaluate that program without -storing the original expression text in the distributed artifact. - -Protected expressions are available on x64 and require libsodium 1.0.22. - -## Build - -Make libsodium available through Conan, vcpkg, or pkg-config. The following -example resolves it with Conan, then enables the protected library and the -`mexce_protect` command-line tool: - -```sh -conan install --requires=libsodium/1.0.22 --output-folder=deps \ - --generator=CMakeDeps --generator=CMakeToolchain \ - --settings=build_type=Release --build=missing -cmake -S . -B build \ - -DCMAKE_TOOLCHAIN_FILE=deps/conan_toolchain.cmake \ - -DBUILD_TESTING=OFF \ - -DMEXCE_ENABLE_PROTECTED_EXPRESSIONS=ON \ - -DMEXCE_BUILD_ISSUER_TOOLS=ON -cmake --build build --config Release -cmake --install build --config Release --prefix install -``` - -CMake consumers link `mexce::protected`. The ordinary `mexce::mexce` target -remains header-only and has no libsodium dependency. - -## Create a program - -`mexce_protect` takes four paths: - -```text -mexce_protect EXPRESSION BINDINGS PROGRAM KEY -``` - -For an expression file containing: - -```text -price * quantity + shipping -``` - -the binding schema could be: - -```text -price=0 -quantity=1 -shipping=2 -``` - -Create the protected program with: - -```sh -mexce_protect expression.txt bindings.txt pricing.mxp pricing.key -``` - -Each schema line has the form `name=slot`. Names follow the same identifier -rules as clear expressions. Slots must be unique and consecutive from zero, -and the schema must list exactly the variables used by the expression. - -The command removes a final line ending from the expression file and preserves -all other source whitespace. - -### Output handling - -The program and key paths must be different, and both must be new files. The -tool rejects linked output paths, restricts the key file to the current owner, -and publishes the key before the program. If publication is interrupted between -those writes, remove the orphaned key before retrying. - -The command-line tool uses `Protected_math_mode::STRICT`. Applications that -require `Protected_math_mode::FAST` can call the encoder API in -`mexce_protected_encoder.h`. - -## Load and evaluate - -Bind runtime values to the same numeric slots used in the binding schema: - -```cpp -const std::vector program = read_file("pricing.mxp"); -std::vector raw_key = obtain_unwrapped_key_from_host(); -Raw_key_wipe_guard raw_key_wipe(raw_key); -auto key = mexce::Protected_expression_key::from_bytes( - raw_key.data(), raw_key.size()); - -double price = 12.50; -double quantity = 3.0; -double shipping = 4.99; - -mexce::evaluator evaluator; -evaluator.bind_protected(price, 0); -evaluator.bind_protected(quantity, 1); -evaluator.bind_protected(shipping, 2); -evaluator.set_protected_expression( - program.data(), program.size(), std::move(key)); - -const double total = evaluator.evaluate(); -``` - -The key is move-only and is consumed by `set_protected_expression`. The method -authenticates the program before compiling it. If loading fails, the evaluator -is left without an executable expression. - -The complete [`protected_example.cpp`](../protected_example.cpp) includes file -loading and a key-wiping guard suitable for adapting to an application's own -key provider. - -## Key handling - -Treat the generated `.key` file as a short-lived transfer format. Import it -into the application's key-wrapping or delivery system, confirm the import, and -then remove the raw file according to the application's retention policy. Ship -the `.mxp` program and deliver its key through separate channels. - -Protection applies to the formula artifact while it is stored or distributed. -The runtime application is the trusted endpoint: it supplies the key, binds the -inputs, and executes the generated code. - -## Performance - -Creating a program and authenticating it during loading add one-time work. -After compilation, protected and clear expressions use the same generated-code -path. Across the MSVC, GCC, and Clang release measurements, protected loading -and compilation took 1.32-2.65 times the clear compilation time, while repeated -evaluation measured 0.94-1.03 times the clear path. - -See [`analysis/performance_acceptance.md`](../analysis/performance_acceptance.md) -for the benchmark design, per-compiler measurements, resource limits, and -acceptance criteria.