research: measure whether Madar reduces work on supported coding tasks

[mohanagy]

User story

As a developer, I want a fair comparison of completing representative tasks with and without Madar, so that I can decide whether adding it is worthwhile.

Status

CLOSED — corrected execution v2 finished with an INCONCLUSIVE outcome (11 September 2026). All nine cells ran. Five review passes have no outstanding finding; two recorded Got passes have correctness withheld after a source audit found an options-absent serialization defect; two reviewers timed out without a verdict. The required native comparison is incomplete, and all six Madar-enabled authors left its context tools unused.

The bounded study is complete. Madar's main usefulness goal remains unproven; this is neither positive qualification nor proof of technical impossibility. No automatic rerun or repair chain follows. See the v2 terminal report for every cell, the static finding, exact versions, timing, usage and limitations.

V1 remains separately inconclusive. Its artifacts and results were preserved, and none of its author scores entered v2. The historical experiment contract below is retained for traceability, with the authorized v2 execution amendment and terminal report recording what actually happened.

Acceptance criteria

• Before author runs, verify the identities below and seal the common task prompts, behavioral checks, commands, dependency identities and execution settings. Record preparation failures.
• Apply the same behavioral correctness criteria and ordinary tools to all conditions. Include setup, indexing, source reading, errors and verification in elapsed time and work records.
• Report all nine v2 scheduled attempts, including failures, unused Madar tools, missing measurements and per-task regressions. V1 results remain preserved separately.
• End v2 with scoped benefit, no demonstrated benefit, or inconclusive using the rules below. No retries, threshold changes, replacement tasks or automatic repair chain.

The completed checklist records execution and transparent reporting, not product success. Missing review verdicts and the final static finding are retained in the terminal result.

Conditions and exact versions

Condition	Identity	Role
N — Native	Codex CLI with ordinary local source-reading, editing and test tools; no Madar server	Primary comparator
M — Madar main	3371ada8425efa7f8cabdac781fa227feaea7a6a	Historical implementation control selected by #753
C — Madar next	2b144504ddf924d64cce51db601bb599be0b6c44	Candidate selected by #753; primary comparison is C versus N

Both Madar commits declare package version 0.32.1, so use commit, tree and built-artifact identities rather than the package label. #742 changed documentation after C; do not silently move C to a newer branch tip. Report M against N and C as secondary observations; do not select a winner after seeing results.

Use Node 22.22.3, Codex CLI 0.153.4, requested model gpt-5.6-sol, reasoning xhigh, service tier default. Node and CLI versions were checked locally on 11 September; model availability was observed in prior execution evidence and is not freshly tested by this plan. Preserve the requested model and any returned model identity; if the service does not report an exact backend revision, disclose that limitation.

Use the included ChatGPT subscription only: no API-key authentication, paid API, purchased credits, reset redemption, billing change or Actions workflow. Verify the authentication route and available subscription allowance in preparation without exposing credentials. Stop if the selected route/settings are unavailable; no silent model or provider substitution.

Reuse the existing direct codex exec --json execution and raw-event capture facilities. Use one fresh author session per attempt and no investigation/review/coding pipeline. Codex CLI owns any future test materialization and task implementation. This issue does not authorize a new evaluator platform.

Frozen task specifications

These are proposed feature requests for local comparison, not claims of upstream maintainer demand or permission to publish patches upstream. Give every author the same task text and ordinary repository instructions. Preparation-only source-navigation notes and prior result artifacts stay outside author workspaces.

T1 — Yup: UTF-8 byte limit

Repository: jquense/yup, commit ff31eee8a2b10c938bb3a22544d9e33957a8cf01.

Add string().maxBytes(limit, message?) to validate the UTF-8 encoded byte length after normal Yup transformations. Support a numeric limit or numeric Yup reference. Empty strings pass a zero limit; absent values retain existing optional/nullable behavior. Use standard UTF-8 replacement for unpaired UTF-16 surrogates. Repeated maxBytes calls replace the earlier constraint. Provide a default locale message and custom string/function messages receiving the resolved maxBytes parameter. Keep max() unchanged. Add focused public API tests and a short API documentation entry. Use browser-compatible code without a new dependency or Node Buffer.

The common checks cover ASCII, accented text, emoji, combining marks, a lone surrogate and empty strings; exact limits and one-byte overflow; sibling/context references; transformed versus strict validation; optional/null values; repeated constraints; locale/custom-message parameters; sync/async behavior and unchanged max(). Limit checks to these public behaviors, not an expected implementation.

T2 — Yup: flatten validation errors

Use the same Yup commit, starting from pristine source independently of T1.

Add ValidationError#flatten() returning { formErrors, fieldErrors }. Collect messages from leaf errors in their existing order. Missing/empty paths belong in formErrors; other paths are literal fieldErrors keys, including items[0].name. Retain duplicates and exclude aggregate parent summaries when children exist. Without children, use the error's own messages. Preserve configured message values without string coercion. Return fresh containers/arrays without modifying the error. Treat proto and constructor as ordinary path keys. Support errors produced with or without disableStackTrace, through synchronous and asynchronous validation. Keep existing construction and serialization behavior unchanged. Add focused tests and a short API documentation entry.

The common checks cover root/field errors, nested object/array failures with abortEarly:false, ordered duplicates, root plus field failures, aggregate-summary exclusion, both stack modes, special path keys, fresh arrays, non-string messages and sync/async parity.

T3 — Got: bounded JSON error projection

Repository: sindresorhus/got, commit 687eb7dcc100ea3e548ebba227173b886789e670.

Add RequestError#toJSON() for structured logging. Return a fresh plain object containing name and code, plus method and url when available, and statusCode when a response exists. The URL retains protocol, host, port and path while omitting username, password, query and fragment. Omit unavailable fields. Exclude message, stack, input, headers, bodies, options, request, response, timings and cause. Do not alter existing error properties or mutate the underlying URL. Existing RequestError subclasses inherit this behavior. Direct calls and JSON.stringify(error) produce the projection. Add focused tests and a short API documentation entry. Keep HTTP execution and retries unchanged.

The common checks cover an error with normalized options and no response; HTTPError subclass fields/status; absent URL; URL filtering without mutation; additional enumerable data and circular cause exclusion; fresh objects; serialization parity; and the existing public localhost HTTP error path. No external endpoint is needed.

These prompts differ from the previously inspected Yup tuple-recursion and class-identity tasks. The repos were in the earlier comparison pool, and wider exposure was not exhaustively audited. Do not relabel them repository holdouts or training-fresh tasks. There are two Yup tasks and one Got task: report that limited diversity.

Preparation and fair execution

One bounded preparation phase must verify source/lockfile hashes, available dependency snapshots and both Madar builds using their pinned locks. Yup is version 1.7.1 and pins Yarn 4.10.0; its observed yarn.lock SHA-256 is b577c728675c5872d952549d8640770d80ad3c34c790272150e1616b19822494. Got is version 16.0.0, requires Node >=22, and has no tracked root dependency lockfile or node_modules in the selected source. Preparation must identify and seal an existing usable Got dependency snapshot, with package versions and content identity; dependency readiness is not established. Do not resolve changing latest versions separately for different conditions.

Materialize the common focused checks outside author access, with one Codex CLI preparation session. Confirm expected missing-feature failures at baseline, validate the expectations against the public semantics and parsed source, and record a small relevant unchanged-regression selection for each repository. Include typechecking of the new public API. A red unrelated baseline check must be named in advance, not discovered and excused after scoring.

If those checks, dependencies, isolation or existing execution facilities cannot be made ready within preparation's limit, stop with an inconclusive preparation result. Do not replace a task, repair Madar or enlarge the harness. The current plan makes no claim that these checks already exist, pass, or fit the execution deadline.

Each attempt starts from a separate pristine source snapshot and the same immutable local dependency supply. No retained Madar graph/index, target build cache, task history, author memory or another attempt's patch enters it. Keep unavoidable OS/provider cache effects visible; cold application state is not a claim of cold hardware or model cache.

N has ordinary tools. M and C have those same tools plus their pinned public Madar stdio MCP server, default-auto graph mode and the same short capability orientation. Preserve each version's shipped defaults; freeze configuration during preparation and do not tune retrieval for individual tasks. Tool use is optional; source reading and follow-up verification are allowed and counted. Disable unrelated personal plugins/memory/hooks equally, preserve common repository instructions and applicable execution rules, and verify tool configuration before authors start. Never expose the planned checks or earlier answers to one condition only.

Run serially in this fixed balanced order:

Task	First	Second	Third
T1	N	M	C
T2	M	C	N
T3	C	N	M

No pilot attempt is a warm-up. No model/session is resumed across conditions. Authors receive their time limit and a direction to submit their best current patch and verification results before it expires. No fixed shell-call count is used as success or as an undisclosed cutoff.

Correctness, time and work

Freeze each submitted patch as-is. One fresh independent reviewer per attempt sees the task, pristine baseline, anonymously labelled patch and common behavioral checks, without the condition, author transcript or other patches. Reviewers assess behavior, regressions, public API/types, requested documentation and scope. Reviewers do not repair patches or coach authors. Run the same prepared checks on each submission and retain their raw results.

A correct completion meets every task behavior, the new API type checks and the predetermined relevant regression checks, with no material behavior regression. An incorrect feature or invented verification result fails. Ordinary ignored build outputs do not themselves fail correctness; deleting or changing source/tests to evade checks does. Keep any failed original submission unchanged.

Measure elapsed time from per-attempt workspace preparation through author completion and its independent verification/review, including provider setup/indexing, tool waits, source reading, author tests and errors. Run each attempt and its review consecutively; include any intervening coordination delay. Also report setup, author and review durations separately.

Record each shared preparation interval once, with disjoint categories. Allocate target-repository preparation equally to its scheduled attempts (six Yup attempts, three Got attempts), and each Madar version's install/build preparation to that version's three attempts. These allocations redistribute measured time; never add an allocated interval again as per-attempt setup or duplicate it in campaign totals. Common check creation and final experimental reporting are study overhead, excluded from the primary task-time ratio but included separately in complete campaign time/work totals. Report both the whole campaign clock and per-attempt clocks.

Record raw input, cached-input, output and reasoning counters when returned, without double-counting cached input or reasoning components. Report tool calls and follow-up reading descriptively. Token counts are work proxies, not dollars; do not translate them using API prices or claim equal monetary cost from missing billing measurements. Review/preparation usage must be included in campaign totals. Report missing counters explicitly; no token-savings claim without comparable counters.

Decision fixed before results

Primary time measure: each task's C/N ratio of task execution, independent verification/review and allocated product setup. Common check creation and final experimental reporting are excluded from this ratio and disclosed as study overhead above. These are practical decision thresholds chosen for this pilot, not an industry standard or a statistical confidence bound. Any benefit claim is about elapsed task time; work proxies and their tradeoffs remain separate reported results.

Apply decisions in this order: first assess the validity of the primary comparison. Invalid common expectations or primary measurement, an unresolved infrastructure interruption, or fewer than three correct N completions makes it inconclusive. If that comparison is valid, any C correctness failure or observed product timeout/noncompletion means no demonstrated benefit; such a failure is not missing timing merely because there is no successful completion time. Compute time thresholds only when all three C and N attempts completed correctly. A failure isolated to secondary M does not automatically invalidate C versus N.

• Scoped benefit observed: all three C and all three N tasks complete correctly; median C/N ratio is at most 0.80; C is faster on at least two of three tasks; no C task is more than 20% slower; timings and required allocations are complete. Actual Madar use must occur in the C tasks supporting the benefit. If the time criteria pass without that use, Madar attribution is inconclusive; report the Madar-available workflow observation separately.
• No demonstrated benefit: execution and measurement are valid, but C has a material correctness regression or does not meet those time thresholds. Preserve individual successes and failures.
• Inconclusive: preparation, common-check validity, missing primary measurements, quota/infrastructure interruption or insufficient correctly completed N tasks prevents the defined comparison. Distinguish a product timeout/crash observed with valid measurement from an ambiguous infrastructure failure; do not discard either.

Report secondary M results even if unfavorable. Fewer tool calls, a correct graph, green retrieval tests or a passing retry cannot substitute for these criteria. Missing secondary token counters limit work claims; they do not silently invalidate a fully measured time-only conclusion. No statistical-significance, repeatability, universal trust or competitor ranking follows from one attempt per cell.

Finite budget and stopping boundary

The future pilot has at most 19 Codex CLI sessions: one preparation session, nine authors and nine independent reviewers.

• Preparation: 45 minutes total, including at most 30 minutes of preparation CLI work.
• Each per-attempt setup plus author: 20 minutes maximum.
• Each independent verification/review: 5 minutes maximum.
• Coordination/final reporting: 30 minutes maximum.
• Entire campaign: 5 hours maximum, including preparation and reporting. Stop sooner if subscription allowance is unavailable.

Stop an attempt at its limit and retain its current result. Continue other scheduled attempts only within the remaining original budget; report any unstarted cells. No retries, author continuations, replacement tasks, threshold edits, API fallback, added sessions or extra diagnostic runs. Missing review at its deadline is unverified, not a pass. A failure does not start fixes under #697/#739/#710/#755/#756.

Before a later authorized execution, attach the compact preparation record containing the exact task/check/command hashes and verified environment identities. Preparation is part of the fixed budget, not a new project. Publishing this plan does not lift the existing execution pause.

Historical #736/#740 outcomes and their original criteria remain unchanged. Other-tool comparisons remain a separate later decision if a useful candidate is demonstrated. No code cleanup, upstream PR, release, issue closure or worktree deletion is included.

[mohanagy]

Execution started — 11 September 2026

The maintainer authorized completion of #754. The published task set, pinned Madar/source/model identities, outcome rules, order, session count and five-hour/45-minute preparation limits remain in force.

Preparation began at 2026-09-10 20:33:58 UTC (11 September in Dubai). Existing ChatGPT login is verified; the account reports 72% of its weekly Codex allowance remaining and no paid credit balance. No API authentication, purchase, reset or Actions dispatch is involved.

Two execution details are corrected before any author or preparation model run:

• The old observer always enforces a command-count limit. Reuse direct Codex CLI JSON capture with the already installed GNU timeout utility for deadline-only process control; no evaluator/runner implementation is needed.
• The bounded search did not find a usable Got dependency snapshot. The earlier assumption that one already existed was wrong. Create one shared dependency installation during this same preparation phase, retain its generated lock and installed-content identities, and reuse it for all Got conditions. This is one pre-run dependency resolution, not separate changing resolutions across conditions. It remains inside the original preparation limit; failure does not authorize a retry or task replacement.

No source/task outcome has been observed. This preparation correction is explicit and does not revise a result, prompt or success threshold. The original issue text and historical experiments are preserved. Nine scheduled task cells will be reported even if preparation prevents them from starting.

[mohanagy]

Execution settings frozen before author runs

• Runtime: Node22.22.3; CodexCLI0.153.4; gpt-5.6-sol/xhigh/default; ChatGPT authentication, no API-key environment.
• Timeout control: existing GNU timeout9.11; no operation-count cap.
• Native: ordinary tools, no configured Madar MCP.
• Madar M/C: use the shipped Codex integration MCP profile, MADAR_TOOL_PROFILE=strict, exposing context_pack and context_expand. The bare server defaults to core graph-navigation tools, but the actual Codex installer explicitly writes strict (both pinned src/infrastructure/install.ts, renderCodexMcpBlock). This choice is made before any author result and assesses the context-pack workflow. No graph-navigation-only result will be represented as context-pack evidence.
• Launch server using pinned Node/provider: serve --stdio --auto-refresh with each attempt's source workspace as cwd. Keep omitted graph positional argument: M resolves graph.json; C resolves canonical graph.madar.
• Public graph preparation: generate . with default extraction and output behavior; retain the generated HTML cost too. No prebuilt graph or task-specific retrieval tuning.
• Installed Codex defaults: startup timeout180s; tool timeout60s. Both consume the 20min per-attempt budget.
• No persistent integration, hooks or user config changes. Ordinary source reads and follow-up verification remain explicitly permitted in every task prompt, regardless of internal context-pack warnings. Tool use is optional.
• Same M/C orientation: "Madar context_pack and context_expand are available to help you locate source-backed context. Use them when useful. Ordinary source reading and verification remain available; verify relevant behavior against source and record missing evidence or tool failures."
• Child workspace-write sandbox permits networking to allow localhost HTTP tests; instructions restrict shell network requests to loopback. Web search and unrelated plugins/memory/hooks are disabled. Outbound source/API lookup is prohibited and any observed violation invalidates that attempt.
• Dependency preparation amendment is recorded in issue comment5625104765. Installed snapshots and generated lock are sealed locally before task runs; setup errors and wall-clock limitations are retained.

[mohanagy]

Preparation complete; author inputs and checks frozen before the first task attempt.

• Both target snapshots match their pinned upstream blobs (Yup: 74; Got: 127). Preparation changed no original source/test files.
• Madar M and C installed from their pinned lockfiles and built successfully. Built distribution manifests are sealed. Both use the Codex integration's strict profile with context_pack/context_expand; ordinary source fallback remains allowed.
• Yup and Got use one sealed dependency snapshot each. Got dependencies were installed once, then an offline lock was captured before author outcomes.
• The preparation CLI initially inherited Node 22.9.0 from the system login profile. Its raw record is retained. All six identical candidate/regression commands were then repeated under the required Node 22.22.3: candidate checks failed only on missing requested APIs; selected regressions passed. All future shells now preserve Node 22.22.3 through a task-local profile.
• Read-only review found T3's automatic URL check compares identity before toJSON rather than after. Its bytes remain sealed. The common reviewer instruction now requires actual before/after URL comparisons after both toJSON and JSON.stringify, identically for all three T3 submissions. Missing this verification cannot yield a pass.
• One preparation CLI session used; zero authors so far. ChatGPT login only; no API keys, purchases, resets or Actions runs.
• Preparation command timing contains some wall-clock output bounds, not precise subprocess durations. We will disclose allocation uncertainty; it cannot support an invented speed-benefit conclusion.

The original 9-cell order, correctness criteria, thresholds, session limits and deadlines are unchanged. No repair/retry cycle follows a failed submission.

SHA-256 frozen identities:

{
  "records": {
    "TASK-PROMPTS.json": "d0b90cc63fbaa1864adb6e865937cb76bd917878e5ccae3049b0dc081696982a",
    "COMMON-AUTHOR-INSTRUCTION.txt": "2291174be9b6027bbd4407f8af7322c0e6d223b12478d90dc04fd35e5d196ea7",
    "MADAR-ORIENTATION.txt": "8bd786f0225c7bd6b4ebfca54bef61c17e4097265c274ef92acf9560465c4343",
    "COMMON-REVIEW-ADDENDUM.md": "2d41f4bc1d9601a699694f4e4cbea0cdd2dc02d75e23a41c62780007e7ad811d",
    "TARGET-SOURCE-IDENTITIES.json": "adb569c3dfc6672d35885e168db1abc08efc0d0cb37448dac0f0327a52e31cd5",
    "DEPENDENCY-IDENTITIES.json": "6329fbead143f9cdb65e92d5513f1df919812e230d0d7fc3b859ce521e4160a6",
    "RUNTIME-IDENTITIES.json": "095a0b1428f11254c29b46d2eb77490d88cbf515d6c612ab095c389b505a76b3",
    "YARN-IDENTITY.json": "42a024b106877741819bb4f278adf695bb4968023f9d027bf5e363dd6e8db216",
    "M-BUILD-MANIFEST.json": "49c207de6c40501159cd08175129ea14304530579dbec05493cb549cfc80f042",
    "C-BUILD-MANIFEST.json": "274a36816a07a5eb8ccccc02e5e695d0aa7f934fe784684f109b9e8a6f74325f",
    "M-MCP-CONFIG-VERIFIED.json": "871b1c9438773cc837946e2ef7ce64ce526f658613caaed7d7f06b6b96ac34da",
    "C-MCP-CONFIG-VERIFIED.json": "961572a95bfd465063b90d3cc86a9890a77666cc7a3259c70f8760601aa9d230",
    "preparation-cli/READY.json": "5921c4543179f7575aa2c7d29403adf941c350a61a17cebb82e428f5c58805da",
    "pinned-node-validation/RESULT.json": "a4d536ff698b706364cd7b2a5e8339c34d010b417768a9dfa1b68b3408470c05"
  },
  "common_checks": {
    "checks/T1/runtime.test.ts": "c411c5a5a5372b452e2217626d0f51f4b90d495e47ec3598de14f137614dccfb",
    "checks/T1/public-api.types.ts": "136b5f8d7614f8d676ead89ab27058f9d7de6d2afb1b6116727051cc60b3b27d",
    "checks/T1/typecheck.tsconfig.json": "b20f350614ed7b73bf846d42febf8195b9218dbf368cda456301fa4c4a674d99",
    "checks/T1/run-candidate.sh": "4f1dc1fe70bb83fdd6ce00fb8c69ef24c70360f3a60593f70a6e67cc9bb3c7e7",
    "checks/T1/run-regressions.sh": "b70b2b30375b910d850fbc60d4fcddea9b2d1ddbdd5d62f9e7cb547b55cbaf01",
    "checks/T1/COVERAGE.md": "5920c7b3750f2f6a948a521093f896afc6800149b1d1645504f789b5586decd2",
    "checks/T2/runtime.test.ts": "64e4d7b32b7f48ae24dc899682549a5a38020bd619a0367619eff2da34c6f13c",
    "checks/T2/public-api.types.ts": "a0bb075c641891d04143d6af449d546229248150c319ca91b1ef044175d43848",
    "checks/T2/typecheck.tsconfig.json": "7fd102628af5eb6e2bed8011f9d09c0862ec167d35fe69825e5a5cc0fe39d7f9",
    "checks/T2/run-candidate.sh": "ee73dcb1af2f4e9f4a5c75f29f9e53ec9bbcd814fb33d02858e68d9c81b702c3",
    "checks/T2/run-regressions.sh": "3cc8987aaa1c6e18fc5a5b3d5367a3c7d8b7b1061e6be723fa992a4e4d810b9e",
    "checks/T2/COVERAGE.md": "ca350f40a8adb9eee0592fa647a59c7eee444c0b1672acd3ecc741f286a74775",
    "checks/T3/runtime.test.ts": "e441bf1d85470451bb8faec5c93fe3d8d2605940cd63fcee03bf00cecf36ae3e",
    "checks/T3/public-api.types.ts": "0013b876e42c9ef506908f0563abd5772510d9ab23637a69852ffab6ff14475c",
    "checks/T3/typecheck.tsconfig.json": "ce1b307795a36d58a402c9a3c744fadf7b11f0869d2db349a02df09a46ec0aa3",
    "checks/T3/run-candidate.sh": "fc1e4b31457a0c6674dee6856fb1767ff7507cf679788b38d54d75fad73b1fd4",
    "checks/T3/run-regressions.sh": "73ce544bf0d65e617cf7a8ccec2fbb1838db6d8b9e7c6622ddd81b0c80b7bb8a",
    "checks/T3/COVERAGE.md": "d1b2abef17debc17fa5aa09978613836ee281811a1fb47114ca819472dabec88",
    "README.md": "0513ee5a9cfed3e51508d56059ed815384e9874a6c855bcdc877e132317c1f91"
  }
}

[mohanagy]

#754 terminal result — Inconclusive

The bounded pilot is concluded. It did not establish whether Madar reduces work. Required shared preparation timings were not captured completely enough to allocate them without overlap. This is an experiment setup failure by the coordinator, not a Madar product failure. The eight remaining author attempts were not launched once that validity defect was confirmed; the already-running Native author was allowed to finish and received its single independent review.

The full nine-attempt comparison was not completed. Closing this bounded research run does not mean Madar is qualified or its main goal is achieved. No speed, token-savings, cost-equivalence, trust-percentage, repeatability or competitor-superiority claim follows.

All scheduled cells

Cell	Outcome	Madar tool use	Interpretation
T1-N	Completed; independent PASS	0; Native condition	Correct on this task; timing allocation incomplete
T1-M	Not started	Not run	Stopped after comparison validity failure
T1-C	Not started	Not run	Stopped after comparison validity failure
T2-M	Not started	Not run	Stopped after comparison validity failure
T2-C	Not started	Not run	Stopped after comparison validity failure
T2-N	Not started	Not run	Stopped after comparison validity failure
T3-C	Not started	Not run	Stopped after comparison validity failure
T3-N	Not started	Not run	Stopped after comparison validity failure
T3-M	Not started	Not run	Stopped after comparison validity failure

Correct Native completions: 1/3. Correct C completions: 0 measured; all unstarted. M completions: 0 measured; all unstarted. Unstarted cells are not product failures. C/N ratios and the median were not calculated. The frozen validity rules select inconclusive, before any benefit threshold is assessed.

Native result retained

T1 added Yup string().maxBytes(limit, message?) in four source/test/documentation files. The fresh reviewer received an anonymous copy, baseline, task and sealed common checks, without the condition or author transcript. It returned PASS with no material findings:

• Common feature suite: 6/6 passed; public API TypeScript check exited 0.
• Predetermined relevant regressions: 6 passed, across sync/async projects.
• Additional review of the author's focused tests: 12 passed, across sync/async projects.
• Source review supported UTF-8 and lone-surrogate behavior, numeric/reference limits, messages, transformations, absence, exclusive replacement, unchanged max(), browser compatibility, documentation and scope.

The author separately recorded 879 passed / 2 skipped, type checking, lint, declaration and browser builds. Those broader author checks were not all independently repeated. Its original submission and generated outputs remain retained. The reviewer initially ran an extra focused command from the wrong directory, then ran it successfully from the candidate; both logs are retained. No source repair or author continuation occurred. The anonymous review source still matches the four submitted changed files.

Measurement failure and other deviations

1. Missing complete preparation timing: redirected-log modification times recorded last output, not process completion. Later polls only bound completion. M/C installs overlapped; M/C builds overlapped each other and dependency copying. The original tool receipts narrowed bounds but cannot reconstruct exact, disjoint ownership. Summing those windows or using last-output time as completion would invent a primary measurement. TIMING-AUDIT-AND-STOP.json retains the exact receipt references and limitations.
2. Git metadata isolation: the first author copy retained the original upstream worktree's .git pointer. HEAD remained the pinned Yup commit, with no observed prior task commits or answer/history retrieval. The shared index bytes subsequently changed, consistent with a Git stat-cache refresh; a read-only cached diff showed no staged semantic changes. Original source files remained unchanged. Full metadata isolation was not achieved; no answer contamination is asserted.
3. Preparation runtime: its initial checks inherited Node22.9. The same sealed commands were revalidated under required Node22.22.3 before the author; all expected missing-feature baseline failures and green regressions reproduced. Both author/reviewer confirmed Node22.22.3.
4. T3 check gap: the automated suite checked URL identity before toJSON. Before any author result, the common review addendum required actual post-call non-mutation verification for every T3 review. No T3 attempt occurred, so no T3 result is claimed.
5. Retained setup errors: the unavailable prior Got dependency snapshot was replaced once during authorized preparation and frozen before outcomes; a failed offline shrinkwrap command, initial read-only source-copy permission failures, an initial MCP inspection flag error and initial offline Yarn-cache miss remain recorded. None caused an author retry or model substitution.

These limitations supersede the earlier readiness declaration. The correct decision was to prevent further author dispatch once primary timing validity could not be established. The lead should have checked the measurement and copy boundaries before starting T1-N.

Direct timings and work

Measured component	Seconds
T1-N source/dependency setup, including its initial copy error	44.290
T1-N author session	326.765
Intervening coordination before reviewer setup	50.207
Reviewer setup	8.189
Reviewer CLI	172.855
Direct attempt cycle, workspace preparation through review exit	602.311

Shared product preparation allocation is incomplete, so the last row is not a complete primary comparison time. Exact partial copy allocations are Yup 1.486s per scheduled Yup attempt, Got 2.388s per scheduled Got attempt, M source 0.216s per M attempt, C source 0.302s per C attempt; they omit other required preparation and are not substituted for it. Common-check creation/revalidation and final study reporting are study overhead. The preparation declaration was made 2159.7s after campaign start. Complete disjoint subcategory attribution is unavailable and is explicitly unresolved rather than zeroed or double-counted.

Whole campaign clock: 2026-09-10 20:33:58 UTC through 2026-09-10 21:23:07 UTC at report sealing: 49.16 minutes. Final GitHub close-out receipts are retained separately. Preparation and review work is included in the following CLI counters:

CLI session	Input	Cached input (subset)	Output	Reasoning output (subset)	Completed command events
Preparation	2,817,803	2,609,920	38,833	13,320	67
T1-N author	785,318	737,920	15,085	7,049	42
T1-N reviewer	204,336	168,320	8,097	3,834	16
Total	3,807,457	3,516,160	62,015	24,203	125

Cached input and reasoning are not added again to their parent counters. These are recorded CLI work counters, not money. Coordinator and read-only collaboration counters are unavailable; no claim of complete account token accounting is made. Source reading and failed commands are included in the raw command records. Three of nineteen permitted CLI sessions were used: one preparation, one author, one reviewer. No retries, resumed authors, replacement tasks or threshold changes.

Environment, cost boundary and artifacts

Requested model gpt-5.6-sol, reasoning xhigh, tier default; Codex CLI 0.153.4; author/reviewer Node 22.22.3. The service did not attest an exact backend revision. M and C were built at the frozen commits 3371ada and 2b14450, but neither was used in an author task.

Only the existing ChatGPT subscription route was used. No paid API, credit purchase, reset redemption, billing change or GitHub Actions run. The account-wide weekly used percentage was 28% at the start and 30% at the last check; that is shared account usage, not exact attribution to this run. The original Madar working tree retains the same main/behind-seven state and five existing untracked paths; no Madar product changes, PR, cleanup or worktree deletion were performed.

Local evidence directory: /Users/mohammednaji/Documents/Codex/2026-09-04/files-mentioned-by-the-user-madar/outputs/issue-754-execution-2026-09-11. Raw CLI events, errors, prompt/job files, all common-check hashes, dependency/build manifests, the original submitted patch, review results, timing audit and the complete nine-row register are retained. They are local artifacts; this comment does not claim they were uploaded to GitHub.

Key SHA-256 identities:

• Original submitted patch: eadfe69f73b9e345a626c574ec8ad318ce6441b28bf34e0222cfda83293283b4
• Original submission file manifest: 575ec3f75808441d0882b71d61f1efef099454fc6e168eac4d884e61f6acad6a
• Independent review result: faf42a5618840f6fbc454d7aa417c4287bd97e58d4a1a35ff8a7f205cec7a33d
• Sealed common-check readiness record: 5921c4543179f7575aa2c7d29403adf941c350a61a17cebb82e428f5c58805da
• Final work/timing record: ff6b27b51f65ef1c502d4c88ab2b0a99c6dec3d200ddd052a5b19c7c828f865d

Disposition: close this bounded pilot as inconclusive, retain its incomplete comparison and failed setup, and make no automatic follow-up changes under other issues. Madar's practical benefit remains unproven by #754.

Verified close-out

GitHub reports #754 CLOSED at 2026-09-10T21:24:31Z. Total campaign elapsed through closure verification: 51.34 minutes, including 131.13s of report publication and close-out after the first report seal. CLOSEOUT-RECEIPT.json records the full clock; its SHA-256 is 21f269fcef09608f724fa7191cc564843e1f46fc9506db91c0fb137c128c05e2. This bookkeeping addition does not change the inconclusive verdict or any attempt result.

[mohanagy]

Authorized corrected execution v2 — 11 September 2026

The user authorized proceeding to the usefulness conclusion without intermediate approval stops. Reopen the existing user story; preserve v1's inconclusive result and every v1 artifact as a separate failed execution. No v1 author result enters the v2 comparison.

V2 changes execution only:

• Codex CLI corrects the existing direct execution workflow to bracket actual process waits with start/end clocks, retain errors/timeouts, run product preparation sequentially and account for each shared interval once. The setup CLI's outer duration is an envelope; nested product intervals are not counted again as study overhead.
• Every source checkout has its own real Git directory/index and detached pinned HEAD. No copied worktree pointer, shared Git metadata or carried task patch/index enters an author workspace.
• A cheap local preflight verifies capture, private checkout metadata, pinned runtime, common-check hashes and the existing baseline expectations before model authors begin. Existing sealed target dependency snapshots are reused and verified; their historical acquisition cost is disclosed, while current verification/copy costs are measured. Both Madar providers are installed/built sequentially from their pinned locks using the local cache.

Retain the same three tasks, N/M/C commits, strict MCP profile, optional Madar use, common behavioral checks, T3 post-call URL non-mutation review addendum, fixed nine-cell order, success thresholds, fresh independent reviewers and no-retry rule. Use fresh authors for every v2 cell, including T1-N. No product fixes, changed task expectations or new evaluator platform.

The original limits apply anew to this authorized v2: preparation45min including one setup CLI of at most30min; each setup+author20min; each review including its setup5min; at most19 CLI sessions; total5h. A failed product attempt remains a result. No patch/review repair loop or budget increase follows. Account all nine scheduled cells, including any unstarted after quota/infra interruption.

ChatGPT subscription authentication only; no paid API, purchases, resets, billing changes or Actions runs. Codex CLI owns implementation. Other issues remain outside this work unless a concrete execution blocker establishes relevance.

V2 output directory: outputs/issue-754-execution-v2-2026-09-11; campaign clock recorded in START.json. No author has started at publication. Final verdict will use the original validity precedence and scoped-benefit/no-demonstrated-benefit/inconclusive rules. Reopening does not erase v1 or establish Madar's usefulness.

[mohanagy]

Madar #754 — corrected execution v2 terminal report

Verdict: INCONCLUSIVE. Madar's intended usefulness remains unproven. All nine fixed cells finished. Seven independent reviewers recorded PASS and two timed out without a verdict. A final source audit then found a concrete missing-options defect in two Got submissions that had received PASS. The final record therefore contains five passes without an outstanding finding, two recorded passes with a static correctness finding, and two unverified submissions. Original review verdicts and patches are preserved.

The comparison did not establish that Madar produces sufficiently trustworthy context or reduces coding work. It also does not establish that Madar's main technical goal is impossible.

All nine results

N is native Codex; M is Madar main; C is Madar next. Elapsed time includes per-cell setup, author work, review and allocated product preparation.

Cell	Final evidence status	Accounted elapsed
T1-N	PASS	9:40
T1-M	PASS	10:17
T1-C	PASS	10:17
T2-M	Unverified: review timed out	15:56
T2-C	PASS	12:42
T2-N	Unverified: review timed out	16:22
T3-C	PASS recorded; static defect found	12:06
T3-N	PASS	10:22
T3-M	PASS recorded; static defect found	11:11

T1 is Yup UTF-8 maxBytes; T2 is Yup ValidationError.flatten; T3 is Got RequestError.toJSON. All nine common behavioral/type and predetermined regression command sets returned success, but those checks did not catch the Got gap below and cannot replace missing review verdicts.

What failed and what the results mean

The required comparison could not be completed validly. T2-M and T2-N completed their automated checks, but their independent reviewers exhausted the original five-minute budget without saving RESULT.json. No pass was backfilled. The frozen decision first requires three correct native completions; T2-N is unverified, so INCONCLUSIVE takes precedence. T1-N also reached its review deadline, but its complete RESULT.json had been saved before that deadline; that recorded pass is retained with the timeout disclosed.

The final source audit found a coverage gap. Existing Got code explicitly constructs RequestError when request options are undefined. T3-M destructures this.options, and T3-C reads this.options.method, without guarding that case. On this path their serializers throw instead of returning the required name/code projection. T3-N uses optional access and its already-recorded got(false) test passed. This is a source-derived defect; no new reproduction, test, patch or reviewer was run. The original common absent-URL fixture and the URL non-mutation addendum used existing options objects and missed this path. Correctness acceptance for T3-M/C is withheld. This concerns generated Got patches and this evaluation's coverage, not a demonstrated Madar graph defect.

Madar's context tools were unused. All six M/C authors made zero Madar context calls and read no graph contents. T3-C demonstrably discovered both callable context tools and left them unused. Other cells have configuration or server-start evidence with weaker proof of the exposed inventory. Non-use does not establish tool unavailability. Passing an optional Madar-enabled session cannot establish retrieval quality or a causal benefit from Madar.

The timings provide no support for the planned benefit. T1, the only C/N pair left without an outstanding correctness/review finding, took C/N = 1.064638: C was 6.464% slower. T3's recorded elapsed ratio is 1.166702, or 16.670% slower, but C has the static correctness finding, so this is only a descriptive timing observation. No qualifying T2 ratio or three-task median is calculated. The original thresholds remain unchanged: all C/N correct, median at most 0.80, C faster on at least two tasks, no C more than 20% slower, and actual Madar use supporting attribution.

Execution boundaries and preserved failures

The exact pins remain M 3371ada and C 2b14450. Yup is ff31eee8a2b10c938bb3a22544d9e33957a8cf01; Got is 687eb7dcc100ea3e548ebba227173b886789e670. Node 22.22.3, Codex CLI 0.153.4, requested gpt-5.6-sol/xhigh/default and strict Madar profile were fixed. No exact backend model revision is claimed.

V1 remains separate and inconclusive; none of its author scores enters v2. V2 preparation reached READY within the original 45 minutes. The setup CLI suffered a transport interruption and resumed the same logical conversation within its original 30-minute envelope. Its generated launcher failed two retained preflight attempts; direct coordination used its existing operation primitives and sealed checks. The launcher and failed attempts remain intact. There were 19 logical CLI conversations and 20 process invocations because preparation used two processes. Each of the nine authors and nine reviewers was fresh; none was resumed or retried.

Private Git directories, pinned pristine source and common dependency supply were verified before authors. Every original submission remained unchanged through review. Madar-generated out/ artifacts were retained in originals and omitted from anonymous review copies. Existing dependency snapshots were reused: v2 counts current verification/copy/build work, while historical v1 dependency acquisition cost is not charged again. OS and provider caches were not claimed cold.

Retained deviations include T1-C and T2-C task-generated temporary output outside their workspace, T2-M navigation into its own parent metadata, blocked Corepack registry lookups, rejected cleanup commands, failed initial checks, and corrections. No other-condition source read was observed in the retained command audit; this was not a syscall or exhaustive network audit. Got repository-wide lint had 31 errors in unchanged baseline files; separate focused lint/type/test successes are not presented as a passing full npm test. T3-N's author typecheck preceded later edits; its independent common typecheck covered the final frozen submission. Two earlier audit false positives were withdrawn: ignored declaration files had not disappeared, and T2-C's successful package test included lint.

Time and usage

Campaign start: 2026-09-11T04:35:34.185030+00:00. Last cell sealed: 2026-09-11T07:03:12.623022+00:00. Through all nine cells: 2h 27m 38.4s. Shared product preparation was 57.258s counted once. Study overhead through the last cell was 2324.265s; final report/publication time is added in the final clock receipt. Preparation's CLI parent envelope is not double-counted with its nested intervals.

Recorded CLI usage, including preparation, all authors and all reviewers: 10,954,150 input tokens (including 9,904,384 cached) and 245,769 output tokens (including 111,278 reasoning). Preparation and three timed-out reviewers lack complete terminal consumption, so these are recorded lower-bound counters. Coordinator and read-only audit model work is not separately metered in these records. They are work counters, not dollars or a token-savings claim.

Only the included ChatGPT subscription route was used. No paid API, purchase, reset redemption, billing change or Actions run was requested. At 07:01:19 UTC, the account-wide weekly window showed 48% used and 52% remaining; that is not an isolated campaign measurement. User Madar root Git status matches its starting state, and the final process check found no remaining campaign CLI/MCP process.

Scope of the conclusion

This is one attempt per cell, with two Yup tasks and one Got task. It cannot establish general correctness percentages, statistical repeatability, competitor ranking or universal trustworthy context. The run has exhausted its fixed attempts. There is no automatic repair/rerun chain. #754 is closed as a completed bounded study with an inconclusive outcome; the main product goal remains open. No Madar production patch, other issue, release, upstream PR or worktree cleanup is part of this result.

Evidence retained locally under outputs/issue-754-execution-v2-2026-09-11. Raw private Codex rollouts are not published. This terminal result preserves v1 and follows the authorized v2 amendment.

Evidence hash index: EVIDENCE-SHA256.json, SHA-256 1ded9ad2fa206c7b0bf9e8f6229aed397034a190ed20cc0261b168f58242aadd (87 retained records).