feat(jobs): add claude-sonnet-4.6, gpt-5.2-codex, and zai/glm-4.7 eval results - #1
Open
DESU-CLUB wants to merge 2 commits into
Open
feat(jobs): add claude-sonnet-4.6, gpt-5.2-codex, and zai/glm-4.7 eval results#1DESU-CLUB wants to merge 2 commits into
DESU-CLUB wants to merge 2 commits into
Conversation
|
…l results
Run artifacts:
- claude-4-6-sonnet (2026-04-22 14:24)
- gpt-5.2-codex (2026-04-22 15:14)
- zai/glm-4.7 (2026-04-23 12:58) re-run; previous run's trajectory
exceeded GitHub's 100 MB file limit
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DESU-CLUB
force-pushed
the
feat/add-models
branch
from
April 23, 2026 22:15
78dda28 to
7f8c7e6
Compare
DESU-CLUB
changed the title
Replace task_35 and task_38 trials in all 4 job runs (gemini-flash, claude-4-6-sonnet, gpt-5.2-codex, glm-4.7) with re-runs against the patched task_35 verifier, and update job-level result.json stats accordingly. Also replace tasks/task_35 with the fixed version: test_final_state.py now calls execute() with validated input directly instead of wrapping in inputData, matching createTool's actual contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DESU-CLUB
force-pushed
the
feat/add-models
branch
from
April 24, 2026 22:57
7f8c7e6 to
0e7db20
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds full run artifacts for three agent/model combinations against the Mastra benchmark task set:
claude-4-6-sonnet(2026-04-22 14:24)gpt-5.2-codex(2026-04-22 15:14)zai/glm-4.7(2026-04-23 12:58)The original
zai/glm-4.7run from 2026-04-22 is not included: one of its trajectories (~380 MB) exceeded GitHub's 100 MB per-file limit so the run was discarded and re-executed. The 2026-04-23 re-run's largest trajectory is ~1 MB, well under the limit.Result overview
Task 35 is an environment/verifier mismatch with the current
@mastra/coretool.executesignature (affects every model) — fix is tracked separately, not part of this PR.Test plan
git ls-tree -r -l origin/feat/add-models | awk '$4 > 100000000'returns empty (no file exceeds the GitHub size limit).site/) renders the three new job rows without errors.🤖 Generated with Claude Code