Reason
During PR #402 (Add validation case val-2j), several issues were observed with the AI agent workflow in TMAP8 that could be improved for future development:
-
Agents consistently forgot to copy session .jsonl files to doc/content/agent/<issue-number>/ before committing, despite the requirement in AGENTS.md. The current instruction is buried in a paragraph and easy to skip.
-
Agents did not recognize that a separate light test case is unnecessary when the heavy case already runs in less than 2 seconds.
-
Context size may be a limiting factor for complex validation cases — reading C++ code, input files, and documentation consumes a large part of the available context.
-
Agents sometimes failed to define acronyms on first use (e.g., ODEs).
-
The symlink step needed to generate documentation figures is not documented in AGENTS.md.
Design
Proposed improvements to AGENTS.md and agent workflow:
-
Pre-commit checklist: Move the JSONL copy and commit trailer requirements from prose into a bold, standalone checklist that agents cannot skim past. Optionally, automate via a Claude Code pre-commit hook in .claude/settings.json or a /commit skill. A possible solution to the large size of .jsonl files is to include a simplified and summarized .jsonl file for each PR.
-
Light/heavy test guideline: Add a rule that a separate light test case is unnecessary if the heavy case completes in under 2 seconds.
-
Context management strategies: For complex validation cases that may exceed agent context:
- Define agent teams in
AGENTS.md to separately read codespace, documentation, and inputs, so sub-agents can summarize and transfer information to the main agent without impacting its context.
- Introduce a "skills" structure within TMAP8 so tasks can be divided into sub-agents with subtasks without requiring large shared context.
- Future models with larger context windows may also address this.
-
Acronym convention: Add a rule to AGENTS.md requiring full definitions on first use of acronyms.
-
Documentation figure symlinks: Add instructions to AGENTS.md for creating symlinks needed to generate documentation figures (e.g., ln -s ~/projects/TMAP8/test/tests/val-2j/comparison_val-2j.py ~/projects/TMAP8/doc/content/verification_and_validation/comparison_val-2j.py).
Impact
Reason
During PR #402 (Add validation case val-2j), several issues were observed with the AI agent workflow in TMAP8 that could be improved for future development:
Agents consistently forgot to copy session
.jsonlfiles todoc/content/agent/<issue-number>/before committing, despite the requirement inAGENTS.md. The current instruction is buried in a paragraph and easy to skip..jsonl. PR Add validation case val-2j: Tritium TDS from Li2TiO3 solid breeder #402 added 9 MB of.jsonldata, which could lead to unnecessary repository growth in future PRs.Agents did not recognize that a separate light test case is unnecessary when the heavy case already runs in less than 2 seconds.
Context size may be a limiting factor for complex validation cases — reading C++ code, input files, and documentation consumes a large part of the available context.
Agents sometimes failed to define acronyms on first use (e.g., ODEs).
The symlink step needed to generate documentation figures is not documented in
AGENTS.md.Design
Proposed improvements to
AGENTS.mdand agent workflow:Pre-commit checklist: Move the JSONL copy and commit trailer requirements from prose into a bold, standalone checklist that agents cannot skim past. Optionally, automate via a Claude Code pre-commit hook in
.claude/settings.jsonor a/commitskill. A possible solution to the large size of.jsonlfiles is to include a simplified and summarized.jsonlfile for each PR.Light/heavy test guideline: Add a rule that a separate light test case is unnecessary if the heavy case completes in under 2 seconds.
Context management strategies: For complex validation cases that may exceed agent context:
AGENTS.mdto separately read codespace, documentation, and inputs, so sub-agents can summarize and transfer information to the main agent without impacting its context.Acronym convention: Add a rule to
AGENTS.mdrequiring full definitions on first use of acronyms.Documentation figure symlinks: Add instructions to
AGENTS.mdfor creating symlinks needed to generate documentation figures (e.g.,ln -s ~/projects/TMAP8/test/tests/val-2j/comparison_val-2j.py ~/projects/TMAP8/doc/content/verification_and_validation/comparison_val-2j.py).Impact
AGENTS.md(formatting and new guidelines, no changes to existing requirements).claude/settings.jsonhook or.claude/skills/commit.md