A clean-room, evidence-first account of building scientific agents across four real competition tasks.
中文说明 · Architecture · Case studies · Research notebook · Reproducibility
AI4S Agent Lab is a public learning and engineering record derived from our 2026 AI for Science competition project. The private GitCode repository is the canonical internal record for versioned source, documentation, and decision/evidence indexes; it is not a complete payload archive. Some original logs, official data, submission artifacts, third-party payloads, and the final-defense Keynote are not stored in GitCode; whether they remain available must be verified separately in their authorized storage locations. Those restricted materials cannot be published as-is. This repository therefore starts from a new Git history and contains only from-scratch public-safe code, synthetic examples, reconstructed evidence, and reviewable documentation. Here, “clean-room” names that publication boundary; it is not a legal claim of independently isolated reverse engineering.
The central lesson is deliberately modest:
A scientific agent is not made trustworthy by adding more agents or more language-model calls. It becomes useful when observations change the next action, scientific tools produce real evidence, validators control promotion and rollback, and every claim is bounded by what the evidence can support.
According to team-held platform and review records, we entered four tracks, advanced to the final after review at rank 6, and obtained a combined score of 1.7414, calculated from the two strongest normalized track scores: protein ensemble 0.9075 and molecule design 0.8339. The public repository reports these records but does not contain an independently verifiable official score archive.
| Case | Platform result | Normalized score | Public lesson |
|---|---|---|---|
| Virtual screening | Mean EF1% 39.70 |
0.8095 |
Measure real throughput before allocating expensive inference. |
| Targeted molecule design | 0.7575 |
0.8339 |
Docking feedback can change the next generation instead of merely ranking the last one. |
| Protein conformational ensemble | 0.7355 |
0.9075 |
A one-run high point, a seeded operating range, and a causal LLM claim are different statements. |
| Neural-operator PDE | 176.46 → 88.23 |
0.5181 |
Both prebuilt task-specific training tools were ruled out of bounds; both subtasks were halved. We accept the ruling. |
These are historical competition results, not a claim that this public repository can reproduce the scoring images or official data. See Results and limitations.
- Read this page.
- Scan the capability ledger.
- Open the molecule-design case and the tool-governance postmortem.
You will see a project that spans system architecture, scientific tooling, failure recovery, evidence design, reproducibility boundaries, and an honest compliance postmortem—not a polished demo that hides the unsuccessful paths.
- Read Architecture and The research loop.
- Inspect the clean-room core and synthetic examples in this repository.
- Read Evidence model and the reconstructed traces.
- Run the commands in Reproducibility.
- Read all four case studies.
- Follow the iteration timeline and negative-result ledger.
- Review multi-agent, context, and memory boundaries.
- Evaluate the claims using the R1–R4 framework and benchmark matrix.
The four competition systems did not share one universal solver, one runtime state machine, one model, or one control depth. The real shared layer was thin: language-model access, JSONL call logging, a competition-spec snapshot, basic validation and packaging patterns, and common engineering discipline.
Each task still required its own representation, scientific instruments, quality gates, budget policy, and output contract. This repository turns those lessons into a cleaner reference architecture:
flowchart LR
A["Task contract"] --> B["Observe and measure"]
B --> C["Propose a bounded action"]
C --> D["Run a scientific tool"]
D --> E["Collect evidence"]
E --> F{"Verifier gate"}
F -->|promote| G["Update best known state"]
F -->|reject or fail| H["Diagnose, retry, or roll back"]
G --> I{"Budget still worth spending?"}
H --> I
I -->|yes| B
I -->|no| J["Validate and deliver atomically"]
The reusable object is the research-control discipline. The scientific instruments remain task-specific.
The public example calibrates a synthetic decay process. It is deliberately small and dependency-free: the point is to inspect the contract, floor, proposals, tool results, verifier decisions, rollback, evidence log, and atomic best-artifact delivery—not to imitate a competition dataset.
python -m venv .venv
source .venv/bin/activate
python -m pip install -e .
python -m ai4s_agent_lab \
--output-dir artifacts/decay_demo \
--iterations 6 \
--run-id local-demo
python -m unittest discover -s tests -vInspect artifacts/decay_demo/evidence.jsonl and best_model.json. A rejected or failed proposal should be recorded as a rollback and must not replace the validated best artifact. The latest observed release status remains in the verification report.
The v0.1 executable targets POSIX systems (Linux and macOS): its evidence lock and durability path use fcntl, fsync, and same-filesystem atomic replace. Windows behavior is not part of the R2 claim.
A scale-and-scheduling case: a large newly mounted workload, a fixed wall-clock budget, a cheap full-coverage floor, and a more expensive 3D path. The important control decision was not “which magical model wins?” but “how much expensive inference can the measured machine actually finish while preserving complete, ordered output?”
The deepest feedback loop in the project: pocket analysis → candidate generation → validity and synthesis filters → real docking → feedback-driven fragment reweighting → another generation → higher-fidelity redocking → route search and validation. Version evidence supports that docking feedback changed later sampling. More precisely, a recoverable internal versioned run-config comparison shows that v115c → v115ga adds only GOAL_AWARE_FRAGS=1 at source level. The platform history records a separate cross-version score observation, v115=0.752650 → v115ga=0.757559 (+0.004909), while contemporaneous notes called that magnitude within evaluation noise. v115c lacks an independently scored platform run, so the score observation is not the platform A/B counterpart of the source-level toggle. The complete scoring image and score-bound log carriers are also missing, and no repeated equal-budget runs estimated variance; the cross-version difference is therefore not a stable independent net-effect estimate. The case also contains useful negative results where plausible additions reduced the score.
A multi-solution and uncertainty case: folding and sampling engines produced candidates, data-dependent gates decided which branches ran, and quality/diversity rules selected an ensemble. The best historical run was about 0.7355; two later seeded runs were about 0.719–0.720. The evidence does not support attributing the best score to successful online LLM control.
The most important failure case. A ReAct-style controller performed real runtime actions and the models trained at evaluation time, but two prebuilt tools already contained task-specific training and metric-alignment logic. Reviewers halved both subtasks. Runtime execution is not the same as runtime scientific invention.
| Capability | Historical implementation | Evidence strength | Public clean-room status |
|---|---|---|---|
| Task understanding | Input discovery, task-specific parsing, resource probes | Code path + stage records | Demonstrated on synthetic contracts |
| Hypothesis generation | Bounded LLM proposals in some tasks; deterministic gates in others | Uneven by task | Interface provided; no universal-agent claim |
| Real experimentation | Docking, folding, sampling, training, validation | Strongest where tool outputs and platform records align | Scientific backends are not redistributed |
| Feedback-driven iteration | Strong in task2; parameter/action feedback in task4; narrower elsewhere | Task-specific | Generic promotion/rollback pattern |
| Multi-agent oversight | Supervisor-like second opinion in later task-specific versions | Advisory, not strongly isolated | Presented as a pattern, not a completed universal system |
| Cross-run memory | Development documents and version control, not a runtime memory service | No unified runtime proof | Future work only |
| Context compression | A bounded recent-message window in one controller; summaries elsewhere were ad hoc | Partial | Not implemented; evaluation plan only |
| Reproducibility | Versioned evidence varied by task | R1–R4 differ | R2 verified for the synthetic core; R4 is not claimed |
The detailed, claim-by-claim version is in Capability ledger.
We do not publish raw competition logs. They can contain official input structure, internal paths, service metadata, and operational details. We also do not manufacture a smooth “perfect run.” Instead, the public evidence layer contains:
- an explicit evidence hierarchy;
- a schema for observation → action → tool result → verification → promotion;
- reconstructed and labeled traces based on source-code paths, tool outputs, version notes, and platform results;
- negative-result records and claim boundaries.
Every reconstructed event has reconstructed: true. It is an explanatory artifact, not an original log line.
This project did not implement a single, strongly isolated multi-agent organization shared by all four tasks. Later task-specific versions included a supervisor role that combined deterministic checks with an optional LLM opinion. In at least one controller, main and supervisor roles shared the same process and model client; the supervisor was advisory rather than an independent security boundary.
Likewise, the competition runtime did not provide a proven cross-session long-term memory system, a Socratic-question memory, or a universal half-hour memory compressor. Development-time knowledge lived in repository documents, progress ledgers, prompts, and version history. See Multi-agent, context, and memory.
We separate four different promises:
| Level | Promise | This public release |
|---|---|---|
| R1 | Source and claims can be reviewed | Verified by manifest, release scan, and review passes |
| R2 | A licensed synthetic example runs end to end | Verified locally and in Python 3.10–3.14 CI on POSIX runners |
| R3 | The full scientific environment can be rebuilt | Not yet claimed |
| R4 | Historical platform results can be repeated | Not claimed |
“Open source” does not automatically grant permission to redistribute model weights, official data, or third-party training assets. Full details: Reproducibility and Third-party notices.
ai4s-agent-lab/
├── src/ and tests/ clean-room control primitives and tests
├── examples/ synthetic, license-safe examples
├── case_studies/ four evidence-bounded technical narratives
├── lab_notebook/ hypotheses, iterations, negative results
├── evidence/ schemas and reconstructed traces
├── benchmarks/ evaluation design, not a claimed leaderboard
├── docs/ architecture, claims, provenance, reproducibility
└── release/ public boundary and verification manifest
The Apache-2.0 license in this repository covers only the original clean-room materials committed here. It does not relicense the private competition repository, team/company materials, official competition data, third-party source code, model weights, or online services.
Human contributors remain responsible for architecture, scientific choices, evidence review, version selection, and release decisions. Development-time programming agents assisted implementation, review, and documentation. Runtime language models, where used, were replaceable control components and should not be confused with the scientific backends that performed docking, folding, sampling, or training.
See Contributions, Provenance, and Third-party notices.
This is a post-competition public reconstruction and learning archive, not the original submission repository. Before citing executable status, check the verification report and the current release tag.