Skip to content

fix(verify): report simulators that produced no results, and fail when none did - #154

Merged
jcschaff merged 2 commits into
mainfrom
fix/verify-report-failed-simulators
Oct 22, 2026
Merged

jcschaff merged 2 commits into
mainfrom
fix/verify-report-failed-simulators

Conversation

@jcschaff

Copy link
Copy Markdown
Contributor

Problem. POST /verify/omex answered COMPLETED with empty results whenever every simulator's run failed on biosimulations.org. OmexVerifyWorkflow skipped each child that had no run (continue), so neither the run ids nor the reasons reached the caller. A client can't tell that from a success. That's the "100% concordance for a run that compared nothing" case Novrus found in #126.

Reproduction (2026-10-22, anonymous, default five simulators).

  • BioModels' own BIOMD0000000012 archive gave workflow compose-api-verification-3214309a-01f8-4a3d-a0a1-0151276e88c4: COMPLETED, with empty sims_run_info and comparison_statistics.
  • Running one child by hand (COPASI 4.45.296, run 6ada2269041d31202094ca8f) ended FAILED with an empty error. The reason was only in GET /logs/{id}: "/root/archive.omex is not a valid COMBINE/OMEX archive … Content element BIOMD0000000012-biopax2.owl is invalid. Location is not a file". BioModels' manifests list files their zips don't contain, and BioSimulators validates strictly.

Changes (on top of #136):

  • VerifyWorkflowOutput.failed_simulators is a new, additive field: one SimulatorFailure {simulator_version, biosimulations_run_id, error_message} per requested simulator that produced no results, in request order.
  • poll_biosim_run_activity reads a failed run's reason from its logs, best effort, using error_from_run_logs: the archive-level exception, otherwise the first SED document's or task's.
  • When no requested simulator produced results, the workflow ends FAILED, with workflow_error naming each simulator's reason. feat(verify): link OMEX hash to verification runs and return run metadata #136's write-once ledger then records FAILED instead of COMPLETED. The statistics activity is still scheduled, so workflows already in flight replay deterministically.
  • The OpenAPI spec is regenerated.

For #126: VerificationStatus.vue already shows workflow_error on FAILED. It would also help to list failed_simulators, including on a partly successful COMPLETED, with links to /runs/{id}, and to leave failed simulators out of the concordance figures (details in this comment).

Tests.

  • tests/biosim_verify/test_failed_simulators.py covers reading the reason from the real failed run's logs (recorded as tests/fixtures/local_data/biosim_failed_run_logs.json), the task-level fallback, failures reported in request order, the FAILED message, and that a partial success isn't a failure.
  • Locally, ruff and mypy (biosim_server tests) are clean. tests/biosim_verify tests/biosim_runs tests/api gives 417 passed, 2 failed.
  • The two failures, test_main.py::test_omex_verify_and_get_output and test_runs_verify_and_get_output, fail the same way on unmodified main with this machine's .env, and are skipped without it. So they come from the local environment, not this change.

🤖 Generated with Claude Code

jcschaff and others added 2 commits October 22, 2026 11:08
…n none did

POST /verify/omex answered COMPLETED with empty results when every
simulator's run failed on biosimulations.org: OmexVerifyWorkflow skipped
each child without a run, so neither the run ids nor the reasons reached
the caller. Reproduced 2026-10-22 with BioModels' own BIOMD0000000012
archive, whose manifest lists files the zip does not hold: all five runs
fail BioSimulators' archive validation (run 6ada2269041d31202094ca8f), and
the run's own error field is empty.

- VerifyWorkflowOutput.failed_simulators (additive): one SimulatorFailure
  {simulator_version, biosimulations_run_id, error_message} per requested
  simulator that produced no results, in request order.
- poll_biosim_run_activity reads a failed run's reason from its logs
  (error_from_run_logs: the archive-level exception, else the first SED
  document's or task's), best effort.
- When no requested simulator produced results the workflow ends FAILED
  with workflow_error naming each simulator's reason, so #136's write-once
  ledger records FAILED rather than COMPLETED. The statistics activity is
  still scheduled, so workflows in flight replay deterministically.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jcschaff
jcschaff merged commit 957feb5 into main Oct 22, 2026
5 checks passed
@jcschaff
jcschaff deleted the fix/verify-report-failed-simulators branch October 22, 2026 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant