Repository navigation
fix(verify): report simulators that produced no results, and fail when none did - #154
Merged
Merged
Conversation
…n none did
POST /verify/omex answered COMPLETED with empty results when every
simulator's run failed on biosimulations.org: OmexVerifyWorkflow skipped
each child without a run, so neither the run ids nor the reasons reached
the caller. Reproduced 2026-10-22 with BioModels' own BIOMD0000000012
archive, whose manifest lists files the zip does not hold: all five runs
fail BioSimulators' archive validation (run 6ada2269041d31202094ca8f), and
the run's own error field is empty.
- VerifyWorkflowOutput.failed_simulators (additive): one SimulatorFailure
{simulator_version, biosimulations_run_id, error_message} per requested
simulator that produced no results, in request order.
- poll_biosim_run_activity reads a failed run's reason from its logs
(error_from_run_logs: the archive-level exception, else the first SED
document's or task's), best effort.
- When no requested simulator produced results the workflow ends FAILED
with workflow_error naming each simulator's reason, so #136's write-once
ledger records FAILED rather than COMPLETED. The statistics activity is
still scheduled, so workflows in flight replay deterministically.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem.
POST /verify/omexansweredCOMPLETEDwith empty results whenever every simulator's run failed on biosimulations.org.OmexVerifyWorkflowskipped each child that had no run (continue), so neither the run ids nor the reasons reached the caller. A client can't tell that from a success. That's the "100% concordance for a run that compared nothing" case Novrus found in #126.Reproduction (2026-10-22, anonymous, default five simulators).
compose-api-verification-3214309a-01f8-4a3d-a0a1-0151276e88c4:COMPLETED, with emptysims_run_infoandcomparison_statistics.6ada2269041d31202094ca8f) endedFAILEDwith an emptyerror. The reason was only inGET /logs/{id}: "/root/archive.omexis not a valid COMBINE/OMEX archive … Content elementBIOMD0000000012-biopax2.owlis invalid. Location is not a file". BioModels' manifests list files their zips don't contain, and BioSimulators validates strictly.Changes (on top of #136):
VerifyWorkflowOutput.failed_simulatorsis a new, additive field: oneSimulatorFailure {simulator_version, biosimulations_run_id, error_message}per requested simulator that produced no results, in request order.poll_biosim_run_activityreads a failed run's reason from its logs, best effort, usingerror_from_run_logs: the archive-level exception, otherwise the first SED document's or task's.FAILED, withworkflow_errornaming each simulator's reason. feat(verify): link OMEX hash to verification runs and return run metadata #136's write-once ledger then recordsFAILEDinstead ofCOMPLETED. The statistics activity is still scheduled, so workflows already in flight replay deterministically.For #126:
VerificationStatus.vuealready showsworkflow_erroronFAILED. It would also help to listfailed_simulators, including on a partly successfulCOMPLETED, with links to/runs/{id}, and to leave failed simulators out of the concordance figures (details in this comment).Tests.
tests/biosim_verify/test_failed_simulators.pycovers reading the reason from the real failed run's logs (recorded astests/fixtures/local_data/biosim_failed_run_logs.json), the task-level fallback, failures reported in request order, theFAILEDmessage, and that a partial success isn't a failure.biosim_server tests) are clean.tests/biosim_verify tests/biosim_runs tests/apigives 417 passed, 2 failed.test_main.py::test_omex_verify_and_get_outputandtest_runs_verify_and_get_output, fail the same way on unmodifiedmainwith this machine's.env, and are skipped without it. So they come from the local environment, not this change.🤖 Generated with Claude Code