verify: 34 /verify/omex workflows stuck IN_PROGRESS, 6 that 500 on GET, and 27 FAILED with no reason (backend v0.10.2 lacks #154) #156

compose-api submitted 96 archives to POST /verify/omex between 19:59 and 20:24 UTC on 2026-10-22. That was the triage of the biomodels-curated-300 suite (docs/plan-verify-oracle.md in compose-api). Each run used include_outputs=true and the default five simulators, at about 4 per minute. 29 completed. The other 67 hit three separate problems, and as of 04:37 UTC on 2026-10-23 none of them has changed.

1. 34 workflows stuck in IN_PROGRESS for more than 8 hours

Every one was submitted between 19:59 and 20:24 UTC, and each GET still returns IN_PROGRESS with workflow_results: null. A single archive normally takes about 1–2 minutes (BIOMD0000000012 on 3 simulators took 22 s per run), so these look stuck rather than queued. Things worth checking in Temporal:

• whether these workflows have a pending activity (a poll_biosim_run_activity waiting on a biosimulations.org run that never finishes?);
• whether the worker is still picking them up.

compose-api polls them every 30 s and gives up 6 hours after the last resume (around 05:30 UTC). Resubmitting would only add to the same backlog.

The 34 workflow ids

• compose-api-prod-comparison-61-2f0f5b20-9054-4656-9004-857d51d13a14
• compose-api-prod-comparison-65-867aca80-bc90-4c48-acdd-5dd297563dbb
• compose-api-prod-comparison-67-d96e3051-286f-4ab9-8ea7-42038f319ac8
• compose-api-prod-comparison-105-b8bf8b37-4bdf-4068-8422-351e5ed779cb
• compose-api-prod-comparison-113-441837aa-5781-41c0-8984-3d55ad2e27b1
• compose-api-prod-comparison-116-9c2dcef0-48fd-483c-b3ed-4c1efff4bf1c
• compose-api-prod-comparison-121-b91e2f65-675f-4f29-a3b4-b6aed1f82d76
• compose-api-prod-comparison-128-db0880e4-179a-4dbb-b9bd-2a42ef0e9574
• compose-api-prod-comparison-145-fa596394-fbd2-48f8-bed7-6ead5f1fb449
• compose-api-prod-comparison-166-f4500a8b-2b3b-4171-9007-84d6f07a1a08
• compose-api-prod-comparison-167-67b3fa20-adda-40d6-b2c1-2564e9454127
• compose-api-prod-comparison-168-310fe394-7352-46b0-a3b2-000cb0e62281
• compose-api-prod-comparison-179-950d560b-4967-40b2-ba0d-50aacb726d78
• compose-api-prod-comparison-180-7bf00b02-24bf-4187-8798-d7f859775f48
• compose-api-prod-comparison-200-d443de36-6370-496a-8157-a38d0104c719
• compose-api-prod-comparison-209-98dc6916-8326-4829-85c4-32a2de9e38ad
• compose-api-prod-comparison-210-95272e07-f99e-4358-99d5-a7f5030676fb
• compose-api-prod-comparison-218-7e71b48a-b7ff-4c5d-ab2f-0ae3a33e5288
• compose-api-prod-comparison-219-41a3f7dd-d58c-4260-8c1c-3354b3dcca03
• compose-api-prod-comparison-220-9377ccbc-a3b8-4b34-84e6-a752c68bb1b9
• compose-api-prod-comparison-221-456dbb96-bbd1-4d2f-9869-65b7beb759da
• compose-api-prod-comparison-222-951b3a5b-f76b-47ab-a350-e6d23d27834e
• compose-api-prod-comparison-230-45ffc3f5-aeef-4737-9e3c-546736650c22
• compose-api-prod-comparison-239-c852f0e3-efb3-419f-b195-cb81e5559de5
• compose-api-prod-comparison-259-b56e1596-8e58-44eb-b7f3-51598a54167b
• compose-api-prod-comparison-280-404ab76e-f95c-4657-a25d-627113d59798
• compose-api-prod-comparison-283-9ab036ae-a206-4678-9720-ddf360572c08
• compose-api-prod-comparison-284-25fd4fd7-ba37-4257-8516-e969e1ab1943
• compose-api-prod-comparison-295-880b8671-c06e-4da8-81a6-d92c11fcdaad
• compose-api-prod-comparison-306-217f5ba4-3873-47d7-adb4-6e973b011e68
• compose-api-prod-comparison-311-0d85eddb-0858-40ea-a125-b1b88cab43a2
• compose-api-prod-comparison-338-0bdf497e-2afe-4d96-8126-f325dbee05a1
• compose-api-prod-comparison-339-1bdefce5-5cda-4d8d-84d2-65de1f8b0889
• compose-api-prod-comparison-341-2a0432ed-c56f-491e-b27b-7a2efec91379

2. GET /verify/{workflow_id} returns 500 for 6 workflows

This happens on every request (compose-api retried each with backoff, and it still answers 500 now). These workflows are probably in a state the GET handler can't read: a Temporal result that fails to deserialize, or a history query timing out.

The 6 workflow ids

• compose-api-prod-comparison-82-8b88a051-e0ba-4e91-9569-382d33cc971e
• compose-api-prod-comparison-83-ab51112d-e104-450d-b367-b11cd9c67e50
• compose-api-prod-comparison-92-b4a12218-d6fd-4fd7-9a39-5305bd730f2e
• compose-api-prod-comparison-211-0c44af1f-d7e9-4144-a059-6d9525589099
• compose-api-prod-comparison-212-69b4e144-b3b3-4e94-94cc-3127e88ff460
• compose-api-prod-comparison-253-e3768dc5-26d3-4330-a231-ad65bfb81f7c

3. 27 workflows FAILED, with no way to tell why

Each one has workflow_error: "Verification workflow ended with status FAILED", no sims_run_info, and no failed_simulators field. Production runs backend v0.10.2, which doesn't include #154 (merged after backend-v0.10.2, and the only backend change since). With #154, each response would name the simulators whose biosimulations.org runs failed and give their log reason. Releasing a backend with #154 is the fix, and compose-api can then re-verify these 27.

The 27 workflow ids

• compose-api-prod-comparison-74-b1753a0b-f9ab-4c1a-82fd-af65a8018e28
• compose-api-prod-comparison-78-2039b721-d5e5-474f-b03b-cdf86329e71e
• compose-api-prod-comparison-99-f0c2bdf1-5ee8-410d-bf25-6d459bdde68c
• compose-api-prod-comparison-132-d19f839d-bb64-4658-826b-138f4628d60b
• compose-api-prod-comparison-134-5c9cdd04-93a3-4e43-9fa4-188cb42e41ad
• compose-api-prod-comparison-175-3161bc13-fa9b-4389-9e96-15b31c216146
• compose-api-prod-comparison-178-eb7a74e9-d411-4a8c-b3c6-90898910f04e
• compose-api-prod-comparison-194-5f4f4ee2-d4b6-48c0-aa2b-a277ea47fe5e
• compose-api-prod-comparison-202-61f71859-1c14-43d9-935d-20c21bb02818
• compose-api-prod-comparison-225-42b3db53-b967-44bd-8531-61e4107e43af
• compose-api-prod-comparison-240-f77fd392-4529-47d9-8cb5-06b4a9d628dc
• compose-api-prod-comparison-245-e0aaf255-60d6-48a4-8ef5-5d6600cd712d
• compose-api-prod-comparison-246-71eb4890-ac3a-4dfc-97a5-35fc0980bc42
• compose-api-prod-comparison-249-e38f9311-40e4-411d-b9cb-913620d30550
• compose-api-prod-comparison-252-31a34bc5-c58a-41db-9314-b024827a98e9
• compose-api-prod-comparison-255-4aac50aa-1524-40d3-b4fa-524dbf73efbd
• compose-api-prod-comparison-256-6eefe6d2-49d0-47fb-97d2-a9f90f7091d7
• compose-api-prod-comparison-263-b08d6504-db56-4279-9c85-407cd80ab8ac
• compose-api-prod-comparison-271-ef7f11ee-36c1-4c6b-9833-0b9eea4fd576
• compose-api-prod-comparison-289-e27990df-4060-4b12-804a-eae983a2943e
• compose-api-prod-comparison-294-9b452587-4b80-4ef7-8573-1b2574904253
• compose-api-prod-comparison-304-4783d71d-2095-472d-ad94-8d997ea0491e
• compose-api-prod-comparison-321-72764c86-7eba-48d9-b95f-67c5c36ddfeb
• compose-api-prod-comparison-330-048bdf99-3799-42bc-9713-112309c0054e
• compose-api-prod-comparison-331-10a5002d-79f9-45ed-9373-b9a2582f4e4d
• compose-api-prod-comparison-332-0736e506-c791-4604-bee4-2b2aa04ad4dc
• compose-api-prod-comparison-351-e59e1881-c892-4b7a-b671-adf99decc82d

Load

This was about 96 × 5 = 480 legacy runs in 25 minutes. Note that compose-api's 4-per-minute pace counts workflow starts, not the legacy runs each one creates. If that's too much for the legacy service, say so and compose-api will lower verify_per_minute (or drop simulators) before the next triage.

🤖 Generated with Claude Code