[NovrusShehaj]

Summary

Opening certain completed verification runs failed in the browser with a CORS error. Behind each one was an HTTP 500 from GET /verify/{workflow_id}. CORS itself was configured correctly. The API crashed while rendering a stored report that contained NaN, and Starlette sent that crash response from outside CORSMiddleware, so it had no CORS headers.

This PR:

1. Renders NaN and ±Infinity as JSON null in any response, instead of failing it.
2. Makes unhandled 500s carry CORS headers, so a future server bug shows up in the browser as a 500 rather than a misleading CORS error.
3. Rejects non-finite comparison tolerances with 422.
4. Adds the investigation record, backend/docs/verify-500-cors-fix-implementation-plan.md.

The full write-up covers the root cause, where the NaN comes from, the four fixes, technical debt and follow-ups, plus a diagnosis runbook. It is in that document.

Root cause

Confirmed from production logs and data:

1. A simulator returned NaN. In run compose-api-prod-comparison-212-… (BIOMD0000000155), COPASI 4.45.296 returned only the first output step of the time course: time 0.0, 0.01, where PySCeS ran to 10.0 in 1001 points. Seven of its 18 variables were NaN at that step, and one was NaN at both points.
2. The NaN entered the report silently. The simulation-data service sends NaN as the string "nan", and Pydantic converts it to a float NaN without warning. calc_stats then scored the all-NaN variable as nan, even when comparing COPASI with itself. Temporal stored the report as is.
3. Rendering failed. GET /verify read the report, and Starlette's JSONResponse (json.dumps(allow_nan=False)) raised:

   ValueError: Out of range float values are not JSON compliant: nan
   

4. The 500 had no CORS headers. Starlette renders unhandled exceptions in ServerErrorMiddleware, which sits outside CORSMiddleware. The 500 had no Access-Control-Allow-Origin, so the browser reported a CORS error and hid the status.
5. The list and the dialog disagreed. The list showed the run as COMPLETED, because the final status is recorded before the response is rendered. The dialog showed PENDING and polled forever.

Scope: six verifications produced 127 of these 500s on 2026-10-22 and 2026-10-23, 85 of them for run 212. Each one later became a normal 404 … no longer retained once its Temporal history expired, about a day after completion.

Changes

backend/biosim_server/common/json_response.py (new)

NonFiniteSafeJSONResponse is the app's default_response_class. It renders NaN and ±Infinity as null, which is also what Pydantic's model_dump_json produces.

• No extra cost for normal responses. It tries Starlette's normal render first and walks the content only if that render raises ValueError.
• Every route is covered, including stored results and other endpoints that return simulator data.
• Other errors still raise as before. Circular content raises ValueError("Circular reference detected") instead of recursing into RecursionError, and content shared without a cycle is kept.

The frontend already handles null in these fields: formatScore shows —, the analytics composables use ?? null, and a null in chart data is drawn as a gap.

backend/biosim_server/common/server_errors.py (new)

ServerErrorCorsMiddleware is registered inside CORSMiddleware, the same pattern as the existing 413 from MultipartBodyLimitMiddleware. It renders an unhandled exception as {"detail": "Internal Server Error"} with status 500, then re-raises it, as ServerErrorMiddleware does:

• uvicorn still logs the full traceback.
• Test clients that raise application exceptions still raise.
• No exception text reaches the client.

@app.exception_handler(Exception) would not work here: Starlette installs that handler on ServerErrorMiddleware, which is outside CORS.

backend/biosim_server/api/main.py

• Wires up the response class and the middleware.
• Rejects non-finite tolerances. rel_tol, abs_tol_min and abs_tol_scale on POST /verify/omex and POST /verify/runs now use allow_inf_nan=False.
  • Before, ?rel_tol=nan started a workflow whose every score was NaN, at the cost of a rate-limit unit and one simulator job per simulator.
  • These values now get a 422 before any upload, ledger write or workflow start.
  • The OpenAPI spec is unchanged (regenerated, no diff).
• Removes the hard-coded https://biosim.biosimulations.org and https://api.biosim.biosimulations.org origins added in this branch's first commit.
  • biosim-gke already allows the frontend through CORS_EXTRA_ORIGINS, and requests from the API host to itself are same-origin.
  • Deploy-specific hosts belong in that env var, as documented above APP_ORIGINS and in backend/CLAUDE.md.

Other

• backend/docs/verify-500-cors-fix-implementation-plan.md: the investigation record described above.
• backend/CLAUDE.md: a note in the CORS section on why error responses must stay inside CORSMiddleware, and that NaN renders as null.
• .gitignore: ignores the machine-local .perch/ directory.

Tests

backend/tests/common/test_json_response.py, 8 tests:

• A report shaped like run 212's (NaN and Inf in score, NaN in sim_run_data values) returns 200 from GET /verify/{id}, with nulls and CORS headers.
• Normal content renders byte-for-byte as before.
• NaN and ±Inf become null at any depth.
• Non-JSON types still raise TypeError.
• Three cycle cases raise ValueError instead of RecursionError, and content shared without a cycle is kept.

backend/tests/common/test_server_errors.py, 3 tests:

• An unhandled exception returns a 500 with access-control-allow-origin and doesn't leak the exception message.
• The exception is still raised to the server.
• No second response is sent once headers are out.

backend/tests/api/test_verification_completion.py, 18 cases: nan, inf or -inf in any tolerance, on either POST endpoint, returns 422 naming the parameter. There is no upload, no ledger call and no workflow start.

The regression tests fail without the fix:

• The NaN report test fails with the production ValueError … nan.

• The 500 test fails on the missing CORS header.

• All 18 tolerance cases fail.

• pytest: the full backend suite passes (backend-ci: 1547 passed, 17 skipped)

• ruff check and mypy biosim_server tests are clean

• CI is green (backend-ci, frontend-ci, smoke, GitGuardian, Snyk)

Deployment

Merging changes nothing in production on its own. Releasing it takes:

• a backend release (backend-v0.10.3)
• a biosim-gke deploy PR, following backend/CLAUDE.md → Deploy

The release also includes the unreleased verify-workflow change from #126 (17e213a). That change doesn't alter the Temporal command sequence, so no worker drain is needed.

Technical debt and follow-ups (not in this PR)

The review of this PR accepted it as an immediate fix and recorded the handling of non-finite values as technical debt: NaN and Inf should be treated as the sign of a bad result, not passed along as a value. The investigation supports that view. Full detail and priorities are in §8–§9 of the document.

• High: calc_stats reports false matches on non-finite input.

  • A NaN at some time points is skipped.
  • A single +Inf makes a variable's tolerance infinite, so [100, inf, -50] vs [1, 2, 3] scores 0 and counts as a match.
  • Negative tolerances make every variable match.

  Non-finite input should become an explicit "not comparable" error. How to represent that should be agreed with reviewers first.

• High: Fix 3, frontend. The run dialog should stop polling after repeated failures, including failures with no status, and show the error instead of PENDING. The plan is in §5.4.

• Medium:

  • Log a warning whenever the null fallback is used.
  • Investigate COPASI 4.45.296's truncated output.
  • Put bounds on the tolerances (zero and negative values are still accepted).
  • Validate values from the simulation-data service when they are ingested, instead of relying on Pydantic's silent "nan" conversion.

• Low / deferred:

  • A lossless encoding for non-finite raw data.
  • Annotating score and values as list[float | None] in the schema.
  • Fix 4: a request ID on 500 responses, planned in §5.5.

🤖 Generated with Claude Code

[NovrusShehaj]

CORs errors present when accessing the specific run

• Link to run: https://biosim.biosimulations.org/utilities/verify-model?tab=lookup&workflow_id=compose-api-prod-comparison-212-69b4e144-b3b3-4e94-94cc-3127e88ff460

[jcschaff]

the NaN and infinity issues in JSON (and the use of NaN as a flag rather than an error condition) has been a thorn in my side for years. Maybe it is okay to push this through right now, but we'll have to revisit it in the future. In some projects I have used custom serialization/deserialization to marshall these special values (NaN, +inf, -inf). push it through now, but I consider this tech debt. for instance, why was there NaN or Inf in the first place, this should have been the result of a bad result. I am a bit strict about these things, because otherwise you end up tripping over these problems much futher down the line, or ignoring them completely.