Skip to content

fix(ml): old schema support - #13

Merged
UsatovPavel merged 3 commits into
mainfrom
fix/HACK-186-equipment-schema
Sep 28, 2026
Merged

UsatovPavel merged 3 commits into
mainfrom
fix/HACK-186-equipment-schema

Conversation

@UsatovPavel

Copy link
Copy Markdown
Contributor

Jira

HACK-186

Production incident

The ML v4 deploy on main started the equipment inference image before the Backend migration. Equipment warmup completed, but each tick then failed with column "journal_is_alarm" does not exist; the container remained starting and stopped producing predictions. The deploy run has the protected 72-hour snapshot already saved.

Fix

  • Equipment queries use a typed NULL for the incident-only raw alarm column. Incident queries still read the real column after Backend migration.
  • The mandatory pre-switch backup can continue when the old container is running and its recent logs prove this exact missing-column failure. It still copies and verifies the cursor, model, reference views and 72-hour inputs before switching images; other unhealthy states block deployment.

Verification

  • python -m pytest -q: 70 passed.
  • bash -n deploy/backup-equipment-window.sh.
  • Added regression tests for the pre-migration equipment schema and guarded recovery backup.

Merge and deploy this hotfix before Backend #66. Confirm inference becomes healthy and writes fresh equipment predictions, then proceed with incident rollout.

@UsatovPavel
UsatovPavel merged commit 2b94709 into main Sep 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant