feat: exit after single cycle for source=run_and_exit workflows by catastrophe-brandon · Pull Request #428 · OpenShift-Fleet/rehor

catastrophe-brandon · 2026-08-12T18:14:05Z

Summary

When instance.yaml sets source: scheduled, the runner now exits after completing one iteration instead of sleeping and looping
This applies to all three preflight outcomes: start (cycle ran), skip (nothing to do), and error
Lets KEDA be the sole source of truth for scheduling -- the pod runs once and exits, KEDA scales it back to zero, and brings it up again at the next scheduled window

Previously, scheduled workflows relied on a cycle-sleep.json file (24h pause) to prevent repeated runs within the same KEDA window. This was fragile across pod restarts -- when the pod came back up, the sleep state was lost and the bot would run again, wasting tokens on duplicate scans.

The source field was already parsed from instance.yaml and logged but never acted on. This change gives it operational meaning.

Test plan

All existing 176 preflight/run tests pass
4 new tests verify exit-after-one-iteration for all three outcomes (start, skip, error) when source=scheduled, and continued looping when source=jira
Deploy a source: scheduled instance and confirm the pod exits cleanly after one cycle

When instance.yaml sets source=scheduled, the runner now exits after completing one iteration (whether preflight skipped, errored, or the cycle ran) instead of sleeping and looping. This lets KEDA be the sole source of truth for scheduling -- the pod runs once and exits, KEDA scales it back to zero, and brings it up again at the next window. Previously scheduled workflows relied on a cycle-sleep file (24h pause) to prevent repeated runs within the same KEDA window, which was fragile across pod restarts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Verify the runner exits after a single iteration for all three preflight outcomes (start, skip, error) when source=scheduled, and continues looping when source=jira. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The previous tests reimplemented the production loop logic, meaning they could pass even if the real code regressed. Now tests call bot.run.main() directly with mocked dependencies and verify the loop iteration count. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

A transient preflight failure (GitHub API timeout, network blip) was causing scheduled workflows to exit immediately with no retry, meaning the next scan wouldn't happen until the next KEDA window (24h gap). Now retries up to 3 times with 30s between attempts before giving up. A successful retry resets the counter and continues normally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

catastrophe-brandon · 2026-08-12T18:35:57Z

when the pod came back up, the sleep state was lost and the bot would run again, wasting tokens on duplicate scans.

This is exactly what's happening in the quality monitor instance, so we've proven the case that the behavior needs to be corrected.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Moving `from bot.run import main` to module level caused argparse to see the CI runner's argv before the sys.argv patch was active. Use a _run_main() helper that imports inside the patched context instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

catastrophe-brandon · 2026-08-12T19:05:03Z

/retest

florkbr · 2026-08-12T19:38:54Z

@Hyperkid123 are you aware of anyone relying on the current behavior for scheduled? I like the idea of relying on Keda versus the sleep but there are some tradeoffs. WDYT

Hyperkid123 · 2026-08-13T12:08:33Z

@florkbr yes, OBSINT agent is using it for watch duty and triage I think.

REHOR-62's e2e harness starts the bot container without BOT_INSTANCE_ID, but bot/run.py has required --instance-id (or BOT_INSTANCE_ID env var) since May. The entrypoint's final `exec uv run dev-bot --label ...` was failing argparse validation immediately after printing the "Credentials configured" handoff line, killing the container right as 40-entrypoint.sh read its logs. Depending on timing, docker logs either caught the line before the container died or raced it, which is why e2e-minimal has been intermittently failing with "entrypoint did not reach final startup stage" across unrelated PRs (seen on PR OpenShift-Fleet#428 and others). Every other invocation path (docker-compose, deploy template, Tekton pipelines) already sets BOT_INSTANCE_ID; the e2e harness was the one place that didn't. Signed-off-by: Vishali <vsanghis@redhat.com>

…avior Other teams depend on the existing source=scheduled behavior (loop with sleep). Introduce keda_scheduled as a distinct type for the exit-after-one-cycle behavior, preserving scheduled as-is. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add a dedicated table explaining the behavioral difference between jira, scheduled, and keda_scheduled source types so new instance owners can pick the right one without guessing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The name emphasizes that the workload runs to completion and exits, rather than tying the concept to a specific scheduler. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

REHOR-62's e2e harness starts the bot container without BOT_INSTANCE_ID, but bot/run.py has required --instance-id (or BOT_INSTANCE_ID env var) since May. The entrypoint's final `exec uv run dev-bot --label ...` was failing argparse validation immediately after printing the "Credentials configured" handoff line, killing the container right as 40-entrypoint.sh read its logs. Depending on timing, docker logs either caught the line before the container died or raced it, which is why e2e-minimal has been intermittently failing with "entrypoint did not reach final startup stage" across unrelated PRs (seen on PR #428 and others). Every other invocation path (docker-compose, deploy template, Tekton pipelines) already sets BOT_INSTANCE_ID; the e2e harness was the one place that didn't. Signed-off-by: Vishali <vsanghis@redhat.com> Co-authored-by: Martin Marosi <marvusm.mmi@gmail.com>

…ift-Fleet#431) REHOR-62's e2e harness starts the bot container without BOT_INSTANCE_ID, but bot/run.py has required --instance-id (or BOT_INSTANCE_ID env var) since May. The entrypoint's final `exec uv run dev-bot --label ...` was failing argparse validation immediately after printing the "Credentials configured" handoff line, killing the container right as 40-entrypoint.sh read its logs. Depending on timing, docker logs either caught the line before the container died or raced it, which is why e2e-minimal has been intermittently failing with "entrypoint did not reach final startup stage" across unrelated PRs (seen on PR OpenShift-Fleet#428 and others). Every other invocation path (docker-compose, deploy template, Tekton pipelines) already sets BOT_INSTANCE_ID; the e2e harness was the one place that didn't. Signed-off-by: Vishali <vsanghis@redhat.com> Co-authored-by: Martin Marosi <marvusm.mmi@gmail.com>

Scheduled error retries now use the same exponential backoff as the non-scheduled path instead of a fixed 30s delay. The give-up path after 3 consecutive preflight errors now calls sys.exit(1) so KEDA can distinguish a failed pod from a successful one. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Expect SystemExit(1) when retries are exhausted, and exponential backoff sleep values instead of the old fixed 30s delay. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

catastrophe-brandon and others added 6 commits August 12, 2026 13:11

test: add tests for source=scheduled exit behavior

5054b1a

Verify the runner exits after a single iteration for all three preflight outcomes (start, skip, error) when source=scheduled, and continues looping when source=jira. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

format: ruff format

7420d14

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: remove unused ScriptResult import

d42e26e

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

catastrophe-brandon mentioned this pull request Aug 12, 2026

fix: skip anti-pattern scan if already scanned today RedHatInsights/hcc-framework-agent-dev#76

Closed

2 tasks

catastrophe-brandon and others added 2 commits August 12, 2026 13:44

refactor: dedupe repeated main import in scheduled-exit tests

ffd4720

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

florkbr reviewed Aug 12, 2026

View reviewed changes

Comment thread bot/run.py Outdated

florkbr reviewed Aug 12, 2026

View reviewed changes

Comment thread bot/run.py

vishsanghishetty mentioned this pull request Aug 13, 2026

fix(e2e): set BOT_INSTANCE_ID for container-e2e bot container #431

Merged

4 tasks

catastrophe-brandon and others added 5 commits August 13, 2026 10:41

Merge branch 'master' into exit-after-scheduled-cycle

b7a38bb

refactor: rename keda_scheduled to run_to_completion

ae0a808

The name emphasizes that the workload runs to completion and exits, rather than tying the concept to a specific scheduler. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

refactor: rename run_to_completion to run_and_exit

88360ac

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

catastrophe-brandon and others added 3 commits August 14, 2026 12:42

Merge branch 'master' into exit-after-scheduled-cycle

e0e3202

test: update scheduled exit tests for backoff and sys.exit changes

21cc5aa

Expect SystemExit(1) when retries are exhausted, and exponential backoff sleep values instead of the old fixed 30s delay. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

catastrophe-brandon changed the title ~~feat: exit after single cycle for source=scheduled workflows~~ feat: exit after single cycle for source=run_and_exit workflows Aug 14, 2026

petrsimon mentioned this pull request Aug 18, 2026

chore: remove obsolete instance /wake API and dashboard wake UI (REHOR-118) #440

Merged

petrsimon mentioned this pull request Sep 7, 2026

docs: refresh Rehor documentation #473

Merged

4 tasks

Navigation Menu

feat: exit after single cycle for source=run_and_exit workflows - #428

feat: exit after single cycle for source=run_and_exit workflows#428
catastrophe-brandon wants to merge 16 commits into
OpenShift-Fleet:masterfrom
catastrophe-brandon:exit-after-scheduled-cycle

catastrophe-brandon commented Aug 12, 2026

Uh oh!

catastrophe-brandon commented Aug 12, 2026 •

edited

Loading

Uh oh!

catastrophe-brandon commented Aug 12, 2026

Uh oh!

Uh oh!

Uh oh!

florkbr commented Aug 12, 2026

Uh oh!

Hyperkid123 commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

catastrophe-brandon commented Aug 12, 2026

Summary

Test plan

Uh oh!

catastrophe-brandon commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

catastrophe-brandon commented Aug 12, 2026

Uh oh!

Uh oh!

Uh oh!

florkbr commented Aug 12, 2026

Uh oh!

Hyperkid123 commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

catastrophe-brandon commented Aug 12, 2026 •

edited

Loading