Skip to content

feat: resilient background job retry & monitoring#667

Open
Manishcs076 wants to merge 1 commit intorohitdash08:mainfrom
Manishcs076:feat/job-retry-monitoring
Open

feat: resilient background job retry & monitoring#667
Manishcs076 wants to merge 1 commit intorohitdash08:mainfrom
Manishcs076:feat/job-retry-monitoring

Conversation

@Manishcs076
Copy link
Copy Markdown

Summary

Implements background job retry with exponential backoff and a monitoring dashboard.

/claim #130

Features

  • BackgroundJob model with 6 status states (PENDING/RUNNING/SUCCESS/FAILED/RETRYING/DEAD)
  • @with_retry decorator for wrapping any function as a retryable job
  • Exponential backoff with configurable max_retries, base_delay, backoff_factor
  • Thread-safe execution
  • Dead letter queue for permanently failed jobs
  • REST API endpoints for stats, recent jobs, dead letter queue, and retry
  • Frontend Job Monitor page with stats cards, job list, and retry button
  • 15 pytest test cases — all passing, zero regressions on existing 22 tests

Endpoints

  • GET /jobs/stats - Aggregate statistics
  • GET /jobs/recent - Recent job history
  • GET /jobs/dead-letter - Dead letter queue
  • GET /jobs/:id - Get job by ID
  • POST /jobs/:id/retry - Retry a dead job

Closes #130

Implements a background job system with exponential backoff retry,
dead letter queue, and a monitoring dashboard.

Backend:
- BackgroundJob model with status tracking (PENDING/RUNNING/SUCCESS/FAILED/RETRYING/DEAD)
- @with_retry decorator for wrapping functions as retryable jobs
- Exponential backoff with configurable max_retries, base_delay, backoff_factor
- Thread-safe execution with locking
- Dead letter queue for permanently exhausted jobs
- REST API: GET /jobs/stats, /jobs/recent, /jobs/dead-letter, /jobs/:id, POST /jobs/:id/retry
- 16 test cases covering decorator, stats, dead letter, retry logic

Frontend:
- Job Monitor page with stats cards (total, success rate, failed, running)
- Recent jobs and dead letter tabs
- Retry button for dead jobs
- Color-coded status badges

Closes rohitdash08#130
@Manishcs076
Copy link
Copy Markdown
Author

Uploading screenshots and video as a proof:
Screenshot 2026-03-28 at 1 16 39 AM
Screenshot 2026-03-28 at 12 37 17 AM

1.mp4

@Manishcs076
Copy link
Copy Markdown
Author

2.mp4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resilient background job retry & monitoring

1 participant