A distributed background task orchestration engine built with Go, Redis, Lua, and Next.js for executing asynchronous and scheduled workloads independently of the main API.
The system provides task submission, delayed execution, priorities, retries, worker leases, dead-letter queues, and multi-tenant task management.
Project goal: Explore the design of reliable distributed background workers, task scheduling, failure recovery, retries, and multi-tenant workload management.
Client
|
| Submit Task
v
+-------------------+
| Go API |
+---------+---------+
|
v
+-------------------+
| Redis |
| |
| Ready Queues |
| Delayed ZSET |
| Task State |
| Worker Leases |
| Retry Metadata |
| DLQ |
+---------+---------+
|
+-------------+-------------+
| |
v v
+-------------+ +-------------+
| Scheduler | | Workers |
+-------------+ | |
| | Worker 1 |
| | Worker 2 |
| | Worker N |
| +------+------+
| |
+---------------------------+
|
v
Task Execution
|
+----------------+----------------+
| |
Success Failure
| |
v v
Complete Retry / DLQ
Long-running or asynchronous work should not necessarily execute inside the request/response lifecycle of a web API.
For example:
Client
|
| POST /operation
v
API Server
|
| Long-running operation
|----------------------------->
|
| Request timeout
X
A background task system separates submission from execution:
Client
|
| Submit task
v
API
|
| Enqueue
v
Redis
|
v
Worker
|
| Execute asynchronously
v
Task Result
This allows API servers and workers to scale independently.
Multiple worker processes can consume tasks concurrently.
Redis
|
+-----------+-----------+
| | |
v v v
Worker 1 Worker 2 Worker 3
| | |
+-----------+-----------+
|
v
Task Handler
Tasks that should execute in the future are stored using Redis sorted-set semantics.
Conceptually:
task A -> execute at 10:05
task B -> execute at 10:10
task C -> execute at 10:20
The scheduler identifies tasks whose scheduled execution time has arrived and makes them available for workers.
A simplified lifecycle is:
+---------+
| Created |
+----+----+
|
v
+---------+
| Queued |
+----+----+
|
v
+---------+
| Leased |
+----+----+
|
+-------+-------+
| |
Success Failure
| |
v v
+----------+ +-----------+
| Completed| | Retryable |
+----------+ +-----+-----+
|
v
Queued
|
|
max retries?
|
v
+---------+
| DLQ |
+---------+
A worker should not permanently own a task if the worker crashes.
The system therefore uses leases to represent temporary ownership.
Conceptually:
Task
|
v
Worker A acquires lease
|
| Worker processes task
|
+---- success ----> complete
|
+---- crash ------> lease expires
|
v
task becomes
available again
This allows the system to recover tasks from failed workers.
The lease mechanism is an important part of the distributed coordination model.
Transient failures should not immediately move a task to the DLQ.
A failed task can be retried according to its configured retry policy.
A typical flow is:
Attempt 1
|
| failure
v
Retry delay
|
v
Attempt 2
|
| failure
v
Retry delay
|
v
Attempt N
|
+---- success ----> Completed
|
+---- max retries -> DLQ
The retry lifecycle separates transient failures from permanently failing tasks.
Tasks that cannot be successfully processed after the configured retry policy are moved to the Dead Letter Queue.
The DLQ provides a place to inspect and recover failed tasks without blocking normal task processing.
Conceptually:
Worker
|
| failure
v
Retry Policy
|
+---- retry available ----> Queue
|
+---- retries exhausted --> DLQ
|
+--> Inspect
|
+--> Replay
Tasks are associated with tenants so that workloads from multiple customers can share the same orchestration infrastructure.
The design considers tenant-aware task scheduling and fairness rather than treating the entire workload as one undifferentiated queue.
Conceptually:
Task Orchestrator
|
+-------------+-------------+
| | |
v v v
Tenant A Tenant B Tenant C
| | |
tasks... tasks... tasks...
This provides a foundation for preventing one tenant's workload from dominating shared worker capacity.
Redis is used as the shared coordination layer.
Different Redis data structures can represent different parts of the orchestration problem:
| Requirement | Redis Structure |
|---|---|
| Immediate task queues | Lists / queue structures |
| Delayed tasks | Sorted Sets |
| Task metadata | Keys / hashes |
| Leases | Keys with TTL |
| Retry metadata | Task state |
| Dead-letter tasks | DLQ queue |
The exact coordination operations are implemented using Redis commands and Lua where atomic multi-step state transitions are required.
Delayed tasks need to be ordered by execution time.
A Redis sorted set naturally represents:
score = scheduled execution timestamp
member = task ID
Conceptually:
ZSET: delayed_tasks
timestamp task
--------- ----
10:05 task-A
10:10 task-B
10:20 task-C
The scheduler can identify tasks whose scheduled time has arrived and move them toward execution.
Distributed task processing often requires multiple state checks and updates to happen together.
For example:
check task state
|
check lease
|
acquire lease
|
update task state
If these operations are independently executed by different workers, race conditions can occur.
Lua scripts allow related Redis operations to execute atomically.
Task
|
v
Worker A
|
| lease acquired
|
X Worker crashes
|
v
Lease expires
|
v
Task becomes eligible
for recovery
Task
|
v
Worker
|
X temporary failure
|
v
Retry policy
|
v
Delayed retry
|
v
Worker
Task
|
v
Worker
|
X repeated failures
|
v
Maximum retries
|
v
DLQ
The system separates task submission from task execution.
API Process
|
| enqueue task
v
Redis
|
v
Worker Processes
|
+--> execute task
|
+--> update state
|
+--> retry / DLQ
This allows the API layer and worker layer to scale independently.
Conceptually, the repository is organized around:
.
├── API
│ └── Task submission and task management
│
├── Scheduler
│ └── Delayed-task promotion
│
├── Workers
│ └── Concurrent task execution
│
├── Redis
│ ├── Queues
│ ├── Delayed tasks
│ ├── Task state
│ ├── Leases
│ └── DLQ
│
└── Dashboard
└── Next.js telemetry / monitoring UI
| Component | Technology |
|---|---|
| Backend | Go |
| Distributed State | Redis |
| Atomic Coordination | Redis Lua |
| Scheduling | Redis Sorted Sets |
| Dashboard | Next.js |
| Background Processing | Go Worker Pool |
| Deployment | Redis + application services |
Clone the repository:
git clone https://github.com/MohiRaghav/distributed-task-orchestrator.git
cd distributed-task-orchestratorConfigure the required Redis connection/environment variables according to the configuration used by the project.
Start Redis and the required application components.
Start the API, scheduler, workers, and dashboard according to the project's local development configuration.
Run the Go test suite:
go test ./...For concurrency-sensitive code, also run:
go test -race ./...Run static analysis:
go vet ./...The project includes a Next.js dashboard intended to provide visibility into task execution and system state.
Useful operational information includes:
- Task state
- Queue activity
- Worker activity
- Retries
- Failed tasks
- DLQ tasks
- Scheduling state
Redis provides a shared, low-latency coordination layer suitable for:
- Queueing
- Delayed scheduling
- Task metadata
- Leases
- Atomic operations
- Lua scripting
Permanent task ownership creates a recovery problem when a worker crashes.
Leases provide temporary ownership and allow the system to reconsider a task after the ownership period expires.
Without a DLQ, permanently failing tasks can continuously consume retry capacity.
A DLQ separates unrecoverable work from the normal execution path and provides an explicit recovery mechanism.
Immediate retries can amplify load during an outage.
Delayed retries provide time for transient failures to recover while reducing repeated pressure on the failing dependency.
- Distributed task execution
- Worker coordination
- Task scheduling
- Delayed jobs
- Redis sorted sets
- Redis Lua scripting
- Leases
- Failure recovery
- Retry policies
- Dead-letter queues
- Multi-tenant workloads
- Queue-based architecture
- Concurrent workers
- API/worker separation
- Background processing
This project is intended as a distributed-systems engineering demonstration.
Task processing systems generally need to reason carefully about failures between:
Task execution
|
v
External side effect
|
v
Task state update
A worker crash between these operations can create ambiguity about whether the external side effect occurred.
Therefore, the system should not be described as providing universal end-to-end exactly-once side effects. Correctness for external side effects may require application-level idempotency, transactions, or other coordination mechanisms.
Potential extensions include:
- Persistent task result storage
- Stronger authentication and authorization
- Per-tenant rate limits
- Worker autoscaling
- Prometheus metrics
- OpenTelemetry tracing
- Persistent event history
- More sophisticated scheduling policies
- Pluggable task handlers
- Stronger distributed failure testing
- Kubernetes deployment
This project is provided for educational and engineering demonstration purposes.