Repository navigation
fix(jobs): serialize concurrent job updates and archives with per-ent… - #58
Conversation
|
👋 Thanks for opening this PR! A maintainer will review it soon. In the meantime, please make sure you've read CONTRIBUTING.md and filled out the PR template, including screenshots for any visual changes. |
Reviewer's GuideThis PR fixes lost updates caused by concurrent job read-modify-write cycles by sharing per-job asyncio locks across requests, while allowing different jobs to proceed concurrently. It also serializes archive operations with updates and adds regression tests covering same-job field preservation and independent cross-job updates. Sequence diagram for serialized concurrent job updatessequenceDiagram
participant RequestA
participant RequestB
participant JobService
participant JobLock
participant JobRepository
RequestA->>JobService: update_job(job_id, department)
JobService->>JobService: _lock_for(job_id)
JobService->>JobLock: acquire
JobService->>JobRepository: get_job(job_id)
JobService->>JobRepository: save(updated_record)
JobService->>JobLock: release
RequestB->>JobService: update_job(job_id, title)
JobService->>JobService: _lock_for(job_id)
JobService->>JobLock: wait for release
JobLock-->>JobService: acquire
JobService->>JobRepository: get_job(job_id)
JobService->>JobRepository: save(updated_record)
JobService->>JobLock: release
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 4 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="services/job_service.py" line_range="233" />
<code_context>
- stored = await self._jobs.save(updated_record)
- logger.info("job updated", extra=log_context(event="job_updated", job_id=job_id))
- return stored
+ lock = await self._lock_for(job_id)
+ async with lock:
+ record = await self.get_job(job_id)
</code_context>
<issue_to_address>
**Archived jobs become active**
When an admin archives a job after `update_job()` reads it but before it saves, `AdminService.set_job_active()` bypasses the per-job lock, so `update_job()` saves its stale active snapshot and reactivates the job.
Make admin state changes use the same per-job synchronization as job updates.
Also at `services/job_service.py:234`, `services/job_service.py:260-261`.
</issue_to_address>
### Comment 2
<location path="core/container.py" line_range="135-136" />
<code_context>
_fallback_registry: SessionRegistry = field(default_factory=SessionRegistry)
live_interview_locks: dict[str, asyncio.Lock] = field(default_factory=dict)
live_interview_locks_guard: asyncio.Lock = field(default_factory=asyncio.Lock)
+ job_locks: dict[str, asyncio.Lock] = field(default_factory = dict)
+ job_locks_guard: asyncio.Lock = field(default_factory = asyncio.Lock)
def registry_for(self, app) -> SessionRegistry:
</code_context>
<issue_to_address>
**Concurrent edits overwrite each other**
When concurrent updates for the same job reach different worker processes sharing PostgreSQL, `job_locks` is process-local, so each `update_job()` acquires a different lock, reads the same record, and saves a full snapshot; the later save discards the earlier edit.
Use database-level locking or optimistic concurrency so updates across workers cannot overwrite one another.
Also at `services/job_service.py:64-65`, `services/job_service.py:233-235`.
</issue_to_address>
### Comment 3
<location path="services/job_service.py" line_range="64-65" />
<code_context>
+ self._locks_guard = locks_guard if locks_guard is not None else asyncio.Lock()
+
+ async def _lock_for(self, job_id: str) -> asyncio.Lock:
+ async with self._locks_guard:
+ return self._locks.setdefault(job_id, asyncio.Lock())
async def create_job(
</code_context>
<issue_to_address>
**Missing IDs exhaust process memory**
When requests use distinct nonexistent job IDs for update or archive, `_lock_for()` inserts a lock before the repository checks whether the job exists and never removes it, so repeated requests leave entries in the lock map and grow memory without bound.
Avoid retaining locks for missing jobs or safely reclaim unused lock entries.
</issue_to_address>
### Comment 4
<location path="tests/test_job_concurrency.py" line_range="26" />
<code_context>
+ )
+
+ await repo.save(record)
+ service = JobService(job_repository = repo)
+
+ await asyncio.gather(
</code_context>
<issue_to_address>
**Per-request locking regression passes**
When the per-request dependency stops injecting the shared locks, `test_concurrent_job_updates_preserve_both_fields` sends both updates through the single `service` created on line 26, so its service-local lock serializes them even if `get_job_service` stops sharing the container locks. The test then passes while separate HTTP requests can still lose an edit.
Exercise concurrent updates through separate services created by `get_job_service` (or otherwise verify that they share the same per-job locks).
</issue_to_address>Sourcery assessment
Approval pending. 4 findings to address first.
Blocking findings: services/job_service.py:233, core/container.py:136, services/job_service.py:65, tests/test_job_concurrency.py:26
| stored = await self._jobs.save(updated_record) | ||
| logger.info("job updated", extra=log_context(event="job_updated", job_id=job_id)) | ||
| return stored | ||
| lock = await self._lock_for(job_id) |
There was a problem hiding this comment.
🟠 High · Archived jobs become active
When an admin archives a job after update_job() reads it but before it saves, AdminService.set_job_active() bypasses the per-job lock, so update_job() saves its stale active snapshot and reactivates the job.
Make admin state changes use the same per-job synchronization as job updates.
Also at services/job_service.py:234, services/job_service.py:260-261.
Prompt for AI agents
In `services/job_service.py` at line 233:
**Archived jobs become active**
When an admin archives a job after `update_job()` reads it but before it saves, `AdminService.set_job_active()` bypasses the per-job lock, so `update_job()` saves its stale active snapshot and reactivates the job.
Make admin state changes use the same per-job synchronization as job updates.
Also at `services/job_service.py:234`, `services/job_service.py:260-261`.| job_locks: dict[str, asyncio.Lock] = field(default_factory = dict) | ||
| job_locks_guard: asyncio.Lock = field(default_factory = asyncio.Lock) |
There was a problem hiding this comment.
🟠 High · Concurrent edits overwrite each other
When concurrent updates for the same job reach different worker processes sharing PostgreSQL, job_locks is process-local, so each update_job() acquires a different lock, reads the same record, and saves a full snapshot; the later save discards the earlier edit.
Use database-level locking or optimistic concurrency so updates across workers cannot overwrite one another.
Also at services/job_service.py:64-65, services/job_service.py:233-235.
Prompt for AI agents
In `core/container.py` at lines 135-136:
**Concurrent edits overwrite each other**
When concurrent updates for the same job reach different worker processes sharing PostgreSQL, `job_locks` is process-local, so each `update_job()` acquires a different lock, reads the same record, and saves a full snapshot; the later save discards the earlier edit.
Use database-level locking or optimistic concurrency so updates across workers cannot overwrite one another.
Also at `services/job_service.py:64-65`, `services/job_service.py:233-235`.| async with self._locks_guard: | ||
| return self._locks.setdefault(job_id, asyncio.Lock()) |
There was a problem hiding this comment.
🟡 Medium · Missing IDs exhaust process memory
When requests use distinct nonexistent job IDs for update or archive, _lock_for() inserts a lock before the repository checks whether the job exists and never removes it, so repeated requests leave entries in the lock map and grow memory without bound.
Avoid retaining locks for missing jobs or safely reclaim unused lock entries.
Prompt for AI agents
In `services/job_service.py` at lines 64-65:
**Missing IDs exhaust process memory**
When requests use distinct nonexistent job IDs for update or archive, `_lock_for()` inserts a lock before the repository checks whether the job exists and never removes it, so repeated requests leave entries in the lock map and grow memory without bound.
Avoid retaining locks for missing jobs or safely reclaim unused lock entries.|
🎉 Thanks for the contribution, @selladuraisomasundaram! This PR has been merged. We appreciate you helping improve OpenHire. |
Summary of Changes
Fixes a lost-update race condition where concurrent
PATCH /jobs/{job_id}operations to different fields could both return HTTP 200 while one edit was silently discarded.Root Cause:
JobService.update_job()performs a Read-Modify-Write cycle:await self.get_job(job_id))await self._jobs.save(updated_record))Fixes #53
When two requests arrived concurrently, both read the same snapshot before either save completed. The second save subsequently overwrote the entire record using its stale snapshot, obliterating the first request's field edits.
Solution:
Following the established per-entity locking pattern from
LiveInterviewService._lock_for:services/job_service.py: Addedlocks: dict[str, asyncio.Lock]andlocks_guard: asyncio.LocktoJobService, with a private_lock_for(job_id)helper. Wrappedupdate_job()andarchive_job()inasync with lock:to ensure the Read-Modify-Write cycle is atomic per job.core/container.py&core/dependencies.py: Addedjob_locksandjob_locks_guardtoServiceContainerso locks persist across incoming HTTP requests.job_id, concurrent updates to different jobs proceed in parallel without blocking each other.Testing
Added
tests/test_job_concurrency.py:test_concurrent_job_updates_preserve_both_fields: Proves that concurrent updates to distinct fields (departmentandtitle) both persist cleanly.test_concurrent_updates_to_different_jobs_do_not_interfere: Confirms independent jobs execute concurrently without interference.Verified against existing test suite: