Skip to content

fix(jobs): serialize concurrent job updates and archives with per-ent… - #58

Merged
DevSidd2006 merged 3 commits into
DevSidd2006:mainfrom
selladuraisomasundaram:fix/job-update-concurrency
Oct 26, 2026
Merged

DevSidd2006 merged 3 commits into
DevSidd2006:mainfrom
selladuraisomasundaram:fix/job-update-concurrency

Conversation

@selladuraisomasundaram

@selladuraisomasundaram selladuraisomasundaram commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary of Changes

Fixes a lost-update race condition where concurrent PATCH /jobs/{job_id} operations to different fields could both return HTTP 200 while one edit was silently discarded.

Root Cause:

JobService.update_job() performs a Read-Modify-Write cycle:

  1. Fetches current snapshot (await self.get_job(job_id))
  2. Merges the patch dictionary into the snapshot
  3. Saves the full record back (await self._jobs.save(updated_record))

Fixes #53

When two requests arrived concurrently, both read the same snapshot before either save completed. The second save subsequently overwrote the entire record using its stale snapshot, obliterating the first request's field edits.

Solution:

Following the established per-entity locking pattern from LiveInterviewService._lock_for:

  1. services/job_service.py: Added locks: dict[str, asyncio.Lock] and locks_guard: asyncio.Lock to JobService, with a private _lock_for(job_id) helper. Wrapped update_job() and archive_job() in async with lock: to ensure the Read-Modify-Write cycle is atomic per job.
  2. core/container.py & core/dependencies.py: Added job_locks and job_locks_guard to ServiceContainer so locks persist across incoming HTTP requests.
  3. Independent Concurrency: Because locks are keyed by job_id, concurrent updates to different jobs proceed in parallel without blocking each other.

Testing

Added tests/test_job_concurrency.py:

  • test_concurrent_job_updates_preserve_both_fields: Proves that concurrent updates to distinct fields (department and title) both persist cleanly.
  • test_concurrent_updates_to_different_jobs_do_not_interfere: Confirms independent jobs execute concurrently without interference.

Verified against existing test suite:

pytest tests/test_job_concurrency.py -v
# 2 passed in 0.28s

pytest tests/test_backend_jobs_candidates_applications.py -k "job" -q
# 66 passed in 6.74s

@DevSidd2006 add the OSCI'26 label to the PR

Closes #53

## Summary by Sourcery

Serialize concurrent job mutations with shared per-job locking to preserve updates and maintain consistent archive behavior.

Bug Fixes:
- Prevent lost updates and inconsistent outcomes when job updates, archives, and concurrent requests target the same job.

Enhancements:
- Serialize mutations per job while allowing operations on different jobs to proceed independently.

Tests:
- Add concurrency coverage for concurrent field updates, cross-job independence, update/archive ordering, and lock recovery after validation failures.

@github-actions

Copy link
Copy Markdown

👋 Thanks for opening this PR! A maintainer will review it soon.

In the meantime, please make sure you've read CONTRIBUTING.md and filled out the PR template, including screenshots for any visual changes.

@sourcery-ai

sourcery-ai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

This PR fixes lost updates caused by concurrent job read-modify-write cycles by sharing per-job asyncio locks across requests, while allowing different jobs to proceed concurrently. It also serializes archive operations with updates and adds regression tests covering same-job field preservation and independent cross-job updates.

Sequence diagram for serialized concurrent job updates

sequenceDiagram
    participant RequestA
    participant RequestB
    participant JobService
    participant JobLock
    participant JobRepository

    RequestA->>JobService: update_job(job_id, department)
    JobService->>JobService: _lock_for(job_id)
    JobService->>JobLock: acquire
    JobService->>JobRepository: get_job(job_id)
    JobService->>JobRepository: save(updated_record)
    JobService->>JobLock: release

    RequestB->>JobService: update_job(job_id, title)
    JobService->>JobService: _lock_for(job_id)
    JobService->>JobLock: wait for release
    JobLock-->>JobService: acquire
    JobService->>JobRepository: get_job(job_id)
    JobService->>JobRepository: save(updated_record)
    JobService->>JobLock: release
Loading

File-Level Changes

Change Details Files
Introduces shared, per-job asyncio locks to serialize read-modify-write operations and archival transitions.
  • Adds lock registries and guards to the service container.
  • Injects the shared lock state into each JobService instance.
  • Protects update and archive operations while preserving parallelism across different job IDs.
services/job_service.py
core/container.py
core/dependencies.py
Adds concurrency-focused regression coverage for lost updates and cross-job independence.
  • Verifies concurrent edits to separate fields on one job are both retained.
  • Verifies updates to different jobs can complete without interfering.
tests/test_job_concurrency.py

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 4 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="services/job_service.py" line_range="233" />
<code_context>
-        stored = await self._jobs.save(updated_record)
-        logger.info("job updated", extra=log_context(event="job_updated", job_id=job_id))
-        return stored
+        lock = await self._lock_for(job_id)
+        async with lock:
+            record = await self.get_job(job_id)
</code_context>
<issue_to_address>
**Archived jobs become active**

When an admin archives a job after `update_job()` reads it but before it saves, `AdminService.set_job_active()` bypasses the per-job lock, so `update_job()` saves its stale active snapshot and reactivates the job.

Make admin state changes use the same per-job synchronization as job updates.

Also at `services/job_service.py:234`, `services/job_service.py:260-261`.
</issue_to_address>

### Comment 2
<location path="core/container.py" line_range="135-136" />
<code_context>
     _fallback_registry: SessionRegistry = field(default_factory=SessionRegistry)
     live_interview_locks: dict[str, asyncio.Lock] = field(default_factory=dict)
     live_interview_locks_guard: asyncio.Lock = field(default_factory=asyncio.Lock)
+    job_locks: dict[str, asyncio.Lock] = field(default_factory = dict)
+    job_locks_guard: asyncio.Lock = field(default_factory = asyncio.Lock)

     def registry_for(self, app) -> SessionRegistry:
</code_context>
<issue_to_address>
**Concurrent edits overwrite each other**

When concurrent updates for the same job reach different worker processes sharing PostgreSQL, `job_locks` is process-local, so each `update_job()` acquires a different lock, reads the same record, and saves a full snapshot; the later save discards the earlier edit.

Use database-level locking or optimistic concurrency so updates across workers cannot overwrite one another.

Also at `services/job_service.py:64-65`, `services/job_service.py:233-235`.
</issue_to_address>

### Comment 3
<location path="services/job_service.py" line_range="64-65" />
<code_context>
+        self._locks_guard = locks_guard if locks_guard is not None else asyncio.Lock()
+    
+    async def _lock_for(self, job_id: str) -> asyncio.Lock:
+        async with self._locks_guard:
+            return self._locks.setdefault(job_id, asyncio.Lock())

     async def create_job(
</code_context>
<issue_to_address>
**Missing IDs exhaust process memory**

When requests use distinct nonexistent job IDs for update or archive, `_lock_for()` inserts a lock before the repository checks whether the job exists and never removes it, so repeated requests leave entries in the lock map and grow memory without bound.

Avoid retaining locks for missing jobs or safely reclaim unused lock entries.
</issue_to_address>

### Comment 4
<location path="tests/test_job_concurrency.py" line_range="26" />
<code_context>
+    )
+
+    await repo.save(record)
+    service = JobService(job_repository = repo)
+
+    await asyncio.gather(
</code_context>
<issue_to_address>
**Per-request locking regression passes**

When the per-request dependency stops injecting the shared locks, `test_concurrent_job_updates_preserve_both_fields` sends both updates through the single `service` created on line 26, so its service-local lock serializes them even if `get_job_service` stops sharing the container locks. The test then passes while separate HTTP requests can still lose an edit.

Exercise concurrent updates through separate services created by `get_job_service` (or otherwise verify that they share the same per-job locks).
</issue_to_address>

Sourcery assessment

Approval pending. 4 findings to address first.

Blocking findings: services/job_service.py:233, core/container.py:136, services/job_service.py:65, tests/test_job_concurrency.py:26


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread services/job_service.py
stored = await self._jobs.save(updated_record)
logger.info("job updated", extra=log_context(event="job_updated", job_id=job_id))
return stored
lock = await self._lock_for(job_id)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High · Archived jobs become active

When an admin archives a job after update_job() reads it but before it saves, AdminService.set_job_active() bypasses the per-job lock, so update_job() saves its stale active snapshot and reactivates the job.

Make admin state changes use the same per-job synchronization as job updates.

Also at services/job_service.py:234, services/job_service.py:260-261.

Prompt for AI agents
In `services/job_service.py` at line 233:

**Archived jobs become active**

When an admin archives a job after `update_job()` reads it but before it saves, `AdminService.set_job_active()` bypasses the per-job lock, so `update_job()` saves its stale active snapshot and reactivates the job.

Make admin state changes use the same per-job synchronization as job updates.

Also at `services/job_service.py:234`, `services/job_service.py:260-261`.

Comment thread core/container.py Outdated
Comment on lines +135 to +136
job_locks: dict[str, asyncio.Lock] = field(default_factory = dict)
job_locks_guard: asyncio.Lock = field(default_factory = asyncio.Lock)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High · Concurrent edits overwrite each other

When concurrent updates for the same job reach different worker processes sharing PostgreSQL, job_locks is process-local, so each update_job() acquires a different lock, reads the same record, and saves a full snapshot; the later save discards the earlier edit.

Use database-level locking or optimistic concurrency so updates across workers cannot overwrite one another.

Also at services/job_service.py:64-65, services/job_service.py:233-235.

Prompt for AI agents
In `core/container.py` at lines 135-136:

**Concurrent edits overwrite each other**

When concurrent updates for the same job reach different worker processes sharing PostgreSQL, `job_locks` is process-local, so each `update_job()` acquires a different lock, reads the same record, and saves a full snapshot; the later save discards the earlier edit.

Use database-level locking or optimistic concurrency so updates across workers cannot overwrite one another.

Also at `services/job_service.py:64-65`, `services/job_service.py:233-235`.

Comment thread services/job_service.py
Comment on lines +64 to +65
async with self._locks_guard:
return self._locks.setdefault(job_id, asyncio.Lock())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Missing IDs exhaust process memory

When requests use distinct nonexistent job IDs for update or archive, _lock_for() inserts a lock before the repository checks whether the job exists and never removes it, so repeated requests leave entries in the lock map and grow memory without bound.

Avoid retaining locks for missing jobs or safely reclaim unused lock entries.

Prompt for AI agents
In `services/job_service.py` at lines 64-65:

**Missing IDs exhaust process memory**

When requests use distinct nonexistent job IDs for update or archive, `_lock_for()` inserts a lock before the repository checks whether the job exists and never removes it, so repeated requests leave entries in the lock map and grow memory without bound.

Avoid retaining locks for missing jobs or safely reclaim unused lock entries.

Comment thread tests/test_job_concurrency.py Resolved
@DevSidd2006
DevSidd2006 merged commit 947e30d into DevSidd2006:main Oct 26, 2026
5 of 6 checks passed
@github-actions

Copy link
Copy Markdown

🎉 Thanks for the contribution, @selladuraisomasundaram! This PR has been merged. We appreciate you helping improve OpenHire.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Concurrent job PATCH requests can discard one edit

2 participants