Develop Integration Guide for PgQueuer with Django

[janbjorge]

Description

Write a concise guide on integrating PgQueuer with Django to leverage PostgreSQL for task management effectively. This guide will serve as a resource for Django developers to implement PgQueuer in their projects, optimizing task management capabilities.
Objectives

• Setup Instructions: Steps to integrate PgQueuer within Django
• Configuration: Adjustments necessary in Django settings for
• Examples: Code examples for defining and enqueuing
• Management Commands: Integration of Django commands for PgQueuer
• Best Practices: Tips for efficient use and optimal configuration.

[baseplate-admin]

Hi, if you want to support django, i think it will be better if you use the postgres driver that django uses under the hood, psycopg

[janbjorge]

Hi, if you want to support django, i think it will be better if you use the postgres driver that django uses under the hood, psycopg

I think i need to make driver abstraction for this to come out nice. Another approach might be to sniff connections parameters from psycopg and try to establish a connection with asyncpg insted (to only be used by asyncpg).

Ive mostly been using fastapi/flask, so Django is foreign to me.

[baseplate-admin]

The problem is asyncpg is a 3.3MB dependency. Size optimization for django is a must for deploying into aws lambda.

I think i need to make driver abstraction for this to come out nice.

Do you have any particular problem that is not implementable using psycopg? pgadmin itself is based on psycopg, so in theory it should implement every postgres feature

[janbjorge]

I used to use psycopg2 back in the day (before it got async). Since most of my projects moved to async using asyncpg was natural at that time. I do not have any objects to psycopg, but i still want to support asyncpg. So i think i need to spend a bit of time to figure out how i want to make an abstraction for the drivers(psycopg/asyncpg etc...) to plug into.

[janbjorge]

Short update, ive added an abstraction layer for database connection. By passing the the a async psycopg connection to a 'psycopg-driver-class'. Still working on a few details to make this neater.

PR: #37

[janbjorge]

Tabling this for now.

[opentyler]

Any luck trying to figure out a way to implement an abstraction for the driver? I'd also love to use this in a django project but prefer not to require asyncpg just for this, would be great to be able to reuse psycopg. Great looking library tho! All the best.

[janbjorge]

Disclaimer: I’ve never worked with Django before; The code was generated by AI - I’ve reviewed it and I feel it makes sense.

---

You can integrate PgQueuer into Django without pulling in asyncpg by leveraging Django’s built-in psycopg support. Simply grab the raw psycopg.Connection from django.db.connection.connection, wrap it in your SyncPsycopgDriver, and let Django manage connection lifecycles.

---

Code Example

import json
from django.http import JsonResponse, HttpResponseNotAllowed
from django.db import connection
from pgqueuer.db import SyncPsycopgDriver, dsn
from pgqueuer.queries import SyncQueries

def get_driver() -> SyncPsycopgDriver:
    if connection.connection is None:
        connection.ensure_connection()
    # connection.connection is now a psycopg.Connection
    return SyncPsycopgDriver(connection.connection)

def enqueue(request):
    if request.method != "POST":
        return HttpResponseNotAllowed(["POST"])
    data = json.loads(request.body)
    entry = data.get("entrypoint", "default")
    payload = data.get("payload")
    raw = payload.encode() if payload is not None else None
    prio = int(data.get("priority", 0))

    driver  = get_driver()
    ids     = SyncQueries(driver).enqueue(entry, raw, prio)
    return JsonResponse({"job_ids": [str(j) for j in ids]})

def queue_size(request):
    if request.method != "GET":
        return HttpResponseNotAllowed(["GET"])
    stats = SyncQueries(get_driver()).queue_size()
    return JsonResponse(
        [
            {"entrypoint": s.entrypoint, "priority": s.priority,
             "status": s.status,     "count":    s.count}
            for s in stats
        ],
        safe=False
    )

Does it make sense? 😅