Nice to have features

[beratdogan]

Hi,

Thanks for sharing this project. Good job!

I was designing a project for inboxing purposes with similar approach. I think I'll not be able to start working in near future. I wanted to share my notes and maybe considered in PgQueuer.

• Archiving options for the events already done after configurable interval
• Partitioning Investigate Table Partitioning #81
• Multiple asyncio workers in consumers
• Event joins (two events waiting each other, process them together)
• Event retries configurable with backoff time
• Kafka like MQs as "producers" (for inboxing)
  -- Handler routing for different types of events
  -- Idempotency keys should be considered
• Built-in support for metrics and traces

Last thing, I wanted to name it as a[io]q (I-OK). You may want to use it. :)

[janbjorge]

Thanks submitting, have you checked out the dashboard command @beratdogan ?

[beratdogan]

Yes I did. Basically I expect it to export as Promehus metrics.

[janbjorge]

Ah, that is an interesting idea. The data i display has decent pydantic models, writing some sort of Promehus converter should be a simple ish task. It also outlines how the event listener can be used to avoid unnecessary polling.

-- EDIT --
At first glance i thought it could be attached to the queuemangner, but since it QM can run have multi-pull instances i think that will mess-up the promehus metrics. Might need a separate service to push metrics?

[beratdogan]

No, ideally just need starting the HTTP server, Prometheus should pull the metrics from there.

[janbjorge]

@beratdogan Something like this? #97

[janbjorge]

@beratdogan Could you elaborate a bit on Multiple asyncio workers in consumers? As of now, you can start as many works are you like (i think the upper limit is the number of connections your db can accept).

[beratdogan]

Looks it's exact same thing with batch_size of QueueManager.run. 🏉