Detect mempool changes for getblocktemplate long polling with a channel

[arya2]

Motivation

We could help miners collect higher fees, shorten the wait-time for transactions to be mined, and make Zebra more efficient by returning from long polling immediately when the mempool changes.

Possible Designs

• Add a channel from the transaction verifier to the RPC server
• Send a message to the RPC server when mempool transactions are verified
• Debounce those messages
• Fetch mempool transactions when new transactions are verified instead of at a fixed interval

Possible Alternatives

We could instead check the mempool more frequently or at a configurable interval.

[teor2345]

Currently Zebra checks the mempool every 5 seconds. But we could check every 1 second (or every hundred milliseconds) if miners want that. Since the mempool is all in memory, this is very cheap. (We're just updating a few atomic refcounts, and creating a vector of pointers.)

Would you mind listing this as an alternative in the ticket?

Also, if the mempool is cleared due to a chain fork, we might want to wait for some transactions to be re-verified before returning from long polling. This doesn't matter with the current 5 second interval, but it will matter with an instant notification or very short interval.

[arya2]

Currently Zebra checks the mempool every 5 seconds. But we could check every 1 second (or every hundred milliseconds) if miners want that. Since the mempool is all in memory, this is very cheap. (We're just updating a few atomic refcounts, and creating a vector of pointers.)

Would you mind listing this as an alternative in the ticket?

Sure!

Also, if the mempool is cleared due to a chain fork, we might want to wait for some transactions to be re-verified before returning from long polling. This doesn't matter with the current 5 second interval, but it will matter with an instant notification or very short interval.

Good point, it should debounce mempool changes up to some time limit. I added that as a step and realized the existing channel is only updated when poll_ready is called so this requires a new channel for notifying the RPC server when a transaction is downloaded and/or verified.