Dashboard/Backend/transactions: added mempool API/WS to fetch transactions from bitcoin node by sethdivyansh · Pull Request #72 · dmnd-pool/dmnd-client

sethdivyansh · 2025-06-08T20:46:11Z

Overview

This PR introduces a Rust-based backend service for streaming real-time Bitcoin blockchain data. Built with Axum, the service connects to a Bitcoin Core node via RPC and ZeroMQ to deliver blockchain events through WebSocket and REST API endpoints.

Add these lines to your config.toml:

# Bitcoin Core Configuration (all fields optional)
zmq_pub_sequence = "tcp://127.0.0.1:28334"  # Defaults to tcp://127.0.0.1:28334 if not set
rpc_allow_ip = "127.0.0.1"                  # Defaults to 127.0.0.1
rpc_port = 8332                             # Defaults to 8332
bitcoin_datadir = "path/to/your/bitcoin"    # Defaults to OS-specific Bitcoin data directory

Prerequisite::

To use ZMQ with Rust, the native ZeroMQ library (libzmq) must be installed on the system. The zmq crate depends on it during compilation.

Installation

Linux (Ubuntu/Debian):
```
sudo apt install libzmq3-dev
```
macOS (with Homebrew):
```
brew install zeromq
```
Windows (using vcpkg):**
```
vcpkg install zeromq
```
Or download prebuilt binaries from https://zeromq.org/download/

API Endpoints

REST Endpoints

GET /api/mempool - Current mempool transactions with full details

WebSocket Endpoints

/ws/bitcoin/stream Real-time Bitcoin events (mempool + blocks)
/ws/jd/stream Real-time NewTemplate event notifications

Event Types

About zmq
The WebSocket stream provides four types of events:
A: New transactions added to mempool with full transaction details
R: Transactions removed from mempool (includes txid and sequence number)
C: New blocks mined with block hash and complete transaction list
D: Block reorganizations with affected transactions

Architecture

Bitcoin Core Node
       ↓
    ZMQ Sequence
       ↓
   Rust Backend ←→ RPC Calls
       ↓
   WebSocket/REST
       ↓
   Frontend Client

ZMQ Listener: Receives sequence events from Bitcoin Core
Event Processor: Enriches events with RPC data (transaction details, block info)
Broadcaster: Distributes events to connected WebSocket clients
REST API: Provides current state endpoints

Error Handling

Automatic Reconnection: ZMQ connection automatically reconnects on failure
Backlog Management: Failed broadcasts are queued and retried

sethdivyansh · 2025-06-15T17:39:03Z

@Priceless-P Please review this when you get a chance.

Priceless-P · 2025-06-16T22:42:45Z

I noticed that it is missing

POST /api/job-declaration - Submit selected transactions for inclusion in the next block
GET /api/job-declarations - Get history of submitted job declarations

The rest of the endpoint seems okay. Let me know when you have them available so I can test them together.

Priceless-P · 2025-06-25T12:09:58Z

        template: NewTemplate<'static>,
        token: Vec<u8>,
-        tx_list_: Seq064K<'static, B016M<'static>>,
+        tx_list_receiver: Arc<tokio::sync::Mutex<TReceiver<Seq064K<'static, B016M<'static>>>>>,


Use roles_logic_sv2 Mutex not tokio Mutex

Priceless-P · 2025-06-25T12:34:10Z

        }
        super::IS_CUSTOM_JOB_SET.store(false, std::sync::atomic::Ordering::Release);
+
+        let tx_list_ = {


There should be a configurable duration (default of 30secs) to wait for a transactions to arrive. If no transaction arrives within that time, fall back to using the transaction from TemplateRx.

sethdivyansh · 2025-06-27T20:26:33Z

@Priceless-P I have made significant refactors in the transaction logic by switching from separate rawtx and rawblock ZMQ topics to the unified sequence topic. This change improves handling of edge cases such as:

Transaction updates via Replace-by-Fee (RBF)
Transaction removal due to abandonment or eviction
Blockchain reorganizations (reorgs)

It also reduced overall complexity.
About zmq: https://github.com/Sjors/bitcoin/blob/sv2/doc/zmq.md#message-format

Priceless-P · 2025-07-03T07:21:16Z

+        );
+
+        if jd_event_broadcaster.send(template_notification).is_err() {
+            error!("Failed to send template notification: no receivers available");


This will always fail because you are not using the receiver anywhere

It will fail until a client is connected to the websocket.
Since this is a Tokio broadcast channel, receivers must explicitly subscribe to the sender in order to receive messages. That subscription happens inside the ws_event_handler() fn in jd_event_ws.rs.

Until at least one receiver is subscribed, any .send() call on the broadcaster will return an error which is expected behavior in this case.

Priceless-P · 2025-07-03T07:22:57Z

        }));

+        // Set up the connection with the upstream for job response notifications
+        if let Err(e) = Upstream::set_job_declarator(&up, &self_) {


why do we need this?

This connection setup is essential for enabling bidirectional communication between JobDeclarator and Upstream. Without it, Upstream can't notify JobDeclarator about job declaration responses, causing the API to wait indefinitely. By injecting a reference to JobDeclarator into Upstream, it enables a complete async response flow, ensuring the API receives timely feedback from the pool via Upstream.

Priceless-P · 2025-07-03T07:32:07Z

Rename dashboard_build to dashboard, move its contents into a build/ subfolder.
Move jd_event_ws.rs, assets.rs, and dashboard.rs from api/ into dashboard/.
Disable bitcoincore_rpc logs by default.
Add doc comments explaining each function’s purpose and behavior.

sethdivyansh · 2025-07-03T11:06:56Z

Rename dashboard_build to dashboard, move its contents into a build/ subfolder.

Move jd_event_ws.rs, assets.rs, and dashboard.rs from api/ into dashboard/.

Disable bitcoincore_rpc logs by default.

Add doc comments explaining each function’s purpose and behavior.

Done with the changes.

…nt to fetch transactions from the bitcoin node

… and confirmed transactions

…nt to fetch transactions from the bitcoin node

…ocket route

…nt to fetch transactions from the bitcoin node

…_tx_list API

… methods

…tabase handling

sethdivyansh changed the title ~~Dashboard/Backend/transactions: added mempool API/WS~~ Dashboard/Backend/transactions: added mempool API/WS to fetch transactions from bitcoin node Jun 8, 2025

sethdivyansh marked this pull request as ready for review June 12, 2025 15:41

sethdivyansh force-pushed the backend/transaction branch from 97ce92d to b785a7e Compare June 13, 2025 17:34

sethdivyansh force-pushed the backend/transaction branch 2 times, most recently from 01decc0 to 71e6e88 Compare June 21, 2025 13:38

Priceless-P reviewed Jun 25, 2025

View reviewed changes

sethdivyansh force-pushed the backend/transaction branch 4 times, most recently from 7e1b8ed to a7b2fd4 Compare July 2, 2025 22:37

Priceless-P reviewed Jul 3, 2025

View reviewed changes

sethdivyansh force-pushed the backend/transaction branch 2 times, most recently from 448d6ac to 739ec91 Compare July 7, 2025 20:19

sethdivyansh force-pushed the backend/transaction branch from 739ec91 to 894ac52 Compare July 23, 2025 13:10

sethdivyansh and others added 11 commits August 23, 2025 19:14

add mempool API and WebSocket support; integrate bitcoincore RPC clie…

463069f

…nt to fetch transactions from the bitcoin node

dashboard/backend/transaction: added api/ws endpoints for unconfirmed…

f6c9dda

… and confirmed transactions

add tower-http dependency and implement CORS support in API

ca22cc2

Flush job cache on new blocks

21ed440

use channel_factory valid jobs

af6cdc6

add mempool API and WebSocket support; integrate bitcoincore RPC clie…

7b85d6a

…nt to fetch transactions from the bitcoin node

Resolved cargo.lock conflict

1bd30a9

backend/jobDeclaration: implemented job declaration post endpoint

5b5b174

Refactor mempool API: Switch ZMQ topics to 'sequence' and update Webs…

3c04a5e

…ocket route

Add Bitcoin RPC integration and configuration options

06bdc2e

add mempool API and WebSocket support; integrate bitcoincore RPC clie…

f6e666a

…nt to fetch transactions from the bitcoin node

sethdivyansh added 8 commits August 23, 2025 19:18

feat: implement end-to-end flow for job_data response via POST submit…

8c35c86

…_tx_list API

refactor: update dependencies

15cc160

Reorganize files and directories; add function documentation

b254d6e

updated dashboard build file

86a0e88

feat: implement Bitcoin RPC client with updated username and password…

78d981b

… methods

feat: jd history

031b050

added auto-selection and storing user settings

26e1c8c

feat: enhance balanced transaction selection algorithm and improve da…

8b5dcc1

…tabase handling

sethdivyansh force-pushed the backend/transaction branch from b3098d2 to 8b5dcc1 Compare August 30, 2025 10:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dashboard/Backend/transactions: added mempool API/WS to fetch transactions from bitcoin node#72

Dashboard/Backend/transactions: added mempool API/WS to fetch transactions from bitcoin node#72
sethdivyansh wants to merge 19 commits into
dmnd-pool:masterfrom
sethdivyansh:backend/transaction

sethdivyansh commented Jun 8, 2025 •

edited

Loading

Uh oh!

sethdivyansh commented Jun 15, 2025

Uh oh!

Priceless-P commented Jun 16, 2025

Uh oh!

Priceless-P Jun 25, 2025

Uh oh!

Priceless-P Jun 25, 2025

Uh oh!

sethdivyansh commented Jun 27, 2025 •

edited

Loading

Uh oh!

Priceless-P Jul 3, 2025

Uh oh!

sethdivyansh Jul 3, 2025

Uh oh!

Priceless-P Jul 3, 2025

Uh oh!

sethdivyansh Jul 3, 2025

Uh oh!

Priceless-P commented Jul 3, 2025

Uh oh!

sethdivyansh commented Jul 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sethdivyansh commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Prerequisite::

Installation

API Endpoints

REST Endpoints

WebSocket Endpoints

Event Types

Architecture

Error Handling

Uh oh!

sethdivyansh commented Jun 15, 2025

Uh oh!

Priceless-P commented Jun 16, 2025

Uh oh!

Priceless-P Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

Priceless-P Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

sethdivyansh commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Priceless-P Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

sethdivyansh Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

Priceless-P Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

sethdivyansh Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

Priceless-P commented Jul 3, 2025

Uh oh!

sethdivyansh commented Jul 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sethdivyansh commented Jun 8, 2025 •

edited

Loading

sethdivyansh commented Jun 27, 2025 •

edited

Loading