Datum Work

Datum Work is a mining-work monitor for AlphaPool, PyBlock, XOR, and your own BLAKE2b block templates. The home dashboard brings these sources together while retaining their source and algorithm information. PyBlock defaults to its BLAKE2b WAVICLES endpoint at b.pyblock.xyz:23115; its SHA256 endpoint is a separate configuration described in the monitor guide.

This project is a fork of bboerst/stratum-work. The original Bitcoin Stratum dashboard remains available at /table. Credit for the original visualization idea belongs to 0xB10C, and for the upstream implementation to its contributors.

Run the Datum Work dashboard locally

Use Node.js 22.14 or newer and Python 3.10 or newer. The monitor uses the Python standard library and needs no Python package installation. From this checkout:

cd web
npm ci
npx prisma generate
npm run dev

Open localhost:3000. In a second terminal at the repository root, start the monitor:

python3 monitor/service.py

The web app reads the monitor at http://127.0.0.1:8810 by default. For a different monitor address, set DATUM_MONITOR_URL when starting the web app. See monitor/README.md for pool endpoints, optional pool authorization, the SHA256 PyBlock override, and your own gateway template configuration.

After installing the web dependencies, you can also start both local services from the repository root:

./run-local.sh --check
./run-local.sh

The helper checks dependencies and refuses to start when either port is occupied. It binds both services to loopback and stops them together on Ctrl+C. Set WEB_PORT=3001 or DATUM_MONITOR_PORT=8811 to use different ports. PYTHON_BIN and NODE_BIN can select installed runtimes. Stop any separately started web app or monitor before using the helper.

The legacy SHA256 view at /table requires the separate Bitcoin stack: configured collectors, RabbitMQ, MongoDB, and a Bitcoin node with RPC and ZMQ. It cannot show live or historical data from a web-only startup. The legacy stack and data flow are documented below.

Development checks

From web/:

npm test
npx tsc --noEmit
npm run build

See Helm deployment configuration for fork-specific image repositories and publishing settings.

Legacy Stratum Work stack

The following architecture and Docker Compose instructions describe the inherited Bitcoin Stratum application. They are separate from the Datum Work monitor service and its BLAKE2b template source.

The inherited application visualizes mining.notify messages from Bitcoin Stratum pools and identifies the pools that mined confirmed blocks.

Architecture Overview

Stratum Work consists of three main components:

1. Collector: Connects to Bitcoin mining pools via the Stratum protocol, captures mining.notify messages, and forwards them to RabbitMQ.
2. Backend: Processes Bitcoin blocks, identifies mining pools, and provides block data to the web application.
3. Web Application: Displays real-time mining notifications and block information in an interactive interface.

Data Flow

1. Multiple collector instances connect to different mining pools
2. When a pool sends a mining.notify message, the collector:
   • Stores the message in MongoDB
   • Publishes the message to RabbitMQ
3. The web application consumes these messages from RabbitMQ and displays them in real-time
4. The backend processes new blocks from a Bitcoin node and identifies which pool mined each block

Technical Details

Stratum Protocol and Mining.Notify Messages

The Stratum protocol is used by mining pools to coordinate miners. The mining.notify message is particularly important as it contains the template for a new block that miners should work on. Each message contains:

• Job ID: A unique identifier for this mining job
• Previous Block Hash: The hash of the last block in the chain
• Coinbase (Part 1 & 2): The coinbase transaction split into two parts
• Merkle Branches: Hashes needed to construct the merkle root
• Version: Block version
• nBits: Target difficulty
• nTime: Current timestamp
• Clean Jobs: Boolean indicating if previous jobs should be discarded

Trustless Data Processing

A key design principle of Stratum Work is trustless data processing. While the server collects and streams the raw data, all decoding, formatting, and visualization is performed in the client browser.

Raw Data

The application provides a Server-Sent Events (SSE) endpoint:

GET /api/stream

This endpoint delivers the same real-time data that powers the web interface, allowing for custom integrations or alternative visualizations.

Data Processing and Visualization

Collector Processing

The collector performs minimal processing to maintain data integrity:

See collector/main.py:294-323 for the notification document creation function.

Web Application Processing

The web application performs extensive processing to make the data more readable and visually informative:

1. Coinbase Transaction Reconstruction:
   See web/utils/formatters.ts:38-45

2. Coinbase Script ASCII Extraction:
   See web/utils/bitcoinUtils.ts:48-76

3. Coinbase Output Analysis:
   See web/utils/bitcoinUtils.ts:95-126

4. First Transaction Extraction:
   See web/utils/bitcoinUtils.ts:129-139

5. Fee Rate Calculation:
   See web/utils/bitcoinUtils.ts:142-168

Backend Block Processing

The backend identifies which pool mined each block by analyzing the coinbase transaction:

See backend/main.py:815-890 for the block processing function.

Backend Analytics Plug‑ins

The backend supports a pluggable analytics system that runs whenever a new block is processed. These analytics inspect the set of mining.notify templates for the block’s height and may emit structured findings that get saved on the block document and forwarded to RabbitMQ.

• Location: backend/analytics/

  • prev_hash_divergence.py – detects pools advertising different previous block hashes
  • invalid_coinbase_no_merkle.py – flags templates with no merkle branches where the coinbase output exceeds the block subsidy
  • pool_identification.py – owns pool identification and the PoolsManager

• Output shape (stored on db.blocks and in the SSE payload):

  • analysis.flags[]: list of small findings with compact metadata
    • Each item: { key: string, icon: string, details: {...} }
    • Example (prev-hash fork):

      {
        "key": "prev_hash_fork",
        "icon": "fork",
        "details": {
          "groups": [
            { "prev_hash": "…", "pools": ["PoolA", "PoolB"] }
          ]
        }
      }

  • analysis.pool_identification: richer object for the winning pool
    • { mining_pool: {...}, method: "address|tag", addresses_considered: [] }

Adding a New Analyzer

1. Create a new module in backend/analytics/, e.g. my_new_analysis.py, and export a function that returns either a finding dict or None:

   from typing import Any, Dict, List, Optional
   
   def analyze_my_feature(templates: List[Dict[str, Any]], logger) -> Optional[Dict[str, Any]]:
       # inspect templates …
       if not interesting:
           return None
       return {
           "key": "my_feature",
           "icon": "star",
           "details": { "…": "…" }
       }

2. Import and append it in backend/main.py inside run_block_analyses(...) (the function already collects and passes the MongoDB mining_notify records for the block height):

   from analytics.my_new_analysis import analyze_my_feature
   
   …
   finding = analyze_my_feature(templates, logger)
   if finding:
       flags.append(finding)

3. Rebuild the backend image so the new module is included (the Dockerfile copies backend/analytics/).

Guidelines:

• Keep findings small (store details needed to render UI, avoid large payloads).
• Prefer deterministic, height-scoped logic using the templates fetched for that block height.
• Use the shared logger for traceability; avoid printing secrets.

Features

• Real-time display of mining.notify messages from Stratum pools
• Customizable table columns for displaying relevant data
• Light and dark mode
• Live and historical data views
• MongoDB integration for data storage and retrieval
• Most 'work' is done client-side for trustless data processing
• Raw data access via /api/stream endpoint

Local Development with Docker-Compose

To simplify local development and testing, you can use docker-compose to run all the components on your machine. This allows you to quickly spin up RabbitMQ, MongoDB, a web application container, and one or more collector containers.

Prerequisites

• Docker
• Docker Compose

Steps

1. Open your Datum Work checkout:

   cd datum-work

2. Customize collectors:

   In the ./docker-compose.yml file, you will see a collector-base service along with one or more collectors (e.g., collector-f2pool). Each collector points to a specific Stratum pool. To add more pools, just duplicate one of the collector services and update the --pool-name and --url fields:

   collector-newpool:
     <<: *collector-base
     command: >
         /usr/local/bin/python main.py
         --pool-name "NewPool"
         --url "stratum+tcp://newpool.com:3333"
         --userpass "someuser:somepass"
         --rabbitmq-host "rabbitmq"
         --rabbitmq-username "mquser"
         --rabbitmq-password "mqpassword"
         --db-url "mongodb"
         --db-name "stratum-logger"
         --db-username "mongouser"
         --db-password "mongopassword"
         --log-level "DEBUG"

   The <<: *collector-base line pulls in all the shared environment variables and dependencies. All you need to do is adjust pool-specific configuration.

3. Build and start the services:

   docker-compose up --build

   This command will:

   • Build the images for the webapp and collector services.
   • Start RabbitMQ, MongoDB, and your configured collectors.
   • Start the web application.

   Once everything is running, the web application should be accessible at http://localhost:3000/table.

4. View Logs:

   As everything starts, you can see logs in the terminal where you ran docker-compose up.These logs will show connections to pools,RabbitMQ events, and requests to the webapp.

5. Stop the services:

   To stop and remove the containers, press Ctrl + C in your terminal. To remove containers, networks, and images created by docker-compose, run:

   docker-compose down

   To also remove the mongodb volume:

   docker-compose down -v