Skip to content

setup.md

Latest commit

 

History

History
300 lines (224 loc) · 9.57 KB

setup.md

File metadata and controls

300 lines (224 loc) · 9.57 KB

Setup Guide

Prerequisites

Dependency Version Notes
Rust 2024 edition edition = "2024" in Cargo.toml
Bitcoin Core 25.0+ Must expose ZMQ and JSON-RPC
libzmq headers 4.x apt install libzmq3-dev · pacman -S zeromq · brew install zmq
protoc 3.15+ Protocol Buffer compiler — required by tonic-build for gRPC codegen

Key crate versions pinned in Cargo.toml:

bitcoin 0.32          corepc-client 0.10     zmq 0.10
tonic 0.12            prost 0.13             dashmap 6
crossbeam-channel 0.5 parking_lot 0.12       clap 4
tracing-subscriber 0.3 (env-filter)          thiserror 2

Release profile ([profile.release]): opt-level = 3, fat LTO, single codegen-unit, panic = "abort", symbols stripped. Benchmark profile inherits release with debug info retained.

bitcoind Configuration

The engine requires two ZMQ topics. Add to bitcoin.conf:

# Raw transaction bytes — payload used to decode fee, vsize, txid.
zmqpubrawtx=tcp://127.0.0.1:28332

# Total-ordered mempool sequence notifications.
# Emits A (added), R (removed), C (block-connected), D (block-disconnected).
# This is NOT zmqpubhashblock — the engine needs the sequence stream for
# ordered add/remove/block events and reorg detection.
zmqpubsequence=tcp://127.0.0.1:28333

# Optional: raw blocks (used for rawblock_endpoint if configured).
zmqpubrawblock=tcp://127.0.0.1:28334

server=1
txindex=1
maxmempool=1000

# Recommended: ensures the mempool applies full-RBF rules unconditionally,
# so the engine's view stays consistent with miner selection behaviour.
mempoolfullrbf=1

Restart bitcoind and verify:

bitcoin-cli stop && bitcoind -daemon
bitcoin-cli getzmqnotifications

Expected output — note pubsequence, not pubhashblock:

[
  {"type": "pubrawtx",    "address": "tcp://127.0.0.1:28332", "hwm": 1000},
  {"type": "pubsequence", "address": "tcp://127.0.0.1:28333", "hwm": 1000}
]

Memq Configuration

All settings live in memq.toml. Every section carries #[serde(default)] and may be omitted — an empty file is valid for a local regtest node.

[zmq]
rawtx_endpoint    = "tcp://127.0.0.1:28332"   # must match zmqpubrawtx
sequence_endpoint = "tcp://127.0.0.1:28333"   # must match zmqpubsequence
rawblock_endpoint = "tcp://127.0.0.1:28334"   # must match zmqpubrawblock (optional)
hwm               = 10000                     # ZMQ high-water mark per socket

[rpc]
url      = "http://127.0.0.1:18443"           # JSON-RPC (initial sync + block reconciliation only)
user     = "bitcoin"
password = "bitcoin"

[grpc]
listen_addr = "[::1]:50051"

[pipeline]
channel_capacity = 65536                       # bounded crossbeam channel; backpressure when full

[hydration]
workers = 1                                    # number of parallel RPC hydration threads

[mempool]
max_total_vsize = 300000000                    # maximum mempool vsize before eviction (vbytes)

[estimation]
decay_half_life_secs = 3600                    # time-decay half-life for stale transaction down-weighting

[metrics]
enabled     = true
listen_addr = "[::1]:9100"                     # Prometheus exposition endpoint

Building

cargo build                    # debug
cargo build --release          # opt-level 3, fat LTO, codegen-units 1, strip symbols
cargo test                     # unit + doc tests
cargo test --test regtest_integration   # end-to-end against bitcoind regtest
cargo bench                    # criterion: fee_engine, mempool, ingestion

CLI Flags

memq [OPTIONS]

  -c, --config <PATH>           Path to TOML config [default: memq.toml]
      --zmq-rawtx <ENDPOINT>    Override zmq.rawtx_endpoint
      --zmq-sequence <ENDPOINT> Override zmq.sequence_endpoint
      --grpc-addr <ADDR>        Override grpc.listen_addr
      --skip-sync               Skip initial RPC mempool sync (starts in degraded mode)

CLI flags take precedence over memq.toml values.

Running

# Default — reads memq.toml from cwd
./target/release/memq

# Custom config
./target/release/memq -c /etc/memq/prod.toml

# CLI overrides
./target/release/memq \
  --zmq-rawtx tcp://10.0.0.5:28332 \
  --zmq-sequence tcp://10.0.0.5:28333 \
  --grpc-addr "[::]:50051"

# Skip initial sync (no RPC needed at startup)
./target/release/memq --skip-sync

Logging

Controlled via the RUST_LOG environment variable (tracing-subscriber with env-filter). Default level: info. Thread IDs are included in every line.

RUST_LOG=info  ./target/release/memq          # default
RUST_LOG=debug ./target/release/memq          # verbose
RUST_LOG=memq::ingestion=trace ./target/release/memq  # per-module

Querying via gRPC

Ten RPCs are exposed on memq.MemqService:

# EstimateFee — fee estimate for a confirmation target
grpcurl -plaintext -d '{"target_blocks": 1}' \
  '[::1]:50051' memq.MemqService/EstimateFee

# GetMempoolStats — current mempool summary (count, vsize, fees)
grpcurl -plaintext \
  '[::1]:50051' memq.MemqService/GetMempoolStats

# GetFeeHistogram — fee-rate distribution bucketed into N bins
grpcurl -plaintext -d '{"num_buckets": 20}' \
  '[::1]:50051' memq.MemqService/GetFeeHistogram

# SimulateBlocks — project the next N blocks from current mempool
grpcurl -plaintext -d '{"num_blocks": 3}' \
  '[::1]:50051' memq.MemqService/SimulateBlocks

# GetTransaction — look up a single mempool transaction by txid
grpcurl -plaintext -d '{"txid": "a1b2c3..."}' \
  '[::1]:50051' memq.MemqService/GetTransaction

# StreamFeeEstimates — server-streaming fee estimates at a fixed interval
grpcurl -plaintext -d '{"target_blocks": 1, "interval_ms": 5000}' \
  '[::1]:50051' memq.MemqService/StreamFeeEstimates

# EstimateConfirmationDistribution — probability distribution of confirmation by block
grpcurl -plaintext -d '{"fee_rate_sat_per_vb": 12.5, "max_blocks": 6}' \
  '[::1]:50051' memq.MemqService/EstimateConfirmationDistribution

# EstimateCpfpFee — CPFP fee estimate for bumping a parent transaction
grpcurl -plaintext -d '{"parent_txid": "a1b2c3...", "target_blocks": 2, "child_vsize": 141}' \
  '[::1]:50051' memq.MemqService/EstimateCpfpFee

# GetFeePressure — current fee-pressure snapshot (no params)
grpcurl -plaintext \
  '[::1]:50051' memq.MemqService/GetFeePressure

# StreamFeePressure — server-streaming fee-pressure updates at a fixed interval
grpcurl -plaintext -d '{"interval_ms": 10000}' \
  '[::1]:50051' memq.MemqService/StreamFeePressure

Prometheus Metrics

Served at GET /metrics on the address configured in [metrics].listen_addr (default [::1]:9100). All other paths return 404. A 5-second read timeout mitigates slow-loris connections.

Counters (cumulative):

Metric Description
memq_ingested_txs_total Transactions ingested from ZMQ
memq_removed_txs_total Transactions removed
memq_blocks_connected_total Blocks connected
memq_zmq_gap_count_total ZMQ sequence gaps detected
memq_hydration_failures_total Fee hydration RPC failures
memq_compared_blocks_total Blocks compared for accuracy
memq_estimate_mempool_total Estimates served from mempool engine
memq_estimate_core_total Estimates served from Bitcoin Core
memq_estimate_fallback_total Estimates served as mempool fallback

Gauges (current state):

Metric Description
memq_mempool_tx_count Current mempool transaction count
memq_mempool_total_vsize Current mempool total vsize (vbytes)
memq_mempool_total_fees_sat Current mempool total fees (satoshis)
memq_template_jaccard_last Last block template Jaccard similarity
memq_template_jaccard_avg EMA of block template Jaccard similarity
memq_fee_prediction_error_last Last block fee prediction error (sat/vB)
memq_fee_prediction_error_avg EMA of fee prediction error (sat/vB)
memq_ready Whether memq is ready (1/0)
memq_degraded Whether memq is degraded (1/0) 
curl -s http://[::1]:9100/metrics

Shared Memory Interface

Memq publishes a compact fee-estimate snapshot to shared memory at /dev/shm/memq. External processes can read it without any IPC overhead by memory-mapping the file.

The layout uses a seqlock pattern for lock-free consistency:

  1. Read the 8-byte sequence number at offset 0.
  2. If the sequence number is odd, a write is in progress — spin and retry.
  3. Read the payload (fee-estimate fields following the sequence number).
  4. Re-read the sequence number. If it changed, the payload was torn — retry from step 1.

A working reader is provided in examples/shm_reader.rs:

cargo run --example shm_reader

Regtest Development

Use regtest for local development and integration testing.

# bitcoin.conf
regtest=1
zmqpubrawtx=tcp://127.0.0.1:28332
zmqpubsequence=tcp://127.0.0.1:28333
server=1

Generate test activity:

bitcoin-cli -regtest createwallet "test"
bitcoin-cli -regtest -generate 101
bitcoin-cli -regtest sendtoaddress $(bitcoin-cli -regtest getnewaddress) 0.1

Run the integration test suite (requires a running regtest node):

cargo test --test regtest_integration

Benchmarks

Criterion benchmarks live in benches/ and inherit the release profile with debug info enabled ([profile.bench]).

cargo bench                    # run all three suites
cargo bench --bench fee_engine # fee engine projection + estimation
cargo bench --bench mempool    # mempool insert/remove/lookup
cargo bench --bench ingestion  # ZMQ message parsing throughput

HTML reports are written to target/criterion/.