cmf_backtest — HFT Backtest Engine

A single-threaded C++20 backtesting engine for replaying tick-level limit-order-book (LOB) data and evaluating algorithmic trading strategies at scale.

---

Project Overview

This project implements a backtesting framework for an HFT simulator, extending an existing C++-based trading platform with historical data replay and strategy evaluation capabilities.

Original Task: Design and Implement a Backtesting Framework for a C++-based HFT Simulator

Objectives

• ✅ Tick-level data input: Zero-copy parsing of LOB and trade CSVs via mmap + std::from_chars
• ✅ Time-controlled replay: Merge LOB snapshots and trade events in chronological order
• ✅ Strategy integration: Five distinct strategies via a Strategy C++20 concept
• ✅ Performance analytics: PnL, drawdown, Sharpe ratio, win rate, fills export
• ✅ Production-grade quality: Unit tests (GoogleTest), benchmarks (GoogleBenchmark), clean architecture

---

What's Inside

Core Components

main.cpp
 ├── args.hpp             CLI config parsing (Config, parse_args)
 ├── types.hpp            Price types, LobSnapshot, Order, Fill, PnlState, Strategy concept
 ├── parser.hpp           MmapFile, LobReader, TradeReader (zero-copy mmap CSV)
 ├── order_book.hpp       OrderBook (pmr maps/deques), OrderBookLike concept
 ├── engine.hpp           try_fill_on_lob() [matching engine], run_backtest()
 ├── strategies/           PassiveStrategy, ReplayStrategy, TrendFollowingStrategy, AvellanedaStoikovStrategy, MicropriceAvellanedaStoikovStrategy
 ├── strategy_factory.hpp StrategyType enum, StrategyVariant, make_strategy()
 └── analytics.hpp        compute_analytics(), export_fills_csv()

Deliverables

Item	Status	Details
Backtest engine	✅	src/engine.hpp with matching logic and fill semantics
Historical data input	✅	CSV via zero-copy mmap (src/parser.hpp)
Strategy integration	✅	5 strategies, C++20 concept dispatch (code/src/strategies/)
Analytics	✅	PnL, drawdown, Sharpe, win-rate, fills CSV (src/analytics.hpp)
Tests	✅	4 files, 50+ unit/integration tests (GoogleTest v1.17.0)
Benchmarks	✅	3 files, end-to-end + microbench (GoogleBenchmark v1.9.5)
Sample dataset	✅	200 LOB + 2.9k trades under data/
Full dataset	📥	1.8GB on Google Drive

---

C++20 Features

This project showcases modern C++ capabilities:

1. Concepts — Zero-cost abstraction for strategy dispatch

template<typename T>
concept Strategy = requires(T& s, const LobSnapshot& lob) {
    { s.on_lob(lob) } -> std::convertible_to<void>;
};

Enables run_backtest<S>() to specialize per-strategy at compile-time, eliminating vtable overhead.

2. std::pmr — Custom memory allocation

std::pmr::unsynchronized_pool_resource pool;
OrderBook ob(&pool);  // Fast arena-backed allocation

Allocator reuse keeps per-run allocations in hot cache (§engineering culture).

3. std::from_chars — Locale-free field parsing

std::from_chars(field.data(), field.data() + field.size(), value);

Zero-allocation, 40x faster than atoi or std::strtod for CSV fields.

4. [[nodiscard]] — Enforce intentional use

[[nodiscard]] int64_t mid_ticks() const noexcept {
    return (asks[0].price + bids[0].price) / 2;
}

Compiler warns if computed value is discarded accidentally.

5. std::variant + std::visit — Type-safe strategy dispatch

StrategyVariant s = make_strategy("trend", config);
std::visit([&](auto& strategy) { run_backtest(strategy, ...); }, s);

Single StrategyVariant dispatches to correct strategy at runtime.

6. Integer digit separators — Readable constants

static constexpr int64_t  PRICE_SCALE = 10'000'000;
static constexpr uint64_t US_PER_DAY  = 86'400'000'000ULL;

---

Trading Strategies

Strategy	Description	Report
passive	Symmetric market maker. Posts bid at best_bid - 1 tick and ask at best_ask + 1 tick. Cancels and re-quotes when the market moves. Pure maker — no taker orders.	passive.md
replay	Replays the trade tape. For each trade event, places an opposing limit order at the trade price and size. Used to test fill mechanics on real market activity.	—
trend	Dual-EMA crossover with Order Book Imbalance filter. Fast EMA period=500 ticks, slow=2000 ticks. Confirmed by OBI (depth=5, threshold=0.1). Sends aggressive limit orders to reach target position. Includes 2500-tick warmup and 100-tick cooldown between position flips.	trend_ema.md
avellaneda_stoikov	Market making strategy based on the Avellaneda & Stoikov model. Dynamically sets bid/ask quotes based on inventory level and market volatility.	avellaneda_stoikov.md
microprice_as	Avellaneda & Stoikov strategy enhanced with microprice-based order placement. Uses full LOB depth information for more precise quote placement.	microprice_as.md

---

Dataset

Input Characteristics

• LOB snapshots: 1,036,690 rows (25-level depth)
• Trade events: ~21.9 million rows
• Time unit: Microseconds (non-uniform: median 3 µs, 75th pct 1,307 µs)
• Sample data: 200 LOB + 2.9k trades bundled under data/
• Full dataset: Download from Google Drive (1.8 GB)

Fill Criterion

Simplest possible: a resting limit order is filled when the market price crosses its level. On each LOB update:

• a buy limit at price P fills when best_ask <= P
• a sell limit at price P fills when best_bid >= P

Orders execute at their own limit price (passive maker assumption). A same-tick look-ahead guard prevents an order from filling on the very LOB snapshot that triggered its placement (lob.timestamp != order.placement_ts).

Trade-tape events do not match resting orders. The Strategy concept exposes only on_lob; strategies that need the trade tape (e.g. ReplayStrategy) hold their own TradeReader& and consume it inside on_lob. See try_fill_on_lob in engine.hpp.

---

Simplifications

Decision-time modelling: The engine does not directly measure strategy decision latency when advancing past L2 order-book states. It assumes the strategy can complete its decision before the next tick arrives. Given the non-uniform tick distribution above, this assumption holds most of the time but breaks down during burst periods (sub-microsecond ticks). Accurate decision-time accounting is planned for a future iteration.

---

Build & Run

Prerequisites

• C++20 compiler (GCC 16.1, Clang 22.1.5 — CI-tested)
• CMake 3.25+
• macOS/Linux (tested on Apple M4 Pro)

Build

mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j

Run on Sample Data

./code --lob data/sample_lob.csv --trades data/sample_trades.csv --strategy trend

Expected output:

LOB rows parsed   : 200
Trade rows parsed : 0
Orders submitted  : 0
Fills             : 0
Final position    : 0
Realized PnL      : 0.00000000
Unrealized PnL    : 0.00000000
Total PnL         : 0.00000000
Max Drawdown      : 0.00000000
Sharpe (daily~)   : 0.0000
Win Rate          : 0.00%
Elapsed           : 0.000 s
Throughput        : 0.4 M events/s
Fills CSV         : fills_0_trend_ema.csv
Maker fee (bps)   : 0.00

Note: Trend strategy produces no fills on sample data because the 2500-tick EMA warmup exceeds the 200 available LOB rows.

Run with Full Dataset

./code --lob /path/to/lob.csv --trades /path/to/trades.csv \
       --strategy trend --strategy passive --strategy replay \
       --strategy avellaneda_stoikov --strategy microprice_as

Example output (trend strategy on full dataset):

=== Backtest Results [trend_ema] ===
LOB rows parsed   : 1036690
Trade rows parsed : 0
Orders submitted  : 469
Fills             : 469
Final position    : 1000
Realized PnL      : 0.35481080
Unrealized PnL    : 0.00840000
Total PnL         : 0.36321080
Max Drawdown      : 1.44900000
Sharpe (daily~)   : 2.9769
Win Rate          : 32.44%
Elapsed           : 0.874 s
Throughput        : 1.2 M events/s
Fills CSV         : fills_0_trend_ema.csv
Maker fee (bps)   : 0.00

CLI Options

Flag	Default	Description
--lob	required	Path to LOB CSV
--trades	required	Path to trades CSV
--strategy	replay	Strategy (passive, replay, trend, avellaneda_stoikov, microprice_as); repeatable
--target-qty	100000	Max position size
--maker-fee-bps	0.0	Maker fee (basis points)
--pnl-sample-interval	1000	PnL snapshot frequency
--latency-profile	none	Preset for the latency model (none, lan, metro, wan, retail)
--md-latency-us, --md-latency-jitter-us	0, 0	Market-data delivery delay (mean ± uniform jitter, µs)
--order-latency-us, --order-latency-jitter-us	0, 0	Order-entry delay (mean ± uniform jitter, µs)
--cancel-latency-us, --cancel-latency-jitter-us	0, 0	Cancel/drain delay (mean ± uniform jitter, µs)
--latency-seed	0xC0FFEE	RNG seed for jitter (deterministic when fixed)

Latency Modelling

Three independent channels can be simulated, each with mean delay + uniform ± jitter:

• MD latency — on_lob receives a delayed snapshot; the strategy acts on a stale book.
• Order latency — submit() actions reach the matching engine after a delay; quotes may miss fast moves.
• Cancel latency — drain() takes effect after a delay; resting quotes can still fill during the cancel window, which models adverse selection on stale quotes.

--latency-profile <name> is a one-shot bundle that fills all six latency fields. Individual --*-latency-us flags placed after the profile override per-field values (left-to-right, last-write-wins).

Profile	mean (µs)	jitter ±(µs)	Scenario
none	0	0	Idealised — instantaneous (default)
lan	50	10	Colocated, same-rack matching
metro	500	100	Cross-DC inside one metro
wan	5,000	1,000	Inter-region, public WAN
retail	50,000	15,000	Retail-broker round trips

Jitter is uniform [-J, +J]. The MD jitter stream is independently re-seeded from --latency-seed so it does not correlate with order/cancel jitter; identical seeds yield identical fill streams. With mean=0 jitter=0 a channel collapses to a zero-overhead fast path.

Visualization & Analysis

After running a backtest, visualize strategy performance with an interactive dashboard:

uv run scripts/backtest_report.py cmake-build-release/out.csv

This launches a Streamlit web app that displays:

Overview Tab

• High-level metrics: number of strategies, best PnL, best Sharpe, lowest drawdown, composite score
• Full results table with all key metrics
• Bar charts: total PnL, Sharpe ratio, turnover, fills by strategy

Efficiency Tab

• Fill rate (%) and PnL per fill/order metrics
• Scatter plot: PnL vs turnover (colored by Sharpe, sized by fills)
• Turnover per fill analysis to assess market impact

Risk Tab

• Max drawdown and PnL-to-drawdown ratio
• Win rate by strategy
• Risk/Return map: drawdown vs Sharpe (colored by PnL)
• Normalized metrics heatmap for quick strategy comparison

Leaderboard Tab

• Composite scoring system averaging normalized metrics (higher PnL, Sharpe, fill rate; lower drawdown)
• Key observations: best/worst performers, fill rates, PnL efficiency

The report helps you understand:

• Which strategy performed best on your dataset
• Risk-adjusted returns via Sharpe ratio and drawdown analysis
• Fill efficiency through fill rates and PnL per fill metrics
• Comparative performance across all tested algorithms

---

Testing

Tests are written using Google Test (v1.17.0) and live in tests/:

• test_backtest.cpp — 6 unit tests for try_fill_on_lob() (fill logic, same-tick guards, multi-order depletion)
• test_order_book.cpp — 20+ tests for OrderBook (submit, match, FIFO, price levels, depletion)
• test_strategies.cpp — Parameterized integration tests (runs all 5 strategies on sample data)
• test_analytics.cpp — 20+ tests for analytics (drawdown, Sharpe, win rate)

Run tests

cmake --build build --target cmf-backtest-tests
./build/bin/cmf-backtest-tests

---

Benchmarking

Benchmarks use Google Benchmark (v1.9.5) and live in benchmarks/. 5 iterations × 3 repetitions per configuration, Apple M4 Pro.

End-to-end backtest (BM_RunBacktest)

Implementation	Strategy	Mean time	Throughput (mean)	CV
OrderBook	passive	861 ms	1.21 M items/s	0.14%
OrderBook	trend_ema	834 ms	1.25 M items/s	0.43%
OrderBook	replay	2,756 ms	8.32 M items/s	1.32%
OrderBookV2	passive	876 ms	1.18 M items/s	0.51%
OrderBookV2	trend_ema	852 ms	1.22 M items/s	0.31%
OrderBookV2	replay	3,251 ms	7.05 M items/s	0.48%

on_lob() per-call latency (BM_OnLob, synthetic LOB, no I/O)

Strategy	Mean latency	Throughput	CV
passive	2.57 ns	389.2 M/s	1.12%
trend_ema	1.75 ns	573.6 M/s	0.56%

OrderBook primitive microbenchmarks (N=64)

Benchmark	Mean latency	Throughput	CV
Submit (64 orders)	5,511 ns	23.2 M ops/s	1.00%
MatchFound	0.691 ns	1.45 G ops/s	0.18%
MatchEmpty	0.237 ns	4.22 G ops/s	0.89%
EmptyNonEmpty	0.236 ns	4.25 G ops/s	1.04%
EmptyEmpty	0.235 ns	4.27 G ops/s	0.36%
Drain (64 orders)	4,391 ns	29.2 M ops/s	0.84%

Run benchmarks

Use benchmarks/run_benchmarks.py to build benchmark binaries, run profiles, and write timestamped outputs under benchmarks/results/:

uv run benchmarks/run_benchmarks.py run --profile raw
uv run benchmarks/run_benchmarks.py run --profile all --running-mode pgo -- --benchmark_filter=OrderBook

Supported profiles are raw, perf-stat, flamegraph, heap, and all. Arguments after -- are passed through to Google Benchmark. Each result directory includes hardware.json with the machine, CPU, memory, OS, and Python details captured for that run.

perf-stat records Linux hardware counter summaries with perf stat. heap records allocation behavior with heaptrack, falling back to Valgrind Massif when heaptrack is unavailable.

Use --running-mode pgo to train an instrumented build, rebuild with the merged PGO data, and run the selected profile against the optimized binary.

Flamegraph profiling uses the platform profiler (sample on macOS, perf on Linux) plus Inferno for stack collapsing and SVG rendering:

cargo install inferno

You can also run the benchmark binary directly:

cmake --build build --target cmf-backtest-benchmarks
./build/bin/cmf-backtest-benchmarks

---

Engineering Culture

Design Principles

• Zero-copy I/O: mmap + std::from_chars eliminates parsing bottleneck
• Compile-time dispatch: Concepts + templates = no vtable overhead
• Modular strategies: Common Strategy interface allows easy extension
• Type-safe numerics: Integer-tick pricing (int64_t) prevents FP errors
• Arena allocation: std::pmr keeps per-run allocations hot

Testing Strategy

• Unit tests: Individual components (OrderBook, analytics)
• Integration tests: Full strategy runs on real data
• Microbenchmarks: Identify hotspots (match, allocation)
• End-to-end benchmarks: Wall-clock validation at scale

Code Quality

• No undefined behavior: Concepts, bounds checks, safe defaults
• Readable constants: Digit separators, named constants
• Fast path annotations: [[nodiscard]], noexcept, inline hints
• Minimal dependencies: Only GoogleTest and GoogleBenchmark (via CMake FetchContent)

---

Architecture Deep Dive

See code/README.md for detailed technical documentation.

See code/SUMMARY.md for full performance analysis, implementation notes, and learnings.

---

Quick Links

• Build: mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release && make -j
• Sample run: ./code --lob data/sample_lob.csv --trades data/sample_trades.csv --strategy passive
• Tests: cmake --build build --target cmf-backtest-tests && ./build/bin/cmf-backtest-tests
• Benchmarks: cmake --build build --target cmf-backtest-benchmarks && ./build/bin/cmf-backtest-benchmarks
• Full dataset: Google Drive (1.8 GB)
• Original task: LinkedIn post

---

Author: Mark Andreev
Language: C++20 | Platform: darwin-arm64 | Build: CMake 3.25+
Date: 2026-04-10 | Branch: dev-20260408