Skip to content

Commit 562edcb

Browse files
Johnson-OgundejiJohnson-Ogundeji
committed
bench: govern benchmark IDs with registry and CI validation
1 parent a3253cd commit 562edcb

12 files changed

Lines changed: 540 additions & 45 deletions

‎.github/workflows/ci.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -419,13 +419,27 @@ jobs:
419419
env:
420420
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
421421

422+
# ---------------------------------------------------------------------------
423+
# Benchmark ID registry validation (all pushes and PRs)
424+
# ---------------------------------------------------------------------------
425+
benchmark-registry:
426+
name: Benchmark Registry
427+
runs-on: ubuntu-24.04
428+
429+
steps:
430+
- uses: actions/checkout@v6
431+
432+
- name: Validate benchmark registry
433+
run: ./scripts/validate_benchmark_registry.sh
434+
422435
# ---------------------------------------------------------------------------
423436
# Benchmark regression detection (push to main only)
424437
# ---------------------------------------------------------------------------
425438
benchmarks:
426439
name: Benchmarks (Ubuntu)
427440
runs-on: ubuntu-24.04
428441
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
442+
needs: benchmark-registry
429443

430444
permissions:
431445
contents: write

‎Benchmarking_Protocols/BENCHMARK_ID_REGISTRY.md‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# Benchmark ID Registry
2+
3+
This document is generated from `Benchmarking_Protocols/benchmark_id_registry.tsv`.
4+
Do not edit this Markdown file by hand. Update the registry source and rerender it.
5+
6+
## Governance
7+
1. Benchmark IDs are global across tracked benchmark suites in `benchmarks/` and `Benchmarking_Protocols/`.
8+
2. Benchmark IDs are immutable once published, unless explicitly deprecated and replaced.
9+
3. Every tracked `BENCHMARK(...)` or `BENCHMARK_ADVANCED(...)` label must exist in the registry.
10+
4. Generic labels such as `write`, `read_all`, or `delta encode` are not permitted.
11+
5. `status=active` entries must exist in code. `deprecated` and `reserved` entries are retained for history and future planning.
12+
6. The CI validator is the enforcement point. A benchmark change is incomplete until the registry and generated documentation are updated.
13+
14+
## Registry Source
15+
- `Benchmarking_Protocols/benchmark_id_registry.tsv`
16+
17+
## Protocol Benchmarks
18+
| ID | Suite | File | Status |
19+
|---|---|---|---|
20+
| `RT1: roundtrip 1M PLAIN` | `phase10_roundtrip` | `Benchmarking_Protocols/bench_phase10_roundtrip.cpp` | `active` |
21+
| `RT2: roundtrip 1M optimal Snappy` | `phase10_roundtrip` | `Benchmarking_Protocols/bench_phase10_roundtrip.cpp` | `active` |
22+
| `RT3: roundtrip 1M PME` | `phase10_roundtrip` | `Benchmarking_Protocols/bench_phase10_roundtrip.cpp` | `active` |
23+
| `RT4: roundtrip 1M PQ` | `phase10_roundtrip` | `Benchmarking_Protocols/bench_phase10_roundtrip.cpp` | `active` |
24+
| `W1: 1K PLAIN uncompressed` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
25+
| `W2: 100K PLAIN uncompressed` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
26+
| `W3: 1M PLAIN uncompressed` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
27+
| `W4: 10M PLAIN uncompressed` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
28+
| `W5: 1M PLAIN Snappy` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
29+
| `W6: 1M PLAIN ZSTD` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
30+
| `W7: 1M PLAIN LZ4` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
31+
| `W8: 1M PLAIN Gzip` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
32+
| `W9: 1M optimal Snappy` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
33+
| `W10: 1M optimal ZSTD` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
34+
| `W11: 1M optimal Snappy PME` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
35+
| `W12: 1M optimal Snappy PQ` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
36+
| `W13: 10M optimal Snappy` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
37+
| `W14: 10M optimal Snappy PME` | `phase2_write` | `Benchmarking_Protocols/bench_phase2_write.cpp` | `active` |
38+
| `R1: read all 1M PLAIN` | `phase3_read` | `Benchmarking_Protocols/bench_phase3_read.cpp` | `active` |
39+
| `R2: projection bid+ask 1M` | `phase3_read` | `Benchmarking_Protocols/bench_phase3_read.cpp` | `active` |
40+
| `R3: read 1M optimal Snappy` | `phase3_read` | `Benchmarking_Protocols/bench_phase3_read.cpp` | `active` |
41+
| `R4: mmap read 1M optimal Snappy` | `phase3_read` | `Benchmarking_Protocols/bench_phase3_read.cpp` | `active` |
42+
| `R5: typed read 1M optimal Snappy` | `phase3_read` | `Benchmarking_Protocols/bench_phase3_read.cpp` | `active` |
43+
| `R6: read 1M optimal ZSTD` | `phase3_read` | `Benchmarking_Protocols/bench_phase3_read.cpp` | `active` |
44+
| `R7: read 1M PME` | `phase3_read` | `Benchmarking_Protocols/bench_phase3_read.cpp` | `active` |
45+
| `R8: read 1M PQ` | `phase3_read` | `Benchmarking_Protocols/bench_phase3_read.cpp` | `active` |
46+
| `R9: read all 10M PLAIN` | `phase3_read` | `Benchmarking_Protocols/bench_phase3_read.cpp` | `active` |
47+
| `R10: read 10M optimal Snappy` | `phase3_read` | `Benchmarking_Protocols/bench_phase3_read.cpp` | `active` |
48+
| `P1: filter symbol 1M` | `phase4_predicate` | `Benchmarking_Protocols/bench_phase4_predicate.cpp` | `active` |
49+
| `P2: filter symbol+exchange 1M` | `phase4_predicate` | `Benchmarking_Protocols/bench_phase4_predicate.cpp` | `active` |
50+
| `P3: time range filter 1M` | `phase4_predicate` | `Benchmarking_Protocols/bench_phase4_predicate.cpp` | `active` |
51+
| `P4: bloom probe 1M` | `phase4_predicate` | `Benchmarking_Protocols/bench_phase4_predicate.cpp` | `active` |
52+
| `P5: filter symbol 10M` | `phase4_predicate` | `Benchmarking_Protocols/bench_phase4_predicate.cpp` | `active` |
53+
| `WAL1: 100K WalWriter` | `phase5_wal` | `Benchmarking_Protocols/bench_phase5_wal.cpp` | `active` |
54+
| `WAL2: 100K WalMmapWriter` | `phase5_wal` | `Benchmarking_Protocols/bench_phase5_wal.cpp` | `active` |
55+
| `WAL3: 100K WalManager` | `phase5_wal` | `Benchmarking_Protocols/bench_phase5_wal.cpp` | `active` |
56+
| `WAL4: 1M WalWriter` | `phase5_wal` | `Benchmarking_Protocols/bench_phase5_wal.cpp` | `active` |
57+
| `WAL5: 1M WalMmapWriter` | `phase5_wal` | `Benchmarking_Protocols/bench_phase5_wal.cpp` | `active` |
58+
| `WAL6: 1M WAL write + read_all` | `phase5_wal` | `Benchmarking_Protocols/bench_phase5_wal.cpp` | `active` |
59+
| `WAL7: 100K WalWriter (encrypted)` | `phase5_wal` | `Benchmarking_Protocols/bench_phase5_wal.cpp` | `active` |
60+
| `FS1: write 100K vectors` | `phase6_features` | `Benchmarking_Protocols/bench_phase6_features.cpp` | `active` |
61+
| `FS2: get latest 1000x` | `phase6_features` | `Benchmarking_Protocols/bench_phase6_features.cpp` | `active` |
62+
| `FS3: as_of 1000x` | `phase6_features` | `Benchmarking_Protocols/bench_phase6_features.cpp` | `active` |
63+
| `FS4: as_of_batch 6 entities` | `phase6_features` | `Benchmarking_Protocols/bench_phase6_features.cpp` | `active` |
64+
| `FS5: history 1000 records` | `phase6_features` | `Benchmarking_Protocols/bench_phase6_features.cpp` | `active` |
65+
| `AI1: DecisionLog 10K records` | `phase7_ai` | `Benchmarking_Protocols/bench_phase7_ai.cpp` | `active` |
66+
| `AI2: InferenceLog 10K records` | `phase7_ai` | `Benchmarking_Protocols/bench_phase7_ai.cpp` | `active` |
67+
| `AI3: verify_chain` | `phase7_ai` | `Benchmarking_Protocols/bench_phase7_ai.cpp` | `active` |
68+
| `AI4: column_view 1M doubles` | `phase7_ai` | `Benchmarking_Protocols/bench_phase7_ai.cpp` | `active` |
69+
| `AI5: EventBus 4P4C 100K` | `phase7_ai` | `Benchmarking_Protocols/bench_phase7_ai.cpp` | `active` |
70+
| `CR1: MiFID2 JSON` | `phase8_compliance` | `Benchmarking_Protocols/bench_phase8_compliance.cpp` | `active` |
71+
| `CR2: MiFID2 NDJSON` | `phase8_compliance` | `Benchmarking_Protocols/bench_phase8_compliance.cpp` | `active` |
72+
| `CR3: MiFID2 CSV` | `phase8_compliance` | `Benchmarking_Protocols/bench_phase8_compliance.cpp` | `active` |
73+
| `CR4: EU AI Act Art.12` | `phase8_compliance` | `Benchmarking_Protocols/bench_phase8_compliance.cpp` | `active` |
74+
| `CR5: EU AI Act Art.13` | `phase8_compliance` | `Benchmarking_Protocols/bench_phase8_compliance.cpp` | `active` |
75+
| `CR6: EU AI Act Art.19` | `phase8_compliance` | `Benchmarking_Protocols/bench_phase8_compliance.cpp` | `active` |
76+
| `I1: Arrow export 1M doubles` | `phase9_interop` | `Benchmarking_Protocols/bench_phase9_interop.cpp` | `active` |
77+
| `I2: tensor wrap 1M doubles` | `phase9_interop` | `Benchmarking_Protocols/bench_phase9_interop.cpp` | `active` |
78+
| `I3: batch tensor 1M x 6` | `phase9_interop` | `Benchmarking_Protocols/bench_phase9_interop.cpp` | `active` |
79+
80+
## Micro Benchmarks
81+
| ID | Suite | File | Status |
82+
|---|---|---|---|
83+
| `encoding_delta_encode_timestamps_10k` | `encodings` | `benchmarks/bench_encodings.cpp` | `active` |
84+
| `encoding_delta_decode_timestamps_10k` | `encodings` | `benchmarks/bench_encodings.cpp` | `active` |
85+
| `encoding_bss_encode_prices_10k` | `encodings` | `benchmarks/bench_encodings.cpp` | `active` |
86+
| `encoding_bss_decode_prices_10k` | `encodings` | `benchmarks/bench_encodings.cpp` | `active` |
87+
| `encoding_rle_encode_bit_width_1_10k` | `encodings` | `benchmarks/bench_encodings.cpp` | `active` |
88+
| `encoding_rle_decode_bit_width_1_10k` | `encodings` | `benchmarks/bench_encodings.cpp` | `active` |
89+
| `encoding_delta_encode_timestamps_10k_vs_plain` | `encodings` | `benchmarks/bench_encodings.cpp` | `active` |
90+
| `encoding_plain_copy_baseline_timestamps_10k` | `encodings` | `benchmarks/bench_encodings.cpp` | `active` |
91+
| `encoding_bss_encode_prices_10k_size_check` | `encodings` | `benchmarks/bench_encodings.cpp` | `active` |
92+
| `encoding_bss_decode_prices_10k_size_check` | `encodings` | `benchmarks/bench_encodings.cpp` | `active` |
93+
| `mpmc_ring_push_pop_single_thread` | `event_bus` | `benchmarks/bench_event_bus.cpp` | `active` |
94+
| `mpmc_ring_shared_column_batch_4p4c_4000_items` | `event_bus` | `benchmarks/bench_event_bus.cpp` | `active` |
95+
| `column_batch_push_1000_rows_and_column_view` | `event_bus` | `benchmarks/bench_event_bus.cpp` | `active` |
96+
| `column_batch_as_tensor_1024x8` | `event_bus` | `benchmarks/bench_event_bus.cpp` | `active` |
97+
| `event_bus_publish_pop_1000_single_thread` | `event_bus` | `benchmarks/bench_event_bus.cpp` | `active` |
98+
| `feature_write_batch_10k_vectors` | `feature_store` | `benchmarks/bench_feature_store.cpp` | `active` |
99+
| `feature_get_latest_1000_calls` | `feature_store` | `benchmarks/bench_feature_store.cpp` | `active` |
100+
| `feature_as_of_mid_range_1000_calls` | `feature_store` | `benchmarks/bench_feature_store.cpp` | `active` |
101+
| `feature_as_of_batch_100_entities` | `feature_store` | `benchmarks/bench_feature_store.cpp` | `active` |
102+
| `feature_history_100_records` | `feature_store` | `benchmarks/bench_feature_store.cpp` | `active` |
103+
| `read_typed_price_column_double_50k` | `read` | `benchmarks/bench_read.cpp` | `active` |
104+
| `read_all_string_conversion_50k` | `read` | `benchmarks/bench_read.cpp` | `active` |
105+
| `read_projection_price_qty_50k` | `read` | `benchmarks/bench_read.cpp` | `active` |
106+
| `read_typed_ts_column_int64_50k` | `read` | `benchmarks/bench_read.cpp` | `active` |
107+
| `read_open_and_num_rows_footer_50k` | `read` | `benchmarks/bench_read.cpp` | `active` |
108+
| `wal_writer_append_32b_single` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
109+
| `wal_writer_append_256b_single` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
110+
| `wal_writer_append_1000_batch` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
111+
| `wal_writer_append_flush_no_fsync` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
112+
| `wal_manager_append_32b_single` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
113+
| `wal_reader_read_all_10k_records` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
114+
| `wal_mmap_append_32b_single` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
115+
| `wal_mmap_append_256b_single` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
116+
| `wal_mmap_append_1000_batch` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
117+
| `wal_mmap_append_flush_no_msync` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
118+
| `wal_compare_fwrite_append_32b` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
119+
| `wal_compare_mmap_append_32b` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
120+
| `wal_three_way_writer_append_32b` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
121+
| `wal_three_way_manager_append_32b` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
122+
| `wal_three_way_mmap_append_32b` | `wal` | `benchmarks/bench_wal.cpp` | `active` |
123+
| `write_int64_10k_uncompressed` | `write` | `benchmarks/bench_write.cpp` | `active` |
124+
| `write_double_10k_bss_uncompressed` | `write` | `benchmarks/bench_write.cpp` | `active` |
125+
| `write_mixed5_10k_uncompressed` | `write` | `benchmarks/bench_write.cpp` | `active` |
126+
| `write_int64_double_100k_10_row_groups` | `write` | `benchmarks/bench_write.cpp` | `active` |
127+
| `write_string_10k_dict_uncompressed` | `write` | `benchmarks/bench_write.cpp` | `active` |

0 commit comments

Comments
 (0)