Skip to content

feat(report): add batch benchmarking, plot generation, and reproducible Docker environment - #6

Open
cherninkiy wants to merge 7 commits into
mainfrom
dev
Open

feat(report): add batch benchmarking, plot generation, and reproducible Docker environment#6
cherninkiy wants to merge 7 commits into
mainfrom
dev

Conversation

@cherninkiy

Copy link
Copy Markdown
Owner

Summary

This PR introduces several major improvements to the benchmarking workflow, reporting, and development environment:

  • Batch benchmarking – a new bench_batch.c and batch mode in scripts/benchmark.sh to measure performance under batch processing.
  • Plot generation & reportsscripts/plot_results.py aggregates sequential and batch CSV results, generates plots, and embeds them into docs/report.md. Figures are also copied to docs/figures/ for portability.
  • Docker supportDockerfile, docker_run helper, and install_deps.sh to provide a reproducible containerized build and benchmark environment with mounted results/docs.
  • Cleanup – removed all references to the obsolete iaik_merkle_tree submodule, updated third_party orchestration and git message template accordingly.

Motivation

  • Batch mode enables performance evaluation under realistic bulk‑operation scenarios.
  • Automated plots and Markdown reports make it easier to analyse and share results.
  • Docker ensures consistent dependencies across local development and CI.

Testing

# Run batch benchmark
./scripts/benchmark.sh --mode batch

# Generate plots and report
python3 scripts/plot_results.py

# Build and run inside Docker
./docker_run

Related

  • Updates README with links to the new report.
  • Portable PNG plots allow viewing without re‑running scripts.

Let me know if you'd like to adjust the tone or add any specific details.

…cript

Add bench_batch.c and update CMakeLists; enhance scripts/benchmark.sh with batch mode
Add scripts/plot_results.py to aggregate seq/batch CSV and render plots; embed figures in docs/report.md; update README; include copied PNGs in docs/figures for portability
Provide reproducible container build and wrapper to run benchmarks with mounted results/docs
Remove iaik_merkle_tree from third_party docs/comments; update git message template and third_party orchestration

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request completes the MVP for the batch PQC benchmarking project by introducing a new batch signing benchmark, a comprehensive reporting and plotting pipeline, and Docker support for reproducible runs. Key changes include the removal of the external Merkle tree submodule and the addition of a detailed final report. Feedback focuses on addressing a potential integer overflow in memory allocation, resolving a conflict between Docker build steps and .dockerignore, improving constant definitions for hash sizes, and ensuring build isolation when using Docker volumes.

Comment thread bench/bench_batch.c
Comment thread Dockerfile Outdated
Comment thread bench/bench_batch.c Outdated
Comment thread scripts/docker_run.sh
…sh size from bits constant

Check cfg->batch_size * cfg->msg_size for SIZE_MAX overflow before malloc; define BATCH_HASH_BITS and derive BATCH_HASH_SIZE.
…ule expectations

Submodules must be initialized on the host; remove git submodule commands that fail due to .dockerignore excluding .git
…nflicts

Mount named Docker volume 'batch-pqc_build' to /opt/batch-pqc/build so host build artifacts don't interfere with container builds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant