Skip to content

Performance benchmarks: measurement scripts and analysis docs (#272) - #281

Merged
AndreasIgel merged 11 commits into
mainfrom
feature/272-performance-documentation
Sep 12, 2026
Merged

AndreasIgel merged 11 commits into
mainfrom
feature/272-performance-documentation

Conversation

@AndreasIgel

Copy link
Copy Markdown
Collaborator

Summary

Closes #272 — adds repeatable performance measurement infrastructure and documents
benchmark results comparing Simple Builder, Simple Minimal Builder, RecordBuilder,
and Lombok.

Changes

run_full_analysis.sh (new, single entry point)

Runs the complete analysis in two parts:

  1. Cross-framework comparison — all 4 builder types measured with wall-time
    only (--no-tracking) for a fair comparison, since RecordBuilder and Lombok
    provide no processor-level metrics.
  2. Formatting-mode analysis — simple-builder run with each formatting mode
    (jdt, lightweight, none) with JSON tracking enabled for processor-level
    phase breakdowns, followed by an automatic comparison.

Portable across platforms: bash (not zsh), no macOS-only constructs — caffeinate
is guarded by command -v, renice failure is tolerated, and only
POSIX-compatible date usage remains. RUNS=5 ./run_full_analysis.sh overrides
the default of 10 runs.

run_performance_measurement.py

  • New --formatting-mode <jdt|lightweight|none> option, propagated via
    -Dsimplebuilder.formattingMode → -Asimplebuilder.formattingMode.
  • New pre-flight check: verifies generated sources use the annotation matching
    --builder-type, failing early with a clear message instead of misleading
    compile errors.
  • Bug fix: the tracking flag was passed to run_one as the
    is_simple_builders parameter, so --formatting-mode (and tracking args) were
    silently dropped whenever --no-tracking was used — every "none"/"lightweight"
    run actually produced JDT output. Verified that all three modes now generate
    visibly different files.

run_full_comparison.py

  • New --label-suffix option to distinguish measurement campaigns
    (e.g. sb-10runs-stability).

Documentation: PERFORMANCE_ANALYSIS.md

  • Explains the measurement methodology: sources generated from the JSON catalog,
    repeated mvn clean compile, wall/compiler time aggregation, optional JSON
    processor metrics, wall-time-only comparison for external processors.
  • Cross-framework results (10 runs): Simple Builder 56.2s, Simple Minimal
    Builder 23.2s, RecordBuilder 7.0s (295 builders), Lombok 7.2s — including the
    caveat that RecordBuilder generates far fewer builders because the test data
    contains fewer records.
  • Formatting-mode results (10 runs, tracked): JDT formatting costs ~17.5s
    (~16 ms/builder, ~33% of wall time), while lightweight (~0.14s) and none
    (~0.03s) are effectively free.
  • README.md links to the benchmark results.

Verification

  • RUNS=1 ./run_full_analysis.sh executed end-to-end: all 7 report directories
    produced, final comparison printed.
  • Generated builders diff-verified per mode: NONE vs JDT = 1561 diff lines,
    NONE vs LIGHTWEIGHT = 1410, LIGHTWEIGHT vs JDT = 520.
  • bash -n syntax check passes.

@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Sep 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@AndreasIgel
AndreasIgel merged commit 7f75c87 into main Sep 12, 2026
8 checks passed
@AndreasIgel
AndreasIgel deleted the feature/272-performance-documentation branch September 12, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Having performance test and comparing this with other processors, adding documentation on it

1 participant