Performance benchmarks: measurement scripts and analysis docs (#272)

README.md

@@ -51,6 +51,8 @@ Simple Builders generates fluent, type-safe builders for your **existing** class
 
 Use Simple Builders when you want fluent, type-safe builders for the classes and records you already have, generated as plain readable source, with no bytecode manipulation and no IDE plugin — and no lock-in: because the builders are ordinary generated Java, you can drop the dependency at any time by copying the generated builder classes into your own sources, and they keep working.
 
+For performance benchmark results comparing Simple Builders, Simple Minimal Builder, RecordBuilder, and Lombok, see the [Performance Analysis Guide](performance-test/docs/PERFORMANCE_ANALYSIS.md#benchmark-results).
+
 ### Doing what other builders advertise — the Simple Builders way
 
 - **Required fields:** Primitive fields and fields annotated with an annotation named `NotNull` or `NonNull` are non-nullable; constructor parameters are builder inputs. `build()` enforces the required/non-null contract with `IllegalStateException` ([configuration details](#required-fields-and-null-safety)).

performance-test/docs/PERFORMANCE_ANALYSIS.md

@@ -15,6 +15,7 @@ processor to measure processing time. Three scripts work together:
 | `run_performance_measurement.py` | Run N compilations and aggregate timing results |
 | `compare_performance.py` | Compare results from multiple measurement runs side-by-side |
 | `run_full_comparison.py` | Run all frameworks end-to-end and compare (convenience) |
+| `run_full_analysis.sh` | Full analysis: cross-framework comparison plus formatting-mode analysis (recommended entry point) |
 
 ## Supported Builder Types
 
@@ -30,7 +31,19 @@ metrics. Builder types without JSON reports only measure overall wall time.
 
 ## Quick Start
 
-Run all four frameworks with N runs each, then compare:
+For the complete analysis (cross-framework comparison plus formatting-mode
+breakdown), run the top-level script:
+
+```bash
+./scripts/run_full_analysis.sh
+```
+
+It runs all four frameworks with wall-time only, then runs simple-builder with
+each formatting mode (`jdt`, `lightweight`, `none`) and JSON tracking enabled,
+and finally prints comparisons. `RUNS=5 ./scripts/run_full_analysis.sh`
+overrides the default of 10 runs per measurement.
+
+To run only the cross-framework comparison:
 
 ```bash
 python3 scripts/run_full_comparison.py --runs 10
@@ -56,6 +69,10 @@ python3 scripts/generate_classes.py --builder-type simple-builder --force
 python3 scripts/run_performance_measurement.py --runs 30 --label sb-30runs --builder-type simple-builder
 ```
 
+`run_performance_measurement.py` also accepts `--formatting-mode <jdt|lightweight|none>`
+to control source formatting for simple-builders types, and `--no-tracking` to
+skip JSON processor metrics (wall-time only).
+
 Results are written to `performance-test/performance-reports/<label>/`. Compare paths
 are relative to that directory (or use absolute paths). The parent directory
 name is used as the column label.
@@ -127,3 +144,85 @@ performance-test/
   "wallTimeOnly": true
 }
 ```
+
+## Benchmark Results
+
+The following results are from a 10-iteration run of
+[`run_full_analysis.sh`](../scripts/run_full_analysis.sh)
+(wall-time-only cross-framework part, equivalent to
+`run_full_comparison.py --runs 10 --no-tracking`).
+All frameworks were measured with wall-time only (no JSON tracking overhead) to
+ensure a fair comparison. Note that the builder counts differ: Simple Builders
+and Lombok generate one builder per class, while RecordBuilder generates fewer
+(295) because the test dataset contains fewer records than plain classes.
+
+### Wall-Time Comparison (10 runs, wall-time only)
+
+| Framework | Builders | Wall Avg (s) | Wall Min (s) | Wall Max (s) | Per Builder (ms) |
+|-----------|----------|-------------|-------------|-------------|-----------------|
+| Simple Builder (`@SimpleBuilder`) | 1079 | 56.2 | 54.1 | 61.6 | 50.2 |
+| Simple Minimal Builder (`@SimpleMinimalBuilder`) | 1079 | 23.2 | 22.8 | 23.9 | 20.3 |
+| RecordBuilder (`@RecordBuilder`) | 295 | 7.0 | 6.9 | 7.1 | 19.6 |
+| Lombok (`@Builder`) | 1077 | 7.2 | 7.0 | 7.5 | 5.6 |
+
+Key observations:
+
+- **Lombok** is fastest per builder but instruments bytecode at compile time
+  rather than generating separate source files, so the comparison is not
+  apples-to-apples.
+- **RecordBuilder** generates far fewer builders (295 vs 1079) because the test
+  dataset contains fewer records than plain classes. However, its per-builder
+  cost (~19.6 ms) is nearly identical to Simple Minimal Builder (~20.3 ms) — the
+  wall-time difference is almost entirely due to the lower builder count, not
+  per-builder efficiency.
+- **Simple Minimal Builder** is ~2.4x faster than Simple Builder. The speedup
+  comes from two factors: fewer generated methods (no collection helpers,
+  conditional logic, supplier/consumer setters, Javadoc, etc.) and
+  correspondingly less source formatting work.
+- **Simple Builder** is the slowest due to its full feature set. The per-builder
+  cost (~50 ms) is dominated by code generation and formatting.
+
+### Processor-Internal Breakdown (with JSON tracking)
+
+For deeper insight into where time is spent inside the Simple Builders
+processor, enable the processor's internal performance tracker with
+`-Asimplebuilder.performanceTracking=true`. This adds minor overhead but
+provides phase-level breakdowns in the JSON report.
+
+A 10-run measurement of Simple Builder (full features, JDT formatting, with
+tracking enabled) shows the following phase distribution:
+
+| Phase | Avg (s) | Share |
+|-------|---------|-------|
+| Config Resolution | 0.08 | <1% |
+| Builder Def Extraction | 0.76 | ~2% |
+| DTO Mapping | 0.04 | <1% |
+| Code Generation | 42.9 | ~93% |
+| **Processor Total** | **46.1** | |
+| **Wall Total** | **55.5** | |
+
+Code generation dominates at ~93% of processor time. This includes Roaster
+source construction, string serialization, and source formatting. The same run
+set compared across the three formatting modes (Simple Builder, 1079 builders)
+shows the following:
+
+| Formatting Mode | Wall Avg (s) | Processor Avg (s) | Formatting Phase (s) | vs NONE |
+|-----------------|-------------|-------------------|----------------------|---------|
+| NONE (raw Roaster) | 41.8 | 30.7 | 0.03 | — |
+| LIGHTWEIGHT | 42.7 | 31.3 | 0.14 | +2% |
+| JDT (default) | 55.5 | 46.1 | 17.6 | +33% |
+
+The Eclipse JDT formatter is expensive: ~17.5s across 1079 builders (~16 ms per
+builder), roughly a third of total wall time. The lightweight formatter, by
+contrast, is nearly free — its regex/string post-processing adds only ~0.1s.
+`none` and `lightweight` are effectively equivalent in speed, so the choice
+between them is about output readability, not performance. For
+performance-sensitive builds (e.g. large generated codebases), `lightweight` or
+`none` is recommended over `jdt`.
+
+### Running the Benchmarks
+
+To reproduce the wall-time comparison, see [Quick Start](#quick-start) above.
+For processor-internal breakdowns, run
+[`run_performance_measurement.py`](../scripts/run_performance_measurement.py)
+without `--no-tracking` to enable JSON reporting.

performance-test/scripts/run_full_analysis.sh

@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+# Run the full performance analysis for simple-builders.
+#
+# This is the single entry point referenced in PERFORMANCE_ANALYSIS.md.
+# It runs two analyses:
+#
+#   1. Cross-framework comparison (wall-time only):
+#      simple-builder, simple-minimal-builder, record-builder, lombok
+#      All measured with --no-tracking for a fair wall-time comparison.
+#
+#   2. Formatting-mode analysis (with processor metrics):
+#      simple-builder with jdt, lightweight, and none formatting modes.
+#      Tracking is enabled so JSON processor metrics are available for
+#      deeper insight into formatting overhead.
+#
+# Usage:
+#   ./performance-test/scripts/run_full_analysis.sh
+#   RUNS=5 ./performance-test/scripts/run_full_analysis.sh   # override run count (default: 10)
+#
+# Output directories:
+#   performance-reports/{sb,mb,rb,lombok}-{N}runs/
+#   performance-reports/fmt-{jdt,lightweight,none}-{N}runs/
+#
+# Expected runtime: ~30-40 minutes (depends on RUNS and hardware)
+
+set -euo pipefail
+
+RUNS="${RUNS:-10}"
+DATE_FMT='+%Y-%m-%d %H:%M:%S'
+
+START_TIME_FMT=$(date "$DATE_FMT")
+
+echo "============================================================"
+echo "  FULL PERFORMANCE ANALYSIS started: ${START_TIME_FMT}"
+echo "  Runs per measurement: ${RUNS}"
+echo "============================================================"
+echo
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+BASE_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
+cd "$BASE_DIR/.."
+
+# --- macOS performance optimizations (skipped on other platforms) ---
+# Prevent system sleep while the analysis is running
+if command -v caffeinate &>/dev/null; then
+  caffeinate -dimsu -w $$ &
+fi
+
+# Run the script and all children at elevated priority (requires privileges; ignore failure)
+renice -n -10 -p $$ 2>/dev/null || true
+
+echo "============================================================"
+echo "  JVM / BUILD CONTEXT"
+echo "============================================================"
+JAVA_VERSION=$(java -version 2>&1 | head -n 1)
+MAVEN_VERSION=$(mvn -version 2>&1 | head -n 1)
+JVM_FLAGS=$(java -XX:+PrintFlagsFinal -version 2>/dev/null)
+HEAP_MAX=$(echo "$JVM_FLAGS" | awk '/MaxHeapSize/ {printf "%.0f", $4/1024/1024; exit}')
+HEAP_INIT=$(echo "$JVM_FLAGS" | awk '/InitialHeapSize/ {printf "%.0f", $4/1024/1024; exit}')
+META_MAX=$(echo "$JVM_FLAGS" | awk '/MaxMetaspaceSize/ {
+  if ($4 == "18446744073709551615") {
+    print "unlimited"
+  } else {
+    printf "%.0fm", $4/1024/1024
+  }
+  exit
+}')
+GC=$(java -XX:+PrintCommandLineFlags -version 2>&1 | tr ' ' '\n' | grep '^-XX:+Use.*GC' | sed -e 's/-XX:+Use//' -e 's/GC$//' | head -n 1)
+echo "  Java:      $JAVA_VERSION"
+echo "  Maven:     $MAVEN_VERSION"
+echo "  Heap:      -Xms=${HEAP_INIT}m -Xmx=${HEAP_MAX}m"
+echo "  Metaspace: -XX:MaxMetaspaceSize=${META_MAX}"
+echo "  GC:        ${GC:-default}"
+echo "============================================================"
+echo
+
+# ============================================================
+# Part 1: Cross-framework comparison (wall-time only)
+# ============================================================
+echo "============================================================"
+echo "  PART 1: Cross-framework comparison (${RUNS} runs, --no-tracking)"
+echo "  Builders: simple-builder, simple-minimal-builder, record-builder, lombok"
+echo "============================================================"
+echo
+
+python3 performance-test/scripts/run_full_comparison.py \
+  --runs "$RUNS" --no-tracking
+echo
+
+# ============================================================
+# Part 2: Formatting-mode analysis (with processor metrics)
+# ============================================================
+echo "============================================================"
+echo "  PART 2: Formatting-mode analysis (${RUNS} runs, with tracking)"
+echo "  Builder: simple-builder"
+echo "  Modes:   jdt, lightweight, none"
+echo "============================================================"
+echo
+
+# Generate simple-builder sources once for all formatting-mode runs
+python3 performance-test/scripts/generate_classes.py \
+  --builder-type simple-builder --force
+echo
+
+for MODE in jdt lightweight none; do
+  echo "============================================================"
+  echo "  simple-builder, formattingMode=${MODE}, ${RUNS} runs (with tracking)"
+  echo "============================================================"
+  python3 performance-test/scripts/run_performance_measurement.py \
+    --runs "$RUNS" \
+    --label "fmt-${MODE}-${RUNS}runs" \
+    --builder-type simple-builder \
+    --formatting-mode "$MODE"
+  echo
+done
+
+echo "============================================================"
+echo "  FORMATTING-MODE COMPARISON (with processor metrics)"
+echo "============================================================"
+echo
+echo "=== FORMATTING MODES: jdt vs lightweight vs none ==="
+python3 performance-test/scripts/compare_performance.py \
+  "fmt-jdt-${RUNS}runs/summary.json" \
+  "fmt-lightweight-${RUNS}runs/summary.json" \
+  "fmt-none-${RUNS}runs/summary.json"
+echo
+
+# ============================================================
+# Summary
+# ============================================================
+echo "============================================================"
+echo "  Full performance analysis complete."
+echo "  Started:  ${START_TIME_FMT}"
+echo "  Finished: $(date "$DATE_FMT")"
+echo "  Reports:"
+echo "    Cross-framework: performance-test/performance-reports/{sb,mb,rb,lombok}-${RUNS}runs/"
+echo "    Formatting-mode: performance-test/performance-reports/fmt-{jdt,lightweight,none}-${RUNS}runs/"
+echo "============================================================"

performance-test/scripts/run_full_comparison.py

@@ -95,6 +95,14 @@ def main() -> None:
         "avoiding the overhead of the processor's internal performance tracker. "
         "This ensures a fair comparison without measurement overhead.",
     )
+    parser.add_argument(
+        "--label-suffix",
+        type=str,
+        default="",
+        help="Suffix appended to the label for each builder type "
+        "(e.g. --label-suffix stability gives 'sb-5runs-stability'). "
+        "Useful to distinguish different measurement campaigns.",
+    )
     args = parser.parse_args()
 
     num_runs = args.runs
@@ -104,6 +112,8 @@ def main() -> None:
 
     for bt in BUILDER_TYPES:
         label = f"{LABEL_PREFIX[bt]}-{num_runs}runs"
+        if args.label_suffix:
+            label += f"-{args.label_suffix}"
         print_section(f"{bt}  (label: {label})")
 
         # 1. Generate classes

performance-test/scripts/run_performance_measurement.py

@@ -133,8 +133,8 @@ def parse_compiler_time(output: str) -> Optional[float]:
     return None
 
 
-def run_one(run_index: int, profile: str, is_simple_builders: bool, report_dir: Path,
-           builder_type: str = "") -> Optional[dict]:
+def run_one(run_index: int, profile: str, use_tracking: bool, report_dir: Path,
+           builder_type: str = "", formatting_mode: str = "") -> Optional[dict]:
     """Run a single clean compile and return the parsed JSON report (or wall-time-only dict).
 
     The report file uses the run_index in its name so that retries overwrite the failed
@@ -153,11 +153,14 @@ def run_one(run_index: int, profile: str, is_simple_builders: bool, report_dir:
         "-Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS",
         "--no-transfer-progress",
     ]
-    if is_simple_builders:
+    is_simple_builders = builder_type in SIMPLE_BUILDERS_TYPES
+    if use_tracking:
         cmd.extend([
             "-Dsimplebuilder.performanceTracking=true",
             f"-Dsimplebuilder.performanceOutputFile={report_file}",
         ])
+    if is_simple_builders and formatting_mode:
+        cmd.append(f"-Dsimplebuilder.formattingMode={formatting_mode}")
 
     result = subprocess.run(
         cmd,
@@ -180,7 +183,7 @@ def run_one(run_index: int, profile: str, is_simple_builders: bool, report_dir:
             builder_count = count_annotated_sources(annotation)
     compiler_time = parse_compiler_time(result.stdout + result.stderr)
 
-    if is_simple_builders:
+    if use_tracking:
         if not report_file.exists():
             print(f"  Run {run_index}: compiled OK but no JSON report found ({elapsed:.1f}s)")
             return None
@@ -432,6 +435,15 @@ def main() -> None:
         "lombok/record-builder), avoiding the overhead of the processor's "
         "internal performance tracker.",
     )
+    parser.add_argument(
+        "--formatting-mode",
+        type=str,
+        default="",
+        choices=["", "jdt", "lightweight", "none"],
+        help="Override the formatting mode for simple-builders types via "
+        "-Asimplebuilder.formattingMode. Only affects simple-builder and "
+        "simple-minimal-builder. Default: empty (use default from profile or JDT, if not defined in profile).",
+    )
     args = parser.parse_args()
 
     num_runs = args.runs
@@ -454,6 +466,34 @@ def main() -> None:
     print(f"Report directory: {report_dir}")
     print()
 
+    # Pre-flight check: verify generated sources use the expected annotation.
+    # If generate_classes.py was run for a different builder type, the Maven
+    # profile won't match and compilation will fail or produce wrong results.
+    expected_annotation = BUILDER_TYPE_ANNOTATION.get(builder_type)
+    if expected_annotation:
+        src_dir = BASE_DIR / "src" / "main" / "java"
+        if src_dir.exists():
+            actual = None
+            for f in src_dir.rglob("*.java"):
+                try:
+                    text = f.read_text()
+                except OSError:
+                    continue
+                for ann in BUILDER_TYPE_ANNOTATION.values():
+                    if ann in text:
+                        actual = ann
+                        break
+                if actual is not None:
+                    break
+            if actual is not None and actual != expected_annotation:
+                print(f"ERROR: Generated sources use {actual} but --builder-type is "
+                      f"{builder_type} (expects {expected_annotation}).")
+                print(f"Run generate_classes.py first: "
+                      f"python3 scripts/generate_classes.py --builder-type {builder_type} --force")
+                sys.exit(1)
+
+
+
     max_retries = args.max_retries
     runs: list[dict] = []
     total_attempts = 0
@@ -466,7 +506,8 @@ def main() -> None:
             if attempt > 1:
                 print(f"  Run {i}: retry {attempt - 1}/{max_retries}...", flush=True)
             run_start = time.time()
-            data = run_one(i, profile, use_tracking, report_dir, builder_type)
+            data = run_one(i, profile, use_tracking, report_dir, builder_type,
+                           args.formatting_mode)
             if data is not None:
                 if "_wallTimeSeconds" not in data:
                     data["_wallTimeSeconds"] = time.time() - run_start