Skip to content

chore: cli benchmark#2560

Open
timofei-iatsenko wants to merge 21 commits into
mainfrom
feat/bench
Open

chore: cli benchmark#2560
timofei-iatsenko wants to merge 21 commits into
mainfrom
feat/bench

Conversation

@timofei-iatsenko

@timofei-iatsenko timofei-iatsenko commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR brings a CLI benchmark

Read readme.md from this PR to know more about it. Also CLAUDE.md has information about how the fixtures are generated.


══════════════════════════════════════════════════════════════
  Lingui Benchmark — Preset: medium
  1000 files · 10000 messages · 5 locales
  Node v24.13.1 · darwin arm64
  Apple M3 Max (16 cores)
══════════════════════════════════════════════════════════════

Running: Extract (full pipeline)...

  Extract (scan → extract → merge → write):
    Babel · 1 worker   █████████████████████████  2.75s  364 files/s
    Babel · 2 workers  ██████████████████░░░░░░░  1.95s  514 files/s
    SWC · 1 worker     █████████████░░░░░░░░░░░░  1.44s  694 files/s
    SWC · 2 workers    ██████████░░░░░░░░░░░░░░░  1.06s  942 files/s ⚡

  Summary:
    SWC · 2 workers is 2.6x faster than Babel · 1 worker
    SWC · 2 workers is 1.8x faster than Babel · 2 workers
    SWC · 2 workers is 1.4x faster than SWC · 1 worker

Running: Extract Template (no merge)...

  Extract Template (scan → extract → write, no merge):
    Babel · 1 worker   █████████████████████████  2.49s  401 files/s ±6.8%
    Babel · 2 workers  █████████████████░░░░░░░░  1.72s  582 files/s ±6.9%
    SWC · 1 worker     ███████████░░░░░░░░░░░░░░  1.09s  921 files/s ±5.8%
    SWC · 2 workers    ████████░░░░░░░░░░░░░░░░░  812ms  1232 files/s ⚡

  Summary:
    SWC · 2 workers is 3.1x faster than Babel · 1 worker
    SWC · 2 workers is 2.1x faster than Babel · 2 workers
    SWC · 2 workers is 1.3x faster than SWC · 1 worker

Running: Compile...

  Compile (read PO → compile ICU → write JS):
    1 worker   █████████████████████████  596ms  83.9k msgs/s
    2 workers  █████████████████████████  588ms  85.0k msgs/s ⚡

  Summary:
    2 workers is 1.0x faster than 1 worker

Running: Pure Macro Transform...

  Pure Macro Transform (no I/O, no catalogs):
    Babel             █████████████████████████  1.64s  611 files/s ±13.0%
    Babel - no macro  ██████████████░░░░░░░░░░░  932ms  1073 files/s ±5.9%
    SWC               ███████████░░░░░░░░░░░░░░  695ms  1438 files/s ±9.3%
    SWC - no macro    █░░░░░░░░░░░░░░░░░░░░░░░░   28ms  36.0k files/s ⚡

  Summary:
    SWC - no macro is 58.9x faster than Babel
    SWC - no macro is 33.6x faster than Babel - no macro
    SWC - no macro is 25.0x faster than SWC

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Examples update

Fixes # (issue)

Checklist

  • I have read the CONTRIBUTING and CODE_OF_CONDUCT docs
  • I have added tests that prove my fix is effective or that my feature works
  • I have added the necessary documentation (if appropriate)

@vercel

vercel Bot commented Jun 1, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
js-lingui Ready Ready Preview Jun 10, 2026 11:19am

Request Review

@github-actions

github-actions Bot commented Jun 1, 2026

Copy link
Copy Markdown

size-limit report 📦

Path Size
packages/core/dist/index.mjs 1.69 KB (0%)
packages/detect-locale/dist/index.mjs 633 B (0%)
packages/react/dist/index.mjs 1.22 KB (0%)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new @lingui/benchmarks workspace providing a CLI-focused benchmark suite (fixture generation + scenario runners) intended to measure extraction/compile/macro-transform performance across different configurations.

Changes:

  • Adds a new benchmarks/ workspace with fixture generation, benchmark scenarios, and console/JSON reporting.
  • Updates the root workspace configuration and lockfile to include benchmark dependencies (SWC, tinybench, lingui-swc, etc.).
  • Documents benchmark usage and fixture generation in benchmarks/README.md and benchmarks/CLAUDE.md.

Reviewed changes

Copilot reviewed 23 out of 24 changed files in this pull request and generated 22 comments.

Show a summary per file
File Description
yarn.lock Adds workspace + benchmark dependency resolutions (SWC, tinybench, lingui-swc, etc.).
package.json Registers benchmarks/ as a workspace.
benchmarks/package.json Defines @lingui/benchmarks scripts and dependencies.
benchmarks/tsconfig.json Adds a TS config for the benchmarks workspace.
benchmarks/README.md Documents running the benchmark suite and scenarios.
benchmarks/CLAUDE.md Developer guide for architecture/fixtures/scenarios.
benchmarks/.gitignore Ignores generated fixtures/results and SWC artifacts.
benchmarks/src/presets.ts Defines benchmark presets (small/medium/large).
benchmarks/src/generate-fixtures.ts Generates synthetic fixtures and catalogs for benchmarking.
benchmarks/src/run-benchmarks.ts Runs benchmark scenarios and writes aggregated results.
benchmarks/src/reporters/console-reporter.ts Renders console output (bars, throughput, summary).
benchmarks/src/reporters/json-reporter.ts Writes machine-readable benchmark results to JSON.
benchmarks/src/scenarios/extract.bench.ts Benchmarks lingui extract across Babel/SWC and worker counts.
benchmarks/src/scenarios/extract-template.bench.ts Benchmarks lingui extract-template across Babel/SWC and worker counts.
benchmarks/src/scenarios/compile.bench.ts Benchmarks lingui compile across worker counts.
benchmarks/src/scenarios/macro-transform.bench.ts Benchmarks pure Babel vs SWC macro transforms (no CLI).
benchmarks/src/utils/run-cli.ts Runs Lingui CLI in a subprocess with error surfacing.
benchmarks/src/utils/config-builder.ts Writes temporary Lingui config files for fixtures.
benchmarks/src/utils/silence.ts Utility to silence console/stdout during iterations.
benchmarks/src/utils/deterministic-random.ts Seeded RNG utilities (currently unused).
benchmarks/src/generators/message-pool.ts Message pool + variability rules for fixture generation.
benchmarks/src/generators/jsx-file-generator.ts Generates TSX files using Lingui React macros.
benchmarks/src/generators/js-file-generator.ts Generates TS files using Lingui core macros.
benchmarks/src/generators/po-catalog-generator.ts Generates PO catalogs by extracting a real template then filling translations.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread benchmarks/src/utils/config-builder.ts
Comment thread benchmarks/src/utils/config-builder.ts
Comment thread benchmarks/src/generate-fixtures.ts
Comment thread benchmarks/src/run-benchmarks.ts
Comment thread benchmarks/CLAUDE.md Outdated
Comment thread benchmarks/src/run-benchmarks.ts
Comment thread benchmarks/src/generators/po-catalog-generator.ts Outdated
Comment thread benchmarks/CLAUDE.md Outdated
Comment thread benchmarks/CLAUDE.md Outdated
Comment thread benchmarks/CLAUDE.md Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Comment thread benchmarks/CLAUDE.md Outdated
Comment on lines +26 to +27
silence.ts ← Suppresses console/stdout during benchmark iterations
deterministic-random.ts ← Seeded PRNG (unused currently, available for future use)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest not shipping these unused things. We can always add them separately if needed

Comment thread benchmarks/package.json
"@lingui/message-utils": "workspace:*",
"@lingui/swc-plugin": "^6.3.0",
"@swc/core": "^1.11.0",
"lingui-swc": "^0.5.2",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not yet an official Lingui extractor, so better to hold off on using it for benchmarks until it goes live.

Comment thread benchmarks/CLAUDE.md

Catalog overlap: 90% of unique messages have pre-existing translations. The remaining 10% simulate newly added code not yet in catalogs (empty translation string).

## Adding a New Scenario

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the example imports buildConfig and silenceConsole, but neither exists in the current code. Scenarios now use writeConfigs() + runLingui()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants