PipSync Signal Bench

A small, synthetic, multilingual benchmark for parsers that normalize trading-signal text.

The public dataset contains fabricated English, German, Spanish, and Portuguese messages. It contains no customer messages, account details, broker data, private prompts, or production telemetry.

What it measures

• Actionable vs. non-actionable classification
• Exact instrument and direction extraction
• Numeric entry, stop-loss, and take-profit fields
• Complete-case and per-field accuracy
• Missing and unknown prediction handling

Per-field denominators include every actionable reference case. Missing signals and actionability misclassifications therefore count as incorrect fields instead of disappearing from the metric.

The benchmark is not proof that a parser is safe to trade. A production system must still enforce ambiguity, authorization, risk, idempotency, and broker-state checks server-side.

Quickstart

PYTHONPATH=src python -m pipsync_signal_bench.cli validate
PYTHONPATH=src python -m pipsync_signal_bench.cli evaluate \
  --predictions examples/perfect-predictions.jsonl

Predictions use one JSON object per line:

{"id":"en-basic-buy","actionable":true,"signal":{"instrument":"EURUSD","direction":"BUY","entryPrice":"1.0850","stopLoss":"1.0820","takeProfits":["1.0900"]}}

Ambiguous, cancelled, analysis-only, or incomplete examples can be labelled non-actionable with "signal": null.

Use your own parser

Run your parser over src/pipsync_signal_bench/data/synthetic-signals.jsonl, preserve each case id, and write predictions in the documented format. The same canonical dataset is bundled in the wheel, so an installed pipsync-signal-bench validate works without a source checkout. The evaluator does not import or call the parser, so it can compare local, hosted, deterministic, or model-backed implementations without receiving credentials.

Development

PYTHONPATH=src python -m unittest discover -s tests -v

Continue with PipSync

This benchmark measures normalized fields; it does not authorize execution. See how the managed platform handles signal sources and review boundaries in the PipSync integration guides.

Licenses

• Evaluator code: Apache-2.0
• Synthetic dataset: CC BY 4.0; see src/pipsync_signal_bench/data/DATA_LICENSE