EventFlowSentry

EventFlowSentry is a research software library for specifying and validating equivalent event-time streaming behavior across heterogeneous engines. It runs the same keyed-window semantics through a Python reference implementation, Apache Beam, Spark Structured Streaming, Apache Flink DataStream, and Kafka Streams, normalizes their output, and reports exact structural or numeric differences.

Version 0.1.2 is an alpha research release. It is installed from source; no PyPI publication or journal acceptance is claimed.

Why It Exists

Streaming jobs can stay healthy while producing plausible but incorrect output. Watermarks, event-time assignment, deduplication state, key selection, schema guards, and window boundaries vary across engine APIs. Record counts and process health often miss errors that redistribute unchanged totals across keys or time windows.

EventFlowSentry provides one output contract for those semantics. It is intended for:

• migration tests between streaming engines;
• CI regression tests for event-time pipelines;
• controlled mutation-testing studies;
• reproducible systems experiments; and
• teaching event-time, state, and window behavior.

Supported Execution Paths

Execution path	API used	Event-time windows	Stateful deduplication	Schema guard	Intended scope
Python reference	deterministic in-process model	yes	yes	yes	expected-output construction
Apache Beam	DirectRunner transforms/state	yes	yes	yes	local native validation
Spark	Structured Streaming	yes	yes	yes	local native validation
Flink	DataStream API	yes	yes	yes	local native validation
Kafka Streams	DSL + TopologyTestDriver	yes	yes	yes	local native validation

FlinkPlatformJob and KafkaStreamsPlatformJob are retained transport and diagnostic jobs from the earlier compatibility harness. Headline native correctness uses FlinkNativeJob and KafkaStreamsNativeJob; the two layers are not interchangeable.

Core Capabilities

• Engine-neutral PipelineSpec and PipelineVariant models.
• Three reusable topologies: windowed sum/count, keyed deduplication followed by aggregation, and schema-guarded aggregation.
• Normalized output keys containing case, topology, business key, and window.
• Label-blind exact validation of missing, unexpected, count-mismatched, and total-mismatched output cells.
• Executable output-level metamorphic relations for arrival-order invariance, duplicate idempotence, schema-rollback exclusion, event-time translation, and key-renaming equivariance.
• Deterministic workloads for payments, IoT, clickstream, auction, and market data, plus a checksum-verified NYC TLC trace adapter.
• Controlled variants for processing-time substitution, missing deduplication, schema rollback acceptance, doubled windows, and partition-key misrouting.
• Internal coarse-check ablations, Wilson intervals, exact paired McNemar tests, and Holm correction.
• CLI, JSON evidence, Docker Kafka/Flink systems checks, and experiment runners.
• Extension points for custom workload functions, specifications, adapters, and validation plugins.

The auction workload is Nexmark-style generated data, not an execution of the official Apache Beam Nexmark benchmark.

Requirements

Core library:

• Python 3.9 or newer
• kafka-python for the Kafka transport backend

Native engines:

• Apache Beam 2.75.0
• PySpark 4.2.0
• Java 17 and Maven 3.9+
• Apache Flink 1.19.1 Java dependencies
• Kafka Streams 3.9.1 Java dependencies

Docker Compose v2 is needed only for the supplementary Kafka broker and Flink capacity check.

Install

git clone https://github.com/goutamadwant/eventflowsentry.git
cd eventflowsentry
python -m venv .venv
.venv/bin/pip install --upgrade pip
.venv/bin/pip install -e ".[platforms,data,dev]"

Run the fast unit suite:

.venv/bin/python -m pytest tests -q -m "not integration"

Run native engine integration tests:

.venv/bin/python -m pytest tests/test_native_engines.py -q

Library Example

from eventflowsentry import (
    PipelineSpec,
    generate_defect_cases,
    reference_outputs,
    validate_equivalent_outputs,
)

case = generate_defect_cases(
    workloads=("payments",),
    seeds=(1,),
    n_events=300,
)[0]

expected = reference_outputs(case.case_id, case.events, case.spec)
observed = reference_outputs(
    case.case_id,
    case.events,
    case.spec,
    case.defective_variant,
)

validation = validate_equivalent_outputs(expected, observed)
assert validation.detected
for violation in validation.violations:
    print(violation.kind, violation.key)

The validator has no defect-label parameter. It receives only expected and observed normalized output maps.

Metamorphic Relation Example

from eventflowsentry import (
    generate_defect_cases,
    make_metamorphic_case,
    reference_outputs,
    validate_metamorphic_outputs,
)

case = generate_defect_cases(
    workloads=("payments",), seeds=(1,), n_events=300
)[0]
relation_case = make_metamorphic_case(case)
source = reference_outputs(
    case.case_id, relation_case.source_events, case.spec
)
followup = reference_outputs(
    case.case_id, relation_case.followup_events, case.spec
)
assert validate_metamorphic_outputs(relation_case, source, followup).passed

The relation object constructs a deterministic follow-up input and transforms the source output into the expected follow-up output. The same comparison is used for native engines. Relation violations in correct implementations are false positives; violations in matching planted variants are detections.

CLI Smoke Test

eventflowsentry backends
eventflowsentry transport-smoke --backend memory --events 25
eventflowsentry transport-smoke \
  --backend jsonl --events 25 \
  --output /tmp/eventflowsentry-smoke.jsonl

Reproduce The Correctness Study

Download and verify the public trace using data/README.md, then run:

.venv/bin/python tools/run_correctness_matrix.py \
  --workloads payments,iot,clickstream,nexmark,marketdata,nyc_tlc \
  --seeds 1,2,3,4,5,6,7,8,9,10 \
  --events 300 --bootstrap-resamples 2000 \
  --tlc-parquet data/raw/yellow_tripdata_2025-01.parquet \
  --results-dir results/native_correctness_full

The registered matrix contains five engines, six workloads, ten seeds, five semantic variants, and matched correct controls: 3,000 engine-case runs. Only claim-bearing summaries, manifests, and case rows are versioned. Large engine input/output dumps and pilot runs are intentionally excluded.

Reproduce The Metamorphic Relation Study

.venv/bin/python tools/run_metamorphic_matrix.py \
  --workloads payments,iot,clickstream,nexmark,marketdata,nyc_tlc \
  --seeds 1,2,3,4,5,6,7,8,9,10 \
  --events 300 \
  --tlc-parquet data/raw/yellow_tripdata_2025-01.parquet \
  --results-dir results/metamorphic_full

Each workload-by-defect-by-seed case evaluates the relation targeted to that defect against both the planted variant and a matched correct control. The Python path is reported as a reference self-check; Beam, Spark, Flink, and Kafka Streams are reported as native adapters. The registered relations are not claimed to detect unknown or unrelated fault classes.

Reproduce The Scaling Study

.venv/bin/python tools/run_scaling_benchmark.py \
  --sizes 10000,100000,1000000 \
  --repetitions 5 --parallelism 4 \
  --results-dir results/scaling_full

All 75 retained runs must exactly match the qualified reference output. These are local harness-capacity measurements, not framework-product rankings; the shared specification boundary is documented in EXPERIMENT_PROTOCOL.md.

Supplementary Docker Check

make docker-benchmark
make docker-down

This check measures Kafka round-trip transport, deterministic generation, and Flink REST capacity. It is not used as native Flink correctness evidence.

Verify Retained Evidence

make verify-evidence

The manifest covers native correctness and metamorphic summaries and case rows, scaling summaries, run manifests, and Docker systems evidence.

Repository Layout

src/eventflowsentry/   Python package
platforms/jvm/         Flink and Kafka Streams native jobs
tools/                 Correctness, native-engine, and scaling runners
tests/                 Unit and native integration tests
docs/                  API, benchmark, protocol, and provenance guides
data/README.md         Public trace download and checksum instructions
results/               Allowlisted claim-bearing evidence only
docker/                Kafka/Flink Compose systems check

See ARTIFACT.md for the review-oriented reproduction sequence, docs/API.md for extension points, and docs/EXPERIMENT_PROTOCOL.md for the frozen study.

Evidence Boundaries

• Mutation score covers the five registered output-manifesting variants; it is not universal production-defect accuracy.
• DirectRunner, local Spark/Flink execution, and TopologyTestDriver do not prove distributed recovery, backpressure, autoscaling, or cloud performance.
• The real NYC TLC trace improves input realism but is not transportation ground truth and is not redistributed.
• Exact validation requires a trusted expected map or reference execution.
• Metamorphic relations are targeted specifications, not a universal substitute for an expected-output oracle.
• Coarse key/count/checksum checks are ablations of the same outputs, not independent commercial competitors.

Contributing And Support

Use GitHub issues for reproducible defects and feature proposals. Include the engine, software versions, operating system, minimal event input, pipeline specification, expected output, and observed output. See CONTRIBUTING.md.

Security-sensitive reports should follow SECURITY.md rather than being posted publicly.

License And Citation

EventFlowSentry is licensed under Apache License 2.0. Cite the evaluated v0.1.2 artifact with version DOI 10.5281/zenodo.21909327. The stable software concept DOI is 10.5281/zenodo.21500845. Add the associated SoftwareX DOI only after the article is published.