Skip to content

perf: parse WebSocket ingress with sonic-rs - #55

Merged
Praying merged 4 commits into
Praying:mainfrom
charles-achilefu:agent/sonic-ws-ingress
Oct 5, 2026
Merged

Praying merged 4 commits into
Praying:mainfrom
charles-achilefu:agent/sonic-ws-ingress

Conversation

@charles-achilefu

@charles-achilefu charles-achilefu commented Jul 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • add sonic-rs as a workspace dependency for ccxt-core
  • parse text and binary WebSocket ingress directly from bytes with sonic_rs::from_slice
  • preserve WsClient's public serde_json::Value receive type
  • enforce the prior serde_json-compatible 127-level nesting ceiling before invoking sonic-rs
  • count rejected ingress through ingress_parse_errors() and emit rate-limited, payload-free warnings
  • cover malformed JSON, invalid UTF-8, excessive nesting, quoted bracket characters, and real text/binary WebSocket frames

Why

WebSocket ingress is a high-frequency JSON parsing path. Using sonic-rs moves decoding to its SIMD-aware parser, and binary frames avoid the previous intermediate UTF-8 String allocation.

Hardening found that sufficiently deep input could exhaust the runtime thread stack before sonic-rs returned its recursion error. The preflight nesting guard preserves the effective limit of the previous serde_json parser and rejects that input before recursive deserialization.

Developer impact

The receive type and downstream exchange parsers remain unchanged. Operators gain a resettable parse-error counter and bounded diagnostics that do not log exchange payload contents.

GitNexus MCP assessment

GitNexus indexed commit e5288aa on this branch and mapped the parser through start_message_loop to connect/reconnect coordination. Its pre-change impact report rated the parser change LOW risk (2 direct dependents, 5 symbols total, one module). Change detection rated the hardening diff LOW risk with no affected indexed execution process.

Validation

  • cargo fmt --all -- --check
  • git diff --check
  • cargo test -p ccxt-core --lib ws_client::tests --all-features -- --test-threads=1 (14 passed)
  • cargo test -p ccxt-core --test ws_property_tests --all-features (10 passed)
  • cargo check --workspace --all-features
  • cargo clippy -p ccxt-core --lib --all-features (changed file clean; 9 pre-existing warnings elsewhere)

The existing reconnect integration suite has 7 passing tests and one unrelated baseline mismatch: WsConfig::default().pong_timeout is 30 seconds while the test expects 90 seconds. The full core suite also contains the pre-existing circuit_breaker::tests::test_circuit_breaker_with_events indefinite wait noted earlier.

@Praying
Praying marked this pull request as ready for review July 19, 2026 06:13

@Praying Praying left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the ingress parser change against the PR description and exercised the workspace, bin, bench, and targeted WebSocket test targets. The core byte-based parsing path is sound, but the issues below should be addressed before merging, especially the payload disclosure and production dependency boundary.

let count = counter.fetch_add(1, Ordering::Relaxed) + 1;
if count % 100 == 1 {
warn!(
error = %parse_error,

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep parser diagnostics payload-free

sonic_rs::Error is not safe to format here: its Display implementation includes a fragment of the input around the error position (roughly 16 bytes). As a result, malformed private/account WebSocket messages can leak payload content into logs, contrary to the payload-free diagnostics guarantee in the PR. Please log only sanitized metadata such as the error category/line/column, frame type, payload size, and counter.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Comment thread Cargo.toml
sonic-rs = { workspace = true }

# Terminal benchmark UI
ratatui = { workspace = true }

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep benchmark-only dependencies out of normal builds

ratatui, crossterm, futures-util, tokio-tungstenite, and the root sonic-rs dependency are used by the benchmark binary, but they are added as unconditional root dependencies. Because src/bin/binance_ws_bench.rs is auto-discovered, ordinary ccxt-rust builds and downstream consumers now resolve and compile the terminal UI stack. Please isolate this target in a benchmark/example crate, or make the dependencies optional and gate the binary with required-features.

self.last = value;
self.total += u128::from(value);
self.count += 1;
if self.values.len() == SAMPLE_CAPACITY {

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Use one sampling window for the displayed statistics

When the deque reaches 4096 samples, the oldest value is removed from values, but it is never subtracted from total and count remains cumulative. Therefore mean covers the full process lifetime while p50/p95/p99 cover only the rolling 4096 samples, despite the UI labeling the table as rolling. Please either maintain a rolling sum/count or label the mean as cumulative.

.collect()
}

// Byte-for-byte equivalent to WsClient::validate_ingress_json_depth.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Benchmark the production depth guard instead of a copy

This helper duplicates WsClient::validate_ingress_json_depth byte-for-byte. A later production fix can leave the benchmark measuring stale behavior without any failure. Please expose the production helper to the benchmark through an appropriate internal/test API, or add a shared helper so the measured code cannot drift.

@Praying
Praying merged commit d621f02 into Praying:main Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants