feat(msgpack): add magic-ID type-discriminant prefix to wire format [PoC]

[GordonYuanyc]

Summary

This is a proof-of-idea PR — a starting point for discussion, not a merge-ready change.

Adds a single leading byte (the "magic ID") to every serialised sketch binary — both the portable cross-language format and the internal Rust-only format — so any byte blob is self-describing. Analogous to Prometheus magic bytes for gauge/histogram types.

Wire format before:

[ <msgpack payload> ]

Wire format after:

[ magic_id: u8 | <msgpack payload> ]

Magic-ID table

Portable (cross-language, Go reads these):

ID	Type
0x01	HllSketch
0x02	CountMinSketch
0x03	CountMinSketchWithHeap
0x04	CountSketch
0x05	DdSketch
0x06	KllSketch
0x07	HydraKllSketch
0x08	SetAggregator
0x09	DeltaResult

Native (Rust-internal, serialize_to_bytes / deserialize_from_bytes):

ID	Type
0x81	CountMin<_, _>
0x82	Count<_, _, _> (CountSketch)
0x83	CountL2HH / CMSHeap
0x84	HyperLogLogImpl<_, _, _>
0x85	HyperLogLogHIPImpl<_>
0x86	DDSketch (native map-format)
0x87	KLL<T>
0x88	KLLDynamic<T>
0x89	KMV

What changed

• src/message_pack_format/magic_ids.rs — all IDs in one place (portable 0x01–0x09, native 0x81–0x89)
• src/message_pack_format/error.rs — new Error::BadMagicId { expected, got } variant
• All 9 MessagePackCodec impls in message_pack_format/portable/ updated
• All serialize_to_bytes / deserialize_from_bytes methods on sketch types in src/sketches/ updated
• The native MessagePackCodec shims in message_pack_format/native/ delegate to the sketch methods, so they pick up magic IDs automatically without any further changes
• The portable KllSketch and HydraKllSketch formats embed raw KLL::serialize_to_bytes() bytes; those call sites pick up the magic byte automatically and continue to round-trip correctly

API

Unchanged — callers still call to_msgpack() / from_msgpack() or serialize_to_bytes() / deserialize_from_bytes(). All round-trips pass.

Open questions / follow-ups

• Should the delta formats (compute_delta / apply_delta_bytes) get their own IDs or share the base sketch ID?
• The Go mirror PR is sketchlib-go#68 — the two must land together once IDs are finalised.
• Backward-compatibility strategy for bytes serialised without the prefix (rolling deploy, version-detection shim, etc.).

Test plan

• cargo test — all 426 tests pass
• Round-trip tests for every sketch type in tests/msgpack_compat.rs
• BadMagicId / Uncategorized error surfaces on wrong-type or truncated input
• Portable KLL / HydraKLL round-trips verified (they embed native KLL bytes)

🤖 Generated with Claude Code

[milindsrivastava1997]

One thought - a 2 digit hex magic ID restricts us to 256 IDs. While this seems reasonable for the short term, it's not out of the question that the number of sketches may go beyond that. We can make it 3 digits, or if we want to ensure complete future proofness (at the cost of some complexity), the magic ID can be prepended by a number that denotes the number of digits in the magic ID. Are there any references in other frameworks as to how this is handled?

[GordonYuanyc]

I agree that a single-byte sketch ID gives us an unnecessary 256-ID ceiling.

Prometheus and OTAP both use lightweight metadata before the raw binary payload so the receiver can interpret the following bytes from the bytes themselves. Prometheus stores a chunk encoding before each chunk payload; OTAP wraps raw Arrow IPC bytes with schema_id/type metadata.

For us, a thin wrapper around the existing MessagePack payload should be enough:

[ "ASK1" | version:u8 | kind_id_len:u8 | kind_id:bytes | msgpack_payload ]

"ASK1" identifies this as an ASAP sketch binary. version is for future wrapper-layout changes. kind_id_len + kind_id avoids the 256-ID limit while keeping small IDs compact. kind_id should be canonical unsigned big-endian with no leading zero bytes.

[GordonYuanyc]

With this wrapper, the serialized bytes are self-describing enough for the deserializer to determine which sketch decoder to use.

[milindsrivastava1997]

Sounds good. This encoding scheme makes sense.