fix: align HTTP projection with binary protocol (nested unknowns + ID format)

[colombod]

Overview

This PR fixes two related mismatches between the cxdb HTTP API and the binary protocol, both causing data loss or type inconsistency when clients read data over HTTP that was written via the binary protocol.

---

Fix 1: Propagate include_unknown into nested type projections

Problem

The HTTP projection engine silently drops msgpack tags that are not listed in the registry descriptor for nested types (AssistantTurn, ToolCallItem, etc.). Extension fields written by newer clients — such as Amplifier adding event_blobs (tag 11 on AssistantTurn) or child_context_id (tag 12 on ToolCallItem) — survive perfectly through the binary protocol but are stripped by the HTTP API.

Concretely:

• Binary write → binary read-back: All tags preserved (the server stores opaque msgpack blobs faithfully)
• Binary write → HTTP read-back: Unknown tags on nested objects are dropped, even with include_unknown=1
• include_unknown=1: The unknown: {} dict is empty for nested objects — the server does not cascade include_unknown into nested type projections

Root Cause

Asymmetric handling of unknown tags between top-level and nested projection:

• project_msgpack (top-level ConversationItem): correctly collected unknown tags into an unknown map when include_unknown=true
• render_type_ref (nested objects like AssistantTurn, ToolCallItem): only iterated over fields declared in the descriptor and discarded everything else, completely ignoring the include_unknown flag

Fix

render_type_ref in server/src/projection/mod.rs now:

1. Collects all msgpack map entries whose tag keys are not in the descriptor's field list
2. When include_unknown is true and there are unknown entries, adds them to an _unknown key on the nested JSON object
3. Uses _unknown (with leading underscore) to distinguish from the top-level unknown key, avoiding naming collisions
4. Omits _unknown entirely when there are no unknown tags or when the option is disabled — full backward compatibility

Tests

Three new tests in server/tests/registry_projection.rs:

Test	Verifies
test_include_unknown_nested_ref	Unknown tags surface in nested ref objects via _unknown
test_include_unknown_nested_array_ref	Unknown tags surface in array ref items via _unknown
test_include_unknown_false_nested	_unknown is absent when include_unknown is false

---

Fix 2: Return u64 IDs as JSON numbers across all HTTP endpoints

Problem

Every HTTP endpoint serialized u64 IDs (context_id, turn_id, parent_turn_id, head_turn_id, session_id, etc.) as JSON strings via explicit .to_string() calls. The binary protocol returns native u64 integers, creating a type mismatch that required every HTTP client to implement custom string-to-integer deserialization (the SseUint64 pattern in both Rust and Go clients).

Fix

• Adds a format_id(id, format) helper that returns either a JSON string or a JSON number depending on the requested U64Format
• Applies it to all endpoints: create, fork, append, search, children, provenance, turns, filesystem snapshots, sessions, and the context_to_json helper
• The GET /v1/contexts/{id}/turns endpoint already accepts ?u64_format=string|number and now applies it consistently to both envelope and payload fields
• All other endpoints default to U64Format::Number, matching the binary protocol

Backward Compatibility

Both Rust and Go clients already handle numeric IDs via their SseUint64 deserializers (which accept both strings and numbers), so this is backward-compatible at the client level. Clients that need the old string behaviour on the turns endpoint can pass ?u64_format=string.

Note on ?u64_format=string backward-compatibility escape hatch

The turns endpoint retains a ?u64_format=string query parameter that lets callers opt back into the old string-serialized IDs. This was kept as a safety net during the transition, but it may be unnecessary — both the Rust and Go clients already deserialize numeric IDs natively, and the SseUint64 adapters handle either format.

If the team prefers, the format_id helper and the u64_format parameter can be removed entirely in favour of always returning JsonValue::Number(id.into()). That would simplify the code further — happy to do that here if preferred.

---

Test plan

• All existing tests pass (cargo test)
• New tests verify nested _unknown propagation for ref and array-ref fields
• New test verifies _unknown is absent when include_unknown=false
• Backward compatibility: no _unknown key appears when there are no unknown tags
• Backward compatibility: SseUint64 deserializers in both clients accept numbers natively

🤖 Generated with Amplifier