feat: Add Configurable Nested KeyValue Support for ClickHouse JSON Export

[brightsparc]

Summary

This PR adds support for properly converting nested OpenTelemetry KeyValueList and ArrayValue structures to native JSON objects/arrays in the ClickHouse exporter, with a configurable depth limit and backwards-compatible default behavior.

Motivation

When exporting GenAI span attributes (e.g., gen_ai.input.messages, gen_ai.output.messages) that contain nested KeyValueList structures, the previous implementation would serialize them as empty strings or raw protobuf JSON, losing the semantic structure. This made querying these fields in ClickHouse difficult.

Before (flat mode):

{"gen_ai.input.messages": ""}

After (nested mode enabled):

{"gen_ai.input.messages": [{"role": "user", "parts": [{"type": "text", "content": "Hello"}]}]}

Changes

1. Unified String Handling with Cow<str>

Before:

pub enum JsonType<'a> {
    Str(&'a str),
    StrOwned(String),
    // ...
}

After:

pub enum JsonType<'a> {
    Str(Cow<'a, str>),
    // ...
}

impl<'a> JsonType<'a> {
    #[inline]
    pub fn str_borrowed(s: &'a str) -> Self {
        JsonType::Str(Cow::Borrowed(s))
    }

    #[inline]
    pub fn str_owned(s: String) -> Self {
        JsonType::Str(Cow::Owned(s))
    }
}

Rationale:

Aspect	Two Variants (Str + StrOwned)	Single Variant (Cow<str>)
Match arms in Serialize	2	1
Code duplication	Yes (same serialization logic twice)	No
API clarity	Exposes implementation detail	Constructor methods hide detail
Runtime cost	Identical	Identical

The Cow (Copy-on-Write) smart pointer elegantly handles both borrowed and owned strings:

• Cow::Borrowed(&str) - zero allocation, just stores pointer+length
• Cow::Owned(String) - takes ownership, same as StrOwned before

2. New JsonType::Object Variant

Added support for JSON objects (ClickHouse named tuples):

Object(Vec<(Cow<'a, str>, JsonType<'a>)>)

Serialization format follows ClickHouse rowbinary JSON spec:

object_code (0x1f) | num_entries | [key_len | key_bytes | value_type | value]...

3. Configurable Nested Conversion

pub const DEFAULT_NESTED_KV_MAX_DEPTH: usize = 10;

#[inline]
pub fn anyvalue_to_jsontype<'a>(
    value: &'a AnyValue,
    nested_kv_max_depth: Option<usize>
) -> JsonType<'a> {
    match nested_kv_max_depth {
        None | Some(0) => anyvalue_to_jsontype_flat(value),
        Some(max_depth) => anyvalue_to_jsontype_nested(value, 0, max_depth),
    }
}

Configuration	Behavior	Use Case
None	Flat mode (default)	Backwards compatibility
Some(0)	Flat mode	Explicit flat
Some(n)	Nested up to depth n	GenAI attributes

4. Separate Code Paths for Performance

Rather than adding depth checks to every recursive call in the common case, we maintain two separate implementations:

Flat mode (anyvalue_to_jsontype_flat):

• No recursion tracking
• No depth checks
• Complex nested types → JSON string fallback
• Optimized for throughput

Nested mode (anyvalue_to_jsontype_nested):

• Tracks current depth
• Checks against max_depth
• KvlistValue → JsonType::Object
• ArrayValue → JsonType::Array (recursive)
• Falls back to JSON string at max depth

Performance Analysis

Why Separate Code Paths?

A single function with if depth_enabled { check_depth() } on every call would add:

• Branch prediction overhead on hot path
• Unnecessary parameter passing
• Code bloat in the common case

By separating, the flat path compiles to tight, branch-free code.

Memory Layout

JsonType::Str(Cow::Borrowed(&str)):
  [discriminant: 8B][tag: 8B][ptr: 8B][len: 8B] = 32 bytes

JsonType::Str(Cow::Owned(String)):
  [discriminant: 8B][tag: 8B][ptr: 8B][len: 8B][cap: 8B] = 40 bytes

The discriminant for JsonType variant + Cow tag fits in one cache line. Access pattern is identical to before.

Benchmarking Expectations

Mode	vs Previous	Notes
Flat (default)	~0% change	Same code paths, single match arm may be faster
Nested	~10-20% slower	Additional allocations for Object/Array vecs

The nested mode slowdown is acceptable because:

1. It's opt-in
2. It produces queryable data (worth the cost)
3. GenAI spans are typically low-volume compared to regular traces

CLI Configuration

New flag added to configure nested KV conversion:

# Enable nested KV with max depth of 10 (recommended for GenAI attributes)
rotel start \
  --clickhouse-exporter-endpoint http://localhost:8123 \
  --clickhouse-exporter-enable-json \
  --clickhouse-exporter-nested-kv-max-depth 10

# Or via environment variable
export ROTEL_CLICKHOUSE_EXPORTER_NESTED_KV_MAX_DEPTH=10

Flag	Env Var	Default	Description
--clickhouse-exporter-nested-kv-max-depth	ROTEL_CLICKHOUSE_EXPORTER_NESTED_KV_MAX_DEPTH	unset (flat mode)	Max depth for nested KV→Object conversion

Transformer Configuration (Internal)

// Default: flat mode (backwards compatible)
let transformer = Transformer::new(compression, use_json);

// Enable nested mode
let transformer = Transformer::new(compression, use_json)
    .with_nested_kv_max_depth(Some(10));

Wire Format

Flat Mode (unchanged)

Simple types: type_code | value_bytes
Arrays: 0x1e | 0x2b | 0x20 | count | [type_code | value]...
KvlistValue: 0x15 | 0x00  (empty string)

Nested Mode (new)

Object: 0x1f | num_keys | [key_str | value]...
  where key_str = 0x00 | len | bytes (no type code for keys)
Arrays: same as flat, but elements can be Objects
KvlistValue: becomes Object

Backwards Compatibility

• Default behavior unchanged: None = flat mode
• Wire format unchanged for flat mode
• All 514 existing tests pass
• New tests added for nested mode specifically

Test Plan

• test_anyvalue_arrayvalue_flat_mode - KvlistValue → JSON string, nested Array → JSON string
• test_anyvalue_arrayvalue_nested_mode - KvlistValue → Object, nested Array → Array
• All serialization byte-pattern tests pass
• cargo test - 514 passed, 0 failed

Future Work

1. Default change: Consider nested mode as default in v2.0
2. Benchmarks: Add criterion benchmarks to track performance
3. Compression: Investigate if nested objects compress better than JSON strings

[brightsparc]

After ingesting some data, I can indeed see the JSON values as a deep structure, but I wasn't able to find a elegant way to say get the last item in an array of input messages eg see below:

select SpanAttributes.gen_ai.input.messages from otel_traces where SpanAttributes.gen_ai.input.messages is not null limit 1 format vertical;

SELECT SpanAttributes.gen_ai.input.messages
FROM otel_traces
WHERE SpanAttributes.gen_ai.input.messages IS NOT NULL
LIMIT 1
FORMAT vertical

Query id: 8132869d-afc4-47bd-9d89-e7c883dd5141

Row 1:
──────
SpanAttribut⋯ut.messages: [{"role": "system", "parts": [{"type": "text", "content": "You are a math assistant. Use the calculator tools to perform calculations. Always use the tools - never calculate in your head. Perform operations step by step, using the result of each step in the next."}]}, {"role": "user", "parts": [{"type": "text", "content": "Calculate: add 5 + 3, then divide the result by 4, then subtract 10 from that."}]}]

1 row in set. Elapsed: 0.009 sec.

however there didn't

EXPLAIN PIPELINE
SELECT JSONExtractRaw(CAST(SpanAttributes.`gen_ai.input.messages`, 'String'), 1)
FROM otel_traces
WHERE SpanAttributes.`gen_ai.input.messages` IS NOT NULL
LIMIT 1;

Query id: fa9e85e5-2bbd-442c-9d96-449d82acacce

   ┌─explain────────────────────────────────────────────────────────────────┐
1. │ (Expression)                                                           │
2. │ ExpressionTransform                                                    │
3. │   (Limit)                                                              │
4. │   Limit                                                                │
5. │     (Filter)                                                           │
6. │     FilterTransform                                                    │
7. │       (ReadFromMergeTree)                                              │
8. │       MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder) 0 → 1 │
   └────────────────────────────────────────────────────────────────────────┘

8 rows in set. Elapsed: 0.014 sec.

f476477eba9c :) EXPLAIN PIPELINE                                                                                            
  SELECT JSONExtractRaw(                                                                                      
      dynamicElement(SpanAttributes.`gen_ai.input.messages`, 'String'), 1                                      
  )                                                                                                            
  FROM otel_traces                                                                                            
  WHERE SpanAttributes.`gen_ai.input.messages` IS NOT NULL                                                    
  LIMIT 1;  

Query id: 6109fc69-a14f-4144-8fe9-ca4fcc5329f2

   ┌─explain────────────────────────────────────────────────────────────────┐
1. │ (Expression)                                                           │
2. │ ExpressionTransform                                                    │
3. │   (Limit)                                                              │
4. │   Limit                                                                │
5. │     (Filter)                                                           │
6. │     FilterTransform                                                    │
7. │       (ReadFromMergeTree)                                              │
8. │       MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder) 0 → 1 │
   └────────────────────────────────────────────────────────────────────────┘

8 rows in set. Elapsed: 0.006 sec. 

Could be worth following up with Clickhouse folks on the best way to implement this.

[mheffner]

Hey @brightsparc, as we discussed I support increasing the depth we can support here.

So a few things here as I start to look at this:

1. We should remove the Dockerfile/Python changes to keep this PR limited
2. I think we can combine some of the implementations a bit to DRY this out, I started a branch for this here: https://github.com/streamfold/rotel/compare/nested-ch-attrs?expand=1
3. When I test with this and the otel-loadgen genai branch (feat: Convert GenAI Attributes to Native OTel Types otel-loadgen#5), I'm seeing export errors when JSON is enabled:

2026-01-30T23:15:06.912204Z  INFO Exporting failed, will retry again after delay. attempt=3 delay=23.415s response=HTTPResponse{Status=501 Not Implemented, Body=DbException(48, "Serialization is not implemented for type Nothing: While executing WaitForAsyncInsert. ", "26.1.1.912"), Metadata=false}
2026-01-30T23:15:06.912296Z  INFO Exporting failed, will retry again after delay. attempt=3 delay=8.802s response=HTTPResponse{Status=501 Not Implemented, Body=DbException(48, "Serialization is not implemented for type Nothing: While executing WaitForAsyncInsert. ", "26.1.1.912"), Metadata=false}

Are you seeing any error messages from the exporter?

I'll look at this some more of the next few days, but I may be busy. I'd like to support this, just need to clean it up a bit and identify where the gaps are.

[brightsparc]

@rjenkins I've pulled in those changes, and removed the docker file, so let me know if you think this is GTG

[mheffner]

@brightsparc I can look again at this Mon