Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,45 @@ shapes were checked against the `dream-pipeline` OpenHands source. Both the
array and `{items: [...]}` input forms produced exact production-equivalent
records in the compatibility fixtures.

## LangSmith adapter

The LangSmith adapter is covered with synthetic edge cases and canonical Run
trees adapted from LangSmith's published Messages-view examples. The official
OpenAI Responses and Anthropic examples are stored with root spans, UUIDs,
timestamps, `extra.metadata`, and hierarchical `dotted_order` values so their
normalized fixtures require no repairs or diagnostics. Fixtures also exercise
LangChain constructor messages, Vercel AI SDK content blocks, repeated history
snapshots, excluded internal runs, bare tool outputs, tool-result matching by ID
and tool name, run ordering, metadata, and missing-timestamp repair.

Read-only validation was also run against two user-provided LangSmith projects
without retaining raw traces or normalized content in the repository. The
sample comprised one two-trace thread with 17 total runs (13 chain, two LLM,
and two tool runs) and one standalone LLM run. This surfaced and fixed two
native variants not present in the original synthetic fixtures: an Anthropic
SSE event stream stored in string-valued `outputs.output`, and a tool call
repeated in both content blocks and the message-level `tool_calls` field.

After those repairs, the combined thread normalized to 15 records with native
tool linkage preserved; its only diagnostic was the expected configured
tool-result truncation. The standalone LLM run normalized to three records
without diagnostics.

The official examples were taken from LangSmith's
[Messages-view trace format reference](https://docs.langchain.com/langsmith/messages-view-trace-format).
All five published integration traces were also exercised directly: LangChain,
OpenAI Chat Completions, OpenAI Responses, Vercel AI SDK, and Anthropic
Messages. This differential check caught Responses item-ID versus call-ID
linkage, mixed stable/semantic message deduplication, bare tool-output objects,
and `ls_message_view_exclude` handling.

Deep Agents and Deep Agents Code have dedicated canonical Run fixtures. For
roots marked `ls_integration: deepagents` or `deepagents-code`, the adapter
normalizes the aggregate `outputs.messages` LangGraph state and ignores
redundant child spans. The fixtures cover reasoning, duplicated native tool-call
representations, tool results, final responses, model extraction, and prove
that Claude Code and Codex integration markers remain on the generic path.

## Deep Agents SDK checkpoints

The `deepagents` fixture is generated by Python
Expand Down
35 changes: 35 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ and is empty when the transcript required no recoverable cleanup.
| --- | --- | --- |
| `claude-code` | Native Claude Code JSONL | `claude-code` |
| `codex` | Native Codex rollout JSONL | `codex` |
| `langsmith` | Canonical LangSmith `Run` records as JSON array or `{ "runs": [...] }` | `langsmith` |
| `letta` | Cloud/API message array or local conversation JSONL (legacy and v3) | `letta` |
| `openhands` | JSON event array or an events-API `{ "items": [...] }` envelope | `openhands` |
| `deepagents` | User-supplied Python LangGraph `SqliteSaver` database plus `threadId` | `deepagents` |
Expand All @@ -130,6 +131,40 @@ not a supported native input. OpenHands inputs are serialized exports; when a
native store uses individual event files, assembling the event array remains
the caller's responsibility.

The LangSmith input is the canonical public `Run` span format returned by the
SDK or `/runs/query`: a JSON array of runs, or the API's `{ "runs": [...] }`
envelope, for one trace or chronological thread. Each run must include `id`,
`trace_id`, `name`, `run_type`, and `inputs`. A trace is represented by a root
run and child runs linked through `trace_id`, `parent_run_id`, and
`dotted_order`; LangSmith does not define a separate flattened-conversation
schema. Runs are ordered by `dotted_order` and `start_time`.

Run `inputs` and `outputs` remain integration-specific. The adapter follows the
LangSmith Messages-view formats for LangChain/LangGraph, OpenAI Chat
Completions and Responses, Anthropic Messages, and the Vercel AI SDK. Repeated
message-history snapshots are deduplicated, while tool runs are linked to the
earlier model tool call by call ID and then by tool name when an integration
omits the ID. Runs carrying the `ls_message_view_exclude` metadata key are
ignored, matching LangSmith's Messages-view behavior. Fetching or exporting
runs from LangSmith remains the caller's responsibility.

Traces whose root metadata sets `ls_integration` to `deepagents` or
`deepagents-code` use the LangGraph aggregate stored in the root run's
`outputs.messages`. Their redundant child LLM and tool spans are ignored, which
prevents accumulated message history from appearing more than once. No other
integration receives this aggregate-root special case; all other traces use the
generic LangSmith run decoder described above.

LangSmith's Anthropic wrapper aggregates stream events before storing outputs,
but that reducer is not exported as a public SDK utility. When canonical Run
data contains a raw Anthropic SSE string in `outputs.output`, the adapter
reconstructs its text, thinking, and tool-input deltas locally.

For a multi-turn thread, combine the runs from each member trace into one input
container before normalization. This preserves history and tool linkage that
cross trace boundaries; normalizing each trace separately can correctly report
an initial tool result as orphaned when its call occurred in the prior trace.

## Normalized records

A trajectory is an ordered array containing:
Expand Down
13 changes: 13 additions & 0 deletions fixtures/langsmith/cleanup/expected.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"records": [
{ "role": "meta", "source": "langsmith", "cwd": "/workspace", "git_branch": "feature/weather", "model": "claude-sonnet-4" },
{ "role": "user", "content": "Check Paris weather.", "timestamp": "2026-07-10T13:00:00.000Z" },
{ "role": "assistant", "content": "I'll check.", "timestamp": "2026-07-10T13:00:01.000Z" },
{ "role": "assistant", "content": null, "tool_calls": [{ "id": "call-vercel", "name": "weather", "args": "{\"city\":\"Paris\"}" }], "timestamp": "2026-07-10T13:00:01.000Z" },
{ "role": "tool", "tool_call_id": "call-vercel", "content": "{\"condition\":\"sunny\"}", "timestamp": "2026-07-10T13:00:03.000Z" },
{ "role": "assistant", "content": "It is sunny in Paris.", "timestamp": "2026-07-10T13:00:04.000Z" }
],
"diagnostics": [
{ "code": "timestamps_interpolated", "message": "Interpolated timestamps for 1 normalized records.", "count": 1 }
]
}
1 change: 1 addition & 0 deletions fixtures/langsmith/cleanup/input.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[{"trace_id":"trace-2","run_type":"llm","name":"ai.doGenerate","start_time":"2026-07-10T13:00:00Z","end_time":"2026-07-10T13:00:01Z","inputs":{"prompt":[{"role":"user","content":[{"type":"text","text":"Check Paris weather."}]}]},"outputs":{"role":"assistant","content":[{"type":"text","text":"I'll check."},{"type":"tool-call","toolCallId":"call-vercel","toolName":"weather","input":{"city":"Paris"}}]},"id":"vercel-1","extra":{"metadata":{"ls_integration":"vercel-ai-sdk","ls_model_name":"claude-sonnet-4","cwd":"/workspace","git_branch":"feature/weather"}}},{"trace_id":"trace-2","parent_run_id":"vercel-1","run_type":"tool","name":"weather","start_time":"2026-07-10T13:00:02Z","end_time":"2026-07-10T13:00:03Z","inputs":{"args":{"city":"Paris"}},"outputs":{"result":{"condition":"sunny"}},"id":"vercel-tool"},{"trace_id":"trace-2","run_type":"llm","name":"ai.doGenerate","inputs":{"prompt":[{"role":"user","content":[{"type":"text","text":"Check Paris weather."}]},{"role":"assistant","content":[{"type":"text","text":"I'll check."},{"type":"tool-call","toolCallId":"call-vercel","toolName":"weather","input":{"city":"Paris"}}]},{"role":"tool","toolCallId":"call-vercel","content":{"condition":"sunny"}}]},"outputs":{"role":"assistant","content":[{"type":"text","text":"It is sunny in Paris."}]},"id":"vercel-2","extra":{"metadata":{"ls_integration":"vercel-ai-sdk","ls_model_name":"claude-sonnet-4"}}}]
21 changes: 21 additions & 0 deletions fixtures/langsmith/deepagents-code/expected.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"records": [
{
"role": "meta",
"source": "langsmith",
"cwd": "/workspace/deepagents-code",
"model": "gpt-5.5"
},
{
"role": "user",
"content": "Reply briefly.",
"timestamp": "2026-07-11T13:00:00.000Z"
},
{
"role": "assistant",
"content": "Done.",
"timestamp": "2026-07-11T13:00:02.000Z"
}
],
"diagnostics": []
}
82 changes: 82 additions & 0 deletions fixtures/langsmith/deepagents-code/input.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
[
{
"id": "trace-deepagents-code-1",
"trace_id": "trace-deepagents-code-1",
"name": "LangGraph",
"run_type": "chain",
"start_time": "2026-07-11T13:00:00.000Z",
"end_time": "2026-07-11T13:00:02.000Z",
"dotted_order": "20260711T130000000000Ztrace-deepagents-code-1",
"inputs": {
"messages": [
{
"role": "user",
"content": "Reply briefly."
}
]
},
"outputs": {
"messages": [
{
"id": "human-code-1",
"type": "human",
"content": "Reply briefly."
},
{
"id": "ai-code-1",
"type": "ai",
"content": "Done.",
"response_metadata": {
"model_name": "gpt-5.5"
}
}
]
},
"extra": {
"metadata": {
"ls_agent_kind": "coding_agent",
"ls_integration": "deepagents-code",
"ls_trace_schema_version": "coding-agent-v1",
"cwd": "/workspace/deepagents-code"
}
}
},
{
"id": "child-code-llm-1",
"trace_id": "trace-deepagents-code-1",
"parent_run_id": "trace-deepagents-code-1",
"name": "ChatOpenAI",
"run_type": "llm",
"start_time": "2026-07-11T13:00:00.500Z",
"end_time": "2026-07-11T13:00:01.500Z",
"dotted_order": "20260711T130000000000Ztrace-deepagents-code-1.20260711T130000500000Zchild-code-llm-1",
"inputs": {
"messages": [
[
{
"role": "user",
"content": "Duplicated child input"
}
]
]
},
"outputs": {
"generations": [
[
{
"message": {
"role": "assistant",
"content": "Duplicated child output"
}
}
]
]
},
"extra": {
"metadata": {
"ls_integration": "deepagents-code",
"ls_model_name": "gpt-5.5"
}
}
}
]
49 changes: 49 additions & 0 deletions fixtures/langsmith/deepagents/expected.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
{
"records": [
{
"role": "meta",
"source": "langsmith",
"cwd": "/workspace/deepagents",
"model": "claude-sonnet-4-6"
},
{
"role": "user",
"content": "What is the weather in Paris?",
"timestamp": "2026-07-11T12:00:00.000Z"
},
{
"role": "reasoning",
"content": "I should check the weather tool.",
"timestamp": "2026-07-11T12:00:05.000Z"
},
{
"role": "assistant",
"content": "I’ll check.",
"timestamp": "2026-07-11T12:00:05.000Z"
},
{
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call-weather-1",
"name": "get_weather",
"args": "{\"city\":\"Paris\"}"
}
],
"timestamp": "2026-07-11T12:00:05.000Z"
},
{
"role": "tool",
"tool_call_id": "call-weather-1",
"content": "Sunny, 22 C",
"timestamp": "2026-07-11T12:00:05.000Z"
},
{
"role": "assistant",
"content": "It is sunny and 22 C in Paris.",
"timestamp": "2026-07-11T12:00:05.000Z"
}
],
"diagnostics": []
}
Loading