Skip to content

Surface chat system_fingerprint and logprobs as response metadata - #1104

Open
PratikDhanave (PratikDhanave) wants to merge 1 commit into
microsoft:mainfrom
PratikDhanaveFork:fix/openai-chat-response-metadata
Open

PratikDhanave (PratikDhanave) wants to merge 1 commit into
microsoft:mainfrom
PratikDhanaveFork:fix/openai-chat-response-metadata

Conversation

@PratikDhanave

Copy link
Copy Markdown
Contributor

The Chat path built its ResponseUpdate with ResponseID/FinishReason/usage but never set AdditionalProperties, so two pieces of response metadata were dropped:

  • system_fingerprint (on the completion / chunk), and
  • per-choice logprobs (returned when a caller requests them via Logprobs/TopLogprobs).

The Python client carries both as chat response metadata (_get_metadata_from_chat_responsesystem_fingerprint, _get_metadata_from_chat_choicelogprobs). The Responses path in this provider already populates AdditionalProperties, so this was also a chat-vs-responses asymmetry — the existing chat fixtures even contain system_fingerprint/logprobs values that nothing surfaced.

Change

  • Populate AdditionalProperties with SystemFingerprint (when non-empty) and Logprobs (when the choice carries content/refusal logprobs) on both the non-streaming and streaming paths. Keys follow the PascalCase convention already used by the Responses path helper (responsesPopulateAdditionalProperties).

Tests

  • TestChatResponseMetadataSurfaced_NonStreaming and _Streaming: assert SystemFingerprint (and, non-streaming, Logprobs) reach the message AdditionalProperties. Both fail before the change (no message carries any), pass after.

The Chat path set ResponseID, FinishReason, and usage but never populated
AdditionalProperties, dropping the response system_fingerprint and the
per-choice logprobs that a caller requesting logprobs would expect back. The
Responses path already populates AdditionalProperties, so this was also a
chat-vs-responses asymmetry.

Populate AdditionalProperties with SystemFingerprint (when present) and
Logprobs (when the choice carries any) on both the non-streaming and
streaming paths, matching the Python client which surfaces both as chat
response metadata.
Copilot AI lite review requested due to automatic review settings September 18, 2026 12:28
@github-actions github-actions Bot added area:provider Changes files in the provider area area:provider/openai Changes files in the provider / openai area size:large At most 300 changed lines across at most 10 files labels Sep 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Streaming logprobs may be overwritten across chunks, and this path lacks coverage.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Surfaces Chat response system_fingerprint and logprobs metadata through AdditionalProperties.

Changes:

  • Propagates metadata for streaming and non-streaming responses.
  • Adds tests for metadata exposure.
File summaries
File Summary
provider/openaiprovider/chat.go Populates Chat response metadata.
provider/openaiprovider/chat_test.go Tests metadata propagation.
Review details

Suppressed comments (2)

provider/openaiprovider/chat.go:227

  • The new streaming Logprobs branch is not exercised: the added streaming test only sends/asserts SystemFingerprint, while all existing streaming fixtures use logprobs: null. Add a streamed chunk containing content or refusal logprobs and assert that the yielded update (and collected message) preserves the Logprobs metadata, so this path cannot regress unnoticed.
				if logprobs := chunk.Choices[0].Logprobs; len(logprobs.Content) > 0 || len(logprobs.Refusal) > 0 {
					additionalProperties = map[string]any{"Logprobs": logprobs}
				}

provider/openaiprovider/chat_test.go:457

  • This streaming test only sends system_fingerprint; it never supplies a non-empty chunk logprobs value, so it would pass if the new streaming Logprobs handling were removed or broken. Add a chunk with content/refusal logprobs and assert that the streamed update (and, if collection is supported, the collected response) preserves them.
func TestChatResponseMetadataSurfaced_Streaming(t *testing.T) {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		w.Header().Set("Content-Type", "text/event-stream")
		_, _ = io.WriteString(w, "data: {\"id\":\"chatcmpl-md\",\"object\":\"chat.completion.chunk\",\"created\":1727888631,\"model\":\"gpt-4o-mini\",\"system_fingerprint\":\"fp_test\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"ok\"},\"finish_reason\":\"stop\"}]}\n\n")
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +225 to +226
if logprobs := chunk.Choices[0].Logprobs; len(logprobs.Content) > 0 || len(logprobs.Refusal) > 0 {
additionalProperties = map[string]any{"Logprobs": logprobs}
@github-actions github-actions Bot added kind:code Changes production behavior or code kind:tests Changes tests, fixtures, or test infrastructure labels Sep 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Scope: user-visible behavior (bug fix)

Changed Go contract: No exported API added/changed. chatClient.run in provider/openaiprovider/chat.go now populates the existing agent.ResponseUpdate.AdditionalProperties field with SystemFingerprint (from ChatCompletion.SystemFingerprint / ChatCompletionChunk.SystemFingerprint) and Logprobs (from Choice.Logprobs / ChunkChoice.Logprobs) on both the non-streaming and streaming chat paths, matching what the Responses path already does via responsesPopulateAdditionalProperties.

Upstream evidence reviewed:

  • Python: python/packages/openai/agent_framework_openai/_chat_completion_client.py, _get_metadata_from_chat_response (system_fingerprint) and _get_metadata_from_chat_choice (logprobs), both merged into response_metadata/chunk_metadata and passed as ChatResponse.additional_properties in _parse_response_from_openai (lines ~999-1015, ~871-895).
  • .NET: No same-repo equivalent — Microsoft.Agents.AI.OpenAI builds on the external Microsoft.Extensions.AI OpenAIChatClient, which is not vendored in this repository, so there is no dotnet/src/Microsoft.Agents.AI.OpenAI code path to compare against for this specific mapping.

Result: Aligned. The Go change closes a genuine chat-vs-responses asymmetry (the Responses path already surfaced comparable metadata via responsesPopulateAdditionalProperties) and matches the Python client's behavior of surfacing system_fingerprint and logprobs as response/message metadata. Key naming (SystemFingerprint, Logprobs) follows the existing Go PascalCase convention used by responsesPopulateAdditionalProperties. No exported Go API surface changed (only internal population of an existing field), so public-api-change is not applicable.

Generated by Go API Consistency Review Agent · copilot · auto · 34.8 AIC · ⌖ 8.27 AIC · ⊞ 9.2K ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:provider/openai Changes files in the provider / openai area area:provider Changes files in the provider area kind:code Changes production behavior or code kind:tests Changes tests, fixtures, or test infrastructure size:large At most 300 changed lines across at most 10 files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants