feat(inbox): make AI observability source available to everyone - #4006
Merged
trunk-io[bot] merged 4 commits intoJul 30, 2026
Merged
Conversation
Remove the staff-only "Internal" badge and tooltip from the AI observability card in the Self-driving source config, so it's presented as generally available rather than internal. Generated-By: PostHog Code Task-Id: 9725e5be-8b91-44f8-8c12-3419a5bf080c
|
😎 Merged successfully - details. |
|
React Doctor found 1 issue in 1 file · 1 warning. 1 warning
Reviewed by React Doctor for commit |
joshsny
approved these changes
Jul 30, 2026
Replace the AI observability "Open" link-out card with an on/off toggle like the other sources, matching the web app's AI observability source card. The toggle governs the project-level `evaluation_report` signal source. `llm_analytics` also carries per-evaluation `evaluation` configs (managed on the evaluations page); the toggle is scoped to `evaluation_report` so those are left untouched. Wired in both the responder roster (Inbox 2.0) and the settings source toggles, plus the parallel core SignalSourceService, with tests covering the scoping. Generated-By: PostHog Code Task-Id: 9725e5be-8b91-44f8-8c12-3419a5bf080c
AI observability signals are always emitted per evaluation report — there's no per-evaluation `evaluation` signal source to guard against — so treat llm_analytics as a plain single-type source like the others. Removes the type-scoping helper and the now-moot scoping tests, keeping the source-type mapping to `evaluation_report`. Generated-By: PostHog Code Task-Id: 9725e5be-8b91-44f8-8c12-3419a5bf080c
Signals only emit once evaluations exist, so the AI observability source card now says to set up evaluations and links to the ai-evals docs. Generated-By: PostHog Code Task-Id: 9725e5be-8b91-44f8-8c12-3419a5bf080c
Radu-Raicea
marked this pull request as ready for review
July 30, 2026 14:05
Contributor
|
Reviews (1): Last reviewed commit: "feat(inbox): point AI observability sour..." | Re-trigger Greptile |
2 tasks
trunk-io
Bot
deleted the
posthog-code/ai-observability-available-to-everyone
branch
July 30, 2026 15:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The AI observability source in the Self-driving inbox config was marked with an "Internal" badge and a "This is only visible to staff users of PostHog" tooltip, even though nothing in the code actually gated it to staff — it already rendered for everyone. It also rendered as an "Open" link-out to the web evaluations page instead of an on/off toggle like every other source.
Changes
evaluation_reportsignal source (AI observability signals are always emitted per evaluation report).SignalSourceService.How did you test this?
signalSourceService.test.tsverifying the toggle creates a config with theevaluation_reportsource type. Full core inbox suite passes (172 tests).@posthog/shared,@posthog/core, and@posthog/uitypecheck clean; Biome check clean on all changed files.Automatic notifications
Created with PostHog from a Slack thread