feat(agent): route wizard cloud runs to the onboarding gateway product - #3855
Merged
Conversation
Setup wizard cloud runs (origin_product "onboarding") fell through to the posthog_code default, so their generations billed against the customer's PostHog Code credits. Route them to the new unbilled `onboarding` product instead. origin_product alone does not authorize the free route: any task:write holder can POST a task claiming it. A run must also carry the wizard_config state key that only the server-side wizard flow stamps, and that the task API refuses to let a caller set. Without the marker the run stays on posthog_code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
😎 Merged directly without going through the merge queue, as the queue was empty and the PR was up to date with the target branch - details. |
Member
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
|
React Doctor found no issues in the changed files. 🎉 Reviewed by React Doctor for commit |
2 tasks
Contributor
|
Reviews (1): Last reviewed commit: "feat(agent): route wizard cloud runs to ..." | Re-trigger Greptile |
charlesvien
approved these changes
Jul 28, 2026
gewenyu99
approved these changes
Jul 28, 2026
Contributor
|
suggestion @rafaeelaudibert maybe just call it Cloud run and detached the Wizard term. I suspect the needs of this agent will drift from the Wizard over time. I don't think how/when is clear but I think this will eventually be a entirely different shaped agent. The core bits (the context and skills) are obviously gonna be the same, but the workflow and end-goal will drift. This just semantically doesn't lock you in. |
rafaeelaudibert
added a commit
to PostHog/posthog
that referenced
this pull request
Jul 28, 2026
…74095) ## Problem Setup wizard cloud runs (the "run the wizard in the cloud" path from onboarding) create a task with `origin_product = "onboarding"`. The agent-server has no mapping for that origin, so it falls through to the `posthog_code` default and every generation the run makes bills into `CreditBucket.POSTHOG_CODE_CREDITS`. Two things wrong with that. We charge a customer's PostHog Code credits for onboarding, which is our acquisition spend and happens before they have decided to buy anything. And wizard traffic is indistinguishable from real Code tasks in `$ai_generation`, so we cannot see what onboarding actually costs. `ai_product` cannot be fixed with a header. The gateway re-asserts it from the route in `_apply_owned_event_properties` precisely so callers cannot spoof it. The route is the only lever, which means a new product. ## Changes A new `onboarding` gateway product, unbilled (`credit_bucket=None`), with an explicit cost ceiling so the spend gauge actually gets published for it. Wizard cloud runs now pin `claude-sonnet-5` instead of inheriting the agent-server's `claude-opus-4-8` default, and stamp `ai_stage="wizard_pr_agent"` so the PR agent's generations stay separable inside the product. The pin is what makes the narrow model allowlist workable. A free product needs a closed door, and `origin_product` is caller-settable: any `task:write` holder can POST a task claiming any origin except the three we already reject. So the task API now refuses `onboarding` too, the same way it refuses `image_builder` and `experiments`. Without that, this PR would ship a free LLM endpoint. That guard is one of two. The other lives in the agent-server: it only routes to the free product when the run also carries the `wizard_config` state key, which only the server-side wizard flow stamps and which the run PATCH allowlist drops. A task that merely claims the origin stays on `posthog_code`. Also fixed one adjacent lie in the gateway README, which still documented a `billable=True` flag that was replaced by `credit_bucket`. > [!IMPORTANT] > This PR is the first half. The agent-server half is PostHog/code#3855, and it must land **after** this one. > The gateway deploys from its own path-triggered pipeline, so until `onboarding` exists in `PRODUCTS` there, any request to that path segment gets a 400 from `validate_product`. Landing this first is inert, since nothing requests the slug until the agent change ships and the sandbox base image rebuilds. > [!NOTE] > Model allowlist includes `claude-opus-4-8` on purpose. The Claude SDK sets it as `fallbackModel` for refusal and overload rescue whenever it is not the primary, so a 403 there would fail the run instead of rescuing it. Bedrock ids are in for the same reason, matching `posthog_code` and `background_agents`. ## How did you test this code? I (well, Claude) ran these locally, all green: - `services/llm-gateway` full suite, 1375 passed. The new product is picked up automatically by the derived `TestServerCredentialConfigInvariant`, which is what forces `requires_server_credential=True` on anything sharing the Code OAuth app. - `products/tasks/backend/tests/test_api.py` plus `posthog/api/wizard/`, 660 passed. - `products/tasks/backend/tests/test_facade.py`, 31 passed. New test coverage and the regression each one catches: | Test | Catches | | --- | --- | | `onboarding` case added to `test_create_task_rejects_internal_origin` | Someone drops the API guard and the free route becomes forgeable by any `task:write` caller. This is the security-relevant one. | | `test_create_wizard_cloud_run_pins_its_model` | The model pin gets dropped, runs go back to the premium default, and every wizard cloud run then 403s at the gateway because that model is not in the product's allowlist. | | `onboarding` added to the two `TestServerCredentialRequirement` lists | The server-credential marker stops being enforced for this product. Those lists are hand-enumerated, so a new product is not covered without it. | Not tested by me: the end to end run against a real sandbox. That needs both halves deployed plus a sandbox image rebuild, so it has to happen after the agent PR lands. What I would check then is that the run's `$ai_generation` events carry `ai_product = "onboarding"`, `$ai_billable = false`, `ai_stage = "wizard_pr_agent"`, and sonnet-5 as the model. 👉 _Stay up-to-date with [PostHog coding conventions](https://posthog.com/docs/contribute/coding-conventions) for a smoother review._ ## Automatic notifications - [ ] Publish to changelog? - [ ] Alert Sales and Marketing teams? ## Docs update Updated `services/llm-gateway/README.md` in this PR: added the product to the registered products table and corrected the `billable` reference to `credit_bucket`. No user-facing docs affected. ## 🤖 Agent context **Autonomy:** Human-driven (agent-assisted) Claude Opus 5 in Claude Code, driven by me. Invoked `/writing-tests` before adding tests. The interesting decision was how hard to lock the free route. The first proposal was a dedicated internal OAuth scope minted only for wizard runs and required by the product, mirroring `signal_scout_internal`. I rejected it as too much machinery for the payoff. What we went with instead is the two-factor gate described above, which reuses markers that already exist and are already protected. Worth knowing while reviewing: the gateway's own `requires_server_credential` flag is set here for consistency and to satisfy the config invariant test, but it is not carrying the weight. It is OAuth-only, it is inert unless `posthog_code_model_gate_enabled` is on, and every sandbox token already carries `internal_run:read` anyway, so it cannot tell a wizard run from any other task run. The API guard and the `wizard_config` check are what actually close the door. One related finding I deliberately left out of scope. `Task.internal` is settable on create and mutable on PATCH, and `isInternal` maps to `background_agents`, which is also unbilled. That looks like the same shape of hole, already open. It may be covered by `requires_server_credential` depending on what `LLM_GATEWAY_POSTHOG_CODE_MODEL_GATE_ENABLED` is set to in production, which I could not verify from here. Worth a look, separately.
This is enough, just gotta confirm it's internal
…g object Replaced multiple conditional checks with a mapping object to simplify the resolution of gateway products based on the origin product. This enhances code readability and maintainability.
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
rafaeelaudibert
force-pushed
the
rafa/onboarding-gateway-product
branch
from
July 29, 2026 03:39
c361058 to
18594cf
Compare
Member
Author
|
@gewenyu99 thank you, you prompted me to change this and it's sooo much simpler now! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Setup wizard cloud runs arrive here as a task with
origin_product = "onboarding".resolveGatewayProducthas no branch for that value, so it falls through to theposthog_codedefault and the run's generations bill against the customer's PostHog Code credits.Onboarding is PostHog-funded acquisition spend and happens before the user has decided to buy anything, so it should not touch their credits. It also means wizard traffic is indistinguishable from real Code tasks in
$ai_generation.Changes
Route
origin_product === "onboarding"to the new unbilledonboardinggateway product.The catch is that origin alone is not proof of anything.
origin_productis caller-supplied: anyone holdingtask:writecan POST a task claiming any origin. Pointing a free product at that field on its own would be a free LLM endpoint.So the free route needs two things, and this PR is the second:
onboardingas a caller-supplied origin.wizard_configstate key. Only the server-side wizard flow stamps it, the run PATCH allowlist silently drops it, and the run-create body has no state field at all, so a caller cannot get it onto a run. Without the marker the run stays onposthog_code, billed exactly as today.configureEnvironmentalready reads arbitrary keys off the fetched run (prewarmed,ai_stage), so the marker check costs nothing extra.Warning
Do not merge this before PostHog/posthog#74095. That PR registers the
onboardingproduct in the LLM gateway.The gateway ships from its own path-triggered pipeline, so until it is deployed any request to that path segment is rejected by
validate_productwith a 400. Merging this first would break every wizard cloud run the moment the sandbox base image picks up the new agent.Order is: merge and deploy PostHog/posthog#74095, then merge this, then let the
agent-v*tag publish and trigger the sandbox base image rebuild.How did you test this?
I (well, Claude) ran these locally, all green:
packages/agentvitest forsrc/utils/gateway.test.tsandsrc/server/agent-server.configure-environment.test.ts, 48 passed.pnpm typecheckonpackages/agent, clean.New cases and the regression each one catches:
resolveGatewayProductrows for onboarding with and withoutisWizardCloudRunisInternal: trueand no markerbackground_agents, which is also unbilled. It must land onposthog_code.configureEnvironmentonboarding URL casesNot tested by me: a real cloud run end to end. That needs the gateway side deployed plus a sandbox base image rebuild, so it can only happen after both PRs land. What I would check then is that the run's
$ai_generationevents carryai_product = "onboarding"and$ai_billable = false.Automatic notifications
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Claude Opus 5 in Claude Code, driven by me. Paired with PostHog/posthog#74095, which adds the gateway product, pins wizard runs to claude-sonnet-5 so they stop defaulting to the premium model, and closes the origin at the task API.
We considered gating the free product on a dedicated internal OAuth scope minted only for wizard runs, and dropped it as too much machinery. The
wizard_configmarker gives the same property using state that is already protected in the places a caller could reach.Note the gateway's
requires_server_credentialis set on the new product but is not doing the work here. Every sandbox token already carriesinternal_run:read, so it cannot distinguish a wizard run from any other task run. The two gates described above are what actually close the door.