Repository navigation
feat: telemetry for the workspace link lifecycle, usage, engine and starter #1513
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
567d961
0c0dd39
b24a189
2e4f6f1
4c8a213
db4b121
d55f1cf
bfd7ecb
e1474e7
f48d0a6
723915f
d1fc903
7a4364a
66590e5
1f775ec
403222f
40807f5
217edbd
cad667a
974489f
e02ae7f
06daf9b
15891ec
6abd47f
335cd92
04d18a5
bb18d87
83e0912
d852acf
28779f5
8aa0eb5
9557d1d
49f3f8f
1d7bfdc
6e46604
07a1c07
7f12f74
8393f46
b641b89
87c9ebd
a590b79
7605dd6
2d21add
ead39c6
fd4d933
e52f105
5cd6d70
cd96a8a
a9b9f7a
4016f59
93b3aac
0908868
81cab6d
5f2733b
1cbcf2b
40026d1
9ceb8a4
ea3fcc7
0298ce9
5718138
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -37,7 +37,7 @@ We collect the following categories of events: | |
| | `error_recovered` | Successful recovery from a transient error (error type, strategy, attempt count) | | ||
| | `mcp_server_census` | MCP server capabilities after connect (tool and resource counts, but no tool names) | | ||
| | `context_overflow_recovered` | Context overflow is handled (strategy) | | ||
| | `skill_used` | A skill is loaded (skill name, source — `builtin`, `global`, or `project`, and trigger — `user`, `auto`, or `suggestion` — no skill content) | | ||
| | `skill_used` | A skill is loaded (skill name, source — `builtin`, `global`, `project`, or `workspace`, and trigger — `user`, `auto`, or `suggestion` — no skill content) | | ||
| | `plan_revision` | A plan revision occurs in Plan mode (revision_number, action: `refine`, `approve`, `reject`, or `cap_reached`) | | ||
| | `feature_suggestion` | A post-connection feature suggestion is shown (suggestion_type, suggestions_shown, warehouse_type — no user input) | | ||
| | `sql_execute_failure` | A SQL execution fails (warehouse type, query type, error message, PII-masked SQL — no raw values) | | ||
|
|
@@ -77,6 +77,37 @@ We collect the following categories of events: | |
|
|
||
| Each event includes a timestamp, session ID, a per-launch correlation ID (`launch_id` — a random value regenerated every process start, not persisted and not derived from your machine or identity; it exists only to group events from the same run), CLI version, and a machine ID (a random UUID stored in `~/.altimate/machine-id`, generated once and not derived from your hardware or identity). When the machine is signed in to Altimate, each event also carries `user_id` and `tenant`, described under [User Identification](#new-user-identification). | ||
|
|
||
| ### Workspace events | ||
|
|
||
| These events describe projects linked to an Altimate workspace. A workspace is identified only by its numeric id (`workspace_id`, or `to_workspace_id` and `from_workspace_id` on a switch). No workspace name, project path, repository remote, skill content, memory content or prompt text is sent. | ||
|
|
||
| | Event | Description | | ||
| |-------|-------------| | ||
| | `workspace_linked` | A project was linked to a workspace and the server confirmed it. `how` is `bind_existing`, `create_and_bind`, `rebind_remote` or `rebind_path`. | | ||
| | `workspace_created` | A workspace was created from the CLI or the TUI. `how` is `create_and_bind`, or `create_unbound` when the link follows as a separate step. | | ||
| | `workspace_link_failed` | A link step failed. `how` names the step, and `reason` is a class: `auth`, `not_found`, `network`, `conflict` or `other`. No error text. | | ||
| | `workspace_unlinked` | A project was unlinked, with the workspace id when the CLI knew it. | | ||
| | `workspace_switched` | A linked project was moved to another workspace. `how` is `rebind_remote` or `rebind_path`. | | ||
| | `workspace_auth_expired` | A workspace call was answered 401: the sign-in has expired. Carries `endpoint` and `status`. | | ||
| | `workspace_api_error` | A workspace call failed. `status_class` is `4xx`, `5xx`, `network`, `timeout`, or `response` (a success status with an empty or unreadable body), plus the HTTP `status` when there was one. A 403 is reported here, not as expired auth. A status the caller handles as an answer is not a failure and is not reported, such as a 404 from the link lookups ("not linked") or from an unlink ("already unlinked"). `GET /users/me` is not a workspace call and is not reported. | | ||
| | `starter_shown` | The "what this workspace provides" starter was served after a link: to the TUI, to the IDE extension's chat, or printed by `altimate-code link`. It counts the starter being served, so a client that closes before rendering it is still counted. Contains the counts of skills and integrations when known, `knowledge` (`off`, `all`, `selected` or `unknown`) and `memory_enabled`. | | ||
| | `starter_action_picked` | The user chose an option from the TUI starter: `action` is `prompt`, `open` or `done`, with `prompt_index` (its position) for a suggested prompt. Never the prompt's text. | | ||
| | `workspace_memory_contributed` | A memory block was written back to the workspace's shared memory, including the existing blocks seeded when a project is linked. `block_count` is per write. No memory content. | | ||
| | `workspace_sync_failed` | Syncing the workspace's skills or memory to this project changed to failing. `kind` is `skills` or `memory`, and `reason` is a class. Sent once per change, not on every retry. | | ||
| | `workspace_engine_attach` | A session's connection to the workspace integration engine settled on an outcome (`attached`, `engine_missing`, `engine_too_old`, `connect_failed`, `disabled` or `unbound`), with the engine version and counts of integrations. Sent when the outcome changes, not every turn. | | ||
| | `workspace_engine_install_prompt` | The offer to install or update the engine: `action` is `shown`, `accepted`, `installed`, `install_failed`, `copied` or `declined`, with the `reason` and whether Node.js was found and its major version. | | ||
| | `workspace_engine_precedence` | Whether the workspace's integrations take precedence over the project's own connections, with counts of shadowed connection types and connections. Sent when the state changes. | | ||
| | `workspace_engine_redirect` | A `datamate_manager` call was refused because the project is linked to a workspace. Contains the tool and operation names. | | ||
| | `workspace_engine_error` | The engine failed at a stage (`attach`, `install` or `precedence`). `message` is masked (paths, credentials and quoted strings are replaced) and capped at 200 characters. | | ||
|
|
||
| Other rows gain workspace fields in a linked project: | ||
|
|
||
| - Every event carries `client: "altimate-code"`, which names the product that sent it. `source` still says how the CLI was launched. | ||
| - Rows of a turn in a linked project carry that project's `workspace_id`: `session_start`, `session_end`, `first_prompt_sent`, `first_job_completed`, `skill_used`, `skill_published`, `memory_injection`, `agent_outcome`, `task_classified`, `task_outcome_signal`, `tool_chain_outcome`, `core_failure`, `error_fingerprint`, `plan_revision`, `sql_quality` and `tool_call`. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Minor (carried over, still open) — the contract says Unchanged since the approval: the Suggestion: either stamp the turn's workspace on the Non-blocking: no query or client reads |
||
| - `session_end` counts the workspace skills, memory blocks and integration calls the session used. `memory_injection` counts the workspace's blocks. `task_outcome_signal` says whether workspace context was used. | ||
| - A call to a workspace integration through the engine is reported as a `tool_call` with `tool_type: "mcp"` and `via_workspace: true`. As for other tools, there are no arguments or outputs, and an error message is masked and capped at 500 characters. | ||
| - An `endpoint` value is a route label with every id and path segment replaced by `:id`, for example `POST /datamate-project-bindings/bind`. It never contains an id, a file path or a query string. | ||
|
|
||
| ### Notes on the review events | ||
|
|
||
| - `degraded` is a fidelity flag, not a warehouse flag. It is set when a review found no reviewable | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,6 +12,8 @@ import { Config } from "@/config/config" | |
| import { Flag } from "@/flag/flag" | ||
| import { InstallationVersion } from "@opencode-ai/core/installation/version" | ||
| import { Log } from "@/altimate/util/log" | ||
| // altimate_change — reads the config telemetry opt-out without an Instance | ||
| import { TelemetryConfig } from "../telemetry/config-state" | ||
|
|
||
| /** | ||
| * Why a failure reason is attached at the rejection site rather than inferred from the message: | ||
|
|
@@ -75,7 +77,9 @@ const log = Log.create({ service: "altimate-plugin" }) | |
| // and `cli_version` are strings, treat `cli_version: "local"` as a valid dev | ||
| // build, and treat the payload as untrusted (anyone can craft a URL). An absent | ||
| // `machine_id` means "do not attribute" — it must never be aliased on. | ||
| export async function buildCliContext(machineIdPath?: string): Promise<string> { | ||
| /** `directory`: the project the command acts on, whose config may opt out; the process's working | ||
| * directory when the command has none (`auth login`). */ | ||
| export async function buildCliContext(machineIdPath?: string, directory?: string): Promise<string> { | ||
| // altimate_change start — honour both telemetry opt-out gates, mirroring | ||
| // telemetry/index.ts::doInit: | ||
| // 1. ALTIMATE_TELEMETRY_DISABLED / OPENCODE_DISABLE_TELEMETRY env vars | ||
|
|
@@ -90,17 +94,25 @@ export async function buildCliContext(machineIdPath?: string): Promise<string> { | |
| } catch (err) { | ||
| // Config was unreadable — NOT the normal path. Server routes run inside | ||
| // Instance.provide() (AsyncLocalStorage-propagated across awaits), so | ||
| // Config.get() resolves during an ordinary browser authorize(). The known | ||
| // exception is `altimate auth login <url>`, which deliberately skips | ||
| // instance bootstrap (ProvidersLoginCommand `instance: (args) => !isAuthProviderUrl(args.target)`); | ||
| // on that path this fires. Fail CLOSED — omit the durable machine_id rather | ||
| // than transmit it for a user who may have opted out via config; a missed | ||
| // correlation beats leaking a stable identifier. Log so a low correlation | ||
| // rate is traceable here instead of being mistaken for a lost reply. | ||
| log.warn("cli_context: config unreadable, omitting machine_id (fail-closed)", { | ||
| // Config.get() resolves during an ordinary browser authorize(). It fires on | ||
| // the paths that skip instance bootstrap: `altimate auth login <url>` and the | ||
| // create-and-link hand-off of `altimate-code link`. Read `telemetry.disabled` | ||
| // from the config files directly: an opt-out there, or files that cannot be | ||
| // read (an opt-out may be behind them), omit the id. Only a config that | ||
| // readably leaves telemetry on defers to the telemetry module's own state | ||
| // (opted out by env, no sink, test runner). | ||
| const state = await TelemetryConfig.disabledState(directory) | ||
| if (state === "enabled") { | ||
| await Telemetry.init().catch(() => {}) | ||
| disabled = !Telemetry.isEnabled() | ||
|
Comment on lines
+104
to
+107
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| } else { | ||
| disabled = true | ||
| } | ||
| log.info("cli_context: config unavailable, read the opt-out from the files", { | ||
| code: (err as NodeJS.ErrnoException)?.code, | ||
| state, | ||
| enabled: !disabled, | ||
| }) | ||
| disabled = true | ||
| } | ||
| } | ||
| let machineId = "" | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor — the contract says
generationgainsworkspace_id; this list (and the code) leave it outThe contract in AltimateAI/altimate-code-analytics#22 says this PR "also adds workspace_id to
generation,agent_outcome,plan_revision, ...". Every other row in that sentence hasworkspace_id?in its type and is stamped from the turn's binding (...turnWorkspacein session/prompt.ts);generation(type at telemetry/index.ts ~271, emitted in session/processor.ts ~1053) has neither. This list is right about the code, so the contract sentence is the stale side.Suggestion: either stamp the turn's workspace on the
generationrow (thread the turn binding into the processor's emit) or dropgenerationfrom the contract sentence. Either way, make the two agree.Non-blocking: the analytics funnel reads
generationonly bymachine_id, so no query or client depends on it; a contract edit or a small follow-up.