feat(analytics): instrument the product funnel and add churn analysis

[KurosawaGeeker]

Area

Web app

Problem

The team lacks a reliable view of where users abandon Mosoo's core journey. Product decisions cannot be grounded in conversion, activation, retention, and churn data when key steps are not consistently instrumented or connected to an analysis dashboard.

Operational runtime telemetry and cost usage are not a substitute for product analytics. We need a product-event contract that measures user behavior without leaking Agent content, prompts, credentials, or other sensitive data.

Proposal

Instrument the core product funnel and connect it to a product analytics/churn dashboard.

Start with a small canonical funnel:

1. Sign up or first authenticated visit.
2. Create or enter an App.
3. Create an Agent.
4. Complete required provider and Environment setup.
5. Start the first Preview Run.
6. Receive the first successful result.
7. Publish an Agent.
8. Invoke the published Agent through its user-facing or API surface.
9. Return and run again within agreed retention windows.

For each event, define a stable name, trigger, actor/session identity, App-scoped dimensions, success/failure outcome, and schema version. Add dashboards for funnel conversion, time to first successful Run, activation cohorts, retention, and churn/drop-off by step.

Product and implementation decisions to resolve:

• Select the analytics destination/vendor and define environment separation.
• Define activation, retention windows, and churn for the current Alpha lifecycle.
• Decide identity stitching rules across anonymous and authenticated sessions.
• Define consent, opt-out, access, retention, and deletion requirements.
• Identify which failure categories are safe and useful as dimensions.

Acceptance criteria:

• A reviewed event taxonomy covers the canonical funnel and has one owner/source of truth.
• Events are emitted once at the user-visible state transition, with deduplication where retries are possible.
• Development, preview, and production data are separated.
• No prompt text, Agent instructions, message content, files, provider keys, environment-variable values, tokens, or raw error payloads are sent.
• Dashboards show step conversion, time to activation, cohort retention, and churn/drop-off.
• The team can filter by safe product dimensions such as entry point, runtime family, provider availability state, and success/failure category without exposing secrets.
• Automated tests or a typed validation layer detect missing/invalid event properties.
• A verification guide explains how to validate events and dashboard data in non-production and production.

Alternatives considered

• Infer product behavior from application logs. Logs are optimized for operations and do not provide a stable product-event contract or funnel identity.
• Instrument page views only. Page views cannot distinguish successful setup and activation from abandoned or failed attempts.
• Send full context to simplify analysis. This is rejected because Agent content and credentials are sensitive and unnecessary for funnel analysis.

Compatibility and migration

Instrumentation must be additive and must not change product behavior when the analytics destination is unavailable. Existing operational telemetry and Cost data remain separate. Analytics initialization should fail safely, respect privacy/consent requirements, and avoid blocking user actions.

Contribution

I can provide product feedback.

Checklist

• I searched existing issues before opening this request.
• I described the problem before the proposed solution.
• I kept this request focused on one improvement.