Skip to content

[Feature] Add PostHog Telemetry to CLI and build internal Founder Dashboard #61

Description

@higagan

To track product-market fit and user drop-off, we need anonymous, opt-in telemetry in the modelfuzz scan CLI. Open-source users are sensitive to tracking, so this must be done ethically.

Tasks:

  1. Integrate the posthog Python library into the CLI.
  2. On the first run, print a clear message: "ModelFuzz collects anonymous usage data to improve the tool. Run with --no-telemetry to disable."
  3. Track key funnel events: scan_started, vulnerability_found, scan_completed, and error_raised.
  4. Do NOT track arguments, API keys, or user prompts. Only track event names and timestamps.
  5. [Phase 2] Build a private internal dashboard (Vercel/Next.js) that pulls data from the PostHog API and the PePy.tech API (for PyPI download counts) into one unified CEO view.

Implementation rules (non-negotiable)

These override anything above that conflicts with them.

1. Genuine opt-in, not opt-out

Task 2 as originally written (--no-telemetry to disable) is opt-out — it collects on the first run before the user has agreed. For a security tool, reviewed by exactly the audience most hostile to tracking, that gap will be noticed.

Use real opt-in instead: MODELFUZZ_TELEMETRY=1, or a first-run prompt that collects nothing until answered. Support an env var either way, because a printed notice is not consent in CI or any headless run where nobody sees it. Default state, with no configuration, must be off.

2. Confined to the scan CLI — never the library path

posthog makes outbound network calls from a package people install specifically to stop unwanted outbound requests. So:

  • Telemetry lives only in the modelfuzz scan code path.
  • @shield_tool, PolicyEngine, the rules, and import modelfuzz must never make a network call. A guardrail decorator that phones home is a genuine problem, not just a bad look.
  • posthog must be an optional dependency, not a runtime dep of the core package.
  • Failures must degrade silently and never delay or break a scan — offline, blocked egress, and DNS failure are all normal.

3. Error categories only, never the exception string

error_raised is the highest-risk event. Exception messages routinely carry URLs, model names, and account identifiers — a real example from this project: an OpenRouter 402 returned multiple KB of JSON containing an account id and a dashboard URL embedding a key hash.

Send a bounded category (auth_error, rate_limit, network_error, truncated_response, unknown) and nothing else. Never str(exc), never a traceback, never a URL, never a model name if it could be a private deployment identifier.

Acceptance criteria

  • With no configuration, a scan sends zero network requests to any telemetry endpoint (verify by packet/DNS inspection, not by reading the code)
  • import modelfuzz and a @shield_tool-wrapped call make no network calls under any configuration
  • posthog is an optional extra; pip install modelfuzz does not pull it in
  • A test asserts no event payload field ever contains str(exc)
  • Telemetry failure (offline / blocked) leaves scan behaviour and timing unchanged
  • The privacy behaviour is documented in the README and SECURITY.md

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions