To track product-market fit and user drop-off, we need anonymous, opt-in telemetry in the modelfuzz scan CLI. Open-source users are sensitive to tracking, so this must be done ethically.
Tasks:
- Integrate the
posthog Python library into the CLI.
- On the first run, print a clear message: "ModelFuzz collects anonymous usage data to improve the tool. Run with --no-telemetry to disable."
- Track key funnel events:
scan_started, vulnerability_found, scan_completed, and error_raised.
- Do NOT track arguments, API keys, or user prompts. Only track event names and timestamps.
- [Phase 2] Build a private internal dashboard (Vercel/Next.js) that pulls data from the PostHog API and the PePy.tech API (for PyPI download counts) into one unified CEO view.
Implementation rules (non-negotiable)
These override anything above that conflicts with them.
1. Genuine opt-in, not opt-out
Task 2 as originally written (--no-telemetry to disable) is opt-out — it collects on the first run before the user has agreed. For a security tool, reviewed by exactly the audience most hostile to tracking, that gap will be noticed.
Use real opt-in instead: MODELFUZZ_TELEMETRY=1, or a first-run prompt that collects nothing until answered. Support an env var either way, because a printed notice is not consent in CI or any headless run where nobody sees it. Default state, with no configuration, must be off.
2. Confined to the scan CLI — never the library path
posthog makes outbound network calls from a package people install specifically to stop unwanted outbound requests. So:
- Telemetry lives only in the
modelfuzz scan code path.
@shield_tool, PolicyEngine, the rules, and import modelfuzz must never make a network call. A guardrail decorator that phones home is a genuine problem, not just a bad look.
posthog must be an optional dependency, not a runtime dep of the core package.
- Failures must degrade silently and never delay or break a scan — offline, blocked egress, and DNS failure are all normal.
3. Error categories only, never the exception string
error_raised is the highest-risk event. Exception messages routinely carry URLs, model names, and account identifiers — a real example from this project: an OpenRouter 402 returned multiple KB of JSON containing an account id and a dashboard URL embedding a key hash.
Send a bounded category (auth_error, rate_limit, network_error, truncated_response, unknown) and nothing else. Never str(exc), never a traceback, never a URL, never a model name if it could be a private deployment identifier.
Acceptance criteria
To track product-market fit and user drop-off, we need anonymous, opt-in telemetry in the
modelfuzz scanCLI. Open-source users are sensitive to tracking, so this must be done ethically.Tasks:
posthogPython library into the CLI.scan_started,vulnerability_found,scan_completed, anderror_raised.Implementation rules (non-negotiable)
These override anything above that conflicts with them.
1. Genuine opt-in, not opt-out
Task 2 as originally written (
--no-telemetryto disable) is opt-out — it collects on the first run before the user has agreed. For a security tool, reviewed by exactly the audience most hostile to tracking, that gap will be noticed.Use real opt-in instead:
MODELFUZZ_TELEMETRY=1, or a first-run prompt that collects nothing until answered. Support an env var either way, because a printed notice is not consent in CI or any headless run where nobody sees it. Default state, with no configuration, must be off.2. Confined to the
scanCLI — never the library pathposthogmakes outbound network calls from a package people install specifically to stop unwanted outbound requests. So:modelfuzz scancode path.@shield_tool,PolicyEngine, the rules, andimport modelfuzzmust never make a network call. A guardrail decorator that phones home is a genuine problem, not just a bad look.posthogmust be an optional dependency, not a runtime dep of the core package.3. Error categories only, never the exception string
error_raisedis the highest-risk event. Exception messages routinely carry URLs, model names, and account identifiers — a real example from this project: an OpenRouter402returned multiple KB of JSON containing an account id and a dashboard URL embedding a key hash.Send a bounded category (
auth_error,rate_limit,network_error,truncated_response,unknown) and nothing else. Neverstr(exc), never a traceback, never a URL, never a model name if it could be a private deployment identifier.Acceptance criteria
import modelfuzzand a@shield_tool-wrapped call make no network calls under any configurationposthogis an optional extra;pip install modelfuzzdoes not pull it instr(exc)SECURITY.md