Skip to content

Run an external SRE dogfooding pilot and publish the findings #190

Description

@hellices

Goal

Validate korvid with external SRE/platform users on real clusters and convert
observed failures into a prioritized product backlog.

Current gap

The repository has extensive automated tests and one maintainer's development
history, but no published evidence that independent users can:

  • install the product without repository knowledge;
  • discover the keyboard workflow;
  • complete normal read/diagnostic tasks faster or more safely;
  • understand the AI evidence and approval boundaries;
  • recover from provider, kubeconfig, RBAC, terminal, or cluster failures;
  • keep using the product after initial novelty.

Code volume, test count, and feature breadth are not substitutes for this
validation.

Pilot design

  • Recruit at least five external participants across SRE, platform engineering,
    and Kubernetes-using application development.
  • Include at least three distinct real cluster environments and one
    RBAC-limited user.
  • Use a fixed task script covering install/setup, navigation, filtering, logs,
    describe, context/namespace switching, one diagnosis journey, and one
    approval-gated write preview.
  • Capture task completion, time to first useful screen, time to first grounded
    diagnosis, errors, abandonments, help requests, and perceived trust.
  • Do not add background telemetry by default. Use consented screen recording,
    observer notes, structured interviews, and an opt-in diagnostic bundle with a
    documented redaction policy.

Acceptance criteria

  • At least five independent users complete the pilot; no repository contributor
    counts toward the minimum.
  • Results cover at least three cluster environments and include terminal, OS,
    Kubernetes version, auth/RBAC shape, and provider/local-model configuration.
  • Each scripted task reports completion rate and median time, with raw
    identifying data removed.
  • Installation and first-run failure modes are ranked separately from feature
    requests.
  • The report includes direct evidence for the top five usability/reliability
    problems and creates or links follow-up issues for each accepted problem.
  • At least two participants repeat a real workflow after the facilitated
    session or explain why they would not.
  • Public documentation states the pilot size and limitations accurately; it
    does not turn participants into unsupported adoption claims.

Out of scope

  • Product analytics without explicit consent.
  • Treating GitHub stars or downloads as proof of successful use.
  • Recruiting only experienced contributors who already know korvid's
    keybindings and architecture.

Metadata

Metadata

Assignees

Labels

concern: testingTest infrastructure, evaluation, and quality measurementdocumentationImprovements or additions to documentationenhancementNew feature or requesthelp wantedExtra attention is needed

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions