User Testing Test Plan

User testing test plan (Midpoint & beyond)

Team: BAGEA (GovBid) — Product Requirements Document (PRD)
As-built app: Next.js dashboard in repo web/ — Supabase auth + data (see Frontend Implementation Plan).

This page satisfies the CS194W “Planning to learn from user testing” deliverable: indexed from the wiki sidebar, used at the Midpoint Review, and paired with User Testing Results after sessions.

---

1. Welcome and orientation (moderator script, ~2 minutes)

Thank the tester for their time. Briefly set context:

• Audience: Small and mid-size government contractors (and similar vendors) who need to discover solicitations and judge fit without a full-time capture team.
• Today’s build: An authenticated web dashboard: sign-in, personalized RFP list, detail tabs (Overview, Source PDF, AI analysis), save to profile, and company profile. Data comes from Supabase (not static mock JSON).
• Success for this session: We learn whether they can orient, scan, and decide “would we pursue this?” using the UI — not whether they already know govcon jargon.

Invite think-aloud: say what they’re looking for, what surprises them, and where they’d give up.

---

2. Environment and logistics

Item	Action
Primary URL	Paste your production or stable Vercel URL here before the session: ________________________
Backup URL	Optional second preview if production is down: ________________________
A/B variant URL	Optional second deployment with different NEXT_PUBLIC_LIST_CARD_LAYOUT (see §4 and web README): ________________________
Display	Print this wiki page or show it full-screen next to the test machine (per course instructions).
PostHog	Confirm the same deployment URL sends events (see §5 and PostHog event taxonomy). Spot-check Live events before the first tester.
Tester accounts	Do not put passwords in the wiki. Pre-create 3–4 Supabase Auth users and share credentials via Slack / 1Password / paper handout only. Turn off mandatory email confirm for a dev project if sign-in must be frictionless in the room.

Privacy: Session replay (if enabled in PostHog) may record the screen. If you turn it on, tell testers once before they sign in.

---

3. Setup — lightweight directions + one pre-baked path

Time on rails: ~4–6 minutes structured, then optional free exploration.

1. Sign in with the persona credential we give you (e.g. “new contractor” vs “heavy saved list”).
2. Open Profile (header), change one field, click Save.
3. Close the profile. Pick one RFP from the list.
4. In the detail panel: read Overview, open Source PDF (if it loads), open AI analysis, click Generate summary once and confirm the result appears under Overview (Generated summary RAG), then Save to profile (or unsave if already saved).
5. Switch to Saved in the header nav and confirm the opportunity appears (or disappears if removed).

Then: “For the next few minutes, explore however you like — filters, search, walkthrough if you notice it — and keep thinking aloud.”

---

4. Test matrix (maps to course expectations)

Area	What we want to learn	Tester tasks
Basic functionality	Auth, nav, persistence	Sign in/out; Dashboard / Saved / History; search and filter sidebar; save and unsave; profile save.
UI A/B (list cards)	Does fit score prominence change trust or scan speed vs headline-first cards?	Half of testers use deployment A (NEXT_PUBLIC_LIST_CARD_LAYOUT=score_first), half B (headline_first). Same timed task: “In 90 seconds, pick the one opportunity you’d most want to pursue; say why in one sentence.” Optional follow-up: “What did you look at first on each card?”
Core value	Does this help a real bid/no-bid?	“Name one pursuit and one hard pass” using Overview (including generated RAG summary if they ran Generate summary) + PDF + AI analysis.
Fast-forward / power user	Clutter, scanability with history	Use a login seeded with many saved_rfps and past projects in Supabase. Task: find a needle in the haystack; use Saved nav.
Performance	Cold load, PDF embed	Optional: two people on same Wi‑Fi; note PDF iframe blank vs “Open in new tab”.

A/B assignment cheat sheet (print):

Tester #	Name	URL / variant
1
2
3

---

5. Feedback mechanism (primary: PostHog)

We use one primary medium for behavioral data: PostHog (see web/README.md and implementation in lib/analytics.ts). Qualitative “why” still matters for grading: add a short verbal debrief (moderator writes 2–3 quotes) and/or GitHub issues labeled user-test, linked from User Testing Results.

PostHog event taxonomy

Events include a property list_card_layout (score_first | headline_first) when the key is set.

Event	When
auth_sign_in_success	Successful password sign-in
auth_sign_in_fail	Failed sign-in (code + message; no password)
auth_sign_up_success	Sign-up with immediate session
auth_sign_up_fail	Sign-up error
auth_sign_up_pending_confirm	Sign-up succeeded but email confirm required
auth_submit_exception	Unexpected error in auth form
rfp_selected	User selects an RFP in the feed
detail_tab_changed	Overview / Source PDF / AI tab
rag_summary_requested	User clicks Generate summary
rag_summary_cached_hit	Cached summary loaded from DB
rfp_save_toggled	Save or unsave (now_saved true/false)
profile_saved	Profile form saved
main_nav_changed	Dashboard / Saved / History
walkthrough_started	Tour started
walkthrough_completed	Finished completion screen
walkthrough_dismissed	Closed tour early (X or equivalent)

Post-sign-in, posthog.identify uses the Supabase user id (no email sent to PostHog from this instrumentation).

---

6. Intervention policy

Default: Moderators do not give UI hints for the first 3 minutes of structured tasks unless the tester asks or is blocked on login (then help once so the session isn’t wasted).

If someone is stuck on a non-auth dead end for 2+ minutes with no learning value, you may intervene with a single nudge — note “moderator hint at mm:ss” in your debrief doc.

---

7. Moderator checklist (day-of)

• PostHog live events firing from the deployed URL
• Tester credentials distributed off-wiki
• Printed plan or second display
• A/B URL table filled in
• Optional: session replay enabled/disabled per team decision
• After sessions: update User Testing Results with themes, quotes, PostHog links, and GitHub issue links

---

8. Seeding tester personas (team ops)

Not pasted in this wiki: passwords. Do the following in Supabase (or SQL migrations):

1. Create Auth users for personas: e.g. tester-new, tester-power.
2. Ensure matching contractors row (user_id) exists (the app auto-creates on first login if missing).
3. Power user: insert many saved_rfps for that contractor_id; add several contractor_past_projects rows.
4. New user: minimal saves, thin profile.

Track which persona is which on the printed credential sheet only.

---

9. After testing (course deliverable: actionable results)

1. Groom findings into GitHub issues (bug, enhancement, user-test).
2. On User Testing Results, add: date, tester count, links to PostHog insights/funnels/screenshots, and links to issues.
3. Tie high-impact items back to the PRD (Google Doc sections) in issue comments.