Usability Testing & A/B
Overview
AI reduces the time cost of quantitative analysis and task measurement dramatically. It does not replace human interpretation of why users behaved a certain way. The most common misuse is treating AI test summaries as conclusions rather than starting points. An AI summary tells you what happened. Your job is to understand why and what it means for the design.
AI workflow
7 steps
- 01
Test Plan Generation
Claude
Prompt: "I am testing [product] with [user segment]. My design hypotheses are [X, Y, Z]. Generate a usability test plan with 5 tasks, success criteria for each, and the key questions we need to answer. Include a realistic time estimate per task." Review and adjust before sending to participants.
- 02
Unmoderated Test Setup
Maze
Maze runs unmoderated tests with completion rates, time-on-task, misclick maps, and AI-generated summaries. Free tier: limited monthly responses. For most fixed-bid projects, this is sufficient. Honest assessment: Maze AI summaries are good for quantitative patterns but miss emotional nuance.
- 03
Live Session Heatmaps
Microsoft Clarity
Free, unlimited heatmaps and session recordings with AI-generated session summaries. Best for post-launch monitoring. Install early - data compounds over time. No response caps, no cost. The AI summary feature identifies sessions with rage-clicks and confusion patterns automatically.
- 04
Pre-Launch Predictive Testing
Attention Insight
Attention Insight uses AI trained on eye-tracking data to predict where users will look on your design. Use before launch to identify if critical CTAs and key content are within the predicted attention zone. Honest: paid-only, but can save one full testing round on high-stakes projects.
- 05
Session Recording Analysis
Microsoft Clarity + Claude
Export session summary data from Clarity. Paste patterns into Claude: "Users in 40% of sessions showed rage-click behaviour on [element]. What are the 5 most likely causes? What design changes would address each?" Use this as hypothesis generation, not diagnosis.
- 06
A/B Test Hypothesis Creation
Claude
Prompt: "Here are two design variants for [screen]. Generate 3 A/B test hypotheses, each with: the change being tested, the expected outcome, the success metric, and the minimum sample size needed for 95% confidence." Use a sample size calculator to verify Claude's estimates.
- 07
Findings Synthesis
Claude + Notion AI
Paste anonymised test observations into Claude: "Categorise these usability observations by severity (critical, major, minor). For each critical issue, suggest one design fix. For major issues, suggest two alternatives." Structure the final report in Notion AI for stakeholder delivery.
Tools
| Tool | Use | Free tier | Client-safe |
|---|---|---|---|
| Maze | Unmoderated testing, AI summaries, task metrics | Limited free responses/month | Yes |
| Microsoft Clarity | Free heatmaps, session recordings, AI summaries | Completely free, unlimited | Yes |
| Attention Insight | Predictive pre-launch heatmaps | Paid only | Yes |
| Claude | Test plan, hypothesis generation, findings synthesis | Free tier available | Yes |
| Loom | Participant think-aloud recording | 25 videos free | Yes |
Constraints
4 constraints
- Honest Assessment: Maze free tier is very limited. For more than 2 tests per month, UserTesting or Lookback are more appropriate but significantly more expensive.
- AI test summaries are pattern-detectors, not insight-generators. "40% of users clicked the wrong button" requires human interpretation: Was the label confusing? Was the visual hierarchy wrong? Was the mental model mismatched?
- Statistical significance: For A/B tests, you typically need 1,000+ users per variant for reliable results. Small sample tests generate directional signal, not decisions.
- Rage-click and dead-click data from Clarity is invaluable for audit and redesign projects. Install on all new projects from day one.