Usability testing: types and how to run them

What are usability tests

Usability tests are structured sessions in which real people — who represent the target user — attempt to complete specific tasks on a digital product while the team observes their behavior and collects their comments. The goal isn't to ask if they like the design: it's to observe whether they can use it.

The distinction matters. When you ask someone "does this website seem easy to use?", the answer is subjective and influenced by social desirability. When you ask them to "try to buy the product in size M in blue and proceed to payment", you observe what actually happens: whether they find the size, whether the filter works, whether the add-to-cart button is where they look for it.

Jakob Nielsen demonstrated in 1993 — and the evidence since then has confirmed — that with just 5 participants you detect approximately 85% of the most serious usability problems. You don't need hundreds of participants to draw actionable conclusions.

Why usability tests impact conversion

Usability problems are direct causes of abandonment. A user who can't find the purchase button, who doesn't understand what each pricing plan includes, or who receives an unintelligible error message at checkout won't convert — and in most cases won't try again.

A Baymard Institute report analyzed more than 40,000 checkout sessions and found that 69.8% of shopping carts are abandoned. Among those causes, a significant portion are usability problems: long forms, mandatory account creation, unclear error messages, or a lack of security signals at the point of payment.

Usability tests identify these problems before they cost sales. And when they're already costing sales, they allow you to diagnose exactly where and why.

Types of usability tests

By participation modality

Moderated tests

A researcher guides the participant through the session in real time. They can ask follow-up questions, ask the user to think aloud, and probe unexpected behaviors. This is the modality that generates the most qualitative insight.

Ideal for: exploring complex problems, understanding the reasoning behind behaviors
Limitation: requires more preparation and analysis time, higher cost per session

Unmoderated tests

The participant completes tasks autonomously, without the researcher present. Sessions are recorded and analyzed afterward. Platforms like Maze, UserTesting, or Lookback allow running this type of test with dozens of participants simultaneously.

Ideal for: validating specific hypotheses at larger scale, when time is limited
Limitation: you can't probe unexpected behaviors during the session

By environment

In-person tests

The participant and the researcher are in the same physical space. This allows observing body language, facial expressions, and behaviors that aren't captured on video. They're especially useful when the product involves physical interactions or when the target population is difficult to recruit online.

Remote tests

The participant is in their usual environment — their home, their office — and the session takes place by video call or asynchronously. Remote tests have greater ecological validity (the user is in their real context) and make it easier to recruit specific profiles regardless of geographic location.

By objective

Discovery tests

Conducted in early phases, before the design is defined. The goal is to understand what problems users have with the current solution or with the process the product is meant to facilitate.

Validation tests

Conducted when a design or prototype already exists and the goal is to confirm whether it solves the identified problems and whether users can complete tasks without significant friction.

Comparative tests

Compare two or more versions of a design to determine which works better for specific tasks. They are the bridge between qualitative usability tests and quantitative A/B tests.

When to run usability tests

The short answer: before investing in development and before assuming a conversion problem is solved by more traffic.

Moment	Why to run a test
Before a redesign	To identify what's working before breaking it
During design	To validate flows before the code exists
After a launch	To detect unforeseen problems with real users
When conversion drops	To diagnose whether the cause is usability or something else
Before an A/B test	To generate hypotheses with a solid qualitative basis

A common mistake is reserving usability tests for major redesigns. The most valuable tests are small and frequent: 5 users per month generate more cumulative learning than 20 users once a year.

How to plan and execute a usability test

Step 1: Define the objective

Before recruiting participants or preparing tasks, define what you want to learn. The best tests have a narrow focus: "Can new users find and activate feature X without help?" is a better objective than "is the website easy to use?".

Step 2: Recruit representative participants

Participants must represent the real user, not the ideal user. If your product is for people over 60 with little digital experience, don't test with 30-year-old web designers. Poorly executed recruitment invalidates test results.

For tests of products with a broad audience, 5–8 participants per segment are sufficient for qualitative tests. For quantitative tests (usability metric benchmarking) larger samples are needed.

Step 3: Design the tasks

Tasks must be realistic, framed in terms of the user's goal (not in terms of the system), and must not give hints about the solution.

Bad: "Click the hamburger menu to see the categories." Good: "You want to see what products we have in the running category. How would you do that?"

Step 4: Execute and observe

During the session, the researcher doesn't help the participant even if they're having difficulty. The difficulties are the data. The "think aloud" technique — asking the user to verbalize what they see, what they're looking for, and what they think — is the most valuable tool for understanding the user's mental model.

Step 5: Analyze and prioritize

After the sessions, the analysis must identify patterns (problems that appear with more than one participant) and prioritize them by severity and task impact. A problem that prevents completing checkout is critical. Unclear text in the returns policy is minor.

Severity	Criterion	Action
Critical	Prevents completing the task	Fix immediately
High	Causes significant frustration or prolonged confusion	Include in next sprint
Medium	Slows the task but doesn't block it	Priority backlog
Low	Minor friction, doesn't affect the outcome	General backlog

Tools for usability testing

Test type	Tool	Approximate cost
Moderated remote	Lookback, UserTesting, Zoom	From €0 (Zoom) to €500/month
Unmoderated	Maze, UsabilityHub, Optimal Workshop	From €75/month
Session recording	Hotjar, Microsoft Clarity, FullStory	From €0
Analysis and synthesis	Dovetail, Notion, FigJam	From €0

Usability tests and CRO: how to connect them

Usability tests and A/B tests are complementary, not substitutes. Usability tests explain the why behind quantitative data. A/B tests validate whether the proposed solution actually improves the metric at scale.

The most effective flow combines both:

Analytics data shows that 60% of users abandon at step 2 of checkout
A usability test with 5 users reveals that the "billing address" field causes confusion because it's unclear whether it's required when the shipping address has already been entered
A version is designed that simplifies or removes that field
An A/B test validates whether that version improves checkout completion rate

Without the usability test, the team could have designed any hypothesis for the A/B test. With the usability test, the hypothesis comes directly from observed user behavior.

To see which elements of your website may be generating friction today, start with an analysis at Scan&Boost. If you want to combine usability research with a conversion optimization program, learn about our CRO agency service.

Adrià Vidal, CRO specialist at Boost.