Usability Testing Playbook

What is a Usability Test?

A usability test measures the ability of someone to complete the tasks required to achieve a goal. For designers of government services this could be navigating a website to register for a class, learning how to apply for a permit, or filing a police report. Usability testing puts us in the shoes of someone who is using our services, so that we can understand and continuously improve the user experience.

Why Do We Need Usability Tests?

Our aim is to make City services easy to use, and work for everyone. Usability tests help reveal the successes, pain-points, and thought-processes a user experiences when trying to navigate a government service or digital platform. We use information from testing to make improvements to the design and functionality of a service or product so it is better adapted to the needs of our users.

How Do We Conduct Usability Tests?

A usability test requires a tester, a moderator, and a script for the tester to follow. Follow the five-step process below to get started.

Step 1: Define Your Goals | What are you testing?

What product or service are you developing or improving that needs to be tested? Select typical tasks and scenarios that a person would follow when engaging with the service/product. The test will help determine how easy or difficult it is to accomplish.

Examples:

A parent wants to rent a picnic table at a park for a birthday party.
A new business owner wants to apply for a business license for her bakery.
A resident wants to remodel her kitchen and bathroom.

Step 2: Find Your Testers | Who are you testing?

Find volunteers that meet a profile that best fits the scenario you are trying to test. Use social media and email promotions to find volunteers to test your prototype or product. Ask your colleagues if they know any experts or regulars who would be a good fit. Consider testing at least 5 users to get a good sample size. Tests typically take an hour per session and are usually done virtually.

Step 3: Create Your Test Script | How are you testing?

The test script provides structure to your test and will help you run each session consistently. The script should include:

Introductory questions to provide context for the test and help make the tester comfortable
- “How do you typically find information about City programs?”
- “Have you planned an event that required working with the City?”
- “What was that experience like for you?”
User scenarios and tasks
- Scenario: Your daughter is turning 5 and you have decided to throw a birthday party at Gerstle Park.
- Task: Navigate to cityofsanrafael.org and reserve a picnic table online.
- Scenario: Your daughter is getting married and she asked you to help plan a wedding at Falkirk Mansion.
- Task: Navigate to cityofsanrafael.org and make a reservation for the Falkirk Mansion for her wedding day.
Questions about the experience
- “How would you describe the experience of renting with the City?”
- “What changes would make the process easier for you?”
- “Would you recommend using the City for an event in the future?

Step 4: Run Your Test Sessions | It’s time to start testing!

Ask permission to record the session and try to make the tester comfortable before getting started.
Assure that you are testing the product and not the user to help ease any discomfort around technology.
Remind him or her to be honest with feedback and to talk through challenges and thoughts out loud as they perform each task.
As the moderator, you should remain silent during each task and refrain from injecting your opinion throughout the process.

Your goal will be to test whether the subject successfully completes the task and measure how long it takes. Success ranks as 1.0 point, failure to complete a task as 0. Partial success can be ranked on a range from 0.1 to 0.9.

After testing is complete ask the tester your follow up questions about the experience.

Step 5: Review & Summarize Your Findings | What did you learn?

With testing complete take time to analyze and summarize the results of your tests. Find insights you gained, areas of improvement and recommendations to take to the product team. It helps to have a deliverable report that summarizes data, observations, and actionable steps the team can take to build off of the outcomes of testing.