The challenges of testing prototypes with real users

Conducting usability testing with prototypes can be challenging as they are not fully functional websites or applications. The level of detail can range from simple paper prototypes to high-fidelity pre-launch prototypes, so how can you overcome these challenges to derive the most value from these tests?

What is a prototype? Why do we use them?

A prototype is an early sample, model, or release of a product built to test a concept or process or act as a thing to be replicated or learned from. UX researchers are used to evaluating new website journeys and designs and allow the product team to detect usability problems before implementing them into the live site or app.

What are the challenges when testing prototypes?

1. Visual design

Users can be distracted by a lack of visual design on a prototype because wireframes and other low-fidelity prototypes are very basic. This distraction can result in users commenting on the lack of design and colour, detracting from the goals of this project. The extent of this challenge depends on the level of detail within the prototype.

How to get around this: Ensure the user is aware at the start of a session that the website they are about to view is at an early stage of development, meaning it will not look and feel like they may expect. The researcher may need to be explicit with some users and directly state that visual design is not the purpose of that specific session.

2. Partial journeys

Prototypes often cover only partial user journeys, meaning that users may have to be dropped into a journey at a specific point and may lose the context of the overall task or what they would be coming on the site to do.

How to get around this: As well as creating tasks which set the context, consider including some time at the beginning of the session for users to explore the prototype, as they would normally do on that website/app, without giving them long enough to discover the prototype journeys. You may also wish to ask questions at the start of the session to position the user in the right frame of mind for what the prototype will allow them to do, providing context alongside the task wording.

3. Security concerns

Sometimes users may be required to enter personal or bank details during the testing session. As we know, users consider security to be important, and prototypes tend to lack a lot of visual design and thus may appear less safe than actual websites. This can cause users to become nervous when entering personal information or refuse to enter any at all.

How to get around this: Ensure users understand that the website is safe and any personal details they enter will not be stored, saved, or used to contact them. Of course, we can always be prepared with dummy details if they do not feel comfortable entering their own, but this might not produce the same results.

4. Click happy

Users may become frustrated when they attempt to click on parts of the prototype which are unclickable and tend to become ‘Click happy’. Instead of looking for the way forward in the journey, they start to look for any option that allows them to move forward, and we lose the chance to observe natural behaviour.

How to get around this: Explain to the user when something isn’t working. For example, the researcher should try to reassure the user that their behaviour is still helpful: "I’m sorry that’s not working for us today, I’ll make a note that you’d want to click there". Similarly, when introducing the session to the user, ensure the user is aware that the prototype is in development and that they may have to be patient with it to avoid anyone becoming click-happy.

5. Time

Testing on prototypes can take more time than testing the same journey on a live website. As well as the challenges noted above, there can be technical challenges such as those we discussed in the earlier article.

How to get around this: Pilot the study and test how long it takes for pages to load and how long it takes for someone external to the project to get through the journeys. Factor this into the session time. Having issues with a prototype and running out of time quickly cause significant disruptions in your testing.

The challenges of testing with prototypes can be frustrating for UX researchers and observers from the product team who want to see how a new journey or design works. It can be easy to become hyper-focused on the prototype journeys that we forget we can also learn about user expectations, language and understanding – all valuable insights that improve a journey design. So, prepare for the challenges, observe the sessions, take notes, be patient, and be prepared to make changes. Keeping these potential issues in mind will help you define your usability tests and understand the results better.

If you are looking to conduct user research to test a prototype or product, please get in touch with our friendly team.