What is Wizard of Oz testing and how can it be used?

The Wizard of Oz method is a process that allows a user to interact with an interface without knowing that the responses are being generated by a human rather than a computer by having someone behind-the-scenes who is pulling the levers and flipping the switches.

This method of research gets its name from the story by L.F. Baum. In the story, the Wizard fools everyone by creating a vision of himself to look powerful by using a set of controls while he hides behind a curtain concealing the reality of the situation.

Wizard of Oz testing – The listening typewriter IBM 1984

The Wizard of Oz method allows researchers to test a concept by having one practitioner – the ‘Moderator’ – leading the session face to face with each user, whilst another practitioner – the ‘Wizard’ – controls the responses sent to the user via the chosen device. In the image above from IBM, we see an example for testing the concept of a ‘Listening typewriter’. The user is sitting in one room talking into a microphone while the ‘Wizard’ sits behind the scenes typing what the user is saying so it appears on the user’s screen as if it was done by the computer.

Why?

As UX researchers we often remind people to test systems at every stage of development, and that includes testing before development has even begun. This can save time, money and those ever so embarrassing moments when products are launched before they are fit and ready for users.

The Wizard of Oz methodology allows you to test users’ reactions to a system before you even have to think about development. This could be a new concept you are unsure will work for your users or a project that would require a substantial amount of effort to create, but you want to learn more before it makes sense to invest the time and money, and it cannot be tested with the usual prototype tools. Wizard of Oz is a flexible approach that allows concepts to be tested and modified without having to worry about potentially tiresome code changes, breaks in a daily testing schedule or full development costs.

How?

The easiest way to conduct Wizard of Oz testing is to build a simple and easy-to-use prototype that allows the ‘Wizard’ to quickly react to the user’s gestures or actions with the designed response with a single click.

Similar to any methodology, creating a Wizard-of-Oz prototype starts with having to determine what you want to test or explore. Then we need to figure out how to fake the functionality needed to give the user a realistic experience from their viewpoint. For example, you could prototype a jukebox without creating the mechanics and use a hidden person to play the selected songs to the customer.

Our experience with the Wizard of Oz methodology

We have used the Wizard of Oz methodology for clients from various clients including Arrow Global and EE and can tailor the approach to meet varying business requirements.

This methodology was used when testing interactive voice response (IVR) and short message/messaging service (SMS) systems. For IVR we saved development time and money by creating an Axure prototype with integrated voice files to playback to the user once they have selected the option that is most suited for their needs. This allowed our clients to see, in real-time, how users navigated their IVR systems, and which prompts had a negative effect on the user’s journey. This also allowed us to run a more complex project, trialling different versions of the same prompt within a single project to see which prompts were most effective for a specific audience.

We used ‘Amazon Polly’ automated voice generator to generate automated voice files that are quick to make and consistent. This accelerated the project process as there was no need to wait for voice files to be recorded by a human. It also saved costs of having voice files professionally recorded before testing showed which versions worked best.

Fill in the form below to speak to one of our insight experts to learn more about the Wizard of Oz research methodology and how we can help you succeed with your projects.