Cognitive Walkthrough and Heuristic Evaluation in the Contemporary Design Process

By Liya Zheng
Published - 6 August 2007

The craft of User Experience Design involves many different expertise working together. Some of us work in small teams with generalist backgrounds, others with larger teams of specialists. Regardless of team structure, some age-old usability methods can still help enhance your contemporary Interaction Design work.

In this article, you will learn:

1. How to incorporate techniques from the traditional Cognitive Walkthrough method along with Context Scenarios to get to solid design concepts quickly.
2. To combine a variation of the Heuristic Evaluation method with Key Scenarios to help you move through refining your design.

Cognitive Walkthrough helps pick the right concepts

Traditionally, a Cognitive Walkthrough (CW) or Usability Walkthrough is a task-based evaluation method done by a team of usability experts to identify flaws in a design. The walkthrough is designed and structured by first finding answers to these questions:

1. Who will be the users of the system?
2. What tasks will they perform on the interface?
3. What is the correct sequence of actions?
4. What does the user expect to see at each step?

Evaluators walk through each path of each task on a prototype to identify where the interface does not meet user expectations. Design flaws are then documented to use as change drivers. The material and analysis work often produce volumes of documentation. The traditional methods take a "fix it" approach rather than one that inspires innovative concepts. If you already know something about Cognitive Walkthrough, why don't you try doing a variation of a it during your ideation stage? Here is how it works.

First, prepare your Walkthrough with a context scenario. A context scenario is a day in the life of your persona. It does not capture the persona's day exactly as it happens currently. Rather, it captures what your persona's day would look like in the "magical" world where he or she is using the product or service you are about to design. Coming up with the context scenario provides a structured way for you to brainstorm your design concepts. It also helps you to role-play your personas and keep their goals in mind. You do a context scenario in enough detail to help you identify the functional components in your design. This should help you, the designer, work out ideas in your head before you start drawing. Verbalizing ideas before drawing them often helps you exercise the discipline of putting "the value of your design" before aesthetics.

Here is a snapshot of a context scenario we did for a sleep technologist software suite:

Donna is a sleep technologist. Her context scenario looks like:

"It's 7AM, Donna comes to work in the morning and the night technician, Benny, passes the studies onto her. They have a quick chat about what Benny thought of the patient's conditions based on his observation throughout the night. Donna puts on her scrub and logs into her SleepTech dashboard. She sees that she has 3 sleep studies to score and 2 patients coming in the afternoon for a sleep study. Also, the doctor has left her a note about setting up appointments with new referrals.

Looks like a busy day! Donna puts down her coffee and starts to score the studies. The first study was an obvious one, a quick look at the patient's medical record and a 2-minute scan of the whole study shows that the patient definitely needs a CPAP. However, she does the due diligence to score the events and puts notes together for the doctor to sign off on. Though she has scored hundreds of studies, it is important to her that they are consistently accurate.

Just when she was about to move onto the next study, Donna got a phone call from a patient to reschedule her appointment this afternoon. She switches to the calendar view and reschedules the patient. She sighs as her day opens up! She retrieves another study to score. The phone rings again and it's night tech Jane. She's not feeling well so she can't come in tonight. Donna quickly calls one of the many part-timers, Nick. He agrees to come in. Donna pulls up the Lab Scheduler and puts Nick in the slot. Donna goes back to scoring her study..."

Donna's Functional Components (from this part of the scenario):

• A technician dashboard.
• An automated study scorer with event highlights.
• A multi-view, patient story mode.
• Lab Management tool.
• A seamless workflow between technicians and doctors.
• Easy multi-task flow: Scheduling tools, Lab Management tool, Scoring tool...
• ...

Visualize the functional components and do a walkthrough at the same time. At this point, you have a very specific set of functionality to work with and a context scenario to help guide you. You are ready to illustrate the experience. If you work in a team, one person can be in charge of the design and the other in charge of doing the walkthrough. However, if you have to work alone, you can do this with some discipline. Once you put a concept on the board, check the concept against the scenario. Give it time to live but pose questions like "Would Donna need to switch between the Scheduler and the Lab management tool or are they separate utilities?", "What is the posture of the Scheduler versus the Scoring Tool?","Would she see the summary first then the details of the study or would she need to see them simultaneously?" Sometimes digging deeper, these problems may actually cause you to rethink your context scenario. Also, think about business goals and how the concept helps with that. "The concept serves Donna's needs, does it meet business goals?", "What about the design would be of concern to business stakeholders?".

When you are pushing hard to come up with innovative concepts, it's easy to get derailed by the cool ones that may not serve the user or business needs. Your design partner can use the scenario to bring you back on track. In the case of Donna, we were able to use this method to derive a concept that helped the client, a medical device manufacturer, to see how to expand their business into the becoming a service provider to existing clients and generate new revenue streams.

Check your walkthrough results with SMEs and Stakeholders. A good way to find out how successful your concepts are is to walk through them with your SMEs and Stakeholders, early. Many designers work in specialized domains like the one described here and Subject Matter Experts can help you move along your concepts with more certainty. Bring them in early to see your whiteboard sketches. Showing half-baked concepts and getting early feedback is better than carrying on to the end only to find out it doesn't work. Having raked through many concepts with your partner's help, you should be well positioned to ask SMEs and stakeholders feedback on the big ideas and solicit their opinion based on their domain knowledge.

Using the principles behind Cognitive Walkthrough in the concepting stage is a good idea because:

• We are able to have usability checkpoints without slowing down the product development process.
• It helps you judge a good concept from bad one before too much effort is put into the details of the design.
• It helps your team to build a library of answers to "why did you do it this way"?
• We turned usability techniques into an inspirational instrument to a designer, making the partnership between Interaction Design and Usability more fruitful towards pragmatic product design creation.

Heuristic Evaluations helps you iterate quickly in the Refinement phase

Jakob Nielsen promotes the use of Heuristic Evaluations as a discount usability evaluation method. The method involves a team of usability practitioners doing a structured evaluation on an interface with set heuristics. It is conducted very much like a user test with an evaluator using the system and an observer taking notes. One key difference is that the evaluator is using Nielsen's 10 heuristics to evaluate the interface. This was useful in the old days when we were struggling to integrate usability into the software development lifecycle . Today, however, we can re-purposed the method to be used in our design process while we refine our concepts.

First, create key scenarios. In the traditional Heuristic Evaluation method, they found success when supplying evaluators with usage scenarios. Here we advocate doing key scenarios on the whiteboard with a partner. A key scenario hones in on one part of your persona's workflow. Often, it is a step by step illustration of how your persona will use your functional components or a few related ones. It illustrates how your product or service will work in detailed steps. The brainstorming exercise to come up with the key scenario helps you think through the design problem and make conscious decisions about how your persona's experience will work.

For example, one of Donna's functional components is "A multi-view, patient story mode.". We did a key scenario like this:

Key scenario: "Donna scores a sleep study."

"Step 1 - Donna comes to work in the morning and the night technician passes the studies onto her. She logs into her computer and sees those studies on her dashboard.

Step 2 - She opens the first study on her list. Even if she shares a computer with the other technicians, her interface is exactly the way she likes to arrange things.

Step 3 - The study opens up with an patient overview that has the patient video, visual and numeric summary of events, and related patient history highlighted.

Step 4 - The machines always scores when the patient gets up for the restroom as a big event, Donna clicks on the false events and says "clean up".

Step 5 - This patient had a lot of events, she only needs a certain number of them to qualify the patient for treatment. She clicks on "Score mode" and sees the waveforms along with machine scores.

Step 6 - She looks at the system generated events list alongside the raw data, and the video of the patient, so she can double-check it's accurateness.

Step 7 - She scrolls through with her mouse and the video is synced to the scrolling on the channels. She sees what the patient is doing next to the spike charts on the channels. Donna leaves the ones she agrees with and corrects some of them.

Step 8 - The patient never fell into REM sleep, she put a note for the doctor.

.... Step 14- Satisfied with her scoring, Donna quickly sends the a report to the doctor's queue for sign-off..."

Identify the "heuristics" you will use to help you stay objective. If you have to design alone, having a set of heuristics in your head will help you be more objective when trying to choose between ideas. If you have a teammate, it helps you to be on the same page so the critique is not about "I don't like it this way". Having a shared appreciation for the heuristics being used can help the team to iterate even faster because you can discuss the quality of a design based on reason rather than preference.

Jakob Nielsen has a list of 10 heuristics that serves some situations well, but it doesn't have to be that cut and dry. It can be as simple as "is this the most simple way to solve this complex problem?" Heuristics can be based on principle, practice or a specialized domain. In "About Face 3.0", Alan Cooper, Robert Reimann and David Cronin offers a set of 50+ design principles based on their years of practice in interaction design. In “The Laws of Simplicity”, John Maeda gives us 10 laws to helps us achieve simplicity nirvana. In "Designing Interfaces", Jennifer Tidwell gives us a survey of effective Interaction Design patterns. When you are designing for people who work in highly specialized fields and domains, certain heuristics already exist in those domains. In Donna's example, she works in a hospital setting, certain colors like yellow and red carry very specific meaning and thus must be respected in our design.

Design with the key scenario on drawing board and think aloud. In a traditional HE, an evaluator uses the product while they're doing the evaluation. Ask your partner to help you flesh out the design on the whiteboard while they roleplay your personas use the key scenario as your starting point. Instead of evaluating your design, have your partner help you think through the design by posing questions to help you think through your reasoning. This collaboration works best when you are working on a shared space like a whiteboard. Of course, you will render the designs on a pixel level later, but all of the interaction problems should be solved by the time you do that.

If you have a partner to collaborate with, think aloud while you design to expose your thinking. "Donna can use a split screen view when she's looking through the machine-scored study, let's give her a summary event list that she can click and correct, she can base that on the channel snapshop next to it!" Illustrate the idea while you think aloud. Your partner should be offering you help on the spot and helping you structure your reasoning by posing questions like "Why a split screen? Is that the best way to show the summary?" However, it wouldn't be helpful is they say "I have a better idea..." which turns the collaboration into a competition. Or if they say "The split screen doens't make sense because...", that just puts a mental block in your head. Doing HE on the spot can be tough unless you and your teammate have similar philosophies in design and if you have ground rules set. Having benefited from having a design partner, I can confidently say that given the right person on the team, the effort to work out collaboration details outweighs the pain of designing alone.

If you are not lucky enough to have a design partner to help you through this process, design one or a couple steps in the key scenario, then pause and pretend it's someone else's design and evaluate it. Again, that's not easy.

Why do Heuristic Evaluations in the refinement phase?

The devil is in the details. Designing with heuristics helps us flesh out details in the design refinement phase:

• Doing evaluations while the design is happening saves time because usability problems uncovered are fixed while the design is being fleshed out rather than afterwards. Problems worth noting can be documented in the design specification so your development team will understand why decisions were made.
• We can rapidly iterate through design ideas without the added responsibility of documenting and reviewing the evaluations with a team of usability experts.
• It ensures that when a design is publicly shown, it is already been through the usability gauntlet. When the design is shared with stakeholders and other team members, the review can focus on other aspects of the design and leverage participants’ expertise like development feasiblity, business viability or Sales concerns.

So, Interaction Designers, User Experience professionals (or whatever you call yourselves), usability is not dead, you just have to find its place in your process. Some fundamental principles of age-old methods can still help guide us get to that perfect design solution.

References:
[1] "Communicating Design Concepts without being skewered" by Steve Calde.

[2] C. Wharton et. al. "The cognitive walkthrough method: a practitioner's guide" in J. Nielsen & R. Mack "Usability Inspection Methods" pp. 105-140.

[3] "Heuristic Evaluation" by Jakob Nielsen.

Credit and Contributions:

I owe it to the design team at Liquidnet, everyday, in helping me become better at my craft. In particular, my friend Arti Acharya contributed to this article as my supportive design partner, patient friend and a vigilant usability practitioner. However, any disagreeable qualities in my expressions here and elsewhere are all owed only to my own limitations and shortcomings and not theirs!

 

Next article: How To Deliver Bad News To Customers

Back to Top