Just Enough Research

by katystreet

By Erika Hall.

I read this book a couple weeks ago, before I finished Seeing Voices, for work so that I could better run usability testing for our app. Because of this I took pretty extensive notes, which I’ve copied below. I really liked the book: it’s short and to the point and the author has just the right amount of sass. She talks not just about how to do user research, but how to do it given that you’re probably not a professional and how to do it in organizations that might not want you to do so. Very practical.

My notes:

A way to classify the purpose of your research:

  • Generative or exploratory: “What’s up with…?”
  • Descriptive and explanatory: “What and how?”
  • Evaluative: “Are we getting close?”
  • Causal: “Why is this happening?”

Setting roles, where roles represent a cluster of tasks. Single people can take on multiple roles or a role could be shared:

  • Author
  • Interviewer/Moderator
  • Coordinator/Scheduler
  • Notetaker/Recorder
  • Recruiter
  • Analyst
  • Documenter
  • Observer

Best practices:

  • Phrase questions clearly: not the questions you’re asking users but the questions you are trying to answer.
  • Set realistic expectations: what’s the question being answered, the methods being used, and the decisions to be informed by the findings.
  • Be prepared: makes everything go cleanly and quickly.
  • Allow sufficient time for analysis.
  • Take dictation: notes or it didn’t happen, and they need to be sufficiently informative for anyone who has to make decisions based on the research.

“To make the best use of your time and truly do just enough research, try to identify your highest-priority questions–your assumptions that carry the biggest risk.”

The process:

  1. Define the problem.
  2. Select the approach.
  3. Plan and prepare for the research.
  4. Collect the data.
  5. Analyze the data.
  6. Report the results.

Screening participants is important. You want to make sure they are the type of person your product/organization is looking to engage. You also want to make sure they will be an active and useful participant: not too terse but also not too distracted and verbose. You can use a simple survey and might want to follow with a quick phone call to see if they are a good match for your study.

Usability testing is basically a directed interview with a representative user while they use a prototype or actual product to attempt certain tasks. Usability testing tells you whether people understand the product or service and can use it without difficulty. It does not provide you with a vision, tell you if your product or service will be successful or which user tasks are more important than others.

A sample structure for an analysis session:

  • Setup
    — Summarize the goals and process of the research.
    — Describe who you spoke with and under which circumstances.
    — Describe how you gathered the data.
    — Describe the types of analysis you will be doing.
  • Analyze!
    — Pull out quotes and observations.
    — Group quotes and observations that typify a repeated pattern or idea into themes.
  • Synthesize
    — Summarize findings, including the patterns you noticed, the insights you gleaned from those patterns, and their implications for the design.
    — Document the analysis in a shareable format.

Do organizational research. Your work doesn’t happen in a vacuum, so interview stakeholders to come up with business requirements. Documentation should include problem statement and assumptions, success metrics, completion criteria, scope, risks/concerns/contingency plans, verbatim quotes (without identifying information), and workflow diagrams. “…improve the odds that changes will be fully understood and take hold.”

Assumptions are insults.

House M.D. was right: Everybody lies. (But don’t break into anyone’s house.)

Ethnography: getting to know people, generally by talking to them. Interview with an interview guide that has a brief description and goal of the study (share the with the interviewee but also use to keep yourself on track), any demographic questions you’ll want to ask, a couple of small talk questions (maybe just for inspiration!), and some questions or topics that are the primary focus. Remember, it’s just a guide. When interviewing, you know nothing. The most fascinating thing is the person you are interviewing. Gather a bit of background information (about the subject area, not the person) beforehand.

Contextual inquiry: getting to know people by following them around and observing them.

Jakob Nielsen’s checklist of ten heuristics (heuristic inspection) which can be done by anyone and should be done and scored as often as possible to ensure a basic level of usability. There is more information about these here.
– System status visibility.
– Match between system and real world.
– User control and freedom.
– Consistency and standards.
– Error prevention.
– Recognition rather than recall.
– Flexibility and efficiency of use.
– Aesthetic and minimalist design.
– Help users recognize and recover from errors.
– Help and documentation.

Find out everything you can using cheap usability tests first: paper prototypes before actual ones, tests in the office before tests in the field. It doesn’t really matter how often you do usability testing, as long as you do it earlier than right before you launch, when it’s expensive to fix things. The most expensive usability testing is the kind your customers do for you by way of customer support.

Ask users to do a task. Have a good facilitator who can put the user at ease but won’t lead them through the process. Take notes and record audio; be careful about videotaping because it’s trickier. Analyze the data by looking for issues that prevent the user from completing the task or just cause difficulty. Don’t forget to note how many users have that issue. At the end, rank the issues in terms of priority and get to work!

How to do qualitative analysis of your data: review the notes, looking for interesting behaviors, actions, emotions, and verbatim quotes. Write your observations on sticky notes and start to cluster the notes based on patterns. Watch patterns emerge. Rearranging is key. This is an affinity diagram. Use this to name patterns and identify user needs that come from them. The final step is to identify the actionable mandate or principle.

Personas: fictional archetypes that represent types of users and can be used to “advocate” for the user experience. Design, business strategy, marketing, and engineering can benefit from a single set of personas. HOWEVER, design targets ARE NOT marketing targets. The user type with the highest value to the business might be be the one most valuable to the design process. Personas should be fictional, drawing on many different users, but archetypal. Use a photo of a person no one knows. Give them a name, role, goals, demographic, quote, behaviors and habits, skills, environment, relationships, scenarios.

Mental models describe how users understand how things work. Strictly speaking there are also the mental models that designers sketch out to create a better world. You can make mental models through user interviews and affinity diagrams. Similarly you can make task/workflow analyses.

Quantitative analysis: split testing! It has math, which makes it more fun, but make sure you know what you’re looking for (set up your experiment and define you requirements and goals) and be careful about finding a local maximum-you can only do so much through incremental changes-and upsetting what is working for current users.

Conclusion: I’m just going to quote some lines from the conclusion of the book! To inspire!

“Questions are more powerful than answers. And asking often takes more courage than sticking with comfortable assumptions.”

“Cultivate a desire to be proven wrong as quickly as possible and for the lowest cost. If you work in a culture that prizes failing fast, there is no faster way to fail than by testing an idea that’s still on the drawing board. Exception maybe checking your assumptions before you even get to drawing.”

“Let curiosity be your guide.”

Aaaaand we’re done. Thanks Erika Hall!