Part IV: User Research 👀

Why User Research

Research is a powerful tool for challenging assumptions and testing hypotheses. If you’re excited about running a pilot, consider what you’d accomplish by first running a quick research sprint. Research can stand alone without a pilot (and often should!), can be run in preparation for a pilot to refine hypotheses, or can be run concurrently (or after) a pilot to capture qualitative and “quantish” insights.

The process is intentionally broad. Trust the process! If you feel uncomfortable, it means you’re doing it right.

Research Steps

What are the steps to research and how much time do I need?

Design and Planning (2 hours)

Bring the team together and define a) assumptions, b) desired outcomes, and c) both a design challenge and research objective.
Walk away with a week-by-week calendar of what you plan to accomplish.

Recruiting (2 weeks)

Define target demographics, cast a wide net through different recruiting channels and run potential candidates through a screener. This process always takes longer than you anticipate. Dive in headfirst!
Reference from recent recruiting efforts

Field Guide (1 week)

Using your assumptions, information from expert interviews, insights from the recruiting process and other context, compile everything you need to be organized in the field. The core document is a script that eases your candidate into a conversation. It also includes defining compensation, compiling release forms, camera and recording devices (if needed), and a variety of other tools to help facilitate conversations.
Reference from recent research.

Interviews (1-2 weeks)

Long-form ethnographic interviews are a powerful tool for testing assumptions. The best interviews are not representative, but test extremes. They should be 90-120 minutes of uninterrupted time in a quiet place.
Your note taker(s) should be prepared to capture as much as they can verbatim. As soon as possible after each interview “download” the key quotes and observations.
Shotlist

Synthesis (1-2 weeks)

The output of synthesis is a set of insights. These sentence-long observations are the hardest you’ve worked on a single sentence. They should be short, specific, and evoke questions.
Reference from recent research.

Research Tools

What other research tools can I draw on?

Diary Study

Prompt a small group of participants to respond to a set of questions once or twice a day. These questions should aim to capture rich media -- video, photos and audio -- which help fill in gaps you wouldn’t capture in a single interview.

Behavior Displacement

Often following a diary study or a ‘treatment’ group in contrast with a diary study, a behavior displacement asks participants to exclusively use a product. You should feel comfortable making your participants uncomfortable and compensating them fairly for the trouble.

Immersion

Spend time with your potential users! Work alongside them, eat with them or if relevant and appropriate arrange a homestay. Take detailed notes as you work to see the world through their eyes.

Surveys

Surveys are a powerful tool for gathering information quickly from a large group of people. They are particularly useful if you have a very specific or quantifiable question that needs to be answered. However, surveys have drawbacks. It is difficult to achieve the level of depth in a survey compared to a 1:1 conversation. In addition, it is important to be very careful with designing surveys and wording questions, as surveys are often susceptible to bias, particularly if there are cultural barriers.

Onboarding Survey

After you select your pilot participants, it’s important to learn a bit more about them! The onboarding survey can be a bit longer and will give you some more insights into who your pilot participants are. Knowing more about your pilot participants is helpful for several reasons.

It helps to get an overall understanding of the demographics of your pilot participants. This will tell you if your group is representative of the population you hope to serve.
Also, understanding the individuals in your study might help you identify interesting folks that you may want to pay special attention to. For instance, we always like to ask about prior cryptocurrency experience among pilot participants so that we can make sure to interview folks who have already dabbled in crypto.

Example:

Here is an example onboarding survey we used in a pilot in Kenya
Here is a template we used to track participants. We create a unique tracker for each pilot we conduct

Exit Survey

Exit surveys are a great way to get feedback on the product and pilot experience from users now that they’ve gone through the process. You might want to ask about issues, misconceptions, and net promoter scores to get a sense of how successful the pilot was. In addition, you may have uncovered interesting facts during the interview process or through observations that you can add some quantifiable meat towards in an exit survey. We typically send out the exit surveys on the last day of the pilot or immediately after - waiting too much longer means that you will likely have a steep drop off in response rates.