PilotKit

Part IV: User Research 👀

Why User Research

aResearch is a powerful tool for challenging assumptions and testing hypotheses.  If you’re excited about running a pilot, consider what you’d accomplish by first running a quick research sprint.  Research can stand alone without a pilot (and often should!), can be run in preparation for a pilot to refine hypotheses, or can be run concurrently (or after) a pilot to capture qualitative and “quantish” insights.  
The process is intentionally broad. Trust the process! If you feel uncomfortable, it means you’re doing it right.

What are the steps to research and how much time do I need?

Design and Planning (2 hours)

  • Bring the team together and define a) assumptions, b) desired outcomes, and c) both a design challenge and research objective.
  • Walk away with a week-by-week calendar of what you plan to accomplish.

Recruiting  (2 weeks)

  • Define target demographics, cast a wide net through different recruiting channels and run potential candidates through a screener. This process always takes longer than you anticipate. Dive in headfirst!
  • from recent recruiting efforts

Field Guide (1 week)

  • Using your assumptions, information from expert interviews, insights from the recruiting process and other context, compile everything you need to be organized in the field. The core document is a script that eases your candidate into a conversation. It also includes defining compensation, compiling release forms, camera and recording devices (if needed), and a variety of other tools to help facilitate conversations.
  • from recent research.

Interviews (1-2 weeks)

  • Long-form ethnographic interviews are a powerful tool for testing assumptions. The best interviews are not representative, but test extremes. They should be 90-120 minutes of uninterrupted time in a quiet place. 
  • Your note taker(s) should be prepared to capture as much as they can verbatim. As soon as possible after each interview “download” the key quotes and observations.

Synthesis (1-2 weeks)

  • The output of synthesis is a set of insights. These sentence-long observations are the hardest you’ve worked on a single sentence. They should be short, specific, and evoke questions.
  • from recent research.

What other research tools can I draw on?

Diary Study

  • Prompt a small group of participants to respond to a set of questions once or twice a day. These questions should aim to capture rich media -- video, photos and audio -- which help fill in gaps you wouldn’t capture in a single interview.

Behavior Displacement

  • Often following a diary study or a ‘treatment’ group in contrast with a diary study, a behavior displacement asks participants to exclusively use a product. You should feel comfortable making your participants uncomfortable and compensating them fairly for the trouble.

Immersion

  • Spend time with your potential users! Work alongside them, eat with them or if relevant and appropriate arrange a homestay. Take detailed notes as you work to see the world through their eyes.

Surveys

  • Surveys are a powerful tool for gathering information quickly from a large group of people. They are particularly useful if you have a very specific or quantifiable question that needs to be answered. However, surveys have drawbacks. It is difficult to achieve the level of depth in a survey compared to a 1:1 conversation. In addition, it is important to be very careful with designing surveys and wording questions, as surveys are often susceptible to bias, particularly if there are cultural barriers.
After you select your pilot participants, it’s important to learn a bit more about them! The onboarding survey can be a bit longer and will give you some more insights into who your pilot participants are. Knowing more about your pilot participants is helpful for several reasons.
  1. It helps to get an overall understanding of the demographics of your pilot participants. This will tell you if your group is representative of the population you hope to serve.
  2. Also, understanding the individuals in your study might help you identify interesting folks that you may want to pay special attention to. For instance, we always like to ask about prior cryptocurrency experience among pilot participants so that we can make sure to interview folks who have already dabbled in crypto.
Example:
  • is an example onboarding survey we used in a pilot in Kenya
  • we used to track participants. We create a unique tracker for each pilot we conduct

Exit Survey

Exit surveys are a great way to get feedback on the product and pilot experience from users now that they’ve gone through the process. You might want to ask about issues, misconceptions, and net promoter scores to get a sense of how successful the pilot was. In addition, you may have uncovered interesting facts during the interview process or through observations that you can add some quantifiable meat towards in an exit survey. 
We typically send out the exit surveys on the last day of the pilot or immediately after - waiting too much longer means that you will likely have a steep drop off in response rates.
Example: is an example exit survey that we’ve recently used.

Resources From Recent Research Projects

External Links

- Highly recommended