Plan Your Analyses
You should have a plan for data analysis before you collect any data. Having an analysis plan up front helps you collect the right data to tell your stories, in a large enough quantity to yield meaningful results, and in a format that is useful and usable for you.
Here’s an example: Let’s say you are doing that blood pressure management app I’ve used as an example already in this chapter. You want your users to take their medication more often so they ultimately get their blood pressure numbers into the normal range. So, of course, you will measure how often they take their medication. Now, you know that people are likely to slip up with behavior change, especially when they’re first starting out. So you want your measurement to be sensitive enough to show small improvements, even if your users still aren’t taking their medication perfectly. Knowing that, how should you measure their medication behavior?
You could have them count the pills left in their bottle at the end of the month and then subtract that number from the original quantity. That would get you a number for each user representing the number of days they took their pills. Or you could ask people at the end of the month to answer one question: “Did you take all of your medication as prescribed this month? Yes or no.” That would get you a yes or no for each user.
Since you know you want to show small improvements and not just total success, the first option is much better. It will let you show changes to the average number of days your users took their medication in a month. A change from, say, 21 days to 23 days will be detectable with your data. If you’d asked the yes/no question, all of the people who have improved their medication behavior but not perfected it would look like failures, when, in fact, they are making positive progress. Having the more granular data will also let you look more closely at the people making progress to understand what’s working for them and where you might be able to improve the product to help them more.
The way you collect data has a huge influence on what type of outcomes story you can tell. Writing the story first and understanding what analyses need to happen for you to tell it will help you ask the right outcomes questions.
Chances are, you aren’t a stats whiz, and you’ll have some questions about how to do your analyses. Depending on the complexity of the project and the type of data you’ll collect, you may want to work with a colleague who has strong quantitative skills or even a data science professional. For simpler projects, you can find excellent resources on how to analyze different types of data for different objectives online.
Once you’ve specified your data, you have a complete outcomes logic map for your project. Figure 2.3 shows what that might look like for the hypothetical high blood pressure product that focuses on helping people take their medication more regularly.
Socialize this document with the other members of your team, especially anyone working on building or marketing the product. Don’t be afraid to tweak it as your product grows and you learn more about your users. As your product makes its way into the world, you can start associating actual data with the outcomes on your map. If they don’t support the story you want to tell, then you know it’s time to look at your product and make some changes. But first, let’s talk about where you can get the data to tell the story.
DIAGRAM BY AIDAN HUDSON-LAPORE.
FIGURE 2.3 A completed outcomes logic map that includes specific data and measurements for baseline, exposure, behavior changes, and long-term outcomes. For more complex products, the map will also be more complex.
NOTE CHANGE IS NOT FAILURE
It can be emotionally hard to get data that suggests something you designed wasn’t quite right for users. But informed changes to a product are a healthy part of the design process. It’s much better to learn early on that a feature doesn’t help people, so you can stop maintaining and iterating on it, than to continue to invest time, money, and resources on a dud. And early research can inspire new features that make the product. Did you know the first iPhone didn’t connect to an app store?
Evaluating for Effectiveness
The most important measure of whether a product is effective is that it produced the changes it was supposed to in users’ behavior. Effectiveness is a fancy way of saying, did the product work? Behavior change interventions differ from other sorts of digital products in that their ultimate intended effects take place off the screen in almost all cases. Constraining your measurement practice to the digital product itself means that you won’t be able to detect the real-world outcomes that are hopefully part of your results.
Fortunately, there are several ways to collect data outside of the product itself that will help you understand the product’s effects. This section doesn’t cover them exhaustively, but I’ve highlighted a few of the most common and effective methods of figuring out whether and how a product works. I’ve also focused on research methods that are less common in UX and design, knowing that many in-depth resources exist for the more typical research toolkit. These methods do require an investment of time and resources above and beyond actual product development and customer acquisition costs, but they offer a huge rate of return if your product is able to attract investors or users with their results.
It also helps not to think of effectiveness research as a one-shot deal. Over the life of your product, you’ll do many studies that each tell a piece of the story. In the interview at the end of this chapter, Cynthia Castro Sweet from Omada Health talks about how to layer research over time; it’s a good reminder not to feel overwhelmed by the idea of designing the perfect study.
The Gold Standard: Randomized Control Trials
Randomized control trials (RCTs) are the gold standard of outcomes testing. Realistically, most companies won’t have the resources to do RCTs on their product, but understanding how they work can help with planning more realistic, scaled-down studies that target similar sorts of understanding. RCTs are in a sense the most scientific way to see the effects of an intervention on people’s behaviors. RCTs are a way to look at both within-subjects and between-subjects changes by looking at changes in people who use a product, as well as comparing them to people who do not. They include three elements, handily summarized in their name: randomization of research participants, a control condition, and a defined trial period.
Randomization means that the people who participate in the study are randomly assigned to either use the intervention or use something else. The randomization is important because, with a large enough sample size in the study, it theoretically eliminates the explanation that it’s something about the person that leads to the results. In the real world, people often self-select into experiences or products. They pick the things they enjoy and avoid the things they don’t. But then if something really works for them, you don’t know for sure that it’s because the product is good. It might be that this person liked a crappy program enough to stick with it long enough to get results. So in an RCT, people are randomly assigned to the product they use.
Control refers to eliminating other explanations for any outcomes. People in the control condition do not use the product, but are assigned to use something else that is reasonably similar. Research has shown that just becoming more aware of a behavior can change it. To rule out the explanation that an intervention works only because people are paying attention, the control condition will also do something that increases their attention. In a sleep-coaching program I worked on, the control