If a regular user test gives you a snapshot, then a longitudinal study provides you with a time-lapse video.
If you want a not-clever mnemonic for remembering what makes a test “longitudinal”, just focus on the word “long”.
A longitudinal study (or longitudinal testing) is a type of research that collects data from the same set of users on multiple occasions over an extended period of time.
The term “longitudinal study” is commonly used in psychology and sociology contexts, where tests might run for many years or even decades. But here we’re talking about software user testing, where longitudinal tests don’t run so crazy-long. After all, you’re not aiming to uncover new universal truths about the human condition to publish in a scientific journal.
You just want to validate and improve your software product. And the longitudinal study is a tool that can help you do that, in some ways that regular user testing cannot.
Longitudinal study vs. regular user test
User testing—whether it’s directed toward usability or user feedback more broadly—tends to occur over a very limited period for each participant. It’s often a single session. For the purposes of this article, we’ll call this “regular” user testing, and we’ll assume you are already familiar with the concept.
Regular user tests provide data about the user’s experience with your product during the test. Feedback focuses on the initial learnability of a product or feature, and the user’s reactions to it, typically while performing set tasks in a limited timeframe. These types of tests and the feedback they yield are extremely valuable. There are, of course, limitations on what these tests tell you.
Longitudinal studies, by contrast, last well beyond a single session, and collect a multitude of data sets from each participant over time. These two fundamental features of the longitudinal study —1) abundant time, and 2) multiple data sets—provide us the ability to discover more about our product and our users than we can with regular user testing.
Some advantages of longitudinal testing
A regular user test shows you a single “snapshot” of the user’s experience with your product. A snapshot can tell you what’s happening in the moment, but isn’t very good at indicating where things are headed. And if there are any twists and turns over the horizon? Forget it. You just can’t see them.
Longitudinal tests collect a series of snapshots to create a time-lapse video of a user’s experience (so to speak). The two fundamental features of the longitudinal study —1) abundant time, and 2) multiple data sets—provide us the ability to discover additional things about our product and our users.
Longitudinal studies can help you to find out things like:
- How users’ attitudes toward your product change over time
- Users’ usage patterns, and how they may change over time
- Long-term learnability and forgettability issues in your product
- How users handle the same tasks as they become more experienced with the product
- How users naturally use infrequently-used features
- How users handle long-term tasks that would not naturally occur in a single session
- How users’ preferences and activity changes after they add a critical mass of their own data
- Etc.
Flexibility
A wide variety of insights are possible due to an abundance of time and data sets, but also because the longitudinal study is a pretty flexible tool. You can get creative and design your longitudinal test however you want to get the data you’re after.
Your longitudinal study may use specific test procedures and/or general usage directives. You might periodically interview the participants, or observe test sessions, or neither. You may collect data on-site or remotely. You might instruct your participants to use the product only on a defined periodic schedule, or you might encourage them use the product as much as they want. You might have testers perform the same tasks at every interval, or you may plan out a progression of different test procedures over time. Data collected may include surveys, or maybe you have each participant keep a usage diary.
And so on.
Some disadvantages of longitudinal testing
Longitudinal studies take longer to produce conclusions than regular user studies do. Not a shocker. A longitudinal study is also, not surprisingly, more difficult and more expensive to execute—but not just because it requires more planning and data collection. Longitudinal studies also require additional work in selecting participants (particularly if you are recruiting participants yourself) and are sensitive to participant attrition.
Participant selection and participant attrition are two sides of the same coin when it comes to a longitudinal study. Getting a long-term commitment from potential participants is always going to be more difficult than securing limited, one-time participation. In a regular user test, you can make up for a no-show participant by adding another relatively quickly (if you decide it’s even necessary). For a longitudinal study you’ll need to recruit more participants than your ideal minimum right at the outset—enough to account for drop-outs. The longer the study, the more participants you’ll lose by the end. And it’s not just a quality-of-participants issue; there are innumerable reasons why good people may drop out of your study somewhere in the middle. Life just gets in the way sometimes.
It is also worth noting that participation in a longitudinal study may itself influence the behavior or attitudes of a user. This is not a concern that is particular to longitudinal studies. But keep in mind that your participants, intentionally or unintentionally, might act as they believe they are expected or obligated to act. And that may include continuing to use your product well beyond the point they would have stopped, or other aspects you may be trying to measure.
Learn about how BetaTesting can help your company launch better products with our beta testing platform and huge community of global testers.
BetaTesting Blog 