When beta testing your app or website with real users, you’ll want to save your brand experience questions for the end of the session. Here’s why.
What are your users’ impressions of your product? How do they feel about it?
Do they think of your product as fun? Corporate? Simple? Friendly? etc.?
These types of questions relate to your users’ brand experience, which is something you can verify.
Your users’ brand experience with your digital product includes their attitudes, feelings, emotions, and impressions of that product. (Brand experience also affects—and is affected by—external interactions, your larger brand, and your company as a whole. But for this article we’ll focus on individual products.)
Brand experience is generally important, though it may not be a high priority for every project or every company. Your product team, however, might find it important enough to directly and specifically evaluate. This might be because:
- You want to collect brand-related feedback and experience data for future consideration; or
- You have specific goals regarding brand-related attributes you need to verify are being met, so that your product can be successful.
Early testing vs beta product testing
If it’s vital to your business goals that your product elicits particular reactions from your users—then you are surely designing your product with the intent of giving those impressions and drawing out those emotions.
But you can’t just rely on your intent. You need to test your product to see if real users’ interpretation of your brand matches your intent.
Versions of such testing can occur early and late in the project lifecycle.
Your visual design is key to your users’ first impression of your product.
Fortunately, you can evaluate your users’ brand impressions rather early in a project, before your team writes a line of code. Once you have high-fidelity design mockups, or a high-fidelity prototype, you could, for example, run 5 second tests with your users to verify that your design makes the right first impression.
Beta testing (or later)
Within your product, your visual design is not the only thing that makes a brand impression on your users. The rest of your design—navigation, animation, interactions, etc.—as well as your content contribute to the user’s opinions and reactions as well.
Once your product reaches Beta, you have an opportunity to verify your target brand traits in a deeper way, after your users interact with your working product.
Emphasis on after.
Brand experience assessments at the end of beta sessions
In your beta test sessions, the best time to collect brand experience data is at the end… and only at the end.
Why at the end?
Users can get a pretty good brand impression from a simple glance at your visual design. But interaction with your product will affect your users’ brand perception as well.
For example: an interface whose visual design gives off strong trustworthy and professional vibes might see those brand traits eroded by wonky UI interactions and amateurish text.
Kathryn Whitenton of Nielsen Norman Group gives this more straightforward example: “a design that appears simple and welcoming at first glance can quickly become confusing and frustrating if people can’t understand how to use it.”
After performing some realistic tasks in your product, your beta test users will have formed a deeper impression of your product’s brand traits. In your beta testing, the cumulative effect of your product’s design and content is what you want to capture. You maximize this by assessing brand experience at the end of the test session.
Why not assess at the beginning AND the end?
The instinct to evaluate twice and compare the data is a good one. Knowing where the brand impression starts and seeing how it changes after your users performing beta tasks might help you focus in on what you need to improve.
But here’s why you shouldn’t assess at both the beginning and the end, at least not in the same session with the same user:
When you ask users to describe and/or quantify their reactions to your design, they will then consciously form and solidify opinions about your design. Those opinions—while not unmovable—would self-bias the user for the remainder of the test.
Thus, asking about brand experience at the beginning (or in the middle) is likely to affect the opinions you’d get from those same users at the end.
However, if you do wish to compare “both kinds” of data, you could instead:
- Split beta testers into two groups: those who are evaluated at the beginning, and those who are evaluated at the end; or
- Compare your end-session beta test data with your previous pre-beta visual design impressions data (assuming you have some); or
- Re-use some of your pre-beta visual design test participants in your beta testing. If it’s been a few months (for example) between the visual design tests the beta tests, those participants probably don’t remember many details of the earlier testing, and the impact of self-bias would be minimized.
One last thing: watch out for bugs
You’re testing a beta product, so of course there might be some bugs your users encounter. Keep in mind that bugs may affect you users’ impression of the product for a number of brand traits you may care about.
Factor the impact of beta bugs into your post-testing analysis, and/or retest with subsequent product iterations.
Learn about how BetaTesting can help your company launch better products with our beta testing platform and huge community of global testers.