-
Why Is Beta Testing Important?
Why is beta testing important now more than ever before? The product development lifecycle has changed dramatically over the past 10 years, and beta testing is no longer a phase that definitively ends shortly before product launch.
Product launch was traditionally the end of the product development lifecycle: (1) concept (2) design (3) build (4) launch. In this view, beta testing is something that lived and died in stage 3 (build and test). In contrast, with the rise of agile testing and continuous improvement, beta testing is more important than ever and can now be considered a process that not only occurs pre-launch, but also during the ongoing continuous improvement process prior to launching new features, design changes, or other product improvements.
Prior to launching a new feature or a new product, the aim of the alpha and beta phases is to steadily increase the probability that the product will succeed when it is launched. Continuous improvement is an ongoing effort and philosophy that continues in perpetuity with an aim to constantly evaluate and improve products over time. All of these test phases depend on feedback from real people, using the actual product in unique environments, though each phase is driven by definitive processes and objectives.
Sometimes, there is confusion over the distinction between alpha and beta testing. These two concepts are often interchangeable. However, they can be viewed as two consecutive phases of product testing with unique goals.
Alpha Testing & Beta Testing Important Differences
Alpha testing seeks to evaluate product quality and ensure readiness for beta. The focus of these tests is to find bugs and affirm the product generally works according to the expectations of the product development team. This phase occurs after preliminary QA testing and prior to beta testing. Ideally, the beta phase should occur when the product is about 65%-85% complete—which means that it has adequate stability for technical testers, but likely not feature complete.
Typically, the alpha phase runs 1-2 weeks on each test cycle and may continue for numerous test cycles—varying with the number issues found by the testers find and the number of new features in the release. For some teams, the entire alpha phase can even be 3 to 5 times the length of the beta phase that follows. The stakeholders that typically take interest in the alpha phase include the engineering, product management, and quality assurance teams.
In alpha testing, the testing participants ideally include a range of customers and potential users that are capable of providing meaningful technical feedback during the test. The involvement of employees in alpha testing can also help tighten internal cohesion and prepare staff for live-support after launch. Alpha testers should expect a product containing many bugs, induces crashes or significant failures, and may be missing elements or exhibit glitchy features.
The objective of alpha testing is to identify critical problems that are not testable in the lab and also capture show-stopper bugs that are likely to thwart upcoming beta testing. A product is ready for beta when it meets requirements and design specifications, all primary features function correctly, and the testers no longer find blocking issues.
Beta Testing Process
About the beta testing process
Beta testing evaluates the level of customer satisfaction and verifies readiness to release or deploy a new product, feature, or improvement. Beta tests typically include task and survey distributions to guide users in their engagement, and allow each user to discover new and changed product features. The goal is to gather feedback and make a judgement as to whether or not this is ready to be released into the wild.
Typically, a beta testing phase may run from 1-12 weeks, which may include many smaller iterative cycles. This may vary widely according to the type of product that is under development. For example, accounting system may require more than a month for a single cycle. By contrast, it may be quite sufficient to test a newsreader product on one-week test cycles—with new participants each week. More beta cycles may be necessary if additional features are added toward the end of the project.
Key stakeholders in beta testing include people with user experience (UX), quality management, and product management expertise. In beta, it’s important to recruit fresh, independent users that are drawn from the target market for your product. Great value comes from this vantage, since such testers can provide more objectivity and additional insights on product usage. This feedback can be a vital boost to the success potential for the product, and help provide greater value to your current and prospective customers.
Beta testers should expect a nearly feature-complete product that may still have some bugs, crashes, and incomplete documentation. The aim is to identify and fix critical / important issues, and suggest user experience improvements that are achievable before launching the product.
Beta testing seeks to improve success of the upcoming product launch or release, by providing evidential recommendations for product improvement and a comprehensive perspective of customer experience. Also, future product development is heavily influenced by beta testing outcomes.
A product is ready to move into a continuous improvement phase when typical target-market users are quite comfortable with the user interface, are satisfied with how the product functions, and indicate overall satisfaction with their experiences in using the product.
Agile beta testing important for continuous improvement
With the rise of agile, iterative development approaches, together with the advent of continuous delivery, conventional notions of a beta testing phase—in which product development ends while customers evaluate the software—is fading away. Today’s most product teams are focused on customer feedback, analyzing data, and working to continually improve the user experience by launching new builds weekly. This is what makes beta testing important today as a key component in a continuous improvement strategy.
Erlibird does not view each testing phase as having a strict beginning and end. We focus on simple, agile, and iterative testing phases—rather than one-and-done. We also strongly encourage and cultivate Continuous Improvement following any product launch. Beta testing important new features, design changes, or any major product changes is something that should occur with every release.
Agile development is flexible and high-speed, and therefore requires a beta testing approach that is also fast and flexible. Iterative testing is frequently conducted with a unique set of testers for each iteration, to ensure each participant can engage and provide feedback with open eyes, without the influence of their previous experiences. However, for some products, if makes more sense to re-engage the same set of testers continually, so that each user can receive a build after each sprint and provide updated feedback over time before the the final release ships to paying customers.
Why Is Beta Testing Important For The User Experience?
Like you, we at Erlibird encounter some surprising bugs. And sometimes we don’t fully delight our users. We’ve come to realize that users typically don’t have much patience in a competitive marketplace. If you ship something buggy, they won’t come back.
Whether you are launching a website, a new app, or a new mobile device, it is important that you have independent users test the product thoroughly prior to shipment. Beta testing serves multiple purposes, though all of those initiatives lead to one thing – improving customer experience.Beta testing important new products is also known as user acceptance testing—UAT—and occurs near the end of the pre-release product development cycle. Though often neglected or glossed over, this validation activity reveals critical insights about how customers will perceive, engage with, and operate the software. It assesses how the product meets user expectations. In other words, beta testing move the development team very far along to answering the penultimate product question: Is it shippable?
The cost to ship bad software will always be quite high. If a prospective user downloads your application, there is a very short interval of time to convince that user of its value. In this short duration of time, this user will become a customer only if they perceive that your software will help solve a problem or meet a need, according to John Egan, Pinterest’s growth engineering manager:
“For activation, it all comes down to: Does the user get enough value from your product? For early stage startups, this means to figure out how to reach product-market-fit. And if you’re past that stage, it comes down to being able to communicate your product value to your users. When someone downloads your app, you have a couple of minutes to convince them that this is something they need to use on a regular basis. In this short time they need to understand how the app will help them accomplish whatever the product value is.”
It’s always important to remember that the penultimate goal of any product is to build something that is better than other alternatives and either solves a problem or meets a basic human need. Beta testing helps you achieve that top-level product development goal. When done properly, beta testing helps you iron out the wrinkles and find deficiencies prior to launching your product.
Gain Insights About Real-World Usage
Beta testing provides insights into product functionality, and also helps you better understand user experience. Going beyond lab performance tests, beta testing reveals whether or not the same level of performance is achievable in actual user environments. Many products need to perform well in hundreds of various environments and many different usage contexts.
By presenting the software to users that stand well outside the insular community of developers, beta testing serves to identify elements of functionality that are all too easily overlooked in the lab. In addition, your team benefits from feedback that can be very useful for improving future product versions—or even spawn ideas for entirely new products.
Why Is Beta Testing Important For Your Product?
Beta testing is more important now than ever before. There are many different reasons to beta test, but be sure to keep the following goals in mind:
- Customer Feedback – Beta testing important new products and features is one of the best ways to determine if your product provides value and solve a problem or satisfies a basic human need.
- Quality – Will the product function flawlessly on various devices and environments?
- Usability – Will users be able to accomplish what they want in an intuitive and enjoyable way?
- Performance – Will your product operate quickly and efficiently in the hands of real users?
- Save Money – Save money by fixing problems before they occur.
- Make Money – Improve usability and provide more value to users, leading to increased conversions and higher revenue.
- Vetting Ideas – Prior to launching a new product or feature, it’s important to put it into the hands of new users.
Learn about how BetaTesting can help your company launch better products with our beta testing platform and huge community of global testers.
-
How Netflix Does A/B Testing (And You Can, Too)
Netflix is big. Your company is—most likely—not quite as big. So, when I tell you that Netflix uses A/B testing in numerous ways to improve their service and enhance their business, you might say, “That’s nice for them, but how does that help me and my decidedly smaller business?”
If you said that, I’m glad you did. Because the different ways that Netflix does A/B testing are things that you can do, too—or can inspire you to do something similar.
Hey, real quick: what is A/B testing?
A/B testing is a live experiment that compares two versions of a thing, to find out which version works better. Version A is the one your users normally experience, and Version B is the one you think might more effectively accomplish a particular goal. You run the A/B test on your live product by diverting a portion of your users to Version B, while the rest of your users continue to use Version A. You collect the results from both groups during the test, and then use that data to determine which version performed better.
You’ll see what I mean in the examples below.
1. A/B Testing to increase user sign-ups (and other macro goals)
Netflix wants to add new paying customers to their service (of course), so they consider modifications to their website’s registration page. Whenever their product design team comes up with an improvement that they predict will lead to more user sign-ups, they prepare an A/B test to confirm their hypothesis.
One such hypothesis went like this (paraphrased for our purposes): “If we allow non-members to browse our content before signing up, then we’ll get more people to join the service.” The idea came from actual user feedback, and it made sense to designers at Netflix.
So, in 2013, the product design team tested their hypothesis using an A/B test. They built an alternate working version of the registration page that allowed content browsing (Version B). When they started their test, some visitors to the live site were unknowingly directed to Version B, while the rest saw the original design (Version A), which didn’t allow content browsing. Throughout the test, Netflix automatically collected data on how effective Version A and Version B were in achieving the desired goal: getting new user sign-ups.
Netflix ran this test 5 times, each with different content-browsable Version B designs pitted against the original non-browsable design. All five Version Bs lost to the original design. The design team’s hypothesis turned out to be incorrect. However, Netflix learned valuable lessons from the tests. They also saved themselves bigger problems by testing their assumptions first, instead of simply rolling the change out to everyone. As Netflix Product Designer Anna Blaylock stated at a talk in 2015, “The test may have failed five times, but we got smarter five times.”
How does this apply to me?
Trying to get more users to sign-up for your service is an example of a “macro conversion” goal, and this is perhaps the most common application of A/B testing. Macro conversions goals are the “big” goals that usually represent the primary purpose of the thing you’re testing, whether it’s a website, app, or marketing email.
Examples of macro conversion goals:
- Get users to use your site’s Contact form
- Get users to complete a product purchase
- Get users to respond to your sales email
This kind of A/B testing is something you can do, too. Even though Netflix’s example involved testing a design with a whole new feature (browsing), the design changes that you test will often be much simpler—the text label on your call-to-action button, for example.
Sure, your modest app doesn’t get as many user hits as Netflix does, but you can still set up and run A/B tests of your own. Just like Netflix, you’ll run your test as long as necessary to collect enough data to determine which design is more effective. For Netflix, this is often several weeks. You may need to let your test run for several months.
2. A/B testing to improve content effectiveness (and other micro goals)
Netflix also runs A/B tests to optimize “micro” conversion goals, like improving the usability of a particular UI interaction.
For example, Netflix ran a series of A/B test experiments to determine which title artwork images (a.k.a. “thumbnails”) would be most effective to get users to view their content.
First, Netflix ran a small test for just one of their documentary titles, The Short Game. The test involved assigning a different artwork variant to each experimental test group, and then analyzing which variant performed best—and by how much.
As Netflix’s Gopal Krishnan wrote:
“We measured the engagement with the title for each variant — click through rate, aggregate play duration, fraction of plays with short duration, fraction of content viewed (how far did you get through a movie or series), etc. Sure enough, we saw that we could widen the audience and increase engagement by using different artwork.”
With this initial A/B test, Netflix established that significant positive results were possible by optimizing title artwork. Netflix then went on to run more elaborate tests with larger sets of content titles. These tests measured additional factors as well, to verify that the success of the optimized artwork was actually increasing total viewing time, and not simply shifting hours away from other titles.
I can A/B test like that, too?
Yes! If you have a hypothesis about a design change that might improve your users’ experience, then you can set up an A/B test to try it out.
You’ll want to be careful to:
- Minimize the differences between Version A and Version B. If there are multiple or unrelated design changes on the same screen, you won’t be sure which design element was responsible for better performance.
- Make sure you can measure the results before you start the test. Determine what data would indicate success, and make sure you can automatically collect that data. Micro conversion goals are often not as trivial to track as macro conversion goals. You may need to set up a custom Goal in Google Analytics, or what have you.
- Consider the bigger picture. Your redesigned button might get more click conversions, but perhaps users are also leaving your site sooner. Think about how your design change may affect your other goals, and make sure you’re collecting that data, too. Consider running additional tests.
3. A/B testing custom algorithms
A core part of the Netflix user experience is that much of the UI is customized for each user. Most of that customization is accomplished via algorithms. Algorithms can be rather simple or extremely complex, but their output can still be A/B tested.
Remember a few paragraphs ago, when Netflix tested which thumbnails would work best for all their users? That’s already old news. In 2017, they moved to a system that selects thumbnails for each user based on their personal preferences. For example, if Netflix knows you like Uma Thurman movies, you’re more likely to get the Pulp Fiction thumbnail with Uma Thurman staring back at you, making it statistically more likely you’ll watch it. This personalization is driven by an algorithm that Netflix can improve over time.
Any time Netflix wants to adjust one of their personalization algorithms (or their adaptive video streaming algorithms, or their content encoding algorithms, or anything) they A/B test it first.
Do I need to test algorithms?
You may not actually be employing any custom algorithms in your app or website. If you are, however, you should seriously consider running an A/B test for any algorithm adjustments you make.
Let’s say your company makes a news aggregator app that recommends articles for the user to read. Currently, each article recommendation is based on the user’s pre-set preferences and their overall reading habits.
If you had reason to believe that your users prefer reading articles of the same type in batches, you could modify the algorithm to give higher priority to articles that are similar to the article the user just read. You could run an A/B test to see if your new algorithm is more effective at achieving your goal of increasing app usage.
Before you can run such a test, you’ll need to actually implement the new version of the algorithm (while not interfering with the current algorithm). This may be trickier than your average design change, so tread carefully.
More things to consider:
- Make sure your Version B implementation has been sufficiently software-tested. Bugs in your new version will negatively affect the results and may require you to throw out the test.
- Try to ensure that your Version B latency is very similar to the original’s. If your new algorithm takes noticeably longer to process than the original, then your A/B test will largely be testing the user experience implications of the slowdown, and not the output of the new algorithm vs. the old.
Conclusion
Netflix is a monster at A/B testing. They’ve got data scientists, design teams, and engineers working on potential service improvements all the time, and they A/B test everything before it becomes the default experience for their massive user base. They know it’s too risky not to.
You and your company probably do not have the resources to build the impressive experimentation platform that Netflix has. Not yet, anyway.
But you, too, can reap the benefits of A/B testing for your products, at your own scale and at your own pace. And you can take Netflix’s core lesson to heart: always be looking for ways to optimize your products.
Learn about how BetaTesting can help your company launch better products with our beta testing platform and huge community of global testers.
-
Neura / Beta Testing Case Study
Neura delivers high performance technology by optimizing battery consumption.
Perhaps no technology will impact our future more than Artificial Intelligence. If 2017 was the “Year of The Bot”, 2018 may be the Year the Bot finally earns the title of “Smart”. A record breaking holiday sales season saw the Echo Dot become the #1 selling product on Amazon, while Google has sold more than one Google Home per second since the product debut in late October.
In recent years, AI technology has progressed from a joke, to incredibly smart. Now, it’s hard not to be impressed as our devices begin to integrate seamlessly into our lives, understand what we say, and actually improve our experience interacting with the products around us. Despite the advances, it’s only just the beginning and the future for AI technology is bright.
One company leading the charge is Sunnyvale, California based Neura, Inc.
Neura is a personal AI solution that changes the way digital products interact with people. Neura adds user awareness to apps and devices. This enables their customers to deliver a more personalized experience to their users based on their behavior, habits, and activity. With Neura, digital products can boost engagement and retention by personally reacting to critical moments throughout the user’s day.
As an ingredient technology embedded into other software, it’s vitally important that Neura operates flawlessly, fast, and efficiently. When Neura needed a partner to manage research and testing for battery consumption in the real world, Neura CTO Triinu Magi turned to BetaTesting.com.
“Because Neura’s APIs are accessed via its SDK, it is very important to keep our resources consumption on the user’s device as low as possible,” said Magi. “Keeping a low battery consumption is imperative to our customers as it has great impact on the user experience. We work very hard to be able to deliver our services with high performance levels and minimum battery usage (less than 1%). We run many tests internally and keep improving our performance but before we roll out a new version to our customers, we must check how it performs in the ‘real world’ on a variety of devices and users in different locations. Our ability to check the battery consumption is limited to the devices that we have and the internal testers we can use.”
While testing products in-house or with automated testing tools can prove very helpful, there is no replacement for real-world testing on a wide array of devices and environments. BetaTesting.com helped recruit users within Neura’s target market, and tested the SDK for multiple days (both weekdays and weekends) in order to thoroughly test the product in the real world.
“From experience we’ve learned that some issues are detected only after experiencing the SDK in different locations and devices. We need to make sure that problems are solved so they don’t happen with our customer’s users,” said Magi.
BetaTesting’s test results allowed Neura to quickly diagnose and improve battery consumption, leading to multiple test cycles in order to optimize battery performance.
“During the first test cycle, we received valuable information that led to product changes. We went on to add another test cycle. The process was very easy and convenient. BetaTesting can move very fast and adapt to our changing needs. We informed BetaTesting that we want to lengthen the test and they were fast to respond and react with a sufficient number of testers. The testers that completed the test were good – completed all the questions and provided a lot of feedback. It helped us understand better how the product works in the real world. We improved our battery consumption and also our monitoring capabilities,” said Magi.
As a company that focuses on continually improving the user experience, Neura has found a consistent partner in BetaTesting.com.
“We have worked with BetaTesting a few times in the past and have been pleased with the process. BetaTesting offers a good value compared with the competition. They provide a guarantee to the numbers of testers that will finish the cycle and professionally manage the process, minimizing the resources required for us to manage it. The BetaTesting platform is also very easy to use (as a test manager). The results are shown immediately and clearly. The ability to connect with the user to ask specific questions is also valuable to us.”
Learn about how BetaTesting.com can help your company launch better products with our beta testing platform and huge community of global testers.
-
Xerox / Beta Testing Case Study
Xerox Transportation launches mobility pilot after beta testing in select cities across the US.
It shouldn’t be a surprise that the company most responsible for creating the modern personal computer is continuing to invest in research and development projects capable of changing our world. Since 1970, through its R&D labs, Xerox has built a distinguished reputation as a innovative technology leader, helping invent the Graphical User Interface, Ethernet, laser printing, and even Object Oriented Programming.
Now Xerox is focused on revolutionizing the way we travel. Since 2010, Xerox Innovation Group, spread out across four research centers globally, has been designing new technologies to enable smarter mobility. Xerox is helping cities all over the world to optimize their transit systems and traffic flows, and get people where they need to be, quicker and with less hassle.
A central part of this initiative is the Xerox Mobility Marketplace – an ambitious vision for a unified travel subscription for city dwellers and visitors, branded for a specific city, integrating all the different modes of public and private transportation seamlessly into one easy-to-use app.
Leonid Antsfeld is in charge of Business Development for Transportation in Xerox Research Center Europe. As the Product Owner responsible for Xerox Mobility Marketplace, Antsfeld faced a unique challenge to recruit the right beta testers for the first real-world tests for the city-specific apps in Los Angeles and Denver.
It was important that testers represented a range of demographics and device types, and a mix of public and private transportation users and not only those that primarily relied on their car.
“The first question was– where do we find testers?” asked Antsfeld. “The key challenges were defining and recruiting the right audience, determining which features to test, what questions to ask, and how to interpret the answers. We had to analyze and understand the feedback – what is important and what is less important?”
When searching for a beta testing partner, Xerox turned to BetaTesting.com. “BetaTesting left the best impression and they were recommended by our partners that used the service in the past,” said Antsfeld.
From the early stages of planning through execution, BetaTesting worked with Xerox to design a successful beta testing campaign. BetaTesting helped define the target audience, create the beta tasks and feedback questionnaire, recruit and coordinate with the testers, and summarize and help analyze the results.
“The BetaTesting.com campaign was extremely helpful, literally from the first minutes of the launch when we immediately started to receive instant feedback through the app,” said Antsfeld. “Besides fixing many bugs, we also learned what potential users liked and disliked in the app, and what direction they would like to see it going in the future.”
Today the future for Mobility Marketplace looks bright and the vision is more important now than ever. Urban Millennials are interested in simplifying their life with flexible, environmentally-friendly travel options and participating in the sharing economy through the use of car and bike-sharing services. Just as important, there are huge advantages for cities that prioritize the availability of different transport choices: pollution reduction, improved health and happiness, reduced accidents, and the potential for sustainable population growth with a high quality of life.
“We took testers’ feedback very seriously and are already working on integrating requested functionality in the next versions,” said Antsfeld. “We aim to have bi-weekly updates with bug fixes and minor improvements and once a quarter to release major versions updates with new functionality. We are definitely are planning to continue to work closely with BetaTesting before any major release.”
Learn about how BetaTesting can help your company launch better products with our beta testing platform and huge community of global testers.
-
CapperNetwork / Beta Testing Case Study
Capper Network collects valuable user experience feedback from fantasy sports players during website’s beta testing phase.
These days, it pays to be a good at predicting sports. Each year, an estimated $10 billion changes hands through March Madness NCAA brackets, mostly between friends, coworkers, and family. Fantasy sports is rapidly growing too. In 2018, more than 60 million people play fantasy sports in the US and Canada, generating an estimated $7 billion annually. Sports predicting has become a hobby for both casual and dedicated sports fans.
Capper Network is a startup capitalizing on this evolving industry by developing a web platform for sports predictors (also known as Handicappers, or Cappers).
“Capper Network is geared to find out exactly what teams and players handicappers are accurate at predicting,” said CEO Tucker Harris. “Our users can prove their expertise at predicting certain teams and players, which will increase the demand to see their predictions and analysis on those teams and players.”
When searching for a beta testing partner, CapperNetwork turned to BetaTesting.com to help design and execute a successful beta campaign.
“My team researched various beta testing services before ultimately deciding to go with BetaTesting. We were impressed with the list of previous happy customers that had worked with BetaTesting.” said Harris.
For the beta test, it was extremely important to recruit fantasy sports players that would be interested in the platform and have the context and experience to provide valuable feedback.
“Since our concept is so unique, we really wanted to see what sports fans thought about our platform in general. Fortunately, BetaTesting.com attracted beta testers that were very close to our exact target market (many still use the site today),” said Harris.
By analyzing the feedback reporting provided by BetaTesting, the CapperNetwork team discovered several trends and opportunities to improve the product, along with many ideas to influence the future product road map.
“We received a ton of good ideas for enhancing and expanding our product. We also changed some things we knew about but the feedback really helped prioritize what needed to be done sooner than later. The main problem was our landing page. We thought it was pretty good but it was a common turn off. We realized its importance and have factored in the feedback to form our new landing page which is currently being designed,” said Harris.
Moving forward, CEO Tucker Harris plans to continue to use BetaTesting services for future test iterations.
“The process could not have been better. We got what we paid for and more. The BetaTesting team was easy to get a hold of and very professional. We really appreciate it and will be coming back for more! The next tests we run with BetaTesting will be more focused on particular features instead of general user experience. We will most certainly be using BetaTesting’s services again,” said Harris.
Learn about how BetaTesting can help your company launch better products with our beta testing platform and huge community of global testers.
BetaTesting Blog 