-
How Sengled Beta Tested iOT Smart Lights in the Real World: Case Study
The smart light market is currently worth over $8.5 billion and is growing over 22% each year. With new products, apps, and integrations available every day, it is important for smart light manufacturers to create amazing customer experiences to get consumers to buy and repurchase bulbs.
So when Sengled needed to test a new smartbulb model prior to public release, it was important to select the right beta testing partner. Sengled turned to BetaTesting due to our ability to handle logistics, shipping, and multi-week tests with a community of targeted real-world users.
BetaTesting helped Sengled get their smartbulb in the hands of real-world testers to provide feedback on the user experience, bugs, and integrations with other devices.
Sengled bulbs were shipped to over 100 testers across the US, selected based on a series of screening criteria, such as their previous experience with IoT products and whether they had a Google Home or Amazon Alexa product in their homes. Testers agreed to test the bulbs for a month, and were guided through a series of tasks and surveys to collect their feedback each week.
First, testers were asked to share their feedback about the packaging, unboxing, and setup experience. This included installing the bulb and connecting it to wifi, and completing the setup process through the Sengled app.
Through this process, Sengled received valuable insights into several specific technical issues affecting customers during the setup process. At times, bulbs were not connecting to certain routers and wifi networks, and users were confused at certain points of the onboarding process.
Finding and correcting these issues before launch proved very valuable. Even one or two small improvements early in the setup process can lead to significant decreases in returns and support costs, and increases in customer satisfaction when the product is live.
“The BetaTesting team and their beta testers have passion in new products, and were willing to work with our special requests. We’ve been very fortunate to work with this group. And certainly hope to work with Beta Testing group in the future.” said Sengled Product Manager Robert Tang.
As the month-long test continued, Sengled’s team worked with BetaTesting to design new surveys to test various features of the app and bulb: dimming, power reset, creating schedules and scenes, and more. Sengled was able to collect qualitative and quantitative feedback about which features used loved, where there was confusion in the app’s user flow, and much more.
For example, one important feature that was tested was the voice control option through Google Home and Amazon Alexa. Sengled was able to collect hundreds of voice commands and feedback from testers about the voice control experience, and which phrases Google or Alexa understood.
Real User Feedback:
I asked Google home to “turn on living room light on” and it worked every time. I asked google to “turn living rooms lights off” and it worked every time. I asked google to “Turn Living room light bulb brightness to 50%” and it worked every time. The only command I could not get to work was to set a schedule using voice commands. I tried all kinds of variations like “hey Google set living room light to come on at 7pm” or “hey Google can I set a schedule for living room lights?” The most common answer Google would give would was “I am sorry I can not do that yet”. – Dave Oz
Through the BetaTesting summary reports, Sengled received segmented analysis of Google users versus Amazon users, and how each group perceived the product after using voice commands in their home for weeks.
Sengled also received over 60 bug reports during the test, detailing issues related to connectivity, creating an account, app design, bulbs not working correctly, and much more. Each bug report included the device and operating system of the tester, and many had photos and videos to make it easy to understand the issue and let Sengled’s development team address them quickly.
Sengled found that the new features they had created for this product, such as the ability to setup light schedules and use voice controls, were extremely popular with users. The net promoter scores for the product were very high and Sengled received a range of feedback to continue improving their bulbs for the future.
Learn about how BetaTesting can help your company launch better products with our beta testing platform and huge community of global testers.
-
McAfee + BetaTesting Partner to Beta Test a New Antivirus Product for PC Gamers
Recent studies show that the gaming industry is expected to grow 12% per year from 2020-2025, with North America being the fastest growing market. This is driven by the rise of new gaming and technology platforms, readily available and cheap internet access across the globe, and the advent of new technology like augmented reality.
McAfee, the market leader in antivirus software, wanted to develop a new product focused on the unique security needs of gamers. Gamers have very different technology needs in comparison to other consumer groups. Their gaming systems are often complex, and include cutting edge hardware, memory, graphics cards, and processors to give them a leg up during gaming. Gamers demand high performance, and “high performance” is not typically a phrase used to describe antivirus software, known traditionally as being bulky and slow.
“We were in the need of a unique audience to satisfy the needs of the gamer security product, and BetaTesting did a great job in finding the right cohorts at the intended scale,” said Rajeshwar Sharma, Quality Leader at McAfee.
McAfee set out to develop an antivirus product that could provide maximum protection without affecting gameplay for this market. The McAfee team worked with BetaTesting to design a multi-month beta test before the Gamer Security product launched, for the purpose of collecting user experience feedback and functional/exploratory bug testing over multiple test cycles.
First, BetaTesting recruited over 400 PC gamers matching specific demographic and interest criteria required by McAfee: A unique mix of real-world operating systems, devices, and locations. Participants were recruited through the existing BetaTesting community of 250,000 testers and supplemented with custom recruiting through BetaTesting market research partner networks.
McAfee’s product team and user research teams worked closely with BetaTesting to design multiple test flows and collect feedback from hundreds of gamers. The test included feedback around the installation process, the user interface, and the back-end telemetry data of the antivirus product being used in the real-world.
Testers also provided information about which games they played, their hardware configurations, operating systems, and more, in order to connect all of that data to the performance of the Gamer Security product.
During the final rounds of testing, McAfee also provided marketing materials and screenshots to testers to get feedback about their messaging and content to see what resonated best with their target market.
The research team also set up moderated interviews with testers who had provided the most insightful feedback. Testers connected via video chat with the McAfee team to discuss their in-depth opinions about the product, their willingness to pay, and much more.
Overall, the test helped McAfee understand the value of the product to their target market. The feedback was overwhelmingly positive about how the antivirus software didn’t affect gameplay, and testers loved the simple installation, straightforward UI and performance data. The McAfee team was able to launch the product on time – with good reviews online.
“BetaTesting is an amazing team, who were very helpful, diligent and thorough from immediately understanding the requirements of our unique software to contract negotiations, and to overall execution of the program to the scale of our expectations. They were available, prompt and clear in all communications. I would highly recommend them!” – Rajeshwar Sharma, Quality Leader at McAfee.
McAfee received over 200 bug reports during the test, detailing issues related to installation and update issues, crashes, performance issues during certain games, data mismatches, and more. Each bug report included the device and operating system of the tester, and many had photos and videos to make it easy to understand the issue and let McAfee’s development team address them quickly.
The McAfee Gamer Security test was a comprehensive test that underscores the capabilities of BetaTesting platform and managed services. The BetaTesting team coordinated with different departments and stakeholders within the McAfee team, and the test design focused on everything from onboarding to back-end data collection to marketing content. Finally, testers provided hundreds of bug reports, qualitative and quantitative data, and moderated interview feedback to make this test – and the new product launch – a success for McAfee.
Learn about how BetaTesting can help your company launch better products with our beta testing platform and huge community of global testers.
-
Test Anything! See some recent examples of our beta tests.
We’ve run a lot of tests recently across a wide range of industries and products, and wanted to share some of them to highlight how you can test anything with our platform and community:
📺 a streaming media device with testers across the US
📱 a small business CRM and business phone line app for freelancers and business owners
📷 a drone + camera with automated tracking for existing power users
🎮 a new game on Steam with hundreds of players worldwide
👶🏼 an iOS app to help kids with speech therapy
🛡 a desktop antivirus product designed for gamers
🔒 a desktop VPN product tested in 10 different countries for usability and security
💵 an online web app for a business funding and lending platform
🚗 a machine learning app crowdsourced tester data to track travel behavior
… and many more.
We’d love to help you collect feedback fast and improve your products. Get in touch with our team and we’ll help you design your first test.
-
Jira Integrations are now live!
We’ve been getting this request for a long time and it’s finally live! BetaTesting is integrated with Jira to push bugs from your tests directly into your Jira Server or Jira Cloud account.
It’s simple to map fields from our bugs to various fields in your Jira account, such as title, description, priority level, attachments, and more.
Within each test, you can choose your own integration, so different tests can connect to different development teams.
Contact us for a demo to learn more!
-
Beyond Beta Tests: Collect Data for AI and Machine Learning
Over the past few months, we’ve seen more clients leverage the BetaTesting community (200,000+ participants) to collect real-world data to improve AI and machine learning within their product:
Crowdsource pictures to improve image detection algorithms.
A large car parts company in Europe crowdsourced pictures of car interiors from BetaTesting users to tag images for debris, helping their machine learning algorithm learn if a car was clean or dirty. Within days, they were able to collect thousands of pictures from different vehicle types around the world to improve their product.
Track phone sensor data (e.g. locations, battery, acceleration).
Another customer had users share their schedules throughout the day (at home, driving to work, at work, sleeping, etc) and matched it to their phone sensor data to learn more about their customers and improve their AI algorithms to anticipate customer behavior.
Gather user ratings to improve recommendations engine.
A number of companies have improved their recommendation engines for content by asking a wide variety of testers to rate content and give user feedback around preferences and categories to make their products more personal and targeted for each customer.
Collect software telemetry for PC app over several weeks.
An antivirus product for desktops asked users to beta test their product and provide back-end log files to track the software telemetry internally within their team and learn more about how the product behaved across various devices and operating system versions.
—-
AI and machine learning products need more data than ever, and our platform + community is the perfect mix to source large datasets or hard-to-find niche users to make your products more accurate and powerful. This includes pictures, speech inputs, mobile usage, location data, and much more.
As a team, we are excited to see these opportunities to connect to users in different ways to improve products beyond beta tests. Our community of over 200,000+ people is perfect for gathering any data you need to improve your products. With our tools, you can find the right demographic anywhere around the world and get the exact data you need – within hours or days.
-
BetaTesting Test Design: How to Setup Your First Test Process
One of the most common questions we get asked about our test design process is: “How should I setup my beta test?” How many tasks should I give testers? How many surveys should I setup?
There are a lot of different ways to setup your beta test, and it is usually driven by what you hope to achieve from your test.
QA / Bug Reports / Functional Testing
For technical testing, where you are trying to identify bugs, specific test design scenarios that cover your entire product are usually best. For example:
- Create an account in our app and fill out your profile.
- Go to Settings, click on “invite a friend”, and add their email address to the app.
- Go to ‘Newsfeed’, upload your first picture, and add location tags to it. Submit a bug report if you are unable to complete any of these tasks.
Asking for specific tasks helps testers stay focused and helps our clients find bugs across different operating systems, phones, and browsers that can be resolved quickly.
Usability or UX Testing
If you are more interested in general usability and UX/UI testing, we recommend keeping the test design process more high level and letting users explore your product more organically. For example:
- Create an account and begin using our app.
- Try to connect our app to your Google Home or Alexa.
- Use voice commands to control the settings in the app and share what voice commands you tried.
- Continue using the app every day for the next week and then answer our final survey.
Higher-level test tasks give testers more freedom to explore the app on their own. This provides a larger variety of use cases and opinions from testers. When you want to learn about your app’s design, ease of use, or net promoter score, a test designed to let users figure things out on their own will help you get more accurate scores.
-
Professional Beta Testers vs Real-world Beta Testers
When you run a beta test for your product, there are tradeoffs to consider between professional testers and real-world beta testers. A lot of companies use professional beta testers to help companies test their products, and those testers have deep experience dealing with technical tests that require a deeper dive into a product to find issues.
Before we changed our name to BetaTesting, we built Erlibird as a community of early adopters interested in finding new tech products and providing feedback for them. It made our community a diverse mix of tech-savvy early adopters and regular people. Since then, we have focused on building a community of real-world testers across the world with a variety of tech experience.
We believe using real-world testers leads to better feedback and results for most tests. We are able to recruit testers that actually match the audience for your product, with a true mix of platforms, operating systems, connection speeds, and more that help customers find real bugs and get more accurate UX and UI feedback.
In most cases, companies need quick feedback to iterate their products, see usability videos to learn where users might be getting confused, and test the validity of their new features and products. In those instances, a real-world community makes more sense than a community of professional testers.
-
Sending Effective Beta Testing Invite Emails
At BetaTesting, we’ve sent out close to 5,000,000 beta testing related emails. Many of these have been beta testing invite emails designed with the specific purpose of communicating test details concisely and getting high levels of engagement from our community of testers. We’ve A/B tested, measured results, and redesigned our emails countless times based on those results.
With all of this data, we’ve learned that there are several factors that can influence the effectiveness of your beta testing invite emails. The pay-off is clear: Sending better invite emails will help you attract the right testers that are more likely to become active users of your product now and in the future. In this article, we’ll take a look at some things you can do to improve the effectiveness of your invite email:
1. Work Within Bounds
Understand that your beta testing invite email open and click rates, and your larger beta participation rates are primarily driven by the value of your product.
This is an obvious one (and maybe not the type of advice you’re after right now) but it’s very important to keep in mind that all the marketing/messaging/magic in the world can’t make up for a product that simply isn’t valuable / fun / interesting. If you build an exciting product that is fun and useful, some people are going to get in line to test it. On the other hand, if people aren’t intrigued by your product, the invite email isn’t going to change that. So, when designing the invite email, the goal is to fine-tune the messaging to simply and quickly communicate your product’s value. If you find yourself instead communicating something else (e.g. value your product doesn’t really offer) or leaving important information out in order to increase open/click rates, it’s a sign that you may need to change your product.
2. Communicate Your Product’s Value in One Short Sentence
When communicating your product, LESS WORDS > MORE WORDS
Think of this as the email version of your elevator pitch. Invest some time and effort in coming up with one simple sentence that accurately describes your product and catches people’s attention. Some helpful tips:
- Mention what problem your product solves, or how it might benefit the user.
- Avoid marketing lingo.
- Run A/B tests with different language.
When working with our clients, we have seen that when faced with the challenge of communicating something clearly, everyone’s intuition is to communicate more: more words, longer sentences, links, screenshots, etc. Unfortunately, this has the opposite effect from what is intended because most people are not going to spend the time to read every word you’ve written. More words ends up being more confusing (not more nuanced and complete as you expected).
3. Provide Key Information
What you should include in every beta testing invite email.
Besides communicating a short exciting sentence about your product, here are some other things to include:
- Expectations: Be upfront about how much time you expect the testers to spend engaging / providing feedback. Most people are busy, and they want to know the required time commitment before making a decision.
- Incentives: Let users know what’s in it for them. Although the primary motivation for most testers is to help you launch successfully, they still expect to be compensated for their time and effort. Knowing the incentives upfront will help them decide if the test is a good fit for them.
- Exclusivity: If you are running a private beta with limited invites, mention how many spots are available. The opportunity to be one of the few people that get early access to your product might make it more attractive for potential testers.
- Prerequisites: If there are any prerequisites to apply for your test, you should mention them in the email. (At BetaTesting, we help our clients filter for prerequisites through demographic targeting, and an additional screening survey for refined targeting on any criteria like interests, lifestyle, etc)
- Call to action: Be clear about what you want people to do, and include a “Call to Action” button in your email. For example, if you want people to apply for your private beta, your CTA button could be “Apply for Early Access” (which links to a landing page where users can apply for the test).
4. Measure The Results
A/B test several versions of your invite email to optimize your results.
It is unlikely that you will get the results you want on your first try. You will probably need to refine / re-design your emails several times until they are optimized. If you have a large enough list (thousands of people), you can send a few different versions of your invite to different groups. Each group should be hundreds in the minimum, but ideally thousands in order to obtain reliable data. After you run a few of these tests, you’ll be able to see what works best and you can stick with that version. If you don’t have a big enough list, run some tests with Google Consumer Surveys, or hit the pavement the old fashioned way and ask people on the street.
Finally, remember that while these tips can help you craft better beta invite emails, this is only one small part to achieving high levels of beta engagement. For example, if you are building an email list before you get to the beta stage, it’s much better to stay in touch with your audience and get them excited by regularly sending them interesting content and updates (not just waiting for one big beta launch email).
Learn about how BetaTesting can help your company launch better products with our beta testing platform and huge community of global testers.
-
Small Changes, Large Impact: How Beta Tests Improved Amazon's App.
When most people plan to run a beta test, they think big. We need to test the entire app for usability. We have to find all the bugs before we launch. Lets rebrand our entire site.
While thinking about the entire product and entire user flow is necessary, sometimes it’s the smallest things that confuse users. A small change to your copy can drive more conversions. A small update in your design might make a key feature more clear. This is where some of the most important changes to the usability and user experience of your product come from.
This example really drives the point home.
Even the best e-commerce company in the world makes assumptions and mistakes that lead to user confusion. To customers in rural India, the magnifying glass icon for search was a ping-pong paddle!
Most of us have biases about product design and assume certain icons are universally known to everyone to mean something specific. But as more and more people get online, in remote and rural areas of the world, those biases need to be tested. Everyone, from Amazon to new startups, needs to beta test their apps and get in-depth feedback to figure out if your product is creating confusion you never thought about before.
-
ErliBird is now BetaTesting.com!
ErliBird is now BetaTesting.com!
Today, we’re excited to announce that ErliBird is officially changing our name to BetaTesting.com.
We launched in 2012 as a way to connect early adopters with new tech startups. While the name “ErliBird” was a great fit then, it doesn’t reflect who we are now.
Over the last 6 years, we’ve moved away from simple product discovery, and built a community of over 150,000 enthusiastic beta testers that love engaging with products and providing real world feedback. We’ve worked with some of today’s most exciting startups and established companies (check out our customers page). Our community has submitted many thousands of surveys, usability videos, and bug reports to help companies big and small launch and improve their apps, websites, and hardware.
To reflect our growth and evolution, we wanted to find a new name that reflected who we are now – and most of all, keep things simple. And BetaTesting.com is as simple and direct as it gets.
Going forward, nothing will change for our users. Your login, access to tests, payments, and everything else will stay the same. We will continue to invite you to tests you qualify for, so thank you for all your support over the years as we have grown.
Cheers,
The Erlibird BetaTesting Team
Learn about how ErliBird can help your company launch better products with our beta testing platform and huge community of global testers.
BetaTesting Blog 