BetaTesting powered testing and research for an innovative children’s worksheet product, earning 10M+ downloads and a successful launch on Amazon.
“Magic Worksheets featuring Disney were launched in the App/Play Store and on Amazon with dramatic success, earning a 4.7 / 5 stars on iOS and > 10M downloads on the Play Store”
Recent studies show that the educational games market is growing 20%+ per year, driven by the rise of e-learning during the pandemic, inexpensive global internet availability, and parents seeking engaging ways to teach their young children.
Disney has been a leader in this category for years, and has been able to make unique games based on their large library of intellectual property and characters that kids love. Osmo is an award-winning educational system for the iPad and Android tablets. Now the two companies have partnered together to create an innovative product combining the Osmo tablet with real-world worksheets powered by the Early Learn app (by Osmo parent company BYJU’s) along with engaging content and IP from Disney.
Osmo / BYJU’S first approached BetaTesting multiple years prior to the launch of the worksheet product. Already a staple in India, BYJU’S was interested in bringing their popular product to the US and further expanding globally. Initial testing was focused on testing the BYJU’s Early Learn app.
Initial Early Learn App Testing & User Research
The primary goal for the test was to connect with parents/children in the US to measure engagement data and collect feedback on the user experience and ethnocentric issues around the content and quizzes in the app. If there were any issues like confusing language or country-specific changes required (locations, metric system vs imperial system measurements, etc), it would be important to address those issues first.
Recruiting
The Early Learn team worked with BetaTesting to recruit the exact audience of testers they were looking for: Parents with 1st, 2nd, and 3rd graders with quotas for specific demographic targeting criteria – household income, languages spoken at home, learning challenges, and a geographic mix of regions across the US.
BetaTesting used its existing community of over 400,000+ testers, along with custom recruiting through our market research partners to find over 500 families interested in participating in the test. BetaTesting worked closely with the Early Learn user research team to design and execute the tests successfully.
Test Process & Results
Each child was asked to complete various “quests” in the app, which were educational journeys taken by their favorite characters. Each quest included videos, tutorials, and questions around age-appropriate modules, such as math, fractions, science, units of measurement, and more.
Children completed quests for over 3 weeks, and parents facilitated the collection of comprehensive feedback about the quality of the content, age appropriateness, and any parts of the user experience they found confusing or lacking in engagement.
The Early Learn team also collected in-depth feedback from parents about their opinions regarding technology, educational apps, and perceptions towards different educational approaches. The test helped the team develop insights into how different user personas approach their children’s education and how various factors impact parent’s decision-making regarding technology and educational usage and purchase behavior.
The feedback was overwhelmingly positive, and children seemed to love their favorite characters taking them on an educational quest. The Early Learn team also found dozens of bugs that needed to be fixed, and changes to the user experience and onboarding experience to make the games and quizzes easier and more enjoyable to play.
Ongoing User Feedback & QA Testing
After the initial test proved successful, BYJU’s continued to iteratively test and improve the app over 12+ months through the BetaTesting platform. Testing focused on both UX and QA: User experience tests were conducted with parents/children, while a more general pool of testers focused on QA testing through exploratory and functional “test-case driven” tests.
“The testing results have been very helpful to fix bugs and issues with the app, especially in cases where we need to replicate the real environment in the USA for the end user” – BYJU’s Beta Manager
Here are some examples of the types of tests run:
- Collect user experience feedback from parents/children as they engage with the app over 1 week – 4 weeks
- Ensure video-based content loaded quickly and played smoothly without skipping, logging bug reports to flag any specific videos or steps that caused issues
- Test the onboarding subscription workflow for free trials and paid subscriptions
- Complete specific quizzes at various grade levels to explore and report any bugs / issues
- Test content on WIFI or only on slower cellular coverage (3G / 4G)
- Testing on very specific devices in the real world (e.g. Samsung S22 Galaxy devices) to resolve issues related to specific end-user complaints
Testing continued on a wide range of real-world devices, including iPhones, Android phones, iPads, Android tablets, and Fire tablet devices, which are popular for families with young children.
Development and Testing for an innovative Worksheet product
As the Early Learn app neared readiness, another team at BYJU’s was responsible for the development and testing of an innovative new worksheet product. The product would combine the Osmo system with a new mode in the Early Learn app (worksheet mode) to allow children to complete real-life worksheets which were automatically scored and graded. Engaging characters provided tips and encouragement to provide an engaging environment for learning.
Initial testing for the Worksheets product included:
- Recruiting 50+ families with children spanning various grade levels (PreK, K, 1, 2, and 3).
- Coordinating and managing product shipment to each family
- Recording unboxing videos and the initial setup experience on video
- Daily / weekly feedback and bug reporting on engagement and technical issues
- Weekly surveys
Results:
Testing revealed that families loved the worksheets product, and children were captivated by the engaging learning environment. However, there were bugs and user experience issues to resolve related to setup, as many users were confused about how to set up the product and use it for the first time. There were also technical issues related to the accuracy of the camera, and correctly detecting and grading worksheets. Lastly, a percentage of the magic worksheet markers seemed to be damaged in shipment or simply not strong enough to draw lines that were readable by the camera.
All these issues were addressed and improved through subsequent iterative testing cycles. Additional tests also included the following:
- Tests to capture the entire end-user experience, from reviewing product details online (e.g. on Amazon), purchasing the product, and receiving it in the mail.
- Tests with premium users that opted into the paid subscription.
- Continued QA and UX testing with targeted groups of real-world users.
Ongoing testing revealed that the product was ready to launch, and the launch was a massive success.
Learn about how BetaTesting can help your company launch better products with our beta testing platform and huge community of global testers.
BetaTesting Blog 