In this new digital age where the number of apps can seem infinite, it’s important to consider whether or not the application you are considering has been shown to lead to actual learning. This is where research comes in.
Was the app created based on research and learning science? Are educators and researchers involved in developing the app and content? Has the company conducted any research on how well it works? If so, was this research conducted in-house or by a non-biased third party?
How does a company know if its language app really works?
In May of 2023, the Educational Testing Service (ETS) published an article on evaluating the impact of Ed Tech products. The article argues that it is essential for digital learning tools to be grounded “in evidence-based research to promote effectiveness in achieving learning outcomes” (Coggeshall, 2023, p. 4) and details the processes ETS follows to ensure all its products are based, tested, and evaluated on research.
The article encourages companies to create a body of research on their product to produce an “efficacy narrative” i.e., “a collection of studies, reports, and case studies that describe the effectiveness of an intervention across a range of settings, populations, and outcomes” (Coggeshall, 2023, p. 8). Companies can collect several types of evidence, which range from least to most rigorous (Figure 1), to determine the effectiveness of their product.
Figure 1.
IMMERSE continuously gathers data on the efficacy of our product, and we have partnered with 29 universities around the world to ensure that we have non-biased, third party academic research. Our university partners are not the only way we ensure our platform is grounded in research, though. We collect every type of evidence outlined in Figure 1 to ensure that our language-learning platform delivers real results.
IMMERSE’s Efficacy Narrative
Research can play a significant role in two parts of the discovery and ideation process for any Ed Tech company: ensuring the product meets the needs of users and that it is based on rigorous research.
To make sure a product meets users’ needs, it is necessary to understand their existing behaviors and pain points. At IMMERSE, our Product team does this by conducting regular user interviews, surveys, and observations that allow us to understand our users’ learning journeys and how our platform can be improved to help them achieve their goals.
At the same time, a wealth of scientific research has already been conducted on language learning.
Whenever IMMERSE develops a new product feature, such as assessments or gamification, we tap into this research to ensure the solutions we are designing are known to improve learning.
Our Research team gathers peer-reviewed research studies and pulls out and synthesizes key insights and recommendations for our Product team. The Product team then keeps this research in mind when designing new features to make sure every new product update and feature is grounded in learning sciences and will support our users’ language learning.
The next step up in the ETS framework is feedback from experts in the field. At IMMERSE, this process begins internally, when experts from our Education team join Product meetings to give feedback on new prototypes and verify they align with best practices in education.
After a feature is developed, our university research partners also play a role in giving product feedback. Specifically, some of our earliest research partners from the International House Manchester, Technological University Dublin, and Kyung Hee University primarily conducted studies evaluating IMMERSE’s pedagogical strengths and weaknesses. Our Product team was then able to use this feedback to improve our platform.
Gathering feedback from actual learners is the important next step in evaluating the efficacy of an educational product. Again, IMMERSE collects this data both internally and externally. Internally, we gather feedback from users via one-on-one and focus group interviews, surveys, testimonials, and app reviews.
Our external research partners from Aoyama Gakuin University, The University of Sheffield, and Kanda University of International Studies have also conducted qualitative research studies with users to gather feedback on their perceptions of IMMERSE as a learning tool.
This allows us as a company to see our app from a users’ perspective, showing which features help them learn and achieve their goals and which could serve them better with a few improvements.
The highest levels of evidence in the ETS framework are quantitative research studies. Rigorous quantitative studies allow companies to statistically evaluate the relationship between product use and learning outcomes. Ideally, these include a randomized controlled trial so a company can isolate the effect of their product. Ideally, they are conducted by third-party external researchers and published in peer-reviewed academic journals.
The IMMERSE Research team conducts its own internal analyses of how learning outcomes are driven by the amount of time students spend on our platform and what types of activities they engage in. Our external university research partners, however, are the ones who conduct control-experimental research studies evaluating how effective our platform is for language learning.
By partnering with external researchers, we achieve two main goals: 1) getting unbiased, third-party data to inform our product development and marketing and 2) scaling the amount of research conducted on IMMERSE. Since university researchers are strongly motivated to publish their work in peer-reviewed journals, credible research articles on Immerse get distributed in the language learning field.
For example, partners like Kyung Hee University, the University of Illinois at Urbana-Champaign, and Chuo University have published data on how IMMERSE improves students’ learning outcomes in journals such as Education and Information Technologies, the RELC Journal, and the Journal of Global Informatics. This research has helped IMMERSE better understand how our immersive environments, object interaction, and avatar use impact students’ learning processes.
Both virtual reality and artificial intelligence are developing at a rapid pace, and CALL (Computer Assisted Language Learning) researchers are constantly shedding new light on how these emerging technologies both support and hinder learning.
At IMMERSE, all of this work contributes to our efficacy narrative and understanding of how we can best help our users achieve their learning goals.
For more research on the effectiveness of VR and IMMERSE for language learning generally, please visit the IMMERSE Research page.
