Measuring the effectiveness of VR skills training is vital to understanding its usefulness in training the future of vital industries. In 2022-23, Transfr conducted two experiments in its Louisville, KY laboratory. Participants were local community members with no prior experience performing the tasks they were trained on. The experiments were designed to assess learners’ ability to transfer knowledge from VR to the real world and retain that knowledge over time.
A fun part of this research was seeing people excel at tasks they’d never tried before. When using the real objects, participants would often refer back to the VR lesson, saying things like, ‘Oh, this is the part the virtual coach said I had to be careful with. Afterwards, they’d often say, ‘I remembered more than I thought I would!’ That s a testament to the quality design of Transfr's VR simulations.
Learners were able to transfer most of what they learned in VR to the real world. When given real tools and objects, the average participant was able to accurately perform 75% of the steps in cleaning a paint gun and 66% of the steps in fabricating a flexible fluid line. Even individuals who had never used VR before (28% of participants) were able to complete more than 70% of the paint gun task and more than 60% of the fluid line task. Learners reported the sims were highly usable and produced a strong sense of being present in the virtual training environment.
Learning only matters if people remember what they learn and this research showed that VR training improves knowledge retention. One week after their training sessions, participants could accurately perform significantly more of the task that they’d learned in VR vs the task learned from a slides-based presentation (t(15) = 2.17, p < .05).
The average participant retained a majority of what they’d been taught in VR versus forgetting a majority of what they’d been taught in the slides. Peak performance was 86% following VR learning versus only 71% following slides-based learning.
In both experiments, participants were asked to rate their agreement with the statement, “I think I would like to use training simulations like this frequently” on a 1-to-5 scale with 5 indicating strong agreement. Responses were overwhelmingly positive. The average response was 4.0 (SD = 1.2) for the paint gun simulation and 4.2 (SD = .82) for the fluid line simulation. Fully 40% of participants responded with the strongest possible agreement.
Testing our learning content and products in a lab is rare in ed tech industries, unlike the medical field. We observe and analyze people's interactions and experiences in a well-designed experimental setting to measure their learning. This is how we bring science into the product improvement process.
