key: cord-258269-ig8i9278 authors: Philippe, Stéphanie; Souchet, Alexis D.; Lameras, Petros; Petridis, Panagiotis; Caporal, Julien; Coldeboeuf, Gildas; Duzan, Hadrien title: Multimodal teaching, learning and training in virtual reality: a review and case study date: 2020-10-31 journal: Virtual Reality & Intelligent Hardware DOI: 10.1016/j.vrih.2020.07.008 sha: doc_id: 258269 cord_uid: ig8i9278 It is becoming increasingly prevalent in digital learning research to encompass an array of different meanings, spaces, processes, and teaching strategies for discerning a global perspective on constructing the student learning experience. Multimodality is an emergent phenomenon that may influence how digital learning is designed, especially when employed in highly interactive and immersive learning environments such as Virtual Reality (VR). VR environments may aid students' efforts to be active learners through consciously attending to, and reflecting on, critique leveraging reflexivity and novel meaning-making most likely to lead to a conceptual change. This paper employs eleven industrial case-studies to highlight the application of multimodal VR-based teaching and training as a pedagogically rich strategy that may be designed, mapped and visualized through distinct VR-design elements and features. The outcomes of the use cases contribute to discern in-VR multimodal teaching as an emerging discourse that couples system design-based paradigms with embodied, situated and reflective praxis in spatial, emotional and temporal VR learning environments. Digital teaching and learning embrace active pedagogy and learner-centred approaches. The basic assumption is that learners are unique and therefore learning should be personalized. As argued by Reigeluth et al. [1] , learner's centred activities value intrinsic motivation [2, 3] as well as metacognition [4] for a more personalized and meaningful learning process. For achieving an active, situated and embodied learning experience, meta-cognitive knowledge seems to enhance learning [5] . In line with this assumption, one can consider specific pedagogical methods such as active learning, learning by doing, collaborative learning, problem-based learning and game-based learning. Prince indicates for instance that active learning has a positive but weak effect on students' academic achievement [6] . Vernon and Blakes metaanalysis indicates that problem-based learning may yield metacognitive learning over traditional methods [7] . This is also supported by Dochy et al. meta-analysis that concludes for robust positive effects on student's skills without studies reporting negative effects [8] . Those conclusions are also in line with Walker and Leary's meta-analysis [9] . Based on previous meta-analysis and systematic reviews [10] [11] [12] [13] [14] [15] , serious games [16] may also be efficient in various contexts, if combined with informed learning instructions. In line with the idea of "Active pedagogy" and "Learner-centred approaches" the concept of "Multimodality", was developed in the early 2000s [17, 18] . Multimodality refers to "multiple" modes of representation, with combined elements of print, visual images and design. This shift from paper-based education to multimodal education involves rethinking the way in which teaching, and learning are designed, approached and practiced. This promotes the way in which pedagogies, content and technology are designed and used to enable multimodality to take place in a variety of contexts and social relationships [19] . Whether digital tools are employed to scaffold new ways of learning or they are just reproducing traditional modes of teaching and learning is still an open research question [20] . Digital tools are at least as efficient as non-digital tools, when integrated into a consistent pedagogic strategy, with clearly defined training objectives and instructions. Therefore, the purpose is not to replace traditional methods by digital tools. Rather, one should consider digital tools for the actual value they can bring, for instance regarding the personalization of the content, in combination to non-digital tools and methods. Digital tools are part of the toolbox available for the trainers and they can contribute to the development of multimodal, active and learner-centred approach. In this context, Virtual Reality (VR) was recently introduced to the consumer market [21] and deployed in digital educational interventions [22] . Education and training are pointed out as promising fields of VR implementation [23] . The advent of immersive and high-fidelity digital technologies such as virtual reality may supplement or enhance analogue learning spaces as modes of expression. For example, Öman and Hashemi suggested that technology may be used to increase students' communicative and collaborative skills instead of focusing on how to use the technology from a technical perspective [24] . VR is shown to offer self-regulated and multiple learning choices compared to other learning platforms which lead to high quality of experience and retention [25] [26] [27] [28] [29] . However, the use of VR in the context of multimodality for teaching and learning has not been investigated yet. As an example, a search using the terms VR and multimodality and teaching or learning for any peer reviewed article published between 2010 and 2020 on Science Direct (Elsevier) led to 0 hit. The IEEE database only retrieved 5 articles. The first one relates to the use of listening strategies by learners in Second Life VR environment [30] . A second one relates to multimodality in VR but without teaching and learning purpose [31] . The three other papers are totally unrelated to the topic of interest [32] [33] [34] . This shows that Virtual Reality and Multimodality are rarely tackled scientifically as a connected issue. We propose that multimodality may be investigated as a teaching and learning process in terms of ways of enacting it within a VR's semiotic domain as a design space that affords constructivist and activity-based learning. Multimodality may be viewed as the vehicle for students to design and orchestrate their own modes of learning that are meaningful to them in the form of multimodal ensembles and semiotic resource [35] . Such multimodal ensembles may include images and language along with more static modalities resembling frozen actions [36] such as classroom objects and equipment incorporating desks, tables, displays, chairs, books and chalkboards. There is a plethora of evidence on how multimodality may be deployed in traditional learning environments [37] [38] [39] , in blended learning spaces [40] [41] [42] and online learning spaces [43] [44] [45] . There are constellations of evidence on employing multimodality as means to experience teaching and learning in digital and analogue semiotic domains. Yet, there is infrequent substantiation on how meanings of words, images, communication processes, teaching strategies, roles and learning activities may be situated in a VR semiotic domain. This paper attempts to illuminate the application of multimodal teaching and learning as a pedagogically rich strategy that blends, couples and combines constructivist-led practice. Such strategy constitutes distinct ways of thinking, acting and interacting. Essentially, it allows to experience teaching and learning in unique ways. In Section 2 hereafter, we present an overview of the literature supporting how VR may be used in learning and teaching as means to increase learning efficiency, based on recent developments in designing and representing learning content and activities through VR. In Section 3, we propose a methodology for integration of VR into multimodal learning path and we illustrate this methodology in Section 4 with eleven different VR case-studies resembling different design approaches to map multimodal learning with VR features and components. Finally, in Section 5, we evaluate this proposition considering the current situation of the pandemic and how VR and emerging XR (eXtended Reality) technologies can contribute to reshaping the learning and teaching fields while physical distancing tends to be normalized. Digital learning technologies aim to help learners increase their capacity for innovation, leadership, multiand inter-disciplinary collaboration, emotional intelligence, critical skills, collective problem identification and solving skills, in a participatory environment [46] . Multimedia resources and tools in these environments include, e. g.: videos, interactive images, recorded presentations, online quizzes, discussion forums (synchronous and asynchronous), visual representations of learner data to describe progress (summative analysis) and what learners are doing to learn [47] . The increasing use of multimedia in education and training offers the possibility of presenting content in multiple representations (text, images, video, audio, ubiquitous media) to accommodate different teaching and training strategies, learning outcomes, assessment methods and feedback mechanisms. Key aspects are the use of a range of tools, resources and services in a pedagogical manner to enhance the students' experience. The integration of multimedia learning into different modes of learning seems to encourage learners to develop a more flexible approach based on inquiry and information retrieval. Hazari [48] and Mayer et al. [49] argue that student learning is deeper and more meaningful when a range of interactive tools and resources are deployed rather than using text alone. Shah and Freedman [50] list the benefits of using visualizations in learning, such as (1) external representation of information, (2) deeper learning, (3) triggering learners' attention and concentration by making information more complete, thus simplifying ill-defined concepts and ideas. These tools can be serious games, and virtual reality. These technologies allow us to go beyond standard forms of written and spoken language [17] . VR environments are being envisaged as a medium that aid learners' efforts to be active. Learners consciously attend to, and reflect on, critique leveraging reflexivity and novel meaning-making leading to a conceptual change. There is an enlightening research spectrum on measuring the instructional effectiveness of immersive virtual reality [51] [52] [53] , understanding trainers' conceptions of teaching in VR [54, 55] , students' attitudes towards VR [56, 57] and trainer professional development using VR [58] [59] [60] . There are also studies that investigate associations between specific pedagogies and theories 423 with VR [61, 62] embroiling constructivist models of learning with the premise to individually and collaboratively construct knowledge and experience through critical learning and thinking. In summary, these previous works indicate that learners respond to the environment with an interaction-reaction opposition: they build their own knowledge and social interactions are primordial to that building [63] . The advent of VR head-mounted displays has encouraged the development of an array of VR environments particularly used for augmenting the student learning experience [23] . In this context, VR may be broadly described as an experience in which students interact within a 3D dimensional world with body and gestural movement, experiencing interactive content such as images and sounds [64] . A distinctive characteristic of a VR system is that students can interact and manipulate objects by emulating how objects are manipulated in the real-life [65] . Commentators and researchers alike have attempted, through metaanalyses and systematic reviews, to discern the processes, strategies and methods that are most likely to be aligned with VR tools, elements and features (e.g. for Language Learning) [66] . There is consensus that VR may promote activity based and student-centred learning, as proliferated in the constructivist learning paradigm, whilst attaining student motivation, self-regulation and self-assessment. A key advantage of learning using VR is that students can view objects and content from multiple perspectives and thereby situating learning in line with subject contexts. For example, VR allows rich learning environments such as factories or real-life like working spaces, particle physics events and brain anatomy. It affords opportunities to learn through interacting with virtual objects leading to creating new cognitive schemata tied to situated learning instances. In conjunction to this, spatial perception and cognition such as acquiring navigation and localization skills within a VR environment may be codified and represented as thirdperson symbolic experiences increasingly supporting and amplifying a sense of social and selfpresence [67, 68] . Cooper and Thong highlights four distinctive elements of VR as an educational tool: (1) Experiencing as the ability of students to respond physically and emotionally to a range of stimuli, (2) Engagement as the multisensory experience that may enhance student's engagement, (3) Equitability as ways of responding to sameness and differences in schools and (4) Everywhere as offering exciting possibilities in relation to location, timeliness and how the learning process emerge [69] . Usual VR advantages listed in the literature are embodying, acting, repeating [70] , and increasing motivation while learning compared to other media [26, 71] . Yet, leaners' acceptance and learning instructions' creation are still restraining large adoption [65, 72] . In education and training this can be explained by lack of on-campus experiments [73, 74] . Globally, previous meta-analyses (2 studies), systematic reviews (1 study) and reviews (2 studies) documented VR efficiency for learning [26, 70, 71, 75, 76] . But, according to Lanier et al. experimental quality in VR is sometimes questionable due to the methodological challenges faced on the study design, data analysis reporting and disseminating the knowledge gained [77] . Within the issue of learning with VR in the context of a multimodal path, the ability to create efficient teaching and learning environments as well as strong experimental proofs depends on design principles that are applied. Despite the considerable uptake of VR for learning and teaching, there is little, if any, evidence on designfocused studies that illuminate in-VR elements and features that focus on the affordances and constraints as well as the dynamics perpetuated to support multimodal teaching using VR. An early study from Dickey [78] investigated the potential of a VR to support activity-based multimodal teaching and learning through an evaluative case study. VR elements that afforded an activity-based and multimodal approach to teaching included an in-VR chat tool as the primary means for presenting a concept for discussion. Responsive feedback and interaction with the students were the main learning affordances along with multimodal information presented as visual illustrations. Another affordance was granting unique names for students to establish unique virtual identities for maintaining control over the learning environment. Such virtual identities were inextricably connected with avatar representations [68] . Pre-selecting, modifying or creating new avatars helped students to distinguish their virtual appearance and learning about and coming to appreciate design in their efforts to apply design principles for creating their avatars by manipulating avatar objects, shapes, colours and attributes. Kinesthetics and point-of-view aspects for avatars to interact with objects, within the virtual learning space were directly linked with the provision of an 'avatar' mode for individual and collaborative activities. An integrated web-browser was also viewed as a feature that can instigate multimodal learning especially when connected to in-VR learning objects via sensors for allowing students to make relationships between the VR object and its underlying information found on the Web. This inter-connection between in-VR objects and information about them on the Web alludes to the employment of distinct semiotic principle [35] . Learning occurs through interrelations within and across multiple sign systems (symbols, objects, images, facts, information) as an inter-related and connected knowledge from different semiotic domains. More recently, Doumanis et al. investigated the impact of a multimodal learning interaction of gamified tasks in a collaborative virtual world [79] . Multimodal interactions within the VR seemed to improve learning in comparison to the non-multimodal control group. Specifically, the multimodal interactions observed improved students' ability to generate ideas thus facilitated a sense of presence and immersion with the VR condition. Doumanis et al. triggered three types of immersion (e. g. spatial, emotional and temporal) aligned with VR features. Navigation in the VR world with speech control and virtual representation with an avatar along with access to information, user grouping, textual communication and dialogue log were central features for encouraging active, multimodal and critical, as opposed to passive and unimodal, learning. Principles of active learning were embedded in the use of the VR features creating certain dynamics and controlling essential features. For example, teachers should have full control of the VR classroom in terms of controlling student navigation or a "proxy option" to temporarily take control of a student's avatar as means to facilitate their effort to learn the system or grouping students for collaborative in-VR projects having students working in teams and taking on assigned roles. Therefore, VR allows more interaction opportunities for learners with peers, content (e. g. information) and objects (digital assets). Designing in-VR group dialogic learning experiences enables for distributed knowledge and collaborative problem-solving, encouraging perception of thinking and reasoning as inherently social processes. Designing VR elements creates learning situations for students to think with others by using and manipulating VR tools and places emphasis on the distributed knowledge product generated by a web of students working for resolving a common problem. Innocenti et al. developed a virtual environment for learning how to play musical instruments [67] . Similarly, to Dickey [78] and Doumanis et al. [79] , navigation elements is a key multimodal feature as means to provide spatial orientation cues for learning and usability aspects. For example, to mitigate VR sickness while students are interacting and manipulating 3D objects for designing a prototype or researching an object, a virtual locomotion technique [80] may be induced to offer natural, usable and efficient ways for multimodal driven activities to be navigated through and enacted in the VR environment. Navigation elements need to be tied to in-VR collaborative scenarios for aiding students to perform tasks, set by the teachers or by peers, for practicing the intended learning outcomes. Collaborative scenarios may encompass pre-determined designs of the VP-space such as scaffold for helping the students to move within the VR environment, progressively learning how to interact with objects, making the in-VR 425 goals clear and distinguishable and encouraging exploration, inquiry and observation of how the different modes reveal intended meanings. VR is a semiotic domain that triggers students to learn in different ways, applying an array of developmental skills and competencies as they move from one VR scenario to the next. The role of the teacher and the student within the VR space are changing depending on the scenario, as the general premise is that there is no single master of knowledge. Rather each member takes roles with associated skills to master such as being the researcher, developer, designer or project manager in different settings. In that way, people with varied skillsets and dexterities have the possibility to exchange their roles and learn from each other. As opposed to designing unimodal online learning environments (e.g. creating a moodle page for students to download content, developed by the teacher), VR scenarios may be designed in a way that afford both a change in practice but also a change in identity [81] . This can be done by distributing and re-distributing students into diverse VR groups, switching different roles interchangeably and sharing knowledge mastered from participating in previous VR groups through employing reciprocal problem-based in-VR learning scenarios. The premise is that there is no 'master' of knowledge in the sense that knowledge construction and especially knowledge building is a collaborative process through a network of people that distributes pervasive multimodal information, roles and responsibilities. Multimodality as a context-based and situated learning instantiation may be designed for and represented through collaborative VR, as part of XR. The most common collaborative VR features are: (1) focusing attention, (2) connecting learners to the learning materiel [82] . Collaborative VR [83] may encourage multiple perspectives on a given phenomenon through conversation and interaction and joint construction of knowledge [65] (it echoes the four pillars of learning [84] ) by providing feedback to facilitate the adoption of learning reflexes [23] and monitoring of scenario development; allowing Distant learning [85] . Such collaborative practices in VR may encourage the formation of social identities and viewing knowledge as a social construct developed through a network of individuals having common goals and interests [86] . By employing Collaborative VR teachers may design virtual places that afford collaborative learning processes that take social interactions into account offering a more diverse and richer forms of dialogue that would be challenging to design or construct in other learning environments [87] . Individually or collaboratively, VR allows for learning instances to be embodied (to be represented by an avatar) [88] . Such embodiment gives a unique dimension illuminating a learning by doing orientation rather than only passively memorising and acquiring information [70] . This offers richer and more diverse forms of dialogue and interaction between students and contextualized learning objects for vicarious forms of learning [89] . Being immersed in VR creates a sense of presence, it allows learning from each other and adapting performance in response to meaningful pseudo-natural feedback [90] generated from interactions with 3Dobjects. Reflecting on this pseudo-natural-occurring feedback may cause to transform a daunting learning experience to a harmonious learning situation [91] which can improve learning effectiveness compared to other modalities [92] . VR allows unique teaching and learning experiences which, by design, makes it interesting to implement in existing multimodal paths. Currently, the industry may not always seize the opportunity to apply such design principles, yet VR is starting to be introduced in multimodal paths across different subject areas and disciplines. The purpose of this section is to illustrate the variability of VR application into existing learning paths with examples coming from the field, schools, universities and companies in France and Singapore. Some were implemented in France, others in Singapore, or both and in other countries as well. Accordingly, the games are available in several languages, as illustrated in the following Figures 1 to 12 . Each example, connected to a general purpose, is described according to certain in-VR teaching and learning goals: knowledge transmission, practicing, feedback, evaluation. For each goal, we identify the activity which can be implemented. We distinguish the paths according to the nature of the VR experience that is integrated: Serious games, Simulations, Collaborative VR (see Table 1 ). (1) Serious Game [93] section: we describe several use cases integrating a serious game in VR, single player, including feedback to the player. The user is facing a situation with a non-playing character, embodied with an avatar or only a voice. Feedback are provided immediately during the game as well as at the end of the game and help the user to improve. Such application is particularly relevant to train soft skills, such as how to behave with a client, a patient, a colleague. (2) Simulation [94] section: we describe several use cases integrating Simulation in VR, which purpose is to provide a relevant representation in VR of a target system to be learned (a machine, an organ, a network, etc.). Such a tool is relevant to train users to interact with said system and learn procedural sequences or gestures. (3) Collaborative VR [95] section: we describe several use cases integrating Collaborative VR, i. e. a virtual environment where participants can join and are embodied with a personalized photorealistic avatar. Participants can be distant or in the same room. They can share immersive content, such as interactive 3D models, 360°videos or role-playing game. Such paths, embedding VR experiences along with other activity-oriented multimodal activities, could infer specific in-VR features, representations and visualizations mapped with intended learning outcomes. In addition, the association of multimodal activity-based teaching strategies would enable both individual and collaborative practices in the wider semiotic VR domain within which they occur. Such deployments are being evaluated through quality of experience in order to collect user's state with such apparatuses [96] . The VR tool is proposed as a Practicing activity opening a seminar to foster equality behaviour by managers, regarding gender, disability and diversity. In gender awareness scenarios, the user can play whether a man or a woman. The game points discrimination and stereotypes through dialogue choices the learner must complete. It is part of a general company awareness policy regarding gender inequalities, aiming at understanding how many stereotypes about women drive once thoughts in the work environment, and how much these stereotypes influence our choices when it comes to promoting women. Other scenarios relate to physical and cognitive disabilities as well as sexual orientation. Figure 1 illustrates 2 situations and the 3 options that are offered to the player, following the last statement of the Non-Playing Character (NPC), in English, as deployed in Singaporian company. In this example, we aim at generating empathy by playing the role of some else, a woman or a disabled person. Using the same approach, we propose another application, which will be integrated as part of a Medical and Dentistry and resulting from a French-Singapore partnership. The purpose is to play the role of a child in the medical environment to better understand his.her point of view and anticipate his.her fear and anxieties. This application will be used for Practicing. Figure 2 illustrates the point of view of the child as a patient in different situations: on the dentistry chair (left) and during a discussion with the doctor and a parent (right). Clues in the field of view support the role of the child, such as the view of the child body (Figure 2, left) . 427 The VR tool is proposed as a Practicing activity opening a course in a business school to train students to customer relationship management in a shop. The VR game, including an individual feedback, is played in the classroom and followed by a general discussion with the teacher. Then evaluation takes place using traditional paper and pen methods, in the classroom. This program organization is very close to the previous one, although it takes place at school. Figure 3 illustrates the overall view of the shop (right Figure 1 Single user serious game in VR for social norms and behaviour changes relating to discriminations. picture) and a zoom on a particular shop area, juice machine, where actions are required from the player (left). The VR tool is proposed as a Practicing exercise on top of other applied exercises of a one-week training program for future shop managers. Learners play one half of a day a serious game training them to the management of costumers' satisfaction. The purpose is to be able to answer questions for the best customer service experience possible. The game allows learners to memorize typical issues customers can encounter in respect with companies wording and policy. Figure 4 illustrates the shop where the game takes place (left) and the client NPC the player is interacting with during the game (right). The red dot represents the player gaze direction allowing the interaction with the environment, such as the selection of answers. The VR tool is proposed as a Practicing exercise embedded in a path for developing skills for front desk employees in the hospitality industry. The overall program also includes several individual debriefing sessions with a trainer, who can be distant from the trainee, flashcards to contribute to the retention. The entire path is validated through an assessment quiz certifying learning (retention). Figure 5 illustrates several situations of the game: with 2 client NPCs and 4 possible answers from which the player has to select (left, in English) and a single client NPC (right, in French). A "pause" button is shown as well as the score, showing the player progression in the game. This score contributes to immediate feedback to the player, indicating whether the selected answers are good or not regarding the quality of the customer relationship management. The VR tool is proposed as a Practicing and Evaluating activity, used as a closing applied exercise. The serious game consists in playing a thief trying to steal sensitive and confidential information from PCs, USB memory device but also paper notes. The purpose, as a player, is to steal as many information as possible during a short period of time. Learners' identify sensitive information unprotection behaviours. The game also provides feedback and an evaluation of the player. In this context, the VR game is added to a face-to-face (physical) training program for any worker in companies for them to learn cybersecurity rules. Figure 6 illustrates some objects the player can interact with to determine those that can present a risk regarding cybersecurity, these objects are identified with yellow circles. This learning scenario is implemented in France, in Singapore and other countries. The VR tool is provided as a Practicing and Evaluating activity among multiple applied exercises in a training program for operators on production line in pharmaceutical industry. The VR consists in a simulation of a true machine with the actual procedure for performing assembly task. Each step is written and an agent (a voice) is guiding the player step by step through the process. The purpose is to allow learners to get confronted with the actual machine and pieces before doing it in real life. Several modes allow progressive learning: a guided and a semi-guided mode for Practicing and non-assisted mode for Evaluation. The training program also includes several activities, collective and individual. Figure 7 shows game interface elements: the different steps of the game are presented on the left picture, showing the player is currently playing step 1, instructions are presented to guide the player (here "take the grease tube and grease the joints"), the objects required for the action are highlighted with blue halo (here the joints and the grease tube), on the right picture the player can see his/her hand manipulating the spanner. The VR tool is proposed as a Practicing exercise for two players integrated into a training program for developing skills communication skills along an inter-job process. In two different VR environments, one learner is driving the train while the other learner is communicating at distance from the Control place to give indications to the driver. During that time, the trainer is supervising the simulation without intervene. The trainer is annotating the scenario so he can do a debriefing of specific points with learners after the simulation. The purpose is to make learners memorize procedures while communicating and driving a train. Figure 8 illustrates the game environment. The VR tool is provided as a Practicing tool, which can be used by neurosurgeon students to get used to brain anatomy and 3D manipulation. VR is added to help student to improve their 3D navigation skills, it consists in merged neuroimaging from patients' brain, with addition of artefact mimicking tumours. Students have to identify their precise localization and size. A collaborative mode allows teachers and students to use the application collectively. Such application contributes to downing and replacing part of real brain tumour surgery training. Figure 9 shows the user interface with a measurement tool (left) and the control panel to set the various display parameters (right). The controllers are represented in the virtual environment to facilitate the manipulation and interactions. The collaborative VR tool is provided for Knowledge transmission as an innovative medium to share contents relating to process engineering in a common virtual environment, with students and teacher located in the same classroom. The course takes place in an engineering school. The VR is added as a new mode of the pedagogical multimodal path, integrated with traditional lecture, group assignment, and pen and paper evaluation. Figure 10 shows the setup of the students in the classroom (left) and the setup in the virtual classroom (right). Virtual Reality & Intelligent Hardware 2020 Vol 2 Issue 5：421-442 The collaborative tool is provided for Knowledge transmission as a new mode of the pedagogical multimodal path, allowing to share immersive content (3D interactive models and 360°videos and pictures). The purpose of using VR is to allow students to better understand the actual dimensions of equipment used in particle physics (i.e. colliders) and have a more realistic representation of the experimentation site, without physically visiting Switzerland. During a regular session, high-school students follow introductory courses to particle physics in the morning, then individually sit in front of a computer to identify and sort real particle-collision events. They finish by sharing and piling up their results (together and with other classes around the world), to reach and experience the statistical methods and thresholds of particle discovery. VR is used to lecture students about the real size and lay-out of the gigantic apparatus that allow particle discoveries: an interactive 3D sketch of the CMS (Compact Muon Solenoid) detector (the lecturer and students are teleported to scale on the CMS detector) with particle collision events and then 360°images and video to show the "real" images. The main advantage is to show an environment that cannot be illustrated in a regular classroom, because of its size and inaccessibility. Figure 11 shows the setup of the students in the classroom (left) and the setup in the virtual classroom (right). A view of the virtual environment is shown on the screen in the physical classroom for non-VR participants. The 3D model is shown at real scale, so that the students can better measure it. The Collaborative VR tool is added at different time points for Knowledge transmission, Practicing and providing Feedback as a communication tool in a multimodal path. The general purpose of the training program is to promote learners' leadership skills. It is implemented in an international company with managers from several countries. The use of VR allows participants to connect more easily and regularly without needs for traveling. The trainer can control participants' audio, allowing everyone to hear each other or isolating tables from each other for specific activities. This program has notably been implemented with an international company with 34 managers from 13 countries in Asia-Pacific area, including Singapore. Figure 12 illustrates the view of the virtual environment from the desktop interface of the trainer. It shows the participant list and allows the management of the participants groups (the "tables"). In this paper, we highlight the connection between VR learning and multimodality and the instruction concepts related to it. This paper concentrates on VR but new hardware allow varying degree of "virtuality" [97] . New generations of VR HMDs also offer AR modules with a "see-through" capability for rapid switch from computer generated environment to real environment and any mixture thereof. tracking, voice tracking and body tracking. Haptics and brain computing are also fields undergoing rapid progress and will be launched soon. All these technologies enable Natural User Interface [100] . This comes along with deployment of 5G networks and possibility for streaming and cloud-based application architectures. This is already used for gaming, this will soon benefit to VR. Shadow, a cloud gaming platform, is launching what it calls a "VR Exploration Program", i.e. a closed beta test for VR streaming [101] . These new generations of HMDs combined with opportunities of beyond-5G networks will allow the development of genuine Mixed Reality experiences with fully natural interactions. Changing the habits may take time, said Alvin Graylin, CEO of HTC China, but for the benefit of users. Hardware companies' strategies thus strongly impact the design of learning and teaching solutions that can be developed in France, in Singapore and globally. This paper concentrates on learning and teaching but in the COVID-19 crises context, other uses of XR are wildly discussed. Prior to that crisis, VR was still emerging in some sectors. Several factors are limiting the adoption, including the cost of equipment and complexity of implementation. From a business point of view, these limitations to adoption may be observed both in France and in Singapore. In contrast, adoption of mobile technologies is much more significant, in line with the smartphone mobile equipment rate. While the future of VR was highly associated with entertainment and gaming in some recent market studies [102] , one may expect that these trends will be impacted by the current context of physical distancing and climate issues. Therefore, it could be assumed that the evolution of the VR and XR technologies will support the development of new experiences for remote working, authentic and content rich mediated learning and healthcare contexts. Limiting people travels is a key aspect of the strategy to control pandemics, such as constraints as an opportunity and proposed a virtual version of the event. It has been the case of the V 2 EC 2020, the Virtual VIVE Ecosystem Conference that was planned by HTC in China in March [103] and Laval Virtual planned in France in April [104] . These are in line with preliminary attempts such as the IDC conference hold in the Netherlands in September 2019, with the IBC 360 Live system initiated by Tiledmedia and Intel to live stream the five-day IBC 2019 conference globally in 8K, 360°virtual reality [105] . Of note, the use of VR for remote interaction, as described in use case #11 allows to significantly reduce traveling costs. As a consequence, the initial investment for individual equipment can be rapidly amortized. The impact of sanitary constraints has been significant, and it may last for 18 to 24 months more from now at a certain level. This will induce deep changes in habits notably with the support of technologies precluding a return to what the world was before COVID. Some describe a fully digital world, where XR technology will be the must-have, that's the position of Alvin Graylin, HTC China CEO. At least for the coming years, an adoption phase will be characterised by coexistence in hybrid systems, for instance with organisation of physical events with virtual version, increased use of home office and flex office, blended learning with on-campus and remote learning, etc. More than 70% of the world's students' population have been impacted by the pandemic and the closures of schools, estimates UNESCO [106] . The next school year may also be significantly impacted. This situation has revealed strong inequalities among Teaching systems regarding their digital maturity and their capacity to ensure pedagogical continuity. Although digital tools can bring significant advantages, for remote learning context as well as with the personalisation of education that undoubtedly bring support to students in difficulties, a strong private-public partnership will be essential to design and transform the Education system, says Marie-Christine Levet, CEO of Educapital, the first European Edtech VC fund [107] . This transformation will require support and training of the teachers and parallel adaptation of the technologies to their specific needs and requirements. Additionally, to hold the promise of a digital society, for learning, working and any other activities, XR technologies must be fully inclusive, and ensure accessibility to everyone, despite disabilities, would they be cognitive or physical. In this context, experimentation is the key to evaluate how technologies can be integrated into the teaching practice, especially in a multimodal perspective. This paper stipulated a review on designing and using VR for multimodal teaching, learning and training. VR and XR are highly versatile tools, which can be used for collaborative or individual activities, with distant or physically related participants at any step of the learning process: knowledge transmission, practicing, feedback and evaluation. We have described three main types of tools based on VR: Serious games, mostly for soft skills training, Simulation, mostly for procedural learning and Collaborative VR for immersion with innovative content and to facilitate interactions between distant participants. These are only a few examples and the advent of technologies will allow to combine them more easily (e. g. collaborative Serious Game and Simulation). The concept of Adaptive learning and Deep learning are also progressing along which will allow to create even more personalized and interactive learning experience as well as to provide indicators of individual progression to the trainer. However, we also identify a lack of robust evaluation framework to evaluate how these tools can be used in an optimized way and generate relevant synergies with existing tools, for the benefit of users: trainers and trainees. Our purpose is therefore to increase researchers' attention in terms of designing VR experimentations that could be inspired by the use cases we have described. Moreover, insights from this paper motivate us to list a certain number of research issues that need to be tackle. They are particularly aiming to apply scientific findings to the filed by pushing specific matters: -How to design learning in VR based on a multimodal strategy? -How to collect and analyse user data obtained in VR to discern a multimodal path? -How to determine whether a VR system should replace or complement an existing multimodal teaching and learning intervention enacted in the classroom? -What learning taxonomies may be mapped as means to enable in-VR multimodal teaching and learning for hybrid or purely online and distance forms of teaching and learning? A gap seems to be prevalent between scientific knowledge and industrial practices when VR is deployed in multimodal learning paths. Communicating about use cases and choices that have been made might help to create a framework facilitating the process of mapping multimodality to in-VR features and components. The authors declare no competing financial interest. Instructional-design theories and models. The Learner-Centered Paradigm of Education Intrinsic motivation Toward a tripartite model of intrinsic motivation Metacognition and learning: conceptual and methodological considerations. Metacognition and Learning Effectiveness of learning strategy instruction on academic performance: a meta-analysis Does active learning work? A review of the research Does problem-based learning work? A meta-analysis of evaluative research Effects of problem-based learning: a meta-analysis. Learning and Instruction A problem based learning meta analysis: differences across problem types, implementation types, disciplines, and assessment levels Serious game-based and nongame-based online courses: learning experiences and outcomes A serious game for amplifying awareness on multimodal teaching: game design and usability study From visual simulation to virtual reality to games Digital games, design, and learning: a systematic review and metaanalysis Serious Games: mechanisms and effects Origins of serious games Multimodality and literacy in school classrooms Multimodal discourse: the modes and media of contemporary communication Theories and practices of multimodal education: the instructional dynamics of picture books and primary classrooms. The Reading Teacher Visual fatigue impacts on learning via serious game in virtual reality Reality-creating technologies as a global phenomenon State of the art of virtual reality technology New applications Design and redesign of a multimodal classroom task -implications for teaching and learning Virtual reality training in laparoscopic surgery: a systematic review & metaanalysis A literature review on immersive virtual reality in education-state of the art and perspectives A systematic review of virtual reality in education Educational virtual environments: a ten-year review of empirical research Enhancing our lives with immersive virtual reality Listening strategy applications by learners under the context of multimodality Multimodal interface for temporal pattern based interactive large volumetric visualization Multi-modality behavioral influence analysis for personalized recommendations in health social media environment Deep multimodality learning for UAV video aesthetic quality assessment What video games have to teach us about learning and literacy Modal density and modal configurations Teaching poetry through collaborative art: an analysis of multimodal ensembles for transformative learning The effect of multimodal learning models on language teaching and learning. Theory and Practice in Language Studies Implications of Multimodal Learning Models for foreign language teaching and learning A Literacy Pedagogy for Multimodal Composing: Transforming Learning and Teaching Application of multimodal learning in online English teaching Engaging literature through digital video composing: a teacher′s journey to"meaning that matters Creating meaning through multimodality: multiliteracies assessment and photo projects for online portfolios. Die Unterrichtspraxis/teaching German The distance learning of foreign languages: a research agenda Authentic assessment in online learning: moving beyond text to celebrate multimodal measures of student achievement Learning, teaching, and scholarship in a digital age web 2.0 and classroom research: what path should we take now? Educational Researcher Innovating pedagogy 2016: open university innovation report 5 Journal of information systems education. Investigating Pedagogical Value of Wiki Technology Learning science in immersive virtual reality Investigating the effectiveness of desktop virtual reality for teaching and learning of electrical/electronics technology in universities Virtual health education: scaling practice to transform student learning: using virtual reality learning environments in healthcare education to bridge the theory/practice gap and improve patient safety Teachers′ conceptions and their approaches to teaching in virtual reality and simulation-based learning environments. Teachers and Teaching Teacher perceptions of learning affordances of multi-user virtual environments Teaching with virtual reality: crafting a lesson and student response. (report) Investigating learners′ attitudes toward virtual reality learning environments: based on a constructivist approach Improving teaching and learning of computer programming through the use of the Second Life virtual world Teaching training in a mixed-reality integrated learning environment Fostering learning through interprofessional virtual reality simulation development A constructivist approach to virtual reality for experiential learning. E-learning and Digital Media Emboldened by embodiment: six precepts for research on embodied learning and mixed reality Learning theories and educational paradigms Understanding virtual reality: interface, application, and design. Presence: Teleoperators and Virtual Environments A review of the use of virtual reality head-mounted displays in education and training. Education and Information Technologies Language learning in virtual reality environments: past, present, and future. Educational Technology & musical genre learning in primary education First-and third-person perspectives in immersive virtual environments: presence and performance analysis of embodied users Implementing virtual reality in the classroom: envisaging possibilities in STEM Education Enhancing our lives with immersive virtual reality A systematic review of virtual reality in education Virtual reality challenges in education and training Effects of virtual presence and learning outcome using low-end virtual reality systems Integration of virtual reality in secondary STEM education Educational virtual environments: a ten-year review of empirical research Virtual reality training in laparoscopic surgery: a systematic review & metaanalysis Virtual reality check: statistical power, reported results, and the validity of research on the psychology of virtual reality and immersive environments Teaching in 3D: pedagogical affordances and constraints of 3D virtual worlds for synchronous distance learning The impact of multimodal collaborative virtual environments on learning: a gamified online debate Virtual locomotion: a survey Situated learning: legitimate peripheral participation Teachyverse: collaborative Elearning in virtual reality lecture halls Collaborative virtual environments: an introductory review of issues and systems Virtual reality applications to work The state of immersive technology research: a literature analysis A literature review on immersive virtual reality in education: state of the art and perspectives. In: eLearning and Software for Education (eLSE) Implicit learning through embodiment in immersive virtual reality A taxonomy of virtual worlds usage in education Virtual reality headsets-a theoretical and pragmatic approach How immersive is enough? A meta-analysis of the effect of immersive technology on user presence The effectiveness of training in virtual environments A meta-analysis with examination of moderators of student cognition, affect, and learning outcomes while using serious educational games, serious games, and simulations An analysis of VR technology used in immersive simulations with a serious game perspective Affordances of virtual reality for collaborative learning Virtual classroom ' s quality of experience: a collaborative VR platform tested Augmented reality: a class of displays on the reality-virtuality continuum Brave NUI world: designing natural user interfaces for touch and gesture Marché Des Technologies Immersives (VR/AR/MR) HTC conducts its first virtual "VIVE Ecosystem Conference" (V²EC) fully in VR allowing global attendees to join virtual sessions Laval Virtual 2020