key: cord-0058503-drlhhw0p authors: Simonetti, Marco; Perri, Damiano; Amato, Natale; Gervasi, Osvaldo title: Teaching Math with the Help of Virtual Reality date: 2020-08-26 journal: Computational Science and Its Applications - ICCSA 2020 DOI: 10.1007/978-3-030-58820-5_57 sha: 2ee9a12a80922be5b86cc0fdddfa460e4d519977 doc_id: 58503 cord_uid: drlhhw0p In this work we introduce a learning system based on VR (Virtual Reality) for studying analytical-geometric structures that are part of the curriculum in mathematics and physics high school classes. We believe that an immersive study environment has several advantages over traditional two-dimensional environments (such as a book or the simple screen of a PC or tablet), such as the spatial understanding of the concepts exposed, more peripheral awareness and moreover an evident decreasing in the information dispersion phenomenon. This does not mean that our teaching approach is a substitute for traditional approaches, but it can serve as a robust tool to support learning. In the first phase of our research we have sought to understand which mathematical objects and which tools to use to enhance the teaching of mathematics, in order to demonstrate that the use of VR techniques significantly increase the level of understanding of the mathematical subject being studied by the students. The system which provides for the integration of two machine levels, hardware and software, was subsequently tested by a representative sample of students who then provided feedback through a questionnaire. Much progress has been made in the field of VR since it was first introduced in the early 1970s. In many fields it has found countless applications such as entertainment [1] , teaching [2] , tourism [3, 4] , manufacturing [5, 6] , networking and communications [7, 8] , microelectronic and high performances hardware industries [9, 10] , e-commerce [11] , medicine [12, 13] . Today, we have reached a certain maturity in VR technologies. In this work we will focus on the areas concerning simple VR and AR (Augmented Reality) experiences in the teaching of mathematics. During our work, we were guided by a specific goal: the possibility of giving a real and visual form to the abstract objects of mathematics. This represents a further development in the visual representation of mathematical concepts, which in the course of history has evolved from the primitive use representations of simple counting objects, such as the tally sticks, through the elegant structures of the symbolic algebra of the seventeenth century, to the imposing constructions of mathematical analysis and modern geometry, to get to the current and amazing views of numerical analysis through computer graphics. In this context, we set out to investigate the possibility of extending students' understanding of the concept of link between an algebraic-set structure and its geometric representation on an orthogonal Cartesian space (function). In the first phase of the work, our attention was directed to a limited number of functions, that are used in senior high-school classes, such as the representation of trajectories in the Cartesian plane and simply surfaces in the threedimensional space. The very first proposed functions are as follows: • y = sin x The sine function is well known to students who are currently using it to solve analytical and geometric problems. It is also commonly used in physics to model periodic phenomena such as sound and light waves or changes in average temperature during the day or the year [14] . As showed in Fig. 1 This function is a classic example of a saddle surface that it is a smooth surface containing one or more saddle points. Saddle surfaces have negative Gaussian curvature which distinguish them from convex/elliptical surfaces which have positive Gaussian curvature and are very important in the study of non-Euclidean geometry and in the theory of general relativity [15] . As showed in Fig. 2 • z = xy This function is an example of hyperbolic paraboloid [15] . As showed in Fig. 2 • z = ln(x 2 + y 2 ) This function is an example of bi-dimensional logarithm, useful to describe astrophysical objects. As showed in Fig. 3 • z = sin(x 2 + y 2 ) x 2 + y 2 This function is an example of bi-dimensional dumped sine, useful to describe objects in fluid dynamics, electronics and telecommunications. As showed in Fig. 4 Stimulating multiple sensory dimensions during the learning process of a concept or idea makes the learning more effective, because our mind needs to experience the reality that surrounds it through multiple experiential levels. For example, through looking, listening and touching build a complex conceptual structure that bears and strengthens our knowledge [16, 17] . This multi-sensorial learning is also important in special-need situations (autism, dyslexia). An immersive study environment has several advantages over traditional twodimensional environments (such as a book or the simple screen of a PC or tablet), as it gives the spatial understanding of the concepts exposed, more peripheral awareness or more useful information bandwidth and decreasing in the information dispersion phenomenon [18] . Several works have been proposed to achieve the purpose, especially in the natural sciences: chemistry [19] , biology [20] , physiology [21] , physics [22] . The idea may therefore be to create digital environments to enhance skills, knowledge and competences in math, which in turn can decrease anxiety and improve results [23] . Mathematics can be considered one of the most difficult subjects for many students. A recent study asks questions about traditional way of learning and recommends more active and attractive learning approaches [24] . Studies have shown that immersion in a digital environment can improve education in several ways, because, as mentioned earlier, multiple perspectives are activated [25] ; this fact has a significant value in all areas of mathematics [26] . In several works it has been seen that despite the mathematical simplicity of the concept of a function [27] , many students find it difficult to relate its analytical form to the relative graph, as if the intermediate layers existing between the set concept and the analytic-geometric one prevented a clear understanding of the link [28, 29] . It has recently been proven that the use of software capable of explicitly representing the analytic-geometric link existing in the functions can help students develop a positive mindset towards mathematics itself, in terms of attitude, motivation, interest and competence [30] , but we believe that new researches need to be done to grasp the profound implications that exist. We have created two different user experiences. One uses VR, while the other uses AR. In both cases the graphic engine used is the same, Unity3D. This software allows the composition of virtual environments starting from basic elements called Assets which the scene is composed with. It also takes care of rendering, real-time lighting calculation and user interaction management. The fundamental tools that have been used are the following: • the game objects, i.e. the basic elements that make up the scene you want the user to view. • the scripts, code files written in C# language through which you can execute predefined tasks, such as managing the appearance of objects on the displayed area, or the camera movement as a key on the keyboard is pressed. • the colliders, that prevent intersection or collision between the character that the user is controlling and the objects in the scene The shapes are generated with two C# scripts that allow for the modelling of any mathematical function in two or three dimensions. The first script generates the vertices. The second script receives in input a list of vertices and then generates a three-dimensional figure. Let us suppose we want to render a three-dimensional function, for example: f (z) = x * y. As we all know, a standard mathematical function like this is defined in the continuum space, so if we wanted it to be visually represented we would need an infinite number of points: the aim is therefore to make the image as plausible as possible in a discreet environment, by appropriately choosing the points to be drawn. To do that, it must be chosen a well-defined length along the X, Y and Z axis, and the number of points (i.e. the number of vertices) that compose the graph must be predetermined. In other words, we need to define a grid of points that will define the level of maximum detail we want to achieve. In addition, we must bear in mind that the greater the level of detail, the more calculations the user's device will have to perform in order to display the object on the screen. Defining a grid of points is equivalent to defining a sampling rate. This is the same as when you are processing an electronic signal (e.g. an audio signal) and want to convert it from a continuous signal to its discrete representation. If the number of samples is too low, we can in fact obtain Aliasing, obtaining an inaccurate representation of the mathematical function we want to show. It is possible to run the program already with the code just described. However, this requires recalculating all forms at runtime each time. To improve performance, we have therefore saved the forms generated with Unity3D inside the filesystem so that they can directly be reloaded at program start, with no need to recalculate all objects from the beginning. We then processed the shapes with Blender, a software designed to process models and three-dimensional objects, in order to reduce polygonal complexity without changing their information content. In other words, the complexity of the figures in terms of vertices has been reduced but an user who observed them would not notice any difference. This is possible using Blender, a software made to process models and three-dimensional objects. Finally, we have included them again in the Unity3D project. The VR environment is generated and compiled by WebGL technology. This means that the application is compatible with all devices (computers or smartphones) on the market since the environment is usable through a web browser. The graphic quality of the scene adapts according to the computational power of the device, while remaining undemanding in terms of hardware requirements. The scene can be observed through a virtual reality viewer, such as HTC Vive, or through a normal computer monitor. The user has the possibility to move around the virtual environment using mouse and keyboard. Inside the environment are visible three-dimensional geometric shapes that support the learning of mathematical functions that are otherwise difficult to draw. The AR environment uses the Vuforia framework. The program created is an apk, installable on Android smartphones with 7.0+ operating system. Vuforia is an SDK that allows you to analyze the video stream recorded in real time by the phone camera. Vuforia allows you to create a database of markers (called Vumark). These have manually been associated to the game objects of the scene. When one of the markers present in the database is framed by the camera of the user's device, Vuforia tells Unity to show on the scene (and then on the user's screen) the game object associated with the framed Vumark. Moreover, this SDK manages the spatial orientation of the object according to the user's position with respect to the Vumark. If we frame a Vumark and move around it, the object associated with it will rotate as well, allowing us to appreciate it in a realistic way. In this example, we want to represent a three-dimensional function within a virtual world created with Unity3D. The creation of the figures is done through two scripts. First of all we define the resolution, that is the level of detail, that the figure must have. If for instance we set a resolution equal to 100, then we will have a matrix of 100 × 100 points. In this way we will scroll the variable X and the variable Y along the grid. Then we define the step, i.e. how much space must elapse between one point and the next. The step variable has been fixed at 0.1f. By means of a nested double "for cycle", where the first takes care of the X variable and the second the Z variable, we can calculate the Y value in the grid. The code we made allows us to obtain the list of points that make up the figure. The next step is to calculate the list of triangles. Triangles are a fundamental element in computer graphics. They specify how the points are interconnected to each other and how they should be represented on screen. Each "game object" and its figure have an associated script of this type. The difference among the various scripts therefore remains in their ability to calculate the getY function. Every time it is necessary to change the graph to be represented, it is sufficient to specify how to calculate Y to obtain the change of the figure shown; in all cases, the only thing to be done would be to scan the grid (in our example a 100 × 100 matrix) and recalculate the correct values. A fundamental divergence from this is in the case of two-dimensional functions, like the periodic sine function which is mentioned in Chap. 2. In that case only one "for cycle" was sufficient and a depth fixed at 0.05f. In this way we can represent a two-dimensional function as if it were a tube, which allows us to observe it better when we move around it. The next step is the generation of the three-dimensional mesh from the vertices and triangles calculated in the previous step. To do this, a generic script has been created, which can be recalled from all the codes present in the Unity3D project. Since a three-dimensional object has been generated inside the program, it is necessary to calculate how the light should behave in order to make it visible to the user's camera. The hard work would therefore seem how to light the object in the right way: that means that it is necessary to calculate a light intensity value for each polygon which makes up the entire object structure. This calculation, which can be complex, is carried out very quickly in Unity3D. The calculation is carried out in two phases. At the beginning, a call to a function integrated in the Unity libraries doubles up triangles in the figure, so each of them has got a "specular twin" (i.e. a mirror image), with the normal straight line to the surface with opposite direction to its homologous, so that flow of incident light can correctly be obtained. Finally, a mesh object is created and vertices, normals and triangles are assigned to it. At the end of this operation the figure (also called mesh) is ready to be shown on screen. We have then split the work in two different sections, involving Virtual Reality and Augmented Reality respectively. In the first case a room, has been created, and inside the room the three-dimensional figures have been positioned, as shown in Fig. 5 . The project has then been compiled in WebGL in order to be easily used by a web browser and not have any dependence on a specific operating system (Windows, Linux, Android, iOS, etc.). As far as the use with augmented reality is concerned, Vuforia software has been used instead. Vuforia is a framework that integrates within Unity3D. Vuforia allows you to create projects that use augmented reality by providing all the functions essential for operation on mobile phones. In particular we focused on smartphones with Android operating system. In the first phase of our research we tried to understand which mathematical objects and which tools to use to enhance mathematical teaching, having sensed that the use of VR techniques significantly increases the level of understanding of the subject studied by the students. We would like to expand our research towards the field of immersive learning, in particular those applications that allow the user to be immersed in virtual worlds that increase brain stimulation during the learning phase. Our approach aims to give the student a wider environment of objects to study and focus on, by selecting among them those of greatest interest and didactic utility. Furthermore, we find it interesting to know the degree of absorption and empathic response of students to the system: sensations, disturbances, emotions. Students' feedback will be valuable in order to further enhance the experience and make the AR and VR environments more interactive and attractive. That's why the next step we are considering is to allow high school students to evaluate the quality of the work done by filling out a questionnaire. If the number of students is high enough, we can draw objective conclusions and understand how much virtual reality can impact their perception of mathematics. We are therefore going to select a statistically significant and representative sample, homogeneous by social and cultural level, in order to obtain a set of coherent and indicative answers. As regards generating functions, the current system is able to display objects that are only statically compiled at compiling-time: for the future we intend to design a dynamic system for the generation of mathematical functions that allows the mathematics teacher to draw randomly any three-dimensional or twodimensional graph without the need to print a new Vumark every time. What we want to achieve is a dynamic platform that allows us to understand how the choice of a function and a complete immersive experience in the mathematical object itself (including its specific characteristics and properties) impacts students' learning. Guidelines for web usability and accessibility on the Nintendo Wii Analysis of use of virtual reality technologies in history education: a case study Virtual reality and tourism: fact or fantasy? New realities: a systematic literature review on virtual reality and augmented reality in tourism research Augmented reality for manufacturing planning Virtual reality applications in manufacturing process simulation Exploiting virtual reality for network management VRPN: a device-independent, network-transparent VR peripheral system The AES implantation based on OpenCL for Multi/many core architecture A simulation framework for scheduling performance evaluation on CPU-GPU heterogeneous system Applying virtual reality for trust-building ecommerce environments Nu!RehaVR: virtual reality in neuro telerehabilitation of patients with traumatic brain injury and stroke Chapter twenty -virtual and augmented reality in medicine A History of Mathematics Advanced Calculus Physiological measures of presence in stressful virtual environments Exploring the benefits of immersion in abstract information visualization Virtual reality: how much immersion is enough? A virtual reality laboratory for distance education in chemistry Use of virtual-reality in teaching and learning molecular biology A virtual reality electrocardiography teaching tool Teaching physics using virtual reality The math anxiety-math performance link and its relation to individual and environmental factors: a review of current behavioral and psychophysiological research Active learning increases student performance in science, engineering, and mathematics Immersive interfaces for engagement and learning MAT3D: a virtual reality modeling language environment for the teaching and learning of mathematics The simplicity, complexity and complication of the function concept Development of the process conception of function The function concept: comprehension and complication Using Desmos to draw in mathematics