key: cord-0057977-hc0ger4i authors: Parkhomenko, Anzhelika; Gladkova, Olga; Zalyubovskiy, Yaroslav; Parkhomenko, Andriy; Tulenkov, Artem; Kalinina, Marina; Henke, Karsten; Wuttke, Heinz-Dietrich title: Virtual Environments for Smart House System Studying date: 2021-02-12 journal: Educating Engineers for Future Industrial Revolutions DOI: 10.1007/978-3-030-68198-2_52 sha: 2e6ab35695a18315202578ce9a57c46f20b6dd08 doc_id: 57977 cord_uid: hc0ger4i Today, virtual worlds are used not only in the gaming industry. Virtual, augmented and cross-reality also allow to organize effectively 3D environments that provide the effect of immersion and user interaction with objects and processes of the learning environment. The combination of physical objects and virtual models is a modern trend in the online lab development. Based on this approach, online experiment becomes more visual and interesting for students. In addition, it reduces queues for experiments on the real equipment. The paper presents the results of the development of an interactive web-oriented virtual model that expands the Smart House & IoT remote laboratory functionality as well as 3D virtual environment of Smart House for the virtual reality helmet. The proposed solutions will motivate students to study home automation technologies and the features of Smart Home systems realization. Nowadays Ambient Assisted Living is a relevant and promising area [1] . The activities of today's IT professionals are directly related to the realization of the concepts of Smart Cities and Smart Houses. Thus, the practical studying of Internet of Things (IoT) technologies and Smart House systems (SHS) development is an urgent educational task. The showrooms and demonstration stands from well-known manufacturers in the branch of SHS development are usually created for advertising purposes. They are not oriented to students' training and their cost is quite high for universities [2] . Remote laboratories (RL) are Internet resources that have already become popular in the world, especially in the conditions of COVID-19 pandemic quarantine. They are free and available to all interested users who have access to the Internet. Existing RLs are used for training in various fields of knowledge -from image processing [3] to the design of embedded and control systems [4, 5] . However, there are many open issues regarding the improvement of RL services, in particular user interfaces, and the development of unified software and hardware of RL [6] [7] [8] [9] . For example, in the Smart House & IoT remote laboratory [10] the user works in an interactive mode with a real demonstration stand, which can be viewed online with a webcam [10] . Multiple users can simultaneously perform certain experiments that correspond to specific Smart House subsystems (Illumination control, Access control, Safety control and others). This leads to the appearance of the users queues. To give complete system control to one user (for example, to study the interconnected operation of different subsystems of the house (Presence-Lighting, Climate-Ventilation-Heating, etc.), the administrative module needs to block access to experiments for all other users. Therefore, the remote laboratories for the study of SHS have their specifics that developers must take into account. Users queue is not a problem for virtual labs, in which 2D or 3D models are used to create virtual reality environments (VRE). To increase the attractiveness of online labs, a virtual and real parts of functionality can be combined for better operation. Due to this, laboratories move into the hybrid laboratories category. A well-known example of such a laboratory is GOLDi (Grid of Online Lab Devices), which was created at the Technical University of Ilmenau, Germany [11] . Implementation of an online educational laboratory based on modern virtual, augmented and cross reality environments expands the frames of the learning process as it makes experimenting faster, safer, more visual and fun as well as it provides the possibility of simultaneously work of many students [12] . Thus, the goals of this work are research and development of specialized VRE for studying issues of Smart House systems development and features of home automation technologies realization. Investigations have shown that in the works [13] [14] [15] [16] [17] [18] the results of the development of simulators and virtual environment for SHS rapid prototyping are presented. All of them have specific features of realization, advantages and disadvantages. A multi-purpose scenario-based simulator [13] gives 2D virtual environment for house plan development and home appliances placement in it. It can be used as a tool for real system prototyping at the beginning stages of SHS design. However, the lack of 3D environment for character moving and interacting with the model reduces the visibility of the proposed system. The virtual environment for rapid prototyping of the intelligent environment [14] allows to design the interior of the house and to place the sensors. The user can work in the 3D environment and two modes of the operation are available for him. In the first mode, the user can control the inhabitant and interact with different objects in the house. The second mode is based on scenarios usage to control the inhabitant`s behavior. However, with this approach, the user can`t get practical experience with actuators and real home automation equipment. OpenSHS is an open Smart Home simulator [15] that is intended for forming a Smart Home data set by implementing several stages: design of the initial virtual environment, events modeling and aggregated data set generating. However, the presented tool is an outline that makes it not so interesting for the user. In work [16] the authors describe tools for virtual design of different configurations for SHS to find out the best variant of topology without the application of real equipment. The paper [17] presents an environment for modeling typical actions of smart home residents based on uncontrolled teaching methods and associative rules. This approach can be used to detect changes in the behavior of the house residents, which may indicate, in particular, health problems. At the same time, this approach requires the generation of big data sets by their long-term collection from sensors, and it can be problematic for students because of the limited time of the learning process. The authors of the paper [18] propose tools for modeling the operation of the software on virtual devices in a virtual 3D environment to control changes of input parameters and to format the corresponding aftereffect. Nevertheless, the application doesn`t provide an online working mode. It is well known that virtual reality (VR) has entered the mass market with appropriative gadgets. Accordingly, various applications were created for the SHS studying, based on the usage of VR glasses and helmets. We have analyzed the possibilities of several virtual reality environments for SHS modeling [19] [20] [21] . Electronic House 3D [19] is the software for a smart home virtual environment that allows a person to move freely around the virtual house. The Electronic House works with Oculus Rift on a Windows PC platform. The application has a user-friendly interface. The Electronic House operates at speeds of up to 30 fps. This application gives the possibility to interact with objects in the VE to study their features. Virtual Home [20] is a software of the house virtual environment, which allows a person to move around the house using teleportation at predetermined points. Live Home works with HTC Vive on the Windows PC platform. The application does not provide a very user-friendly interface. Live Home operates at up to 60 fps. This application allows a person only to inspect the house without interacting with it. Planner 5D [21] is a virtual home environment software that gives the possibility of a person moving around the house. GearVR from Samsung, which runs on the Android platform, is used as the hardware. This program has a low level of graphics and it is also poorly optimized. However, the application has a user-friendly interface, which provides convenient interaction with the environment. The virtual environment has no smart home sensors. Planner 5D runs at up to 20 fps. This program allows a person to plan and inspect a future housing place. The experience of simulators and virtual reality environments implementation for studying the SHS that is presented in other authors' works, in particular, their possibilities and approaches to VRE realization is useful for the further improvement of our RL Smart House & IoT. Thus, we can conclude that the development of several types of VRE can expand the functionality of our RL and make it more interesting for students. Further integration of VREs with real equipment of RL will help to make it more accessible and clear for users and move it to the hybrid laboratory category. Based on the research of solutions proposed by other authors, we have decided to realize two different approaches to VRE implementation. The first approach is based on the development of interactive web-oriented models for VRE. This can be done, for example, using Mozilla A-Frame markup language using JavaSctipt, HTML5, WebGL. Such a web application provides the ability to play VR content on users' computers or smartphones and they don`t need expensive additional equipment in this case. The second approach to the development of VRE is based on the usage of game engines. The most popular game engines for VR development are Unreal Engine 4 (EU4) [23] and Unity 3D [24] . Both have a wide range of capabilities and they are reliable tools for 3D environment control, importing content (3D models, images, sound, video), as well as programming of interactivity and gameplay. The Unreal Engine 4 is more optimized in terms of calculations, it gives a more reliable image, but it is more difficult in studying. Unity provides wide opportunities for commercial games creation, as well as it remains more intuitive and effective for novice developers. The created interactive web-oriented VRE for our RL Smart House & IoT (Fig. 1 ) is based on a WebGL platform in combination with HTML5 and JavaScript. Unity 3D game engine and Autodesk 3Ds Max 3D environment were used for its realization [22] . The application runs on the Android mobile platform and provides functions of character control using a helmet and a controller, calibration of the helmet, as well as interaction with the interface using the controller (Fig. 2) . The product was developed on the Unity3D platform which features were used for the VRE optimization: • Texture compression was set at Adaptive Scalable Texture Compression (ASTC), that provides the best balance between quality and file size. The textures, that are used for the user interface, were not compressed to ensure the quality of the visualization. • The automatic API of graphics has been disabled and installed on the OpenGLES 3 API, which is supported with all mobile devices that are currently running Oculus VR. • Multi-threaded rendering has been activated. It moves graphical API calls from the main thread to a separate worker thread. • Static batching was turned on, and all immovable objects were marked as static. Because static batching merges meshes that are marked as static into one large mesh if they have the same material. When several meshes are connected, they can be drawn with a single draw call. It works due to the extra memory and storage space. • Stereo rendering method. Single-pass rendering has been enabled. Instead of duplicating calls for each eye buffer, objects are transferred once to the left eye buffer and after that they are duplicated to the right buffer automatically. When an object is duplicated in the buffer of the right eye, the corresponding vertex position changes and viewing-dependent variables, such as mapping, are applied. • Backend scripting. The backend script was exposed to IL2CPP, which works faster on the device, but at the same time increases the assembling time. Using these optimization techniques, the indicator FPS was increased from 10 to 25 frames per second. The developed VRE (Fig. 3) provides interaction with subsystems of Lighting control, Security control, Climate control, Irrigation control, etc. It gives the possibilities for students to gain knowledge of the structural and functional features of SHS, as well as the principles of systems`control. Today the developed web-oriented VRE of Smart House is available on our RL website and it allows students to get useful information about SHS components and to simulate control of different SHS subsystems. The VRE Smart House for virtual reality helmet also works stably, it gives students useful experience with scenarios of Smart House inhabitants' activity as well as motivates them to study home automation technologies and systems. The practical works, based on the application of created VREs allow students to gain experience with SHS design technologies, human behavior simulating as well as creating control scenarios for SHS. The proposed practically-oriented teaching method, based on the usage of Smart House VRE, allows lecturers to give students the knowledge and practical skills for successful application in future professional activities, valuable practical experience in the field of home automation systems and technologies. Future work will be aimed at the realization of extra functionality for VRE, in particular at the creation of multiple characters (or several Smart House inhabitants) for web-oriented VRE. Disruptive innovation in the era of global cyber-society: with focus on Smart City efforts Investigation and development of demonstration system for training in the field of home automation technologies An image processing online laboratory within the iLab shared architecture Internet-based technologies for design of embedded systems Remote laboratory for teaching of control systems design as an integrated system Laboratory as a Service (LaaS): a novel paradigm for developing and implementing modular remote laboratories Online lab infrastructure as a service: a new paradigm to simplify the development and deployment of online labs Towards a microRLMS approach for shared development of remote laboratories Implementation of reusable solutions for remote laboratory development Integrated complex for IoT technologies study The remote lab cloud "GOLDilabs.net Online laboratories for supporting international student collaboration in merging realities A multi-purpose scenario-based simulator for Smart House environments The virtual environment for rapid prototyping of the intelligent environment OpenSHS: open smart home simulator Smart home simulation system Data-driven generation of rule-based behavior models for an ambient assisted living system UbiREAL: Realistic Smartspace simulator for systematic testing Electronic House 3D for Windows PC Virtual Home free software (3D or VR Virtual model for remote laboratory Smart House & IoT Unreal Engine Blueprints Visual Scripting Projects: Learn Blueprints Visual Scripting in UE4 by Building Three Captivating 3D Games Holistic Game Development with Unity: An all-in-One Guide to Implementing Game Mechanics, Art, Design and Programming