key: cord-0967604-hl4k06l8
authors: Razzaq, Saad; Shah, Babar; Iqbal, Farkhund; Ilyas, Muhammad; Maqbool, Fahad; Rocha, Alvaro
title: DeepClassRooms: a deep learning based digital twin framework for on-campus class rooms
date: 2022-01-07
journal: Neural Comput Appl
DOI: 10.1007/s00521-021-06754-5
sha: d4e723210eb87cdea7430db5689b5a8347949cf3
doc_id: 967604
cord_uid: hl4k06l8

A lot of different methods are being opted for improving the educational standards through monitoring of the classrooms. The developed world uses Smart classrooms to enhance faculty efficiency based on accumulated learning outcomes and interests. Smart classroom boards, audio-visual aids, and multimedia are directly related to the Smart classroom environment. Along with these facilities, more effort is required to monitor and analyze students’ outcomes, teachers’ performance, attendance records, and contents delivery in on-campus classrooms. One can achieve more improvement in quality teaching and learning outcomes by developing digital twins in on-campus classrooms. In this article, we have proposed DeepClass-Rooms, a digital twin framework for attendance and course contents monitoring for the public sector schools of Punjab, Pakistan. DeepClassRooms is cost-effective and requires RFID readers and high-edge computing devices at the Fog layer for attendance monitoring and content matching, using convolution neural network for on-campus and online classes.

Distance education (DE) from the United States Distance Learning Association defines it as The organizational framework and process of providing instruction at a distance. Distance education takes place when a teacher and student(s) are physically separated, and technology (i.e., voice, video, data, or print) is used to bridge the instructional gap'' (USDLA, Glossary, p. 45) [1] . DE is a multidisciplinary field, has reformed the definition, design, and delivery of educational content. It is being revolutionized to accommodate new modes of educational delivery, learning domains and principles of learning, learning processes and outcomes, and new academic roles and entities [2] . DE is the need of the current era. Therefore, in this perspective, modern technologies need to monitor the attendance and content delivered in remotely located institutes and online classes. In addition to this, it assists in comparing contents provided on different campuses of the same institutes situated at remote locations. Quality education is a critical aspect of the growth and development of any country/nation, and it has long-lasting impacts on a country's dynamics.

In the current situation of Covid-19, many educational institutes across the globe are closed to safeguard the life of students, teachers, and supporting staff. As Covid-19 spread out, several people interact closely with an unknown positive carrier of the disease. In this uncertain situation, one cannot close out educational institutes for a long time. This closure will affect students' time and create critical financial problems for families attached to educational institutes. Parents are not willing to payout when educational activities are closeout for a long time [3] . Continuing education in such epidemics is a critical challenge for educational institutes across the globe. Online distancebased education is a crucial solution to a crucial situation that leads to a hopeless situation for students, parents, and educational institutes. The provision of the DeepClass-Rooms framework that includes scholarly content monitoring, student attendance, and online content delivery is a dire need in the current closeout of educational institutes. This framework will help monitor quality in online classes and remote institutes when classes will be held regularly.

Internet of things (IoT)-based devices are getting involved in different segments of life and are creating ease for humans [4] . Electronic gadgets, Smartphones, and advancements in data speed and affordable packages are the key factors behind the increased use of IoT-based devices. Things at the edge generate significantly large amounts of data. Fog computing has recently arisen as the paradigm to address edge computing needs in the (IoT) and Industrial Internet of things (IIoT) applications. These devices ease the life where control and monitoring are required. Like with medical gadgets, a patient's history can easily be monitored and stored. With the help of sensors, Arduino, and Raspberry Pi, IoT devices' usage is more frequent [5] . In context to Smart classrooms, various efforts and attempts are being made initially to make the environment of classrooms friendly and Smart. The initial transition from conventional boards to dust-free boards, multimedia, audio-visual aids, and Smart classroom boards with all the electronic gadgets is part of the change.

Making all classrooms a Smart classroom is a relatively expensive task. It would not be possible in Third World Countries (TWC), where the provision of education facilities is minimal and is maintained with minimal infrastructure and resources. Lots of efforts are being made to develop a framework and prototype for Smart classrooms. Some of them are discussed in the related work, i.e., Sect. 2.

In this paper, a digital twins framework will help the higher education institutes monitor the remotely situated institutes with effective attendance monitoring and content matching. Attendance of students, teachers, and staff will be monitored using radio-frequency identification (RFID) or any other supporting sensors. It will be verified by an edge device linked with Fog. At the same time, the final summary of class monitoring is being stored at the cloud storage. RFID or related sensors are widely used for attendance and academic monitoring, payment, animal tagging, postal tracking, airline baggage reconciliation, road toll management, control track manufacturing systems [6] , collecting museum visitor behavior [7, 8] , campus intelligent security system [8, 9] , etc. Another part of this framework is content monitoring. All the contents planned to be covered in a particular subject during a prescribed time frame are stored in a database. An application will be developed in an android platform that will capture the keywords of the contents mentioned on classroom boards in any class lecture and matches it with the stored content using a convolutional neural network. By implementing and deploying this proposed framework, institutes will require the RFID and supported readers, Smart cards, internet connectivity, LEDs, and computer systems where attendance and course contents will be stored. Using these sensors, we are also framing this DeepClassRooms concept into digital twins. We can easily monitor the attendance and content delivery in both online and physical modes of classes. Moreover, digital twins environment can be opted for by any business model where performance monitoring is required like manufacturing, retail, banking, using relevant sensors, and IoT-based devices.

The rest of the paper is organized as follows. Section 2 illustrates the related work, Sect. 3 contains the proposed framework, and Sect. 4 contains the discussion and conclusion.

IoT-based electronic gadgets, wearable devices, and devices are getting fundamental importance and role in the automobile industry, health, medical, education, Smart video surveillance [10] , and other sectors. Different efforts have been made to design a Smart classroom environment. Atabekov [11] proposed the framework to interconnect chairs wirelessly. They are thereby developing Smart chairs to monitor and store the occupant's detail and presence by sending it to the cloud. It is beneficial in-class interaction through indoor localization (through which we can get reliable indoor positioning) and dynamic generation of the seat map of students. The system is based on Arduino.

Another effort is being made for developing a Smart classroom using Raspberry Pi [12] . The proposed system consists of four modules (Smart projector with PowerPoint presentations, gesture-based PPT control, Smart attendance system, and Google voice assistance system). These modules are implemented on a low-cost single microcontroller platform using Raspberry Pi. They have tried to substitute laptops by proving the features of uploading lecture slides but limiting only to PowerPoint slides. In contrast, the different settings on separate boards were made for controlling various devices. Secondly, the user must be very accurate in gestures to control the lecture slides; however, a Smart attendance feature is appropriate. Also, in Google Voice Assistance speaker has to take care of words phonetics, sound pitch, and quality to have accurate results. Most of the time, in classes, we don't have such a noisefree environment.

Further, the author is silent about the quality of results achieved by these four modules. The application area of Raspberry Pi is broad. Thereby, a classroom prototype was designed by Huang et al., at Ming Chuan University (MCU), Taiwan [13] . It was a layered Raspberry Pi based integration model. It was utilized along with the web application interface to implement a context-aware energysaving Smart classroom application. It encapsulates three parts in the proposed prototype of a context-aware Smart classroom, i.e., a model for technology integration and supporting measures for the operation of Smart classroom architecture. The accumulated results highlighted the feasibility of the proposed architecture for developing Smart campuses through context-aware Smart classrooms. At the same time, features of authentication, authorization, and data exchange details were not defined in the application layer. Therefore, the data exchange scheme and security rules were to be incorporated in future work. In contrast, the result accumulated from the research of Kuppusamy [14] highlighted that it provides a solution for performance and security risks. He proposed a Smart education model, i.e., IoT architecture, to Smart education (IoTASE). The research aimed to track the learning experience of a learner through e-Learning architecture. IoTASE is designed to integrate the functional components, i.e., management, service, organization, IoT process, virtual entities, IoT communication service, and security. It contains a layered architecture to support functional components on the corresponding layers, i.e., learning data collection layer, learning data integration layer, retaining information access layer, and learning application layer.

Moreover, it contains Smart objects for designing complex learning scenarios. Future work incorporates a secure framework and supports interconnectivity through blockchain technology and big data analysis. Further, a teacher-student interaction (TSI) framework based on a monitoring technology, i.e., Raspberry Pi, was proposed by Mahmood et al. [15] . It monitors students' reactions to a camera mounted in a classroom to capture students' facial expressions. These expressions are recognized by the decision tree algorithm and are stored in the cloud in image format. These IoT-based classrooms help the teacher improve the quality of the content delivered by observing real-time student feedback through the stored images. It lacks the functionality to monitor and report the content provided by an instructor. However, the research can be further used for various surveillance and security-based applications.

An emotionally aware AI-based Smart classroom is proposed by [16] . The system can make real-time feedback/suggestions to the class presenter to improve the presentation by updating their content, mode of expression, and style based on listeners/crowd scores. They have used algorithms, i.e., deep learning, high-performance GPU computing, multimodal sensing, and emotion recognition, to analyze the presenter's audio and visual presentation. This system may opt for professional training, graduate, and postgraduate level classes where listeners can give appropriate and mature feedback against declaration but not for schools lacking such hardware infrastructure and the mature crowd to have relevant quantifiable feedback. A similar sensor-based Smart system termed ''Sensate Benches'' was proposed by Arora et al. [17] . The sensors were implemented on the benches of the classroom and connected through a multi-touch interactive system. It helped resolve issues faced during interactive learning, i.e. (resource sharing and multi-personal interaction) by ensuring less complicated information accessibility. It provided the features of gesture detection, automatic calibration, etc., along with interactive learning. In the future, they aim to explore sensor-based technology further to enhance the features of Sensate Benches. Yang et.al [18] investigated Smart classrooms from both pedagogical and technological perspectives by an extensive scale survey conducted in 135 schools participated by 13,495 students in China. They have considered five dimensions (resource, environment, enhancement, management, and presentation). The survey results revealed that the availability of digital resources (Internet and electronic devices) in the classroom for each student is mandatory for an improved learning experience. Along with the availability of electronic resources in classrooms, various checks on browsing must be followed to stop students' deviation during lectures. Smart classrooms may improve the learning experience of students, along with improved pedagogy and examining strategies. Mahesh G et al. performed a research survey to monitor and control mobile phones usage at education institutions. An application for attendance is also developed to take it at the start of the class session by students themselves, and details are being stored on cloud storage. Institute/campus management can define the list of allowed applications that students can use during campus hours [19] . This framework's issue is the list of applications can be managed through university internet services but cannot be restricted to personal data packages. Hassanin et al. [6] proposed a student attendance management system that primarily focused on easy access and timesaving. RFID readers were installed and connected to the RFID server using LAN. The server generates reports based on the data In [20] , authors used webcams to capture face images. After this, the cropped image is stored in the database. The image matching technique is applied to match the captured image with existing stored images in the database, and in case of a match, attendance is marked. Another attempt to use facial fiducial points from eyes, nose, and lips was performed by [21] . Image-based recognition tasks take time in the recognition process and sometimes have low accuracy due to different reasons, i.e., poor light and issues in the image capturing, beard, and veiled cases, especially in Muslim countries. A Kinect-based assessment for Smart classrooms is proposed by [22] . They have used the Microsoft Kinect sensor for learning a Virtual environment in classrooms.

In this perspective, Fraga-Lamas et al. [23] had proposed the Long-Range Wide-Area Network (LoRaWAN) Fog computing-based architecture. The architecture was deployed on the University of A Coruña (UDC) campus, Spain, to connect the IoT nodes. It provides a remarkable feature to support location-aware applications with low latency, thereby decreasing the computational load of traditional cloud computing systems, i.e., network traffic. Moreover, it contains a 3D Ray-Launching radio-planning simulator that detects minute details. It provides precise radio propagation estimations, i.e., capacity, energy efficiency, and network coverage within the Smart campus scenario. The results were accumulated through stimulator planning on a campus covering 26,000 m 2 . The development of threeapplications based on the proposed framework highlighted a basis for future Smart campus designers and developers. Later, Mrabet et al. [24] proposed combining the Smart management system and vocational guidance system to overcome a heterogeneity issue between education and training in Morocco. The research emphasis is on examining a Smart school guidance system called ''ETC guidance system.'' This system uses IoT devices and RFID Tags to stimulate the student guidance system. It enables students to make a career choice at the right time wisely. Thereby, it aims to reduce the failure orientation rate among students from TWC. The data obtained from the survey of 398 sary school students highlighted results they seek to use in the future and an extensive data set to demonstrate the effectiveness and increase the system's efficiency in the Smart environment. It also includes the development of a Smart vocational guidance environment based on a profiling system. It will analyze the tendencies of students based on their aspirations and choices of studies.

The importance of a student-centered active learning environment over lecture-based learning is heightened by Gunn et al. through a case study of an engineering school in the Middle East [25] . The active learning classroom (ALC) technology was utilized for enhanced student-teacher interaction and effective learning. The feedback from the students and faculty was appreciable. However, the monitoring to ensure the proper utilization of ALC by faculty did not exist alike [17] . Thereby, our research emphasis providing a solution to this problem of monitoring the delivered content.

We will develop RFID-based attendance system and content monitoring system using the cloud, and Fog servers, where attendance authenticated is performed at the Fog layer without generating more traffic toward the cloud layer. The proposed approach can be installed across the network of institutes and can be monitor performance. Attendance is also monitored using face recognition.

In this article, a Deep Learning-based Digital Twin framework is proposed for public sector education institutes of a province of Pakistan. These educational institutes are spread across the province for the initial level of schooling with very limited budgeting and facilities. The education department is striving hard to improve the quality of education in these institutes. The first step in this is to ensure faculty/staff and student's presence during school hours. Different manual methods are currently handling this, and a lot of human effort is involved in monitoring them. By using the attendance recording and monitoring system, incredible time and effort may get saved. Schools census data accumulated from Punjab, Pakistan is available at [26] is visualized through Table 1 .

Detailed information will be stored at the edge device, as shown in Table 2 . Usually, edge devices are capable of small to medium size computation and processing. In this framework, a small computing server or I5 to I7 level desktop/laptop system is proposed at the Fog layer to match the attendance and record it. The content matching task is also performed at the Fog layer. Details of attendance verification and content matching are explained in the following sections. In case of online classes during any epidemics or critical situation such as the current situation of Covid-19, the performance of online classes will also be monitored from the teacher's screen. Contents can easily be matched from the teacher's screen in the same way as in the on-campus classroom. Also, faculty may have to enter the contents covered in the online class at the end of each session.

Cap id RFID reader/or any other supproting sensor captures the Std-Id or St-Tec-Id or take attendance record from Edu-Upscale App in case of online classes match the Cap id in database, if match found then mark the attendance (record the detail of student, staff or teacher accordingly) else Beep the alarm not a valid user or Guest. 

All registered students and employee records are stored at the Fog layer. Campus Id, Std-Id, and St-Tec-Id are captured by the RFID reader/or any other supporting sensor. Captured id (Cap_id) is matched with students or employees credentials stored at the Fog layer's high edge device. Detail design of the information stored is given in Fig. 1 . If details are matched, attendance is recorded and maintained at a High edge device. If the system cannot find the record, system alarms and the security personnel will check the concerned person's authenticity. The attendance verification process is shown in Fig. 2 . Students and teachers will mark their attendance in the first 10 min on Edu-Upscale App online attendance interface in online classes. And it is matched and stored in the database. In online classes, attendance is monitored and recorded at the start and at the end of the session to maintain the maximum student strength from the teachers' screen by taking a screenshot. Also, it can be verified from the list of students logged in during the session. Attendance verification through sensors can also reflect the concept of digital twins. Moreover, the attendance record is being saved on the cloud for later usage and analytic.

A mobile app is designed with the name of (Edu-Upscale). Teacher logins are created, courses and class information are stored at a High edge device at Fog. The login screen and the class summary are mentioned in Fig. 3 . Once login is successful, Edu-Upscale picks the class, subject, number of registered students and presents student detail from the database. On the class summary interface teacher will also enter the lecture title to be covered in this lecture. This title is matched with already stored content on the same date. The same procedure is adopted for both online and oncampus classes. Figure 4 gives a brief overview of how the proposed DeepClassRoom architecture works. First of all, the teacher will log in to the system. Based on the date and time, Edu-Upscale will pick the class and subject against the login. The teacher will enter the Lecture Title. After this step, the Application will capture the contents from the classroom board and the teacher screen in online lectures and require preprocessing. In the next step, this image is given as input to CNN for text classification.

All course contents will be recorded at Fog server (class, date, subject, topics to be cover with image). Details have been mentioned in Table 2 . Initially, all the contents Neural Computing and Applications delivery plans based on (class, date, subject) will be finalized by the academic experts at the start of the academic session. This activity is an extensive and time-consuming to complete all the subject's lecture plans based on the academic calendar. But once it is designed, a very few changes will be required to update the content subject-wise every year. Contents will be matched in a two-way process.

Once the teacher will add the topics covered in a text box, as shown in Fig. 2 . Secondly, by the content captured by Edu-Upscale using Convolutional Neural Network (CNN). Text matching will be performed by the contents data stored in Table 2 with the teacher's lecture title details. Contents monitoring according to the given academic calendar is a part of course contents matching. Edu-Upscale captures the content's image from the classroom board or teachers' online screen and performs necessary preprocessing on it. This image is being used as an input to Convolutional Neural Network for content matching.

Edu-Upscale captures the content's image from the classroom board or teachers' online screen and performs necessary preprocessing on it. This image is being used as an input to convolutional Neural Network for content matching. Table 3 shows an estimated number of records stored at the Fog server for Grade 1 to 12. The calculation is based on the estimated number of working days, excluding all vacations, and the average number of subjects per class. The proposed system will record all of the images of topics covered the CNN model will use that. After required pre-processing on images, the CNN model extracts the keywords for the contents covered in lectures from pictures and reports matched contents for a particular subject. Pseudo-code for content matching is shown in Algorithm 2.

Fog and cloud architecture will support storing all data regarding students, staff, and teachers' attendance. Similarly, student-teacher activities like Lectures, contents delivery, class activities, assignment, class notes, and related material will be available in the cloud. This structure is beneficial as it supports the online learning activities of the respective subject for the students in remotely situated institutes that lack a subject specialist. Also, in the current Covid-19 epidemics situation, almost all educational institutions are closed for an undefined time. Such a centralized online education system will ease educational institutes to run online classes across all educational institutes and monitor them. The online content can be accessed through online web resources using Smartphones, tablets, personal computers, and laptops. The proposed framework is highly effective for monitoring content, attendance, and online and offline lecture delivery.

CNN is widely used for image and pattern classification due to better performance than other classifiers [27] . CNN gives better results due to its share weights and local connectivity [28] . It is also used in high-resolution remote sensing images and contains good results in classifying them [29] . Figure 4 shows the image classification using CNN. An initial model is trained on the contents images and its lecture delivery according to the academic calendar. The first layer is the input layer that holds the image. At the convolutional layer, the system computes the dot products of feature images with filters. Our image size is 20*25*3 (length, width, depth). Initially, we have 15 filters, so we have 20*25*15 dimensions at the output. At the activation function layer, the element-wise activation function is applied to the convolutional layer's output. Famous activation functions like sigmoid, unit step, Tanh, and Relu are applied. At the Pool layer main objective is to reduce the volume size and memory. In the end, a fully connected layer takes input from the previous layer and finds the class score, and displays the output layers equivalent to several classes.

The proposed DeepClassRooms will perform attendance monitoring and content matching. Initially, attendance will be taken through edge devices and performs Attendance Verification and Record Module on the Fog layer. In the first step, only those subjects that are English-based will be taken for content matching. In the second module, contents matching will be conducted through CNN. The system will train the model with 19,320 unique images and will do text extraction. Figure 4 shows the summary of information stored at a high-edge device on the Fog layer in a tabular form. The proposed framework can transfer the summary as given in Table 4 . to the cloud server for monitoring and performance analysis. The system only stores limited information in the cloud that reflects performance indicators to monitor the teacher's regularity and contents coverage.

A DeepClassRooms and content monitoring system has been proposed for online and on-campus classes. It provides a digital twin framework for performance monitoring of teachers and attendance monitoring of students and teachers without human intervention. This can also be used for performance-based increments and the promotion of faculty. Human biases and favoritism can quickly be ruled out if such systems are opted by educational monitoring systems. Comparatively, a very low-cost and easily configurable system is presented that can be opted for and followed by any educational institute. A lot of initial effort is required to train the CNN based on available content images. In the future, we look forward to making it more compatible by expanding it to incorporate the features for multi-linguistic content monitoring and matching support. Smartboards may also be added in the framework where cost is not an issue.

Conflict of interest The authors declare that they have no conflict of interest.

A research about distance education students' satisfaction with education quality at an accounting program

University finances and covid 19 different schools different risks

Handbook of research on the internet of things applications in robotics and automation. IGI Global: International Publisher of Information Science and Technology Research

Internet of things with Raspberry Pi and Arduino

Automation of attendances in classrooms using RFID

Evaluating and collecting museum visitor behavior via RFID

IoT application layer: case studies and real applications, The Era of Internet of Things

Campus intelligent safety supervision system based on RFID

Fog-based crime-assistance in smart IoT transportation system

Internet of things-based Smart classroom environment: student research abstract

Smart classroom using Raspberry PI

A context-aware Smart classroom architecture for Smart campuses

Smart education using internet of things technology. Emerging Technologies and Applications in Data Processing and Management

Raspberry PI and role of IoT in education

Towards emotionally aware AI smart classroom: current issues and directions for engineering and education

Sensate benches-a modern approach to education

Evaluation of smart classroom from the perspective of infusing technology into pedagogy

A smart phone integrated smart classroom

Neural Computing and Applications

Student Attendance System and Authentication using Face Recognition

A conceptual model for automated attendance marking system using facial recognition

A kinect-based assessment system for smart classroom

Design and experimental validation of a LoRaWAN fog computing based architecture for IoT enabled smart campus applications

Smart school guidance and vocational guidance system through the internet of things

Smart education: introducing active learning engineering classrooms in the Middle East (2017) Fourth HCT Information Technology Trends (ITT)

Using convolutional neural networks for image recognition

Applications of convolutional neural networks

Application of convolutional neural network in classification of high resolution agricultural remote sensing images. The International Archives of the Photogrammetry

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations