key: cord-0200286-s7ae73pu
authors: Uddin, Khandaker Mohammad Mohi; Dey, Samrat Kumar; Parvez, Gias Uddin; Mukta, Ayesha Siddika; Acharjee, Uzzal Kumar
title: MirrorME: Implementation of an IoT based Smart Mirror through Facial Recognition and Personalized Information Recommendation Algorithm
date: 2021-02-28
journal: nan
DOI: nan
sha: d4c1a3b51554b55819fb061122596b76dad085ec
doc_id: 200286
cord_uid: s7ae73pu

We are living in the era of the fourth industrial revolution, which also treated as 4IR or Industry 4.0. Generally, 4IR considered as the mixture of robotics, artificial intelligence (AI), quantum computing, the Internet of Things (IoT) and other frontier technologies. It is obvious that nowadays a plethora of smart devices is providing services to make the daily life of human easier. However, in the morning most people around the globe use a traditional mirror while preparing themselves for daily task. The aim is to build a low-cost intelligent mirror system that can display a variety of details based on user recommendations. Therefore, in this article, Internet of Things (IoT) and AI-based smart mirror is introduced that will support the users to receive the necessary daily update of weather information, date, time, calendar, to-do list, updated news headlines, traffic updates, COVID-19 cases status and so on. Moreover, a face detection method also implemented with the smart mirror to construct the architecture more secure. Our proposed MirrorME application provides a success rate of nearly 87% in interacting with the features of face recognition and voice input. The mirror is capable of delivering multimedia facilities while maintaining high levels of security within the device.

Nowadays in this world, technologies are advancing day by day. For this reason, maximum devices need to be updated with smart technology. The smart systems are organized by Artificial Intelligence (AI) and build smart equipment that makes the devices more interactive with the user.

However, the smart device has the capability to easily sense, process and analyze the captured information. In modern days, working professionals are highly busy with their daily work and thus it is quite difficult for them to check daily necessary information including the latest news, To-do list, daily stock market update, social media newsfeed, traffic jam update, weather forecast and so on. This research proposed an IoT based smart mirror that helps the users to receive all this information. In addition, an AI-based face detection method also introduced that ensures a specific level of security for the proposed architecture. Besides, this system will also offer the opportunity for users to check their daily e-mails, helps them by playing preferred audio and video songs with the features of voice input. Currently, significant research works have been conducted in the domain of developing an interactive smart mirror for personal use. Kulovic et al. [1] have developed a smart mirror using the concept of Do-It-Yourself (DIY) which displays the basic information of the user including date/time, weather, and news headlines. In their development, authors have used Raspberry Pi for the development of their system and users have the privilege of adding or removing their information using their phone or tablets. Besides, by presenting the basic information of the user's, Akshaya et al. [2] have developed a two-button supported smart mirror, one for the website and another for the map. The key feature of this research is to access the smart mirror from several areas with an assistance of a user-friendly interface. In other work, Yusri et al. [3] introduced a smart mirror that offers users to control their home appliances using a voice recognition mechanism. Mainly, this application is developed to save the energy consumptions of the users along with showing some basic features including time, traffic, weather, and so on. Specifically for the aged and disabled peoples, Athira et al. [4] have developed a mirror using passive Infrared (PIR) sensors. The designed system highly capable of detecting a person with the support of internet connections. Moreover, basic features including social media notifications, weather information, and time also incorporated with this architecture. In other work, in order to present the basic features and to control the household devices, Hossain et al. 

In this section, the design methodology of this proposed framework has been discussed. The proposed architecture is suitable for the essential functionality of any smart home. On the other hand, this will timely remind the user for a specific task based on the user's preference. Different features of the proposed MirrorME application are illustrated in Figure 1 . MirrorME is designed dedicatedly for two different categories of users; one is general users and other is authenticated users. The authentication process is developed with the support of HOG-SVM based face recognition approach. Regarding the access of different features, there is some personal information recommendation features for any existing users whereas the general users only have the access of some specific basic features. However, the authenticated users also access some of the premium features of smart mirror by using the voice recognition features. Functional and access diagram of general user and authenticated user of MirrorME application is shown in Figure   2 . The template of horizontal direction is = [−1,0,1] and to filter image, its transposition can be used, that is why it is possible to calculate the horizontal and vertical gradients, easily. The expression (Eq. 1) and (Eq. 2) is given as follows

Here, the direction gradient of x is represented by and the direction gradient of y is represented by , the pixel value of ( , ) is denoted by ( , ). The gradient magnitude of ( , ) is denoted by ∆ ( , ) and calculated (Eq. 3) by

And, the gradient direction (Ɵ) of ( , )is calculated (Eq. 4) by

In machine learning, Support Vector Machine (SVM) [12] used as a binary classifier. Classes separated with the largest gap between the support vectors. Here, the borderline instances in a class known as support vectors. In addition, we have extended the SVM with the support of kernel.

Kernels transform data from input space to feature space. The kernel is a mathematical function that takes two arguments and returns the dot product of their value. Let two data points 1 and 2 mapping is denoted by then the kernel will be (Eq. 5)

When input space and feature space are equal then it is known as linear kernel and this linear kernel is used in linear SVM. Mathematically, the equation of linear kernel can be expressed as (Eq.6)

Due to its capability of faster application training therefore, it is highly efficient in highdimensional data applications. The proposed workflow is consisted of detecting faces using HOG and linear SVM, computing embedding and comparing the vector to the database via a voting method. The basic flow of the HOG feature extraction algorithm and the face detection technique of the proposed application is illustrated in Figure 3 . 

In this segment, the design and implementation approach has discussed with the architectural view of hardware and software. The hardware of the proposed framework consists of a various components including two-way mirror, raspberry pi 3B+, camera, microphone, wood frame, display screen, speaker and some others. Following Figure 4 depicts the IoT based hardware configuration diagram of the proposed MirrorME application. 

The Raspberry pi [13] is a small and powerful computer that uses its own OS known as Raspberry pi OS and this operating system is Debian-based. Importantly, this device is the core component of our designed smart mirror.

For the purpose of face recognition, this work needed to use a camera and our proposed algorithm is incorporated with the camera to identify an existing user for accessing the MirrorME application.

Basically, a mirror is a smooth polished surface where the image is created by the refection. In this work, a two-way mirror is used in which one side is transparent and the other side shows its reflection. The concept of using a two-way mirror is similar to using a usual mirror and that same mirror also acts as a functional object.

In order to present the basic information including date/time, weather update, alarm, news headlines, traffic update and so on, a display monitor is essential in this model. Therefore, in the development of the model, LG 14 inch monitor is connected to the Raspberry pi module using HDMA interfacing.

To utilize the features of the voice-based activity, a microphone is needed in the designed model.

A single Bluetooth speaker is also attached to the system to receive feedback. However, all the output results can be received through the speaker.

Based on the above discussion, the complete model of the architectural diagram and framework design approach of the proposed system shown in Figure 5 . Figure 6 . In addition, a stepwise description of the working procedure presented in Table 1 . In this step, after successful detection of existing user, designed application will provide access to all the premium features. STEP 5 Authenticated users has the privilege to access all the features of the application Besides, for the authenticated users, the system will also provide the user interaction features to check their regular email, stock exchange update, and SMS notification. 

This section has discussed testing results based on experiments. In order to test our designed model, we have conducted a result analysis based on the experience users observed during the assessment of this application. This research work has been carried out in the machine intelligence lab (MINTEL) of the Dhaka International University. Ten (10) participants have chosen to participate in this experiment based on previous experience in interacting with smart devices. All the participants were chosen aged between 22-25 years. In order to achieve higher accuracy in the task of face recognition, we have chosen a frame rate of 50 and a resolution of 480 x 360. The resolution 480×360 has been taken because a high-resolution frame takes a higher time to process and a low-resolution frame normally does not provide high accuracy. Among the Ten (10) percipients, there were Four (4) female participants (FP) and Six (6) After the testing phase of face recognition, the same ten (10) participants again interacted with the mirror with the voice input. Each of the participants provides ten (10) voice input to interact with the individual features including YouTube, Alarm, Traffic, and daily schedule. Table 3 presents the testing results of the voice input. However, due to the network connectivity issue, sometimes the voice input took a longer time to function properly. during the time of interaction with the MirrorME. Table 4 shows the average success rate for each individual based on the data of Table 2 and Table 3 . In addition, we also highlighted the three highest average success rate for the second male participant (~97%), fourth male participant (~92%) and third female participant (~90%). A comparative analysis section also designed and shown in Table 5 to compare the features of MirrorME application with other research work. Based on the comparison, we observed that our proposed system contains every feature that considered as a complete interactive smart mirror and has outperformed other work in the specific category. 

This paper demonstrated a smart mirror device with a user-friendly architecture with many impressive features. Following a service-oriented approach, a stable and easy-to-use architecture was also introduced in this article. However, security issues cannot be overlooked in today's world of interconnected devices. Therefore, this system was designed with a strong authentication framework to ensure the system's end-to-end security. Two different features (face recognition and voice input) sets this designed model aside from other similar works. Furthermore, unique features such as information customization for each individual user made this model more convenient and effective. Again, the prototype's advancement has infinite possibilities in the future in the field of medical data analysis including BMI calculations, temperature check and blood pressure indicator.

The most notable aspect of this model is the ability to carry the smart mirror display across the entire house.

Diy smart mirror

Smart mirror-digital magazine for university implemented using raspberry pi

Smart mirror for smart life

Smart mirror: A novel framework for interactive display

Smart mirror for ambient home environment

Smart mirror using raspberry pi for human monitoring and home security

Voice controlled smart mirror with multifactor authentication

Design of the smart mirror based on raspberry pi

Smart mirror design powered by raspberry pi

Smart mirror devices for smart home and business

Histograms of oriented gradients for human detection

An empirical evaluation of linear and nonlinear kernels for text classification using support vector machines