key: cord-0926349-otei8xtg authors: Ma, Congcong; Man Lee, Carman Ka; Du, Juan; Li, Qimeng; Gravina, Raffaele title: Work Engagement Recognition in Smart Office date: 2022-12-31 journal: Procedia Computer Science DOI: 10.1016/j.procs.2022.01.243 sha: fdf66b337c28fcd6c48dc2fbb5d26e4c7324ae8b doc_id: 926349 cord_uid: otei8xtg The COVID-19 pandemic has forced a sudden change of traditional office works to smart working models, which however force many workers staying at home with a significant increase of sedentary lifestyle. Metabolic disorders, mental illnesses, and musculoskeletal injuries are also caused by the physical inactivity and chronic stress at work, threatening office workers’ physical and physiological health. In the modern vision of smart workplaces, cyber-physical systems play a central role to augment objects, environments, and workers with integrated sensing, data processing, and communication capabilities. In this context, a work engagement system is proposed to monitor psycho-physical comfort and provide health suggestion to the office workers. Recognizing their activity, such as sitting postures and facial expressions, could help assessing the level of work engagement. In particular, head and body posture could reflects their state of engagement, boredom or neutral condition. In this paper we proposed a method to recognize such activities using an infrared sensor array by analyzing the sitting postures. The proposed approach can unobstructively sense their activities in a privacy-preserving way. To evaluate the performance of the system, a working scenario has been set up, and their activities were annotated by reviewing the video of the subjects. We carried out an experimental analysis and compared Decision Tree and k-NN classifiers, both of them showed high recognition rate for the eight postures. As to the work engagement assessment, we analyzed the sitting postures to give the users suggestions to take a break when the postures such as lean left/right with arm support, lean left/right without arm support happens very often. The emergence of Internet of Things (IoT) and wearables [1] brings new services and applications to leverage the interconnection of physical and virtual realms [2] . Their application to the workplace results in the notion of smart offices, it was defined as workplaces that proactively, but sensibly, support people in their daily work [3] . Nowadays, many office workers are suffering a state of subhealth caused by the prolonged sitting [4] and excessive sedentary lifestyle. So, detecting the working status could create the conditions of a better work habit, and reduce the possibility of obesity, diabetes and cardiovascular disease [5] . The worker's body posture often reflects their work engagement state, their level of attention, and psychophysical fatigue. Thus, in this paper we propose to analyze the sitting postures of the workers using an infrared sensor placed on the office desk. This approach allows for collecting data related to workers' sitting posture in a privacy-preserving way. The main contribution of the paper is twofold: • A novel privacy-preserving work engagement system is proposed. • A sitting posture recognition algorithm is proposed for predicting eight sitting postures commonly performed during office works. The remainder of the paper is organized as follows. Section 2 discusses the state of the art, with specific reference to the engagement in the field of smart office, smart factory and online learning. Especially, for the student engagement assessment in the online learning, as the fundamental problems are similar to office work engagement monitoring. Section 3 describes the proposed method and the hardware/software design of the prototype system. Section 4 describes the experiments environment setup, data collection and experiment results. Finally, in Section 5 we presents the conclusion and outline future works. Engagement recognition is widely applied in several application domains (see Table 1 ). The main advantages of such monitoring are: • Smart Of f ice -Work stress might cause significant impact on health such as mental disorders; so the recognition of the work engagement could help the users better adjust psycho-physical status. • Smart F actory -In the manual work in factories, predicting engagement during manual labor is helpful to keep the workers in good psycho-physical condition and improve the work efficiency. • Online Learning -In school, teachers may adapt their teaching approach and schedule also taking in consideration students engagement feedback. Work engagement is characterized by a high level of energy, the ability to deal with job demands, an effective connection with work activities, and strong identification with one's work [6] . Shigeta et al. [7] propose a smart office system using a camera and a blood flow sensor to recognizes office workers' mental and physiological states, that was used to improve their quality of life at office. Alexandros et al. [8] focusing on work environment of that worker's stress, anxiety and depression. They propose a novel mood recognition framework that is able to identify five intensity levels for eight different types of moods, in order to ensure work healthy. Munoz et al. [9] designed an emotion aware automation platform for smart offices, a few IoT sensors like facial expression, speech and text analysis, and biometric sensors were deployed in the office environment, and emotions were recognized using semantic technologies. Based on the recognition outcomes, work related tasks would change automatically, e.g. when a high stress level is detected, the workload will be reduced. Patricia et al. [10] describes an observational, analytical, and longitudinal study using wearable technology like smart band to assess occupational stress in different work environment. Yutaka [11] developed a continuous posture-sensing chair to monitor the human physical state, and provide an interactive digital signage to give the user habit-changing reminders. Also, some researchers focused on the environment construction of the smart office in order to give the user a good working place or for energy saving solutions. Li [12] proposed a comprehensive smart office system concentrated on door-access, lighting, illuminating, ventilating, heating, and reconfiguration is designed in order to save energy and promote the satisfactions of the employees. Sirmacek et al. [13] considered the properties of low cost and privacy preservation with low-cost and low-resolution heat (thermal) sensors to detect room occupancy to automate lightning control, ambient temperature, and air circulation. In the context of Industry 4.0, significant research efforts focused on the manual labor works in smart factories. Sun et al. [14] proposed a Healthy Operator4.0 concept focusing on enhancement of human's physical, sensory, and cognitive capabilities. Health-related metrics were used to track an operator's health aspects. Kajiwara et al. [15] estimating human emotion and engagement by inputting pulse wave, eye movements, and movements to deep neural networks. Their method could predict the emotion and engagement during work with high exercise intensity, and could help the manual workers adopt a clear working targets. Because student learning task has strong similarities to office working, in the following we review the state of the art on student engagement in online learning domain. In this context, most approaches are based on the use of computer vision; in addition, interesting results were obtained with pressure seat cushion, and infrared sensors. Previous works indeed showed the relation between emotional/behavioural components and sitting postures [16] . Camera-based methods are successfully used to recognize sitting postures. Kuang et al. [17] proposed a learner's gesture recognition using a camera based method. They fused the improved scale invariant local ternary pattern and the local directional pattern algorithm, and support vector machine (SVM) is used for classification and recognition. Ashwin et al. [18] proposed a hybrid convolutional neural network (CNN) architecture for analyzing the students' affective states. Based on facial expressions, hand gestures and body postures, three different affective states (engaged, bored and neutral) were recognized. Huang et al. [19] proposed bidirectional LSTM and attention mechanism based deep learning method to identify the degree of engagement; they analyze the videos to extract features such as gaze direction, head posture, face movement, facial expression. Zeng et al. [20] proposed a visual analytics system to analyze classroom videos and recognize the engagement of students during the lecture. Zaletelj et al. [21] used the Kinect One sensor to obtain the feature set characterizing both facial and body properties of a student, including gaze point and body posture; results showed that Kinect-based attention monitoring system could predict students' attention in real time. Hu et al. [22] proposed the LEARNSense framework using the accelerometer and heart rate sensor to recognize the basic student actions in class and then infer the learner context, to describe the engagement level. Kim et al. [23] assessed the student engagement level by measuring a thermal infrared image; skin tem-perature was analyzed to monitor the dynamic emotional state, so the teacher could know the concentration level of the students in real time. Pressure sensor cushions [24] can also be used to detect sitting postures [25] , activity [26] , and activity level [27] . Mota et al. [28] showed that sitting behavior could reflect the students engagement; they extracted the sitting postures to determine the affective state and found a relationship between patterns of postural behaviors and affective status while learning. Shibata et al. [29] proposed an emotion recognition model by analyzing the sitting postures using pressure sensors on the chair and accelerometers on the body. Wataya et al. [30] developed an ambient sensing chair to study the relationship between audience body sway and emotional status. Although the above-mentioned prior research has been successful in detecting the engagement status using various types of sensors, the proposed system in this paper we introduce a new kind of infrared sensor that could recognize the postures of the workers. From the worker's postures, we could infer their work engagement, so as to give suggestions for good working experience. In contrast to previous literature, camera-based approaches suffer the disadvantage of privacy issues while the use of wearable devices such as smartwatch, motion, electroencephalogram (EEG), and electrocardiogram (ECG) sensors could make the users uncomfortable. To overcome these limitations, our proposed method exploits infrared array sensors, which are unobstructive and privacy preserving, to monitor the work engagement level of office workers. The sensor we used in our prototype is a MLX90640 infrared array sensor [31] produced by Melexis Corporation. It is a type of thermopile array sensor which could be used to detect the infrared ray emitted from any thermal source. The array consists of 768 IR sensors (also called pixels). Each pixel is identified with its row and column position as P ix(i, j) where i is its row number (from 1 to 24) and j is its column number (from 1 to 32). The sensor is deployed in front of the subject, and the field of view of the sensor is a rectangular pyramid as shown in Figure 1(a) . The temperature range of the sensor could completely cover the human target and the environment temperature. Figure 1 (b) shows the overview of MLX90640 infrared sensor board. The sensor board uses I2C protocol and can be directly connected to a micro control unit with I2C interface. Figure 1 (c) shows an example of infrared output array of the sensor, where a human target is in the detection area. The brighter the color, the higher the temperature. MLX90640 was connected to an Arduino board using I2C protocol; the ESP8266 WIFI module is also connected the Arduino board using UART protocol. The ESP8266 module was used to transmit the raw thermal images data to the server. The hardware design and electronic components of the work posture monitoring device are depicted in Figure 2 . Our primary goal is to create a smart Work Engagement Monitoring System that, by detecting of the postures of the office workers, can help them to adapt the working schedule so to relieve psycho-physical fatigue and, in the long run, improve quality of work products. Figure 3 shows the proposed prototype system. The system consists of Arduino-based data collection devices with infrared sensor and WiFi module attached (one for each worker to monitor), a server to store the data, a computation development board that was used to execute the machine learning or deep-learning based algorithm, and a PC to visualize the work engagement results. Using the Posture Monitoring Device, we collected the data generated by the workers. After data processing, we can obtain the recognized posture results. Then, it is possible to infer the work engagement status of the workers, and send feedback messages to them. For the Arduino-based data collection devices, we use the Message Queuing Telemetry Transport (MQTT) [32] broker for communication, and Mongo database for data storage. MQTT is a messaging protocol, it's an extremely lightweight publish/subscribe messaging transport protocol that widely used in the areas of Internet of Things (IoT). For the work engagement recognition, we will use machine learning or deep neural network classifier. Due to the computational power demand, we choose NVIDIA Jetson TX2 [33] embedded AI computing device. The PC executes a QT-based desktop application where the workers interacts with the system. The communication flow is as follows: • Arduino-based data collection devices transmit the infrared data to the server. • Server receives and stores the data. Additionally, it queries the NVIDIA Jetson TX2 board for processing. • NVIDIA Jetson TX2 board processes the data and provides the results to the server. • The user gets all the information feedback from the server with web socket. Figure 4 shows the processing workflow of the proposed work posture classification algorithm. In the data pre-processing stage, we need to filter the data and interpolate the raw images. This step makes the image more suitable for feature extraction. The feature extraction method we choose is called Histogram of Oriented Gradients (HOG) [34] . HOG can detect the object's edges and give the outline of the shape, which could better represent different postures. The edge detection can outline the shape of the user's working posture, and the histogram values calculated based on gradient orientations can be used as features for sitting posture recognition. It is worth noting that the HOG process requires to fine tune two parameters: • Cell: the raw image is divided into different rectangular pixel regions; we compared 2 × 2, 4 × 4, 8 × 8, and found that 8 × 8 could better describe the properties of the image. • Block: each block contains a group of cells; in our work we set the block size as 3 × 4, without block overlap. Figure 5 shows the result of each step of pre-processing stage about a raw thermal infrared image. The 32 × 24 raw image generated by the user is not suitable for processing, so we need to rotate it to a 24 × 32 image. Then, we interpolate it to a 96×128 image. Finally, using HOG feature extraction, histogram features are generated for classification. Principal Component Analysis (PCA) is used for feature selection. PCA is used to reduce the dimension of the features in order to reduce the computation cost and improve classifier recognition performance. Finally, we use 10-fold Cross-Validation to validate the classifier recognition results. The ground truth fetched for the evaluation process was carried out by taking video recording data (manually annotated with offline visual inspection) in frame per second and the infrared sensor data were collected in parallel with the video recordings. The experiment setup is shown in Figure 6 . The infrared sensor was placed in front of the worker to record their postures. The distance between the sensor node and the user is between 50 and 60cm, and it is easy to place the sensor in a convenient position. Before the experiment started, the user could fixed their sitting orientation in order to ensure their sitting posture can be well captured in the thermal image with highest resolution. At the same time, the laptop camera was used to record their postures. We created a datasets based on our proposed Work Engagement System. We collected data from 12 volunteer subjects (5 women and 7 men). The subjects' weight varied from 45 to 90 Kg, ages between 25-35, all of the subjects are the postgraduates and doctors in our lab. The experiments has been approved by the Human Subjects Ethics Sub-committee (HSESC) of The Hong Kong Polytechnic University. One experiment lasts for 40 minutes for one subject to perform different postures following the description in Figure 7 . During the experiment, each subject is asked to perform their postures naturally and to move slightly to generate variances with the same posture category as shown in Figure 7 . In this way, we can generate a more comprehensive dataset to deal with inter-and intra-.subject differences. All the experiment sessions were video recorded so to manually label the samples of each performed posture with one of the defined postures. Decision Tree and k-NN classifier algorithms are adopted for this evaluation. Our aim is to prove the effectiveness of our Work Engagement System and Algorithm, so any machine learning classifier is suitable. Therefore, we chose Decision Tree as it is easy to construct and realize, and the kNN classifier because it does not require preliminary model construction. Personalized smart office working environment can be established to provide workers health and psychophysical recognition with privacy-preserving devices. Infrared sensors and machine learning allowed us to achieve privacy-friendly digital portraits of workers, to analyze their activities during office work with the ultimate goals to assess the work engagement level in real-time and so promoting better working conditions. This is a preliminary step of our work focused on Smart Office. As a future work, we will assess the work engagement based on the detected postures. In addition, we are planning to implement our system on embedded tiny computers such as Raspberry PI with deep learning circuit board to process the data. Wearable body sensor networks: state-of-the-art and research directions Internet of things: Vision, applications and research challenges Ambient intelligence: The confluence of ubiquitous/pervasive computing and artificial intelligence Lightsit: An unobtrusive health-promoting system for relaxation and fitness microbreaks at work Chronic stress at work and the metabolic syndrome: Prospective study Identifying work-related internet's uses-at work and outside usual workplaces and hours-and their relationships with work-home interface, work engagement, and problematic internet behavior Implementation of a smart office system in an ambient environment Healthyoffice: Mood recognition at work using smartphones and wearable sensors An emotion aware task automation architecture based on semantic technologies for smart offices Study for the design of a protocol to assess the impact of stress in the quality of life of workers Sensing and changing human behavior for workplace wellness A novel design for a comprehensive smart automation system for the office environment Occupancy prediction using low-cost and low-resolution heat sensors for smart offices Healthy operator 4.0: A human cyber-physical system architecture for smart workplaces Predicting emotion and engagement of workers in order picking based on behavior and pulse waves acquired by wearable devices Emotion-relevant activity recognition based on smart cushion using multi-sensor fusion Learner posture recognition via a fusing model based on improved siltp and ldp. Multimedia Tools and Applications Automatic detection of students' affective states in classroom environment using hybrid convolutional neural networks. Education and information technologies Fine-grained engagement recognition in online learning environment Emotioncues: Emotion-oriented visual summarization of classroom videos Predicting students' attention in the classroom from kinect facial and body features A framework for learning analytics using commodity wearable devices Ambient intelligence in a smart classroom for assessing students' engagement levels Adaptive sliding window based activity recognition for assisted livings Smart cushion-based activity recognition: Prompting users to maintain a healthy seated posture Activity recognition of wheelchair users based on sequence feature in time-series Activity level assessment using a smart cushion for people with a sedentary lifestyle Automated posture analysis for detecting learner's interest level Emotion recognition modeling of sitting postures by using pressure sensors and accelerometers Sensing of audience excitation and boredom emotion based on the synchrony of sitting body sway Histograms of oriented gradients for human detection