The Human Behaviour Indicator; A Measure of Behavioural Evolution Abubaker Elbayoudi1,2, Ahmad Lotfi1,∗, Caroline Langensiepen1 School of Science and Technology, Nottingham Trent University, Clifton Lane, NG11 8NS, Nottingham, United Kingdom Abstract Activities of daily living (ADL) or activities of daily working (ADW) may be affected by changes in a person’s health or well-being. Measuring pro- gressive changes in one activity or multiple activities is representative of behavioural variations. By inspecting the trends in multiple activities, it is possible to identify and predict human behavioural changes. We refer to the trends in people’s behaviour as behavioural evolution. In this paper, we propose a novel indicator to measure the progressive changes representing a participant’s behavioural evolution. The proposed indicator presents ac- tivities as a holistic measure, which first combine multi-activities and then measure the progressive changes in the combined activities for each single day. Real data sets were collected from a wireless sensor network and used to examine our proposed technique. As part of this process, we were able to quantify progressive changes for individual and aggregated activities. Our experimental results demonstrated that: (1) the proposed approach can identify and distinguish normal and abnormal behaviours; (2) large data sets gathered from sensors in an intelligent environment represented in various time series can be visualised in a simple and more understandable format; (3) identifying trends in ADLs or ADWs is a relevant means of sharing information with carers or supervisors. ∗Corresponding author: Email: ahmad.lotfi@ntu.ac.uk 1All authors are with School of Science and Technology, Nottingham Trent Univer- sity, Clifton Lane, NG11 8NS, Nottingham, United Kingdom. Ahmad Lotfi (Email: ah- mad.lotfi@ntu.ac.uk ); Abubaker Elbayoudi (Email: abubaker.elbayoudi@ntu.ac.uk ) and Caroline Langensiepen (Email: caroline.langensiepen@ntu.ac.uk ) 2PhD research student who has conducted the research as part of his thesis. Preprint submitted to Expert Systems with Applications October 23, 2018 The following abbreviations are used in this manuscript: ADL Activities of Daily Living ADW Activities of Daily Working AmI Ambient Intelligence EWMA Exponentially Weighted Moving Average EMD Empirical Mode Decomposition HBI Human Behaviour Indicator MA Moving Average MCI Mild Cognitive Impairment PD Parkinson’s Disease SCI Synthetic Composite Indicators WHBI Weighted Human Behaviour Indicator EWHBI Exponential Weighted Human Behaviour Indicator Keywords: Activities of daily living; activities of daily working; trend analysis; moving average; behaviour indicator; behavioural evolution 1. Introduction Measuring progressive change in people’s activities is a subject of re- search interest in the machine learning community and has recently at- tracted more attention (Mahmoud et al., 2014) (Wang et al., 2018) (Nweke et al., 2018). These could be activities of daily living (ADL) or activities of daily working (ADW). A number of medical conditions and their treat- ments are associated with activities disorders such as reduced movement over time. Parkinson’s Disease (PD), for example, is characterised by slowness of movement (Zhan et al., 2016), and some Mild Cognitive Impairment (MCI) causes a slight but noticeable and measurable decline in ADL (Wallace et al., 2017)(Kearns et al., 2017). Understanding progressive changes in human behaviour is the key chal- lenge in formulating an effective intelligent environment. Many researchers have investigated human activity recognition in order to solve the prob- lem of activity extraction and prediction. These studies have used different statistical as well as Computational Intelligence methods and techniques to identify the behaviour of occupants based on temporal data gathered from sensor networks. There are many challenges in using such methods to 2 find the relationship between the gathered sensory data from an intelligent environment and identifying the actual behaviour of the participant. Un- derstanding human behaviour from low level sensory data and interpreting them in a meaningful manner are the main challenges that the research here face. The research aims to identify trends in ADL or ADW and interpret them in a presentable format to the user. Change of behaviour of a participant in an Ambient Intelligence environments (AmI) is an indicator of a person’s social and health status. The paper is primarily concerned with the inter- pretation of progressive changes in the participant’s behaviour. The project is aiming to assess the ADLs or ADWs of a person who lives or works in- dependently in his/her own home or office and present it as a single value for each day. This will provide a holistic view of the monitored person’s behaviour and it will be used to forecast the behavioural changes of the par- ticipant and it will raise an early warning of an abnormal behaviour when it is observed or expected to happen in the near future. This paper, therefore, will employ and assess such approaches to discover suitable methods for constructing a synthetic composite indicator (SCI). In particular, it aims to deliver an intelligent technique as a framework to build a human behaviour indicator (HBI) that can help to enable consistent and transparent assessments and forecasting of the progressive changes in human behaviour. The HBI represents a holistic report based on multi- ple sensors/activities representing progressive changes in the participant’s behaviour. An alternative means of identifying the behavioural changes is by clas- sifying the activities into normal and abnormal groups. The underlying technique is to use supervised or unsupervised classifiers to learn the pat- tern within the activities. There are some challenges to be addressed by this approach. Firstly, human activities are a combination of many interrelated activities and classifying activities individually will not provide a holistic view of the changes. Secondly, the classification would require suitable tech- niques to deal with temporal data classification. To be able to have a measure of progressive changes, the first step is to integrate some pervasive measurements into home and work environments to collect the data representing ADL or ADW. There are already many off- the-shelf products available to collect such data. Once individual activities are recognised, identifying trends in users’ behaviour over a period of time will provide useful information. Trends in people’s behaviour are used to identify progressive changes and predict behavioural abnormalities. We refer to the trends in people’s behaviour as behavioural evolution. For instance, 3 identifying the progressive changes in the behaviour of a person suffering from MCI will allow the caregiver to monitor and intervene when abnormal behaviour is predicted. The ADL pattern will change over time and this is a consequence of the individual’s condition and progression of the disease. Identifying an evolving behavioural pattern will help to predict the trend in the ADL before any abnormalities are identified. As a different example, ob- serving and analysing a person’s work activity in an office environment will provide managers with an analytical tool to measure employee’s productiv- ity. This could be a holistic measure of many activities including computer work, desk work, meetings etc. In this paper, we seek answers to the following questions: • How to extract behavioural patterns of a person’s ADL or ADW by analysing sensory data collected from different sensors in an intelligent environment. • How the sensory data is processed to become a time series and then to identify trends within the data. • How the data is forecast to extract important daily patterns from them and predict the direction of the trends in the data. • How the multiple data that represent multiple activities or events in every individual day are combined in one single datum to represent the participant’s overall behaviour. • How unexpected patterns and anomalies within the combined data can be identified. This paper is organised as follows: related work is presented in Sec- tion 2 followed by the new human behaviour indicator and its constructions introduced in Section 3; three case studies using different data sets are dis- cussed in Section 4 and the conclusions and suggestions for future work are presented in Section 5. 2. Related Work In this section a brief summary of the related work in the context of trend analysis, abnormality detection and activities recognition is presented. 4 2.1. Trend Analysis Trend analysis techniques are used to understand the participants’ gen- eral health variation and behaviour evolution. Analysing the activities of a participant can only be fully understood if the daily activities are examined in terms of person, place and time. Numerous researchers have investigated trend analysis techniques that can be applied to human behaviour activi- ties extracted from intelligent environments. For example, Mahmoud et al. (2011) presented the importance of trends in the ADL of a single elderly per- son. Researchers on the BackHome (Rafael-Palou et al., 2015) and iCarer (Lotfi et al., 2017) projects introduced monitoring systems to send feed- back on any changes in the participant’s behaviour and habits to the carer. Trends in activities of daily living disability in a large sample of community- dwelling are presented in (Yu et al., 2016). A very similar study of trends in disability of instrumental activities of daily living among older Chinese adults is presented (Liang et al., 2017). Trends in activities of daily living among stroke survivors are investigated by Chattopadhyay et al. (2013). Forkan et al. (2015b) used Holt’s linear trend method to forecast the time series data of vital signs such as heart rate or blood pressure. In another effort, Forkan et al. (2015a) used trend analysis to detect trends in their data, but it is not clear which trend analysis technique they used. A linguistic summarisation for describing long-term trends in changes in human behaviour is presented by Ros et al. (2011); the procedure is used to provide information to older adults and their carers or family in language that is easy to understand by applying a measure of similarity to compare behaviours that are adapted over time. There idea is based on clustering and comparing the similarities between the activities in a specific period in order to detect trends. However, the study concentrated on linguistic summarisation more than trend analysis itself. Financial time series are considerably difficult to analyse and predict. Furlaneto et al. (2017) have applied a bias effect on predicting market trends with empirical mode decomposition (EMD) to predict market indices. Bagher et al. (2017) introduced the concept of trend to capture dynamics in user interests for a content-based recommender system. The authors pro- posed a Bayesian nonparametric model to construct the trend distributions. 2.2. Abnormality Detection Suryadevara et al. (2013) applied a double exponential smoothing strat- egy to the activity duration time series in models that detects changes in the behavioural and health-related status of a monitored patient who lives 5 in a smart home. Saives et al. (2015) present a model to improve the auton- omy of medically monitored behaviour changes for patients in a smart home. Smart home environments can perceive long-term changes that may cause health concerns. Such systems alert the carers and family of any important changes in the occupant’s behaviour, such as diet, daily tasks or health. AmI offers such solutions, for example, by using human behaviour recognition to monitor the person’s activities and alert the carer if something abnormal is detected. Human behaviour recognition has been demonstrated to be a valuable key to understand people’s needs (Lara and Labrador, 2013). Probabilistic models such as the Hidden Markov Model (HMM) and Bayesian belief networks can be used to model human behaviour (Liu et al., 2010)(Xie and Wu, 2012). Soft computing and machine-learning techniques such as artificial neural networks (ANNs) and support vector machines (SVM) can also be used in this way (Yin et al., 2008)(He and Jin, 2009). However, all mentioned techniques face difficulties in processing large amounts of low-level sensory data; therefore, it is essential to transform the sensory data into a suitable format that can be processed (Cook et al., 2009). Aran et al. (2016) have investigated anomaly detection in elderly daily behaviour based on a probabilistic spatio-temporal model. They have sum- marized daily behaviour to be able to identify the anomaly. A Bayesian formulation is provided for anomaly detection in (Ordóñez et al., 2015). Ac- tivities of residents were extracted using Bayesian statistics and behaviours were estimated based on three probabilistic features. The proposed ap- proach has been validated using 14-25 days of real data. The research in (Suryadevara and Mukhopadhyay, 2012) proposes to compute a wellness in- dex to capture abnormal behaviour using several weeks of observations from elderly living alone. Their proposed approach is based on fusing of informa- tion gathered from sensors in the environment. The wellness functions are calculated during the runtime of the system as background process taking the activity durations from the respective files. The proposed approach is rather simplistic with no experimental data from a real environment. In a very recent publications, Kim and Cho (2018) have applied a hy- brid of convolutional neural network (CNN) and long short-term memory recurrent neural network (LSTM) known as C-LSTM to web traffic anomaly detection. Similar work is reported in video surveillance systems and recog- nition of abnormal behaviour. Readers are refer to (Mabrouk and Zagrouba, 2018) for a review of the subject area. Based on our survey of the literature, the proposed technique presented in the next section to measure the behavioural evolution is novel and it has not been investigated by other researchers. 6 3. The Human Behaviour Indicator To represent human behaviour evolution through a simple measure re- quires some method of combining different types of data. This will require the construction of a composite indicator which can be viewed as a means of reducing multivariable inputs into a single and meaningful output that can be interpreted by non-experts. Aggregating multivariable data seems a simple process, however “the methods for aggregating vast amounts of empirical data remain rather crude” (Cherchye et al., 2002). After giving a short overview of composite indicators below, the Human Behaviour Indi- cator (HBI) will be explained. 3.1. Composite Indicators The aggregated output usually represents a holistic view using a single numerical score and/or an ordinal rank. To achieve a single output value, the index must go through accurate development steps. These steps involve consideration of included or excluded variables, handling missing data, the given weight for each variable etc. Issues related to generating a composite indicator include: • The similarity process of the composite index elements: There are many steps that may be considered to start the measurement of the index, including making sure that the available variables are appropri- ate, that the variables are sufficient and well defined to characterise and explain all elements, and suitable to develop a new measurement index to examine progressive changes in a certain domain. • The clustering process: Clustering can be applied based on the com- parison of the similarity between different elements involved in the composite index using different clustering techniques. The cluster- ing process could use the distance measurement methods such as Eu- clidean, Squared Euclidean or City Block etc. Researchers have developed many techniques to handle the measurement of changes in different domains; however, in this research, only statistical aggregation techniques are used. There are two major types of aggrega- tion technique: additive (linear) and multiplicative (geometric or nonlinear) (Zhou et al., 2010). Additive aggregation technique can offer compensability between aggregated indicators; if an indicator has poor performance, then an indicator that has a significant value can cover the poor one, which may result in the creation of a biased composite indicator. 7 Figure 1: Proposed framework to create the Human Behaviour Indicator. Using weighted methods to develop a statistical composite indicator is crucial to illustrate the results in a single variable outcome, and choosing the best strategies for weighting variables is still the focus of researchers. Equal Weights (EW) is recommended as the standard technique for constructing composite indicators and (Hopkins, 1991) has used EW in building compos- ite indicators. However, some authors disagree with this approach, such as Cherchye et al. (2007), who raised many issues with using EW. They ar- gued that this could result in composite indicators using a “fundamentally flawed” method. 3.2. The Construction of the Human Behaviour Indicator The schematic diagram of the proposed framework to create the Human Behaviour Indicator (HBI) is shown in Figure 1, and comprises three phases: • In the first phase, the ADL or ADW data is collected and pre-processed. This includes the handling of missing data and the extraction of the features that will be involved in representing the holistic view of the activity. Each feature represents specific information from a single activity; for example, the start time and the duration of each event. 8 • In the second phase of the proposed framework, the data are analysed using trend analysis techniques. • Finally, in the third phase of development, the data created from the trend analysis is standardised to a uniform unit of measurement and then aggregated. The boundaries of the three levels of the holistic view of the monitored person are also generated. These levels are the level of normal activity behaviour, the warning level to represent small changes in activity behaviour, and the abnormal level to detect the outlier activity behaviour. 3.2.1. Date Pre-Processing Data preparation (pre-processing) is an important step because inade- quately controlled methods of data-gathering can result in out-of-range val- ues, impossible data combinations or missing values. Analysing data that is not carefully screened can produce misleading results. Thus, the represen- tation and quality of data comes first, before running the analysis. There are many statistical methods to deal with missing and outlier data; we need to consider which method is suitable to our data. The common methods in- clude: listwise deletion, pairwise deletion, mean/mode substitution, dummy variable adjustment, regression imputation, maximum likelihood, and mul- tiple imputation (Peugh and Enders, 2004) Data examination and exploration before performing and interpreting analysis are very important; therefore, discussing ways to evaluate and un- derstand missing data is the first step that should be taken after formatting the data. The following steps are the principal options for analysing missing data: • Analyse only available data (i.e. ignore the missing data) • Impute the missing data by replacing values, and treat these as if they were observed (e.g. impute the mean, impute based on predicted values using analysis methods) • Impute the missing data and consider that these were imputed with uncertainty (e.g. multiple imputations, simple imputation methods with adjustment to the standard error) • Assume by using statistical models to process the missing data based on their relationships with the available data. In this research, the pairwise deletion method was used, as it was suitable to process our data. It was used to improve the validity of the research 9 results and to reduce the waste of resources caused by missing data. The basic idea for this method is to analyse all cases in which the variables of interest are present, and it attempts to minimise the loss that occurs in listwise deletion. This method keeps as many cases as possible for each analysis and uses all possible information with each analysis. On the other hand, it cannot compare analyses because the sample is different each time (Peugh and Enders, 2004). Table 1 shows samples of missing data in our data sets; therefore, we did not discard the whole data, and only the missing value was discarded from the analysis of the data set. 3.2.2. Trend Analysis In this research, we investigated whether the use of trend analysis tech- niques to smooth data gives better results when interpreting human be- haviour. It is found that the moving average (MA) is a technique that can be used to gain an overall idea of the trends in a data set; it is useful for forecasting long-term trends (Droke, 2001). The MA creates a series of av- erages of different sub-sets of the full data set and can be calculated for any period. For example, if we have several years’ worth of data from monitor- ing an elderly person in his/her smart home, then a moving average of days, months or years can be calculated. Moreover, using long-term monitoring data to calculate a moving average will help us to see trends in this person’s behaviour and forecast his/her well-being situation. Table 1: A sample of real data including missing data. Smart Home Date and Time Sensor Type Location Day1 , 09 : 04 : 17.656 PIR Living Room Missing Data PIR Master Bedroom Day1 , 11 : 06 : 03.796 PIR Kitchen Day2 , 10 : 04 : 17.656 Missing Data Living Room Day2 , 10 : 14 : 10.050 PIR Living Room Day3 , 11 : 45 : 10.766 PIR Master Bedroom Day4 , 12 : 06 : 03.996 PIR Kitchen Smart Office Date and Time Sensor Type Location Day1 , 09 : 04 : 17.656 PIR Chair Day2 , 09 : 05 : 03.766 Missing Data PC Day3 , 10 : 30 : 03.796 Door Out 10 However, the MA has different versions; in this research, it is found the exponentially weighted moving average (EWMA) to be the best technique applied to detect trends in ADL/ADW data. EWMA, also known as an ex- ponential moving average (EMA), is an infinite impulse response filter that applies a weighting factors (for each older datum) that decreases exponen- tially and never reaches zero (Holt, 2004)(Gardner, 2006). The EMA for a series P may be calculated recursively: EMA1 = P1 (1) for t > 1 EMAt = β.Pt + (1 − β).EMAt−1 (2) where: • β represents the degree of weighting decrease, which is a constant smoothing factor between 0 and 1. A higher β discounts older obser- vations faster, • Pt is the datum point value at time t, • EMAt is the value of the EMA at time t. EMA1 can be initialised in different ways, most commonly by setting EMA1 to P1. It can be initialised by setting EMA1 to an average of the first four or five observations. It is very important to initialise EMA1 because it affects the resultant MA depending on the β values. Choosing small values of β make the choice of EMA1 relatively more important than large β values. Higher β values will discount older observations faster (Handbook, 2013). EMAt as a weighted sum of datum points Pt is presented in Equation 3: EMAt = β × (Pt−1 + (1 − β) × Pt−2 + (1 − β)2 (3) ... × Pt−3 + (1 − β)k × Pt−(k+1)) + ...(1 − β)k+1 × EMAt−(k−1) for any suitable k = 0, 1, 2, ..., the weight of the general datum point Pt−i is β(1 − β)(i−1). This can be expressed as Equation 4 to show the steps of EMA towards to the latest datum point, using a proportion of the difference (each time). EMAnew = EMAold + β × (Pcurrent − EMAold) (4) 11 3.2.3. Multivariate Aggregation To identify the overall changes and, ultimately, to forecast changes in the ADL or ADW, a new indicator, the Human Behaviour Indicator (HBI), is introduced. This stage has two main tasks; the first task is to build the composite indicator, which will compute the progressive changes in the behaviour based on the events that are performed in the whole day. The events are the actions that the participant performs during his/her activity. Each event has a start time and a duration. For example, when the person works at the computer and sits on the chair, we will compute the start time of using the computer until he/she stops using it as an event, and, at the same time, the start time of using the chair until he/she leaves it as an event. This idea of using the events gave us the opportunity to work with overlapped events and activities. Therefore, we were able to compare or measure the changes of events on different bases such as daily or weekly. In this research, we used the percentage of changes between the events when compared to each other as the best way to measure the changes. However, this task had three steps, as follows: • There are different ways in which we could aggregate human behaviour events and to calculate the percentage of change of the compared events. As part of our investigation, it is found that Weighted Human Behaviour Indicator (WHBI) expressed in Equation 5 and the Expo- nential Weighted Human Behaviour Indicator (EWHBI) expressed in Equation 6 are the best candidates for data represented in this paper. Each equation has its way to reach the final results but both of them gave the same final results. Therefore, either of them could be used as a start point of the calculations. • Calculate Pri = ∑ (Pi+n) ∗ 2∑ (Pi) ∗ (Pi+n) . (5) • Calculate Pri = (α) ∑ (Pi+n) ∗ 2 (1 − α) ∑ (Pi) ∗ (Pi+n) . (6) where: P is current data point value after using EWMA (i.e. start time, duration) of the activity for i=1,2,3,.. and n is the number of data points used to calculate EWMA. • Calculate the summation of changes in events based on all daily events. 12 totald = ∑ Pri. (7) where: d is the number of each day and i=1,2,3,.. represents the percentage of change of each compared event of that day. • Calculate the percentage of total changes of all daily events. Indexi = (totald/totald−n) × 100. (8) where: i,d=1,2,3,.. and n is the number of data points that are used to calculate EWMA. The second task is to calculate the boundaries of the changes. In other words, the index is centralised to 100, because the indicator will present the percentage of the changes in the overall of daily activities after calculating the percentage of change between the values of compared events’ features (i.e start time and duration of each event). The system has three boundaries on each side of the centre value (100). These boundaries represent three main categories: normal, warning and abnormal. The following steps show the process of calculating the indicator and the boundaries: • Calculate the average of all Indexes’ values. avg = ∑ Index/N. (9) where: N is the total number of data points. • Calculate the normal boundary. NB = 100 ± Avg. (10) where: NB is the normal boundary. • Calculate the warning boundary. WB = AV ± NB. (11) where: WB is the warning boundary and AV is the accepted value to represent the warning situation. • Calculate the abnormal boundary. AB = WB ± Pt. (12) where: AB is the abnormal boundary and Pt is a data value at time t. 13 4. Case Studies and Discussion In this section a brief description of data used in this research is provided before the proposed method of formulating the indicator is introduced. The data sets used for this research are collected from within intelligent environ- ments equipped with some sensors to measure indoor activities of an occu- pant. The indoor movements are represented by the sequences of movement from one place to another. Monitoring the movements of a person occupying the environment and collecting the data representing his/her mobility can detect this person’s transitions. Table 1 shows the format of the data summarisation, which represents two people, each occupying an intelligent environment. The first person occupies a smart home and includes examples of his/her activates based on his/her sequential movements during the occupancy of a specific location. Each behaviour can be represented in two order levels. The first level is the activity sequence, which shows the person’s movements. The second level is the sequence of actions, which shows the occupancy time of the activity that the person has performed. For example, walking to the kitchen is a moving activity and sitting in the living room is an occupancy activity. Activities in a smart home are presented by five main categories: “Bed- room”, “Living Room”, “Kitchen”, “Bathroom” and “Hall” activities, rep- resenting sleeping, socialising, eating, cleaning and moving between places, respectively. Activities in a smart office are presented by four main cate- gories: “Chair”, “PC”, “PIR” and “Door” activities, representing sitting, working, occupancy and going in/out, respectively. In this section, three separate case studies are reported. Two of these case studies have real data sets collected from intelligent environments. The first data set represent the ADL of an elderly person who lives in a smart home and the second data set represents the ADW of a person who works in a smart office. In the third case study, a simulated data set is used; this data set simulates an elderly person who lives in a smart home. Real data were collected using wireless motion sensors, PIR and door sensors. The output values of these sensors are discrete. In the smart home these values represent the occupancy of one area at a time, whereas in the smart office, the values may overlap because of the possibility of using different equipment at the same time; for example, the participant may use the PC while sitting on the chair and occupying the desk. However, the results and discussion presented below are based on one participant in each intelligent environment. 14 4.1. Case Study 1: Combine all events of ADLs extracted from real data of a monitored person who lives in a smart home This study aims to establish whether the proposed technique can show the overview of a person’s status and detect abnormality within the be- haviour of a participant who lives in a real environment. In this case study, the data represents all events of ADLs that are completed in different areas of a smart house. To demonstrate the holistic view of the monitored per- son’s activity status based on the sensor network measurement, the data sets are represented in separate groups. Each group shows the start time and duration of each event in one area. These events represent the occupancy activities in each area of the smart home. For example, Figure 2 and Figure 3 show the start time and duration time of events that represent the occupancy activities for the bedroom and kitchen, respectively. In both figures, a sample of only three days is depicted. These days are the same weekday of three weeks. It is very difficult to see each area’s events in one graph if we are hoping to understand the overall behaviour changes or to interpret the behaviour; however our technique can display the holistic view of the person’s behaviour in one graph. For instance, Figure 4 shows the status of a person based on the collected data for 140 days. As can be seen from Figure 4 in the upper graph, the participant is often in a good situation. Frequently, the percentage of changes in this person’s behaviour are around the average of his/her daily routine. However, on some days we can see abnormality in the percentage of changes, which goes beyond the second boundaries that represent the red area in this graph. To explain what led to this result, the lower graph in Figure 4 shows the number of times the participant visited the areas and the total duration spent in each area, alongside the mean of number of visits and the total duration for each area. We thus have at least an idea of the differences in the patterns of normal and abnormal behaviour, as result of using our technique to interpret the overview of the participant’s behaviour in an appropriate graph of all events that occurred in all areas in the home. In addition, this technique makes it possible to identify the abnormality from low-level sensory signals even if we do not have detailed knowledge of the subject. Table 2 shows a sample of the real data recorded in a smart home, which represents the three level that are used in our technique. This information shows that our technique can measure the changes of human behaviour. 15 Figure 2: Sample of events in the Bedroom over three days. Figure 3: Sample of events in the Kitchen over three days. 16 Figure 4: Holistic view of an elderly person’s behaviour in a smart home. 4.2. Case Study 2: Combine all events of ADWs extracted from real data of a monitored person who works in a smart office The aim of this case study is to investigate whether the proposed tech- nique can show an overview of the person’s status and detect abnormality within the behaviour of a participant who works in a real environment. In this case study, the data represents all events of ADWs that are performed in different areas in a smart office. To demonstrate the holistic view of the monitored person’s activity status based on the sensor network measure- ments, the data set is represented in separate groups. Each group shows the Table 2: A sample of the results of using HBI with the smart home’s data. Places Normal Warning Abnormal T.D N.V.P T.D N.V.P T.D N.V.P Bedroom 561 34 488 35 300 30 Kitchen 203 51 217 56 175 55 Bathroom 97 15 154 11 278 13 Hall 102 69 121 82 113 74 Living Room 477 56 458 72 572 73 T.D: Total duration, N.V.P: Number of visiting a place 17 start time and duration of each event in one area. These events represent the occupancy activities in each area in the smart office. It is very important to notice that the data sets overlapped in their times. For example, when the person is using his/her PC, the PC’s sensor will trigger; at the same time, the chair’s sensor will trigger. Figure 5 and Figure 6 show the start times and durations of events that represent the occupancy activities for the desk and the chair, respectively. In both figures, only a sample of three days is depicted. These days are the same weekday in three weeks (e.g. Mondays). Applying our technique to such data can help to understand the overall behaviour changes or the behaviour interpretation by displaying the holistic view of the person’s behaviour in one graph. For instance, Figure 7 shows the status of a person based on the collected data for three months. As can be seen in the upper graph of Figure 7, the participant is using his/her office normally. However, on some days an abnormality occurs and we can see the change in the percentage of changes levels, which goes beyond the second boundaries that represent the red area in this graph. To explain what led to this result, the lower graph in Figure 7 shows the number of times the participants used the equipment in the office and the total duration of each use, alongside the mean number of usage and the total duration of use of each piece of equipment. Therefore, we have at least an idea of the differences in the patterns in the normal and abnormal behaviour. Table 3 shows that we gained important results from using our technique with data from multiple sensors that sometimes overlapped. The HBI shows the progressive changes in the participant’s behaviour and interprets it in one relevant graph of all events using all sensors in the office. In addition, this technique makes it possible to identify the abnormality from low-level sensory signals, even if we do not have detailed knowledge of the subject. Table 3: A sample of the results of using HBI with the smart office’s data. Places Normal Warning Abnormal T.D N.T.S T.D N.T.S T.D N.T.S Chair 401 83 338 54 303 9 Duration 404 56 513 60 595 67 PC 401 449 291 213 44 48 T.D: Total duration, N.V.P: Number of triggering a sensor 18 Figure 5: Sample of events using the desk over three days. Figure 6: Sample of events using the chair over three days. 19 Figure 7: Holistic view of a person’s behaviour in a smart office. Figure 8: Sample of simulated events in the Bedroom over three days. 20 Figure 9: Sample of simulated events in the Kitchen over three days. Figure 10: Holistic view of simulated activities of an elderly person in a smart home. 21 4.3. Case Study 3: Combine all events of ADLs extracted from simulated data of an elderly person In this case study, simulated data prepared in our previous work (El- bayoudi et al., 2015) were used to demonstrate that the proposed technique can detect abnormalities in simulated data, as well to show the overview of the person’s status. In this case study, the data represents all events of a less mobile person’s ADLs that are simulated to be carried out in differ- ent areas in a smart house. The simulated data are grouped into events to demonstrate the holistic view of the monitored person’s activities. Each group shows the start time and duration of each event in one area. These events representing the participant’s activities in each area in the simulated smart home. For example, Figure 8 and Figure 9 respectively show the start time and duration time of events that represent the occupancy activities for the bedroom and the kitchen. In both figures, a sample of only three days is depicted. These days are the same weekday in three weeks (e.g. Mondays). However, it is very difficult to understand the overall situation of the monitored person by observing each area’s events graph, but we can easily understand the overall behaviour changes or the behaviour interpretation if we have a single graph to show the changes. For instance, Figure 10 shows the progressive changes of a person’s behaviour based on the simulated data for around 150 days. As can be seen from Figure 10 in the upper graph, the participant has some abnormal days based on the percentage of changes in this person’s behaviour. On the other hand, we can see that this person has a more stable situation because the frequency of the percentage of changes in this person’s behaviour are around the average of his/her daily routine. To explain what led to this result, the lower graph in Figure 10 shows the number of visits to the areas and the total duration in each area, alongside the mean of the number of visits and the total duration of each area used as a scale of Table 4: A sample of the results of using HBI with the simulated smart home’s data. Places Normal Warning Abnormal T.D N.V.P T.D N.V.P T.D N.V.P Bedroom 640 13 673 14 701 14 Kitchen 96 19 102 18 104 18 Bathroom 64 10 48 9 48 10 Go Out 79 1 65 1 49 1 Living Room 560 21 550 19 537 22 T.D: Total duration, N.V.P: Number of visiting a place 22 changes. Therefore, we have at least an idea of the differences in the patterns of normal and abnormal behaviour. Again, in this case study, we can see from the results from using our technique, that it can be used to interpret the overview of the participant’s behaviour in an appropriate graph of all events that occurred in all areas in the home. In addition, this technique makes it possible to identify the activity levels that are proposed in our technique (normal, warning and abnormal) from low-level sensory signals as it is shown in Table 4. This study has par- ticularly measured the similarity in binary data sets which represent daily activities. The investigation presented here aims to assess the monitoring and measurements that can identify the changes in the pattern of human behaviour, and to distinguish between normal and abnormal behaviour pat- terns of participants. The HBI technique is proposed and applied to binary time series to measure progressive changes in human behaviour and to de- tect abnormal behaviour. The HBI allows the supervisor/carer to observe any changes to the regular pattern on a daily basis and it can interpret and explain the observed changes. The results from HBI are tested and applied to real data sets in two case studies and they confirm the suitability of the recommended indicator. The major findings of this research in terms of measuring progressive changes in human behaviour activities using binary time series data are listed below: 1. Very limited research is reported in the literature on measurement of changes in human behaviour and analysis of binary time series, in particular for analysing real binary data sets collected from intelligent environments. This research has helped with better interpretation and understanding of binary time series using the investigated techniques. 2. Real data sets collected from sensors in intelligent environments could lead to a large amount of data. This binary data is hard to work with and needs to be converted into time series using proper techniques that must not be misleading about the results. 3. Using EWMA has helped to be able to detect trends in the partici- pant’s behaviour and predict the future of his/her behaviour. 4. The demonstrated results indicate that the combination of prediction and trend analysis techniques produced a better understanding of time series. 23 5. The importance of the extraction of daily events from the raw data (time sequence) as inputs to the used techniques is investigated. 6. The impact of the aggregation of daily events and observation of the behaviour changes using HBI technique is investigated. This technique gives clear results of behavioural changes and interprets them in a suitable graph. The HBI gives superior results even when data are overlapped such as data collected from a smart office. 7. Overall, this research could play a vital role in the fields of observing human behaviour changes with more development. 5. Conclusions and Future Work In this paper, a novel technique for measuring progressive changes in a person’s behaviour presented in the form of ADL/ADW is introduced. The proposed approach generate an indicator referred to as HBI which represent multiple activities of a person. The overall progressive changes and HBI could also be depicted in a single graph. Furthermore, the results presented here show that HBI is a very promising approach to interpret binary data sets collected from intelligent environments. The data sets investigated in this paper were based on single participants who occupied a house equipped with appropriate sensors. The results presented here show that data gath- ered from ambient sensors can provide important information about the patterns and behaviour of the participant. A direct comparison of our proposed approach with other techniques e.g. clustering techniques may not be possible. Most of the existing ap- proaches deal with each activity individually. For example, if we classify the data gathered from sleeping activity only, it may represent abnormality in the sleeping activity. However, when this is studied with other activities including eating (kitchen activity) the overall activity may not be abnormal. Further investigation, in which further work could be undertaken, is the implementation of the approach presented in this paper for multiple users in intelligent environments. The current approach is not tested using data sets representing multiple occupants (when visitors or carers are visiting) or when pet animals are involved. This study will serve as a base for future studies in which a combination of data sets from several sensors can be used. The current study has only examined human behaviour changes for a single user without any disruption to the normal routines. Building an easily interpreted user interface for 24 our system to train and test other techniques is part of our future plan. The progressive changes techniques used in this research could be extended to measure more complex human behaviour. The indicator presented here could also be used as a tool for predicting an activity or the overall behaviour of the user. References Aran, O., Sanchez-Cortes, D., Do, M.-T., Gatica-Perez, D., 2016. Anomaly detection in elderly daily behavior in ambient sensing environments. In: Chetouani, M., Cohn, J., Salah, A. A. (Eds.), Human Behavior Under- standing. Springer International Publishing, Cham, pp. 51–67. Bagher, R. C., Hassanpour, H., Mashayekhi, H., 2017. User trends modeling for a content-based recommender system. Expert Systems with Applica- tions 87, 209 – 219. Chattopadhyay, K., Douiri, A., Sheldenkar, A., Wolfe, C., Rudd, A., Chen, R., 2013. Trends in activities of daily living among stroke survivors: Anal- ysis from the south london stroke register. International Journal of Reha- bilitation Research 2 (2), 6–21. Cherchye, L., Kuosmanen, T., et al., 2002. Benchmarking sustainable devel- opment: A synthetic meta-index approach. Documento preparado en el contexto del programa de investigación en Métodos No paramétricos en Enomomı́a de la Producción, los recursos Naturales y el Medio Ambiente. Cherchye, L., Lovell, C. K., Moesen, W., Van Puyenbroeck, T., 2007. One market, one number? a composite indicator assessment of eu internal market dynamics. European Economic Review 51 (3), 749–779. Cook, D. J., Augusto, J. C., Jakkula, V. R., 2009. Ambient intelligence: Technologies, applications, and opportunities. Pervasive and Mobile Com- puting 5 (4), 277–298. Droke, C., 2001. Moving averages simplified. Marketplace Books. Elbayoudi, A., Lotfi, A., Langensiepen, C., Appiah, K., 2015. Modelling and simulation of activities of daily living representing an older adult’s behaviour. In: Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments. ACM, p. 67. 25 Forkan, A., Khalil, I., Ibaida, A., Tari, Z., 2015a. Bdcam: Big data for context-aware monitoring-a personalized knowledge discovery framework for assisted healthcare. IEEE transactions on cloud computing. Forkan, A. R. M., Khalil, I., Tari, Z., Foufou, S., Bouras, A., 2015b. A context-aware approach for long-term behavioural change detection and abnormality prediction in ambient assisted living. Pattern Recognition 48 (3), 628–641. Furlaneto, D. C., Oliveira, L. S., Menotti, D., Cavalcanti, G. D., 2017. Bias effect on predicting market trends with emd. Expert Systems with Applications 82, 19 – 26. Gardner, E. S., 2006. Exponential smoothing: The state of the artpart ii. International journal of forecasting 22 (4), 637–666. Handbook, E. S., 2013. Single exponential smoothing. URL http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc431.htm He, Z., Jin, L., 2009. Activity recognition from acceleration data based on discrete consine transform and svm. In: Systems, Man and Cybernetics, 2009. SMC 2009. IEEE International Conference on. IEEE, pp. 5041–5044. Holt, C. C., 2004. Forecasting seasonals and trends by exponentially weighted moving averages. International journal of forecasting 20 (1), 5– 10. Hopkins, M., 1991. Human development revisited: A new undp report. World Development 19 (10), 1469–1473. Kearns, W. D., Fozard, J. L., Nams, V. O., March 2017. Movement path tortuosity in free ambulation: Relationships to age and brain disease. IEEE Journal of Biomedical and Health Informatics 21 (2), 539–548. Kim, T.-Y., Cho, S.-B., 2018. Web traffic anomaly detection using c-lstm neural networks. Expert Systems with Applications 106, 66 – 76. Lara, O. D., Labrador, M. A., 2013. A survey on human activity recogni- tion using wearable sensors. Communications Surveys & Tutorials, IEEE 15 (3), 1192–1209. Liang, Y., Welmer, A.-K., Möller, J., Qiu, C., 2017. Trends in disability of instrumental activities of daily living among older chinese adults, 1997- 2006: population based study. BMJ Open 7 (8). 26 Liu, C.-D., Chung, Y.-N., Chung, P.-C., 2010. An interaction-embedded hmm framework for human behavior understanding: With nursing en- vironments as examples. Information Technology in Biomedicine, IEEE Transactions on 14 (5), 1236–1246. Lotfi, A., Langensiepen, C., Moreno, P. A., Gómez, E. J., Chernbumroong, S., 2017. An ambient assisted living technology platform for informal car- ers of the elderly-icarer. EAI Endorsed Transactions on Pervasive Health and Technology 3 (9), 152393. Mabrouk, A. B., Zagrouba, E., 2018. Abnormal behavior recognition for intelligent video surveillance systems: A review. Expert Systems with Applications 91, 480 – 491. Mahmoud, S., Lotfi, A., Langensiepen, C., 2014. User activities outliers detection; integration of statistical and computational intelligence tech- niques. Computational Intelligence. Mahmoud, S. M., Akhlaghinia, M. J., Lotfi, A., Langensiepen, C., 2011. Trend modelling of elderly lifestyle within an occupancy simulator. In: Computer Modelling and Simulation (UKSim), 2011 UkSim 13th Inter- national Conference on. IEEE, pp. 156–161. Nweke, H. F., Teh, Y. W., Al-garadi, M. A., Alo, U. R., September 2018. Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges. Expert Systems with Applications. Ordóñez, F. J., de Toledo, P., Sanchis, A., Feb 2015. Sensor-based bayesian detection of anomalous living patterns in a home setting. Personal and Ubiquitous Computing 19 (2), 259–270. Peugh, J. L., Enders, C. K., 2004. Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of educational research 74 (4), 525–556. Rafael-Palou, X., Vargiu, E., Miralles, F., 2015. Monitoring people that need assistance through a sensor-based system: Evaluation and first results. In: Proceedings of AI-AM/NetMed 2015 Artificial Intelligence and Assistive Medicine (AIME 2015), CEUR Workshop Proceedings. Vol. 1389. pp. 22– 31. 27 Ros, M., Pegalajar, M., Delgado, M., Vila, A., Anderson, D. T., Keller, J. M., Popescu, M., 2011. Linguistic summarization of long-term trends for understanding change in human behavior. In: Fuzzy Systems (FUZZ), 2011 IEEE International Conference on. IEEE, pp. 2080–2087. Saives, J., Pianon, C., Faraut, G., 2015. Activity discovery and detection of behavioral deviations of an inhabitant from binary sensors. Automation Science and Engineering, IEEE Transactions on 12 (4), 1211–1224. Suryadevara, N., Mukhopadhyay, S. C., Wang, R., Rayudu, R., 2013. Fore- casting the behavior of an elderly using wireless sensors data in a smart home. Engineering Applications of Artificial Intelligence 26 (10), 2641– 2652. Suryadevara, N. K., Mukhopadhyay, S. C., June 2012. Wireless sensor net- work based home monitoring system for wellness determination of elderly. IEEE Sensors Journal 12 (6), 1965–1972. Wallace, B., Knoefel, F., Goubran, R., Masson, P., Baker, A., Allard, B., Guana, V., Stroulia, E., 2017. Detecting cognitive ability changes in pa- tients with moderate dementia using a modified ’whack-a-mole’ game. IEEE Transactions on Instrumentation and Measurement PP (99), 1–14. Wang, X., Xu, Y., Hu, H., Liu, M., Li, G., 2018. Feedback-based metric learning for activity recognition. Expert Systems with Applications. Xie, B., Wu, Q., 2012. Hmm-based tri-training algorithm in human activity recognition with smartphone. In: Cloud Computing and Intelligent Sys- tems (CCIS), 2012 IEEE 2nd International Conference on. Vol. 01. pp. 109–113. Yin, J., Yang, Q., Pan, J. J., 2008. Sensor-based abnormal human-activity detection. Knowledge and Data Engineering, IEEE Transactions on 20 (8), 1082–1090. Yu, R., Wong, M., Chang, B., Lai, X., Lum, C. M., Auyeung, T. W., Lee, J., Tsoi, K., Lee, R., Woo, J., 2016. Trends in activities of daily living disability in a large sample of community-dwelling chinese older adults in hong kong: an age-period-cohort analysis. BMJ Open 6 (12). Zhan, A., Little, M. A., Harris, D. A., Abiola, S. O., Dorsey, E., Saria, S., Terzis, A., 2016. High frequency remote monitoring of parkinson’s disease via smartphone: Platform overview and medication response detection. arXiv preprint arXiv:1601.00960. 28 Zhou, P., Ang, B., Zhou, D., 2010. Weighting and aggregation in composite indicator construction: A multiplicative optimization approach. Social Indicators Research 96 (1), 169–181. 29