key: cord-0058976-7ci4b335
authors: Kašys, Karolis; Dundulis, Aurimas; Vasiljevas, Mindaugas; Maskeliūnas, Rytis; Damaševičius, Robertas
title: BodyLock: Human Identity Recogniser App from Walking Activity Data
date: 2020-08-19
journal: Computational Science and Its Applications - ICCSA 2020
DOI: 10.1007/978-3-030-58802-1_23
sha: db0e55ccd16bb3851c38e1f021daaa7370840d9f
doc_id: 58976
cord_uid: 7ci4b335

A person’s identity can be recognized based on his/her biometric data such as fingerprints, voice or gait. A person can also be recognized from his/her gait, which requires having sensors capable of detecting changes in speed and direction of movement. Such sensors are readily available on almost every smartphone model. We perform user identity verification using his/her walking activity data captured by smartphone sensors. To support identity verification, we have developed a mobile application for Android-based devices, which has achieved 97% accuracy of identity verification using data from acceleration, gravity and gyroscope sensors of a smartphone and a linear Support Vector Machine (SVM) classifier. The developed unobtrusive human walking analyser provides an additional active layer of protection, which may invoke a stronger authentication measure (mandatory locking) if a threat threshold is exceeded.

In the age of digital information, ubiquitous computing, smart devices and smart environments, the security of a person's data, devices and/or identity is a priority. To protect valuable data and to ensure that only authorized person(s) get access to user device(s) such as smartphones or tablets, a variety of techniques can be used. Password-based identification is, perhaps, the most popular now, but it suffers from eavesdropping and users avoiding the use of complex but secure passwords. As a result, the passwords have a high probability of being cracked or exposed. Moreover, passwords can be forgotten or lost. Furthermore, mobile phones secured with only a password are still vulnerable to data theft when left in an unlocked mode [1] . This is especially important for smartphone users, since smartphones are used to store personal data, emails, photos but also more and more often the commercial data, when smartphones are used in the context of BYOD (bring your own device), when users bring their own devices to workplace and utilize them for any work-related tasks such as to access privileged company information and applications [2] . Therefore, it is important to prevent smartphone intrusion and theft using human biometrics [3] . Using biometrics is becoming a common and popular method of identifying individuals. Examples are fingerprints, iris, voice, face form. Each of these biometrical characteristics has its own advantages, but also disadvantages. Voice recognition requires a microphone, and cameras -to identify the shape and landmarks of the face. A highresolution sensor is required to read the iris or fingerprints.

Biometrics is a more powerful authentication factor as compared to usual password authentication, but it still has its own disadvantages, such as non-repeatability (the same biometrical features of a user are not identical all the time) and non-revocability (it is a permanent characteristic of the user and can not be changed) [4] . A person can also be recognized from his walking behaviour characteristics (or so-called gait). Such recognition requires sensors capable of detecting changes in speed and direction of movement, i.e. accelerometer and gyroscope. Such sensors can be found on almost every smartphone model currently produced, as well as on several other devices such as smart watches. A person's identity recognition program would allow a person's authentication to allow the use of various security-enhancing systems without additional user intervention (such as passcode entry). Among the various biometric characteristics, gait has been shown to be robust to direct spoofing attacks [5] .

Currently, the study of human parameters for a variety of applications, including security but also health diagnostics, is a very relevant topic for research. The examples include iris [6] , voice [7] , ear biometrics [8] , electroencephalography (EEG) biometrics [9] . Gait (walking) is one of the most natural human biometrics. However, gait-based identification has not yet been solved yet. All biometric technologies depend on the quality of the input signal: if the received signal is weak or distorted, the task of identification becomes more difficult. The main challenge for the development of modern biometric algorithms is to overcome these complex conditions and to obtain as much reliable evidence as possible for the high level of personal identification. In addition, human gait differs from other human biometric properties, for example, fingerprints or voice, since its properties change over time due to aging. The human gait also depends on his emotional state, feeling well-being, fatigue, injuries, illnesses, environmental conditions, the type of shoes, etc. Therefore, the creation of a system based on the parameters of a human gait requires that the system continuously trace the characteristics of the human walking characteristics and continually persist in adapting to the present state of the subject.

Biometric security technology depends on the input signal quality: if the received signal is weak or distorted noise, identification task becomes more difficult. The main challenge for modern biometric algorithms to create is to overcome these difficult conditions and to extract the maximum amount of reliable evidence to pinpoint accuracy to make personal status and identity recognition possible. Moreover, human gait is different from other human biometric such as fingerprint or voice, because its properties over time can change significantly. Human gait depends on the emotional state of well-being, fatigue, injuries, diseases, environmental conditions, shoe type, etc. Therefore, developing a reliable security system based on human gait parameters requires following the human gait characteristics in real-time and constantly adapting to the changes in human condition.

Recently, the application of human gait parameters for security applications has been a subject of extensive research. For example, Cola et al. [10] used the acceleration collected at the user's wrist to learn the user's typical gait pattern, and then used anomalies detected in a set of acceleration-based features to identify a possible impostor. The method has been evaluated with 15 subjects, reaching an Equal Error Rate (EER) of 2.9%. De Marsico [11] investigated biometric identification by gait recognition via smartphone accelerometer, and achieved the EER of 7.69%. Derawi and Bours [12] collected data from five users, and constructed three different gait templates, where each template related to varying walking speeds. The phone learned the individual features at the various walk speeds, allowing the phone to recognize the correct user using the proposed Cross Dynamic Time Warping (DTW) Metric. The Correct Recognition Rate (CRR) was 89.3% and the false positive probability was as low as 1.4%. Hoang and Choi [13] proposed a gait based biometric cryptosystem to enhance smartphone user security and privacy. Gait signals were acquired by using an accelerometer sensor in the mobile device and error-correcting codes (ECC) were adopted to deal with intra-class variability of gait measurements. The results using gait samples of 34 volunteers are false acceptance rate (FAR) and false rejection rate (FRR) of 3.92% and 11.76%, respectively, in terms of key length of 50 bits. Watanabe and Sara [14] created a similar gait capture app for Android devices. They collected gait data for 15 subjects in 5 walking states. From the 3-axes accelerometer data, we they extracted 52 features with high accuracy. Nguyen et al. [15] used the signal captured in different positions of mobile phone: in front pocket and on the waist. Using Principal Component Analysis (PCA) and Support Vector Machine (SVM) allowed to achieve an EER of 2.45% and accuracy rate of 99.14% regarding the verification and identification process, respectively. Chetty et al. [16] used smartphone inertial sensor data based on information theory with feature ranking and random forest, ensemble learning and lazy learning classifiers. The best results were achieved by random forests classifier with 96.3% accuracy. Ferrero et al. [17] used the data acquired from the 3axis accelerometer embedded in a smartphone, and provided the description of walking features in the time (statistical characteristics, moments, root means square of the walking waveform, autocorrelation, cycle rotation metric) and frequency (coefficients of Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT)) domains. Fernandez-Lopez et al. [18] implemented accelerometer-based gait recognition (ABGR) using custom gate cycle detection algorithm and achieved and an Equal Error Rate (EER) of 16.38% to 29.07%. Lai et al. [19] extracted frequency features (coefficients of FFT) from gait data obtained using the acceleration sensor and used Weighted SVM to recognize users, reaching 3.5% EER. Yuan and Zhang [20] used periodogram based gait categorization to identify patterns in the walking periodicity, and convolutional neural network (CNN) for based gait-based identification and achieving an accuracy of over 87%. Anusha and Jaidhar [21] used histogram of oriented gradients (HOG), followed by sum variance Haralick texture descriptor calculated from gradient magnitude image to derive low level features which are used for gait recognition. He et al. [22] suggested multi-task generative adversarial networks for learning view-specific gait features. Zhang et al. [23] used Convolutional Neural Network (CNN) for soft biometrics such as age and gender recognition. A review on gait-based identity recognition methods has been presented by Liu et al. [24] .

Our aim is to analyse complex human gait characteristics for monitoring the state and identity of a human (smartphone user), to propose methods for selecting reliable gait features, and to develop and approve the passive user state monitoring system, running on a portable mobile device (smartphone) and allows the device to distinguish the changes in the state (condition) of the smartphone user (e.g., change of owner in case of attempted theft) according to his/her gait parameters.

Such system would be most relevant to smart device holders who want to prevent unauthorized access to smartphone data without using too cumbersome methods of security. Our approach follows the multi-layered model of security [25] , which calls for different layers of security to address different security challenges. While each layer of security alone does not provide the required defence against attacks, their entirety ensures a higher level of security. Therefore, an unobtrusive human walking analyser could provide an additional active layer of protection, which could invoke a stronger authentication measure (mandatory locking) if a threat threshold is exceeded.

We propose and validate a passive user authentication system that operates on a portable mobile device (smartphone) and allows the device owner to be distinguished from an outsider by its track parameters. Upon detecting a changed owner, the system will take appropriate security measures to ensure the security of the data stored on the device (e.g., blocking access to data, informing the owner about the location of the device, etc.) without waiting for active intruder attempts to connect to the device.

In this paper, the following methodology is applied: data acquisition from smartphone sensors; data preprocessing and denoising; feature generation, feature ranking and dimensionality reduction, activity detection and gait model construction. The identification of a subject is performed by comparing the current gait characteristics of a subject with his gait model learned from previous observations during the training stage. The methodology is visualized in Fig. 1 . 

During data acquisition, the smartphone is kept in a static position -the phone can be located either in the trouser pocket or in the jacket pocket (chest level), which however affects the collected data, even if the same type of activity is performed. In order to avoid this problem, the smartphone needs to be kept in the same constant position with regards to the human body. The raw data provided by the smart phone is accelerometer and gyro data, since these sensors are equipped with most smart devices, but other sensors, for example, will also be taken, if possible, e.g., the magnetic field sensor data, if it helps to more accurately identify identity, but more data does not always mean more accurate results because their information is not relevant.

We have selected gait data in 3 states of walking: State 1 (walking forwards): walking on an even surface and holding the smartphone. State 2 (walking stairs up): going stairs up and holding the smartphone in the pocket. State 3 (walking stairs down): going downstairs and holding the smartphone in the pocket.

The following data set is logged by the developed Android application: Logged time (ms); Raw 3 axes of acceleration data (m/s 2 ); Raw 3 axes of gravity data (m/s 2 ); Raw 3 axes of the gyroscope data (rad/s).

When the data is collected, the primary data processing is carried out -the data is normalized. Removing noise can help increase the accuracy of the results by eliminating unnecessary noisy data, bat also the important information characterising differences between individuals may be smoothed, therefore, the denoising has not been used. A common problem was that the sensor data returned by the device is not acquired at regular time intervals. In that case, a linear time interpolation was used to ensure the equal time interval between two data records.

Another problem was the removal of spikes from the data. The cause for these spikes remained unclear and may be related to phone's hardware behaviour. We have identified the spikes as outliers, which are more than 4 standard deviations (sigmas) away from the signal's mean value, thus guaranteeing that less than 0.1% of data is removed and replaced by the mean of the adjacent values.

In many cases (such as calculation of a mean or a moving average), calculation requires using sliding windows over sensor data. For sliding window, following the suggestion of Primo et al. [26] , we use the sliding window of 100 data points with overlap of 50%. As features, we use a set of statistical and other signal features described in [27, 28] . The full list involves 102 different features. Some of the sample features are presented in Table 1 .

For classification we use Support Vector Machine (SVM) [29] with a linear kernel. 80% of data is retained from training and 20% of data is used for testing. The procedure is repeated 20 times using Hold-Out cross-validation, and the results are evaluated using the accuracy, Kappa and F-score metrics. Accuracy is the number of correct guesses made divided by the total number of guesses made. Kappa value compares an observed accuracy with an expected accuracy (random guessing). F-score combines Table 1 . Representative features of smartphone sensor signals precision and recall, and it characterises the precision (how many samples it classifies correctly) and robustness (it does not miss many positive samples) of the classifier.

The human identity recognizer has been implemented as Android app, called Body-Lock. The app has two parts (see Fig. 2 ): the device part is responsible for collection of sensor data, generation of features, sending of data to server, and user interface. The server part is responsible for storage of data and classification of walking features. The application itself works silently in the background mode.

For our experiments we used walking trajectories of 13 different subjects collected in the office environment (in the SANTAKA valley building of Kaunas University of Technology), which included three different types of trajectories: walking forward, walking upstairs and walking downstairs. All individuals were health without any foot problems. All subjects took the same walking trajectory using a flat-surface corridor about 50 m long, as well as up-stair and down-stairs (see Fig. 3 ).

The even floor of the corridor allowed to capture the usual gait characteristics of the subjects without any obstacles or unevenness of the floor. The data capture sessions were organized during the first half of the day to avoid the effect of tiredness, exhaustion at the end of a business day, or office fatigue due to sitting all day.

The results of feature ranking are presented in Fig. 4 . The most important features refer to the 95th and 99th percentiles of acceleration in the X and Z axes of the device, which means that extreme events such as the speed of sudden movements help to recognize people and differentiate one subject from another.

An example of the distribution of sample data is presented in Fig. 5 . Here the 2D feature space is formed by only two features (axz_p95, the 95th percentile of acceleration sensor value and in the X-Z axis, and ry_75, the 75th percentile of gyroscope sensor value in the Z axis). Even in this low-dimensional feature space, the gait characteristics of subjects can be clearly separated. Figure 6 shows the confusion matrix of the classification results.

Finally, Fig. 7 shows the classification results. We have achieved 97% accuracy of correct identification, while the Copen's kappa was 0.944, and F-score was 94%. 

The limitations of the presented approach towards user identification are as follows:

1) Contamination of data with noise, which may lead to incorrect identification of user.

Some lower-end smartphone models have low-quality sensors, which return unreliable data, which can be observed practically by high variability of a sensor signal even if the phone is laying in the static position without any movement. 2) Significant intra-class variability due to changing condition of the subject (e.g., illness, fatigue or stress [30, 31] may influence his/her gait), different shoes and clothes, different environment and flooring conditions (e.g., field environment can vary from solid pavement to grass, sand or snow). 3) Discriminabilitythe interclass similarities may be large if the sufficiently large number of subjects is explored, thus restricting the discriminability provided by the gait analyser.

One of the key challenges is determining how to achieve required functionality and performance without jeopardizing the user experience and power consumption with the overhead of additional computational burden to the mobile phone. The two-tiered (client-server) approach used in the design of the BodyLock app allowed to move computationally costly operations to the server, thus reducing the performance overhead. However, the data still must be secured during transfer and storage at the cloud server to avoid stealing and impersonation attempts. Gait verification might not reach the same level of reliability as fingerprint or iris recognition. However, it still could be used as a method of continuous (active) authentication for accessing less sensitive content on a smartphone (such as games) or be used in concert with other biometrics (e.g. iris recognition) to provide a more reliable user authentication method.

When combined with other signals such as GPS or WiFi from low energy Bluetooth beacons, it could be used for providing smart localisation [32, 33] services as well.

Biometric authentication is a good solution for smartphone users since it relies on the uniqueness of certain physical traits in humans. By continuously and unobtrusively recognizing the phone's owner, the human walking analyser has potential to improve user authentication on the go. Here we described the development of a mobile app for Android-based smartphones, which performs user identity verification using his/her walking activity data. We have achieved 97% accuracy of identity verification using features generated from data acquired from the acceleration, gravity and gyroscope sensors of a smartphone and a linear Support Vector Machine (SVM) classifier. The results achieved are in line with the results achieved by other authors working in the same area of research. As gait-based authentication technologies move towards being deployed as an extra security layer in portable mobile devices, our work represents a step in this direction.

Continuous authentication of mobile user: fusion of face image and inertial measurement unit data

BYOD: security and privacy considerations

Preventing cell phone intrusion and theft using biometrics

PrivBioMTAuth: privacy preserving biometrics-based and user centric protocol for user authentication from mobile phones

Biometrics systems under spoofing attack: an evaluation methodology and lessons learned

Voice biometrics security: extrapolating false alarm rate via hierarchical bayesian modeling of speaker verification scores

Cross-spectral iris recognition using CNN and supervised discrete hashing. Pattern Recogn

Secure ear biometrics using circular kernel principal component analysis, Chebyshev transform hashing and Bose-Chaudhuri-Hocquenghem error-correcting codes. Signal Image Video Process

Combining cryptography with EEG biometrics

Gait-based authentication using a wristworn device

Biometric walk recognizer

Gait and activity recognition using commercial phones

Secure and privacy enhanced gait authentication on smart phone

Toward an immunity-based gait recognition on smart phone: a study of feature selection and walking state classification

A generalized authentication scheme for mobile phones using gait signals

Smart phone based data mining for human activity recognition

On gait recognition with smartphone accelerometer

Gait recognition using smartphone

Using weighted SVM for identifying user from gait with smart phone

Gait classification and identity authentication using CNN

Human gait recognition based on histogram of oriented gradients and Haralick texture descriptor

Multi-task GANs for view-specific feature learning in gait recognition

A comprehensive study on gait biometrics using a joint CNN-based method. Pattern Recogn

A review of gait behavior recognition methods based on wearable devices

Why multi-layered security is still the best defence

Context-Aware active authentication using smartphone accelerometer measurements

Human activity recognition in AAL environments using random projections

Smartphone user identity verification using gait characteristics

Support vector networks

Gender, age, colour, position and stress: how they influence attention at workplace?

Recognition of human daytime fatigue using keystroke data

Fuzzy logic type-2 based wireless indoor localization system for navigation of visually impaired people in buildings

Smartphone based intelligent indoor positioning using fuzzy logic