key: cord-0592659-mfwr4mki
authors: Ruiz-Garcia, Juan Carlos; Tolosana, Ruben; Vera-Rodriguez, Ruben; Herreros-Rodriguez, Jaime
title: ChildCI Framework: Analysis of Motor and Cognitive Development in Children-Computer Interaction for Age Detection
date: 2022-04-08
journal: nan
DOI: nan
sha: b5d1c3877537c57fd16bef754d144d9255ca01a6
doc_id: 592659
cord_uid: mfwr4mki

This article presents a comprehensive analysis of the different tests proposed in the recent ChildCI framework, proving its potential for generating a better understanding of children's neuromotor and cognitive development along time, as well as their possible application in other research areas such as e-Health and e-Learning. In particular, we propose a set of over 100 global features related to motor and cognitive aspects of the children interaction with mobile devices, some of them collected and adapted from the literature. Furthermore, we analyse the robustness and discriminative power of the proposed feature set including experimental results for the task of children age group detection based on their motor and cognitive behaviors. Two different scenarios are considered in this study: i) single-test scenario, and ii) multiple-test scenario. Results over 93% accuracy are achieved using the publicly available ChildCIdb_v1 database (over 400 children from 18 months to 8 years old), proving the high correlation of children's age with the way they interact with mobile devices.

T ECHNOLOGY has become a very important aspect of our lives in recent decades. In particular, mobile devices play an essential role in our daily basis (e.g., work, relationships, communications, business, etc.). This also affects children, who are exposed to these devices from an early age [1] . Recent studies corroborate this fact [2] , [3] . For example, Kabali et al. conducted a study in [2] with 350 children aged 6 months to 4 years concluding that 96.6% of children use mobile devices, and most started using them before the age of 1 year. In addition, around 75% of children by the age of 4 years already have their own mobile device. Similar conclusions were obtained in [3] , where 422 parents of children aged from birth to 5 years were interviewed and 75.6% of them indicated that their children had already used mobile devices at that age. Moreover, and due to the global pandemic of COVID-19 since 2020, the use of mobile devices was rapidly increased as preschools, kindergartens, and schools were closed down for several months in most countries around the world. As a result, traditional face-to-J.C. Ruiz-Garcia, R. Tolosana and R. Vera-Rodriguez are with the Biometrics and Data Pattern Analytics -BiDA Lab, Escuela Politecnica Superior, Universidad Autonoma de Madrid, 28049 Madrid, Spain (e-mail: juanc.ruiz@uam.es, ruben.tolosana@uam.es; ruben.vera@uam.es).

J. Herreros-Rodriguez is with the Hospital Universitario Infanta Leonor, 28031 Madrid, Spain (e-mail: hrinvest@hotmail.com). 1 https://github.com/BiDAlab/ChildCIdb v1 face education was replaced to virtual learning environments (e-Learning) [4] . From such an early age and throughout their development, children experience different evolutionary stages in which their physiological and cognitive capacities improve through continuous interaction with the world they live. Jean Piaget was one of the leaders on the study of children's motor and cognitive development and, according to his theory [5] , children pass in a fixed sequence through four universal stages of development: i) Sensorimotor (from birth to 2 years), children focus on acquiring knowledge by using their senses to touch, smell, see, taste, and hear the objects around them; ii) Preoperational (2-7 years), their language and thinking are improving, as their motor skills. In addition, at this age children are egocentric in their thinking and it is still difficult for them to empathise with other people's feelings; iii) Concrete Operational (7-11 years) , begin to use more logical thinking to solve problems, starting to improve their empathic abilities significantly; and iv) Formal Operational (11 years to adulthood), gain the ability to use abstract cognitive functions to think more about moral, philosophical, ethical, social, and political issues.

Despite the high technological evolution and the application of it in children scenarios, the assessment of the correct motor and cognitive development of the children is still evaluated using traditional approaches that are manual, time-consuming, and provide qualitative results that are difficult to interpret. This is one of the main motivations of our ChildCI framework [6] : the proposal of automatic methods that quantify the motor and cognitive development of the children through the interaction with mobile devices, using both the stylus and the finger/touch. As a first step to reach that future goal, in this article we first evaluate the discriminative power of the tests proposed in the ChildCI framework, trying to shed some light on the following questions: Is there any relationship between children's chronological age and their motor and cognitive development when interacting with the tests proposed in ChildCI framework? Is there any relationship between the age, the type of test, and writing input (stylus/finger) considered? The answers to these questions could provide very interesting insights for the research community and the proposal of automatic and user-friendly methods to better quantify the development of the children.

The main contributions of the present work are: stylus), as well as the analysis of the motor and cognitive development. • Validate the potential of the different tests included in the ChildCI framework in terms of the motor and cognitive development of the children. We propose a feature set with over 100 global features based on cognitive and motor aspects of children while interacting with mobile devices, some of them collected and adapted from the literature. • Analyse whether there is any relationship between the chronological age of the children and their motor and cognitive development while interacting with the tests included in ChildCI. To shed some light on this, experiments are carried out for the task of children age group detection based on their motor and cognitive behaviors. Different acquisition inputs are considered in the analysis, i.e., stylus and finger. In addition, single-and multipletest experiments are studied. Experiments are performed including several automatic feature selection techniques and machine learning approaches. The remainder of the article is organised as follows. Sec. II summarises previous studies on children's touch and stylus interaction, as well as some sets of features successfully used in literature for different lines of work. In the following, we describe in Sec. III the ChildCI framework and database used in the experimental work carried out. Sec. IV describes the proposed feature set based on motor and cognitive aspects of the children. Sec. V describes the experimental protocol and the results achieved for the task of children age group detection. Finally, Sec. VI draws the final conclusions and future work.

Children's interaction with mobile devices has been evaluated and analysed by multiple research studies in recent decades. Focusing on the first stage of Piaget's theory (Sensoriomotor, 0-2 years) there is not much work on the interaction analysis of children under the age of 2 with touchscreen devices, mainly due to the difficulty of capturing data with children at that age. If we focus on touch mobile interactions, Morante et al. presented a very interesting article in this line in [7] . In that work, the authors analysed the behaviors of children aged from 0 to 2 years. They concluded that children at 1 year of age can use the tap gesture intentionally to perform actions and at 2 years they are already able to understand some gestures such as tap and drag to navigate through apps. In [8] the authors assessed the mobile interaction of children aged 1 to 2 years through the analysis of videos from YouTube while they were recorded interacting with mobile devices. They concluded that children under 17 months tend to use both hands for interaction, an aspect that decreases sharply with age, leading to single-hand use.

Looking at the second stage of Piaget's theory (Preoperational, 2-7 years), several studies have analysed the children's interaction with mobile devices, in contrast with the first stage. For example, in the work presented by Vatavu et al. [9] a database of 89 children aged 3 to 6 years and 30 young adults, was captured. A similar research line was studied by Nacher et al. [10] where the authors proposed a set of 8 different tests on a mobile device in order to measure the ability of children aged 2-3 years to perform touch gestures. The results showed that simple gestures such as tap, drag, and one-finger rotation can be performed by children in most cases. However, performing more complex gestures such as double tap, scale down, long press, and two-finger rotation is strongly influenced by the age of the child, with the older children's group performing them easier and quicker than the younger ones. Chen et al. obtained similar conclusions in [11] . The authors found different children's interaction behaviors with mobile devices by analysing the correlation between factors such as their age, grade level, motor and cognitive development, and how they performed touchscreen interaction tasks (target acquisition and gesture detection).

Interaction with mobile devices is not only done through the use of the finger, but also through an stylus [12] , [13] . In general, writing and drawing require greater motor and cognitive development than simple touch gestures. Children start scribbling around the age of 2 years [14] . In [15] , Rémi et al. studied the way children aged 3-6 years perform scribbling activities, concluding that there are significant differences in motor skills depending on the age. Another interesting work in this line is presented in [16] considering children 6-7 years old. The authors analysed the correlation between the performance of polygonal shape drawing and levels in handwriting performance. The results proved that there are different children's drawing strategies that differ in their writing performance.

The correct analysis and quantification of the motor and cognitive development of the children starts with a good definition of robust and discriminative features for the task. Previous studies in the field of Human-Computer Interaction (HCI) could provide interesting features that, after adapting them, could be very useful to analyse motor and cognitive aspects of the children.

For example, in [17] Ishii et al. developed a simple quantitative method to diagnose tremor using hand-drawn spirals and artificial intelligence. The Archimedes spiral is the reference test for the clinical diagnosis of diseases such as essential tremor or Parkinson. In that study, patients used a stylus to trace a spiral on a printed reference spiral and, by comparing the lengths of the reference spiral and the traced one, the total area of deviation between both was calculated, achieving results with success rates up to 79% in detecting people with essential tremor. In a similar work [18] , Solé-Casals et al. proposed a new set of 34 features using only the x and y coordinate points of the strokes made by patients as they traced the Archimedes spiral using a pen stylus on a graphics tablet [18] . In addition to tremor assessment, in [19] the authors proposed a test paradigms on a graphic tablet using different parameters to automatically quantify tremor characteristics and severity in real-time by extracting three parameters: i) the mean radial difference per radian, ii) the mean radial difference per second, and iii) the area under the curve of the frequency spectrum for the velocity. Tremor is directly related to fine motor actions such as pinching, writing, drawing and other small movements. Therefore, it is interesting to analyse the level of tremor in children as they grow up, because it will be higher or lower depending on their motor skills development.

An interesting article in this line was the work presented by Xu et al. in [20] where a variety of touch gestures were used to enhance the security and privacy of users based on the touch operations performed on their smartphone screens. Through the analysis of touch gestures such as swipe, drag and drop, tap or pinch, among others, the authors proposed a total of 132 features that identify the way each user interacts with the mobile device, achieving an Equal Error Rate (EER) of around 10% for all types of gestures and 1% for the swipe operation where the Largest Deviation Point (LDP) was considered. Another interesting study on this was carried out by Vatavu et al. in [21] , where through the features extracted using the touch coordinates x and y, it was possible to detect the age group of the users reaching up to 86.5% accuracy. A similar study was conducted in [22] , where the authors proposed a novel approach to protect society from online threats through the interaction of 147 participants with six micro-games in an Android app. A dataset of more than 9,000 touch gestures was created, characterising how participants interact with the device and achieving results up to 88% accuracy detecting impostors.

As we presented in [6] , ChildCI is an on-going project mainly intended to improve the understanding of children's motor and cognitive development along time through the interaction with mobile devices. Stylus and finger are used as acquisition tools, capturing data and storing it in our novel ChildCI database (ChildCIdb v1 2 ). This is a database collected in collaboration with the school GSD Las Suertes in Madrid (Spain), which is planned to be extended yearly, allowing for interesting longitudinal studies. To the best of our knowledge, ChildCIdb v1 is the largest and diverse publicly available Child-Computer Interaction (CCI) dataset to date in the topic of the interaction of children with mobile devices. It is composed of 438 children in the ages from 18 months to 8 years, grouped in 8 different educational levels according to the Spanish education system. In addition, during the capture process other interesting information from the children is also collected: i) previous experience using mobile devices, ii) grades at the school, iii) attention-deficit/hyperactivity disorder (ADHD), iv) birthday date, v) prematurity (under 37 weeks gestation). All this additional metadata makes the project more powerful and interesting, allowing for multiple lines of future research. This dataset is considered in the experimental framework of this study.

In particular, 6 different tests are considered in ChildCI, grouped in 2 main blocks: i) touch, and ii) stylus. Each one has a maximum amount of time to be performed and requires 2 https://github.com/BiDAlab/ChildCIdb v1 different levels of neuromotor and cognitive skills to complete correctly. We briefly present each of the tests: • Stylus Block -Test 5 -Spiral Test: a black spiral appears on the screen. The children, using the pen stylus, must draw along the spiral from the inner to the outer part, always trying to keep inside the black line that forms the spiral. The maximum time for this test is 30 seconds. -Test 6 -Drawing Test: the outline of a tree appears on the screen and the children must to colour it in as best as they can. The maximum time for this test is 2 minutes. Examples of the different tests can be seen in Fig. 1 , grouped by age. We include red and green marks along the tests to provide a better comprehension of the children interaction along the different age groups.

In order to shed some light on the questions considered in this study, i.e., 1) validate the discriminative power of the different tests included in ChildCI, and 2) analyse whether there is any relationship between the chronological age of the children and their motor and cognitive development, the experimental framework of this study is carried out for the task of automatic children age group detection based on their motor and cognitive behaviors. Sec. IV-A describes the feature set proposed for each of the tests considered in ChildCIdb v1. Sec. IV-B summarises the feature selection techniques considered. Finally, Sec. IV-C indicates the classification algorithms analysed. During the data collection process, each child performs the set of tests shown in Fig 1. This section presents the proposed feature sets for each test. In total, 111 global features are extracted referring to different types of skills.

• Test 1 -Tap and Reaction Time (Table I) : a set of 5 specific features is proposed. • Test 2 -Drag and Drop (Table II): 28 features are proposed. In particular, 8 of them are proposed in this work and 20 are based on study [20] . • Test 3 and 4 -Zoom In/Out (Table III) : a set of 20 features is proposed, 8 of them proposed in this work, 4 based on the study conducted in [22] and the remaining 8 inspired from [20] . • Test 5 -Spiral Test (Table IV) : for this test, 24 features are proposed. In particular, 3 of them are based on the study conducted in [17] , 14 are inspired from [18] , 2 are based on the results obtained by [19] and the remaining 5 are proposed in this work. • Test 6 -Drawing Test: we consider the 34 features originally presented in [6] . The features proposed in this work cover aspects such as: i) the reaction time to touch the screen, ii) the amount of time touching the target, iii) the number of fingers used, and iv) whether the child finishes the test (the carrot ends up on the rabbit, 4 moles are touched, the rabbit is properly scaled, etc.), among others.

In addition to the 111 global features presented above, 114 features based on preliminary studies in the field of HCI and related with Time, Kinematic, Direction, Geometry, and Pressure information are also considered [23] , [24] , forming a final set of 225 features in total.

The following feature selection techniques are used to choose the most discriminative features from the total set originally extracted.

• Sequential Forward Floating Search (SFFS): this is a feature selection algorithm which searches for the bestcorrelated subset of features using a specific optimisation criteria. On the one hand, the solution offered by this algorithm is suboptimal because it does not take into account all possible combinations, but on the other hand it does consider correlations between features, achieving high-accuracy results [25] . The implementation considered in this study has been provided by the MLxtend library 3 . • Genetic Algorithm (GA): this is a metaheuristic algorithm based on Charles Darwin's theory of evolution. It is presented in our previous work [6] and is mainly inspired on the natural selection process of evolution, where over generations and through the use of operators such as mutation, crossover and selection, a positive evolution towards better solutions occurs. Our public version of this 3 http://rasbt.github.io/mlxtend/ library developed in Python can be found on GitHub 4 . In our experiments, we have considered an initial population = 200, a random generations = 100, a crossover rate = 0.6, and a mutation rate = 0.05.

All classifiers are publicly available on Scikit-Learn 5 .

• Support Vector Machines (SVM): this algorithm builds a hyperplane or set of hyperplanes in a high-or infinitedimensional space that differentiates the classes as well as possible. In our case, the regularization parameter is 0.1, the kernel type is "polynomial" with 3 degrees and the coefficient is "scaled". • Random Forest (RF): this is an ensemble method consisting of a defined number of small decision trees, called estimators. A combination of the estimator's decisions is produced to get a more accurate prediction. In our experiments, the number of estimators is 10, the function to measure the quality of a split is "gini" and the maximum depth of the tree is 75.

The experimental protocol considered in this work is designed with the aim of age group detection based on the children interaction behavior. The following 3 different age groups are considered: Group 1 (children aged 1 to 3 years), Group 2 (children aged 3 to 6 years), and finally Group 3 (children aged 6 to 8 years). ChildCIdb v1 is divided into 2 data subsets: development (80%) and evaluation (20%). The development dataset is used for the training of the age group detection systems whereas the evaluation dataset is used to test the performance of the trained systems, excluding the children considered in the development dataset. In addition, and only during the development stage, a data augmentation technique is used as the data available in Groups 1 and 3 are smaller than in Group 2. This technique is called SMOTE and is publicly available in the Imbalanced-Learn toolbox 6 . To provide a better analysis of the results, k-fold cross-validation with k=5 is used, showing the final evaluation results of the 5 fold cross-validation. All experiments are run on a machine with an Intel i7-9700 processor and 32GB of RAM.

This section analyses the performance of the methods presented above to the children age group detection task based on motor and cognitive behaviors when interacting with mobile devices. The analysis is carried out in two different stages: i) a first test-by-test analysis is performed, and ii) then a combination of tests is conducted to analyse the potential of the ChildCI tests as a whole. The results obtained are measured in terms of Accuracy (%). Average [19] . R n refers to the radial coordinates, θ n to the angular coordinates, and t n to sample-to-sample times in seconds. 

Fractal Dimension (FD) [Higuchi's Algorithm with m=5] 14 Maximum Fractal Length (MFL) log( 19 Mean of Radial Difference Per Radian

Mean of Radial Difference Per Second

Global Maximum Quartile of R n 24 Global Minimum Quartile of R n 1) Single-Test Scenario: Table V shows the results obtained in each test using the different classifiers and feature selectors considered. We first analyse the results test by test using the best results achieved for each one. As can be seen, Test 6 obtains the best accuracy (90.45%), while the worst result is for Test 3 (81.33%). Children can draw the tree in many different ways. Therefore, in Test 6 they have more freedom to interact with the device and, as a result, the variation between groups can be better observed, leading to high accuracy in the age group detection task. Analysing the results according to the classifier studied, SVM always achieves better results than RF, regardless of the feature selector used. In particular, SVM achieves an average accuracy of over 86%, while for RF the average is less than 83%. It is also interesting to analyse the results by the type of feature selectors considered, in most cases, SFFS provides the best results, achieving in Test 1, Test 2, Test 5 and Test 6 rates above 87% accuracy. Nevertheless, GA performs better for Tests 3 and 4, reaching 81.33% and 82.45% accuracy. This proves the potential of our proposed feature selector algorithm that is publicly available 7 .

In addition, we analyse the results according to the writing input used (stylus/finger). Always looking at the best tests for each input method, the results obtained in terms of accuracy are similar, indicating that the input method used is not really relevant for the age group detection task. Finally, for completeness, we use a popular visualization technique, called Uniform Manifold Approximation and Projection (UMAP) [26] , Fig. 2 shows the children age groups formed based on the motor and cognitive features proposed in this study for each of the tests of ChildCI (best configurations of Table V) . Therefore, those children belonging to the same age group and whose motor and cognitive interactions on each test are similar should appear in the same group and placed contiguously to each other. As can be seen in most cases, a point cloud with 3 distinct groups can be seen, indicating a high correlation between the age of the children and the way they interact with mobile devices. However, some children are in a different point cloud to their own. For example, in the results of Test 6 in Fig. 2 , we can see that there are children from Group 1 who are in the point cloud of Group 2. These particular cases could be an indicator that these children have more/less advanced motor and cognitive aspects than their age group. In view of the results obtained, we can shed some light on the key questions analysed in this study. First, the validation of the discriminative power of the different tests included in the ChildCI framework. The results achieved in Table V there is any relationship between the chronological age of the children and their motor and cognitive development.

The point clouds shown in Fig. 2 indicate that there seems to be a good relationship between the motor and cognitive features proposed in this study and the chronological age of the children. 2) Multiple-Test Scenario: In ChildCI framework there are 6 tests in total, so we have 6 pre-trained machine learning models, one per test (previous experiment). This experiment analyses the potential of considering all the ChildCI tests together. We consider combinatorial operations, and all possible test combinations, the number of combinations is indicated by the following equation:

The number of total observations is represented by n whereas x refers to the number of selected elements. We combine the individual tests into groups of 2, 3, 4, 5 and 6 tests (all tests together). In total there are 57 possible combinations: i) 15 combinations in groups of 2 tests, ii) 20 combinations in groups of 3 tests, iii) 15 combinations in groups of 4 tests, iv) 6 combinations in groups of 5 tests, and v) 1 combination of all tests together.

In particular, each of the machine learning models trained during the development stage generates 3 probabilities (one for each children age group, values between 0 and 1) whose sum cannot exceed 1. In our experiments, the highest value determines the group to which the child belongs according to the classifier. Then, to combine the tests, an average of the probabilities generated for each set of grouped tests is calculated, with the highest value again determining the age group associated with the child. 

(90.45% in Test 6). Therefore, we can ensure that a combination of the best tests is a good practice to obtain better results for the children age group detection task.

The proposal of automatic methods that quantify the motor and cognitive development of the children through the interaction with mobile devices is one of the main motivations of our ChildCI framework. As a first step to reach that future goal, this study proposes a comprehensive analysis evaluating the discriminative power of the tests presented in our ChildCI framework, and an analysis of whether there is any relationship between the chronological age of the children and their motor and cognitive development.

For each of the tests considered, a robust set of features representing cognitive and motor aspects of children during interaction with mobile devices is presented. The experimental framework of this study is carried out for the automatic children age group detection task based on similar motor and cognitive behaviors.

The results achieved shed some light on the questions and contributions analysed in this study. Indeed, there is a relationship between children's chronological, motor and cognitive age and the type of test they perform when interacting with mobile devices. Fig. 2 shows a high correlation between the age of the children and the way they interact with the devices, denoting that how children perform the tests can give a rough indication of their chronological age group. Nevertheless, 100% accuracy is not achieved in the age group detection task because children's evolution is a maturation process. It means that children of the same age group may have more/less advanced motor and cognitive aspects depending on their development.

In addition, the potential and discriminative power of the tests included in the ChildCI framework is proved. The results achieved in Table V demonstrate that ChildCI tests are able to measure different children motor and cognitive features for the different ages. That indicates both the correct design of the tests, discussed and approved by specialists such as neurologists, child psychologists and educators, and their inherent applicability to other research problems around e-Learning and e-Health.

Future works will be oriented towards: i) relate children's interaction information with mobile devices to the other metadata stored in ChildCIdb (school grades, ADHD, previous experience using mobile devices, prematurity, etc.), ii) presentation of new versions of the database analysing longitudinally the evolution of children when performing the different ChildCIdb tests, iii) take advantage of ChildCIdb's potential in other e-Health and e-Learning research areas and problems.

Research in Child-Computer Interaction: Provocations and Envisioning Future Directions

Exposure and Use of Mobile Media Devices by Young Children

Exposure to and Use of Mobile Devices in Children Aged 1-60 Months

Child-Computer Interaction in Times of a Pandemic

The Psychology of the Child. Basic books

Child-Computer Interaction with Mobile Devices: Recent Works, New Dataset, and Age Detection

Children's Evolving Capabilities in Their Interaction with Touchable Devices from Birth to 2 Years Old

Look, My Baby Is Using an IPad! An Analysis of YouTube Videos of Infants and Toddlers Using Tablets

Touch Interaction for Children Aged 3 to 6 Years: Experimental Findings and Relationship to Motor Skills

Multi-Touch Gestures for Pre-Kindergarten Children

Examining the Link between Children's Cognitive Development and Touchscreen Interaction Patterns

SVC-onGoing: Signature Verification Competition

DeepWriteSYN: On-Line Handwriting Synthesis via Deep Short-Term Representations

The Role of iPads in Pre-School Children's Mark Making Development

Exploring the Kinematic Dimensions of Kindergarten Children's Scribbles

Analyses of Pupils' Polygonal Shape Drawing Strategy with Respect to Handwriting Performance

Spiral drawing: Quantitative analysis and artificial-intelligence-based diagnosis using a smartphone

Discrete Cosine Transform for the Analysis of Essential Tremor

A Digital Assessment System for Evaluating Kinetic Tremor in Essential Tremor and Parkinson's Disease

Towards Continuous and Passive Authentication via Touch Biometrics: An Experimental Study on Smartphones

Child or Adult? Inferring Smartphone Users' Age Group from Touch Measurements Alone

Techno-Regulation and Intelligent Safeguards

Feature-Based Dynamic Signature Verification Under Forensic Scenarios

Towards Mobile Authentication Using Dynamic Signature Verification: Useful Features and Performance Evaluation

Preprocessing and feature selection for improved sensor interoperability in online biometric signature verification

UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction