key: cord-0656133-228oyhvh authors: Chandaliya, Praveen Kumar; Akhtar, Zahid; Nain, Neeta title: Longitudinal Analysis of Mask and No-Mask on Child Face Recognition date: 2021-10-29 journal: nan DOI: nan sha: 02ec86f0140b5375b799f8ace8b419787bbc43bf doc_id: 656133 cord_uid: 228oyhvh Face is one of the most widely employed traits for person recognition, even in many large-scale applications. Despite technological advancements in face recognition systems, they still face obstacles caused by pose, expression, occlusion, and aging variations. Owing to the COVID-19 pandemic, contactless identity verification has become exceedingly vital. Recently, few studies have been conducted on the effect of face mask on adult face recognition systems (FRS). However, the impact of aging with face mask on child subject recognition has not been adequately explored. Thus, the main objective of this study is analyzing the child longitudinal impact together with face mask and other covariates on FRS. Specifically, we performed a comparative investigation of three top performing publicly available face matchers and a post-COVID-19 commercial-off-the-shelf (COTS) system under child cross-age verification and identification settings using our generated synthetic mask and no-mask samples. Furthermore, we investigated the longitudinal consequence of eyeglasses with mask and no-mask. The study exploited no-mask longitudinal child face dataset (i.e., extended Indian Child Longitudinal Face Dataset) that contains 26,258 face images of 7,473 subjects in the age group of [2, 18] over an average time span of 3.35 years. Due to the combined effects of face mask and face aging, the FaceNet, PFE, ArcFace, and COTS face verification system accuracies decrease approximately 25%, 22%, 18%, 12%, respectively. Nowadays, face recognition systems under face mask is getting much more momentum. For instance, the work in [1, 2] evaluated verification performance on both real and synthetic masks. It was later extended in [3] to analyze the human experts and automatic recognition systems on unmasked, real masked, and synthetic mask on adult dataset. Besides face mask, face aging is also a vital co-variate that negatively affect automated face recognition systems, especially for child subjects. For example, Deb et al. [4] fused COTS and FaceNet [5] scores, and attained 80.56% and 53.33% verification accuracy, respectively, for a time lapse of 1 and 3 years between enrollment and probe images for subjects of age [2−18] years old. The work in [6] investigated five top performing COTS matchers, two government matchers and one open-source face recognition system on Wild Child Celebrity and LFW [7] datasets, and obtained maximum of 78.20% and 85.2%, respectively, verification and Rank-1 identification accuracy. The study also showed each algorithm's negative bias towards children compared to adult face samples. There exist several mask face datasets [8, 1, 9, 10] but they mainly contain adult faces and caucasian and north Asian demography. To the best of our knowledge, no study has explored the combined effect of aging and mask on face recognition when the subjects are children. Also, no prior works explicitly evaluated the longitudinal identification performance when probe samples are with mask and gallery samples are without mask and vice versa. Therefore, this work analyzes the practical covariates (e.g., elapsed time, age, sex, with mask and no-mask, and eyeglasses with mask and no-mask). Namely, we present a longitudinal study using one of the largest, deepest, and longest (in terms of number of subjects, number of images per subject, and time spans of subject images) child face dataset. Thus, this study investigated the above-mentioned directions by extending Children Longitudinal Face (CLF) [4, 11] dataset. We simulated the synthetic mask over all face images by using open source tool Masked Face-Net [8] on the children dataset while keeping faces' longitudinal nature, as also shown in Fig.1 . There are several venues where child face recognition systems are needed, e.g., finding missing children [4, 12] , de-duplication of identification documents (e.g., minors passport validation and diver license) [13, 14] and school attendance during COVID-19 pandemic with mandatory mask [15, 16] . The contributions of this study are as follows: • A longitudinal child dataset with face samples with synthetic masks. • We conduct extensive longitudinal performance analyses of three top-performing public and one COTS face recognition systems on face images of children with mask. There is no such longitudinal study of children to our knowledge. • We provide an extensive comparative evaluation of child longitudinal face verification and identification under joint and disjoint gender with and without face mask. Additionally, we analyze gender bias with eyeglasses and masks. • FRS performance degrades with increasing time between gallery and probe samples. Using a face mask together with aging causes such declines. The manuscript is organized as follows. Section 2 details the extended longitudinal child dataset used in this study. Section 3 presents the experimental setup. Section 4 discusses experimental results. Finally, the conclusion and future works are presented in Section 5. All publicly available datasets have age range [14−80] . But there are no publicly available longitudinal masked face recognition datasets specific to children. Therefore, we used extended Children Longitudinal Face (ECLF) dataset that contains 26, 258 face images of 7, 473 subjects in the age group of [2, 18] . The average number of images per subject is 3, which were acquired an average over time lapse of 3.35 years. The ECLF dataset is comprised of 14, 057 (53.53%) boys and 12, 201 (46.48%) girls. Statistics for the dataset are shown in Table 1 . In Fig. 2 shows main steps of MaskedFace dataset generation. It is worth noting that some faces of ECLF dataset were not able to be processed (1, 550 boys' and 55 girls' images) because of larger pose and illumination. Hence, the resulting MaskedFace-ECLF contains 24, 653 masked face images (12, 507 boys and 12, 146 girls) of 7, 457 subjects (3, 732 boys, 3, 725 girls). To analyze real-world scenarios of child longitudinal study with mask and no-mask, following four face recognition protocols were investigated, as shown in Fig.3 . BCADGNMDV vs. CADGMDV: This protocol evaluates cross-age face verification performance under no-mask and mask with disjoint gender influence. It is done by performing 1:1 comparison, where enrollment image (first acquired image at youngest age) is compared to subsequent face images of the same subject at greater age than enrollment. We named this protocol Baseline Cross-Age Disjoint Gender No-MaskeD Verification vs. Cross-Age Disjoint Gender MaskeD Verification (BCADGNMDV vs. CADGMDV). BCAJGNMDV vs. CAJGMDV: To compare joint effect of gender and aging with mask and no-mask, we used same 1:1 cross-age verification strategy as in (BCADGNMDV vs. CADGMDV) but with joint gender. We named this cross-age protocol as Baseline Cross-Age Joint Gender No-MaskeD Verification vs. Cross-Age Joint Gender MaskeD Verification (BCAJGNMDV vs. CAJGMDV). BCANMDG&NMDPI vs. CANMDG&MDPI and BCAN-MDG&NMDPI vs. CAMDG&NMDPI: This protocol simulates two real time identification cases: (i) time of reopening school and (ii) missing child recognition in pandemic. In former case, the gallery set faces were with no-mask and the probe set faces were with mask. We perform cross-age identification comparison between all gallery enrollment samples of all subjects and probe nonenrollment samples. Particularly, we conduct joint gen- T1 T2 T3 T4 T5 T6 Avg T1 T2 T3 T4 T5 We present the cross-age verification, cross-age identification, and verification performance achieved by the four FRS. In [4] . Moreover, we analyzed the skin tones of boys and girls by selecting a 3 × 3 patch from forehead of the subject, then we averaged the patch values as a skin tone indicator. The average skin tone indicator for boys and girls, in the used dataset, is 166.07 and 176.59, respectively. Namely, the girls' skin tones are lighter than boys, and it has been reported in many studies, e.g., [17] , that face systems attain better performances on lighter skin subjects. Also, we found that more boy subjects are with eyeglasses than girls that may be another variate negatively impacting the face systems. Among FaceNet [5] , PFE [18] , ArcFace [19] and COTS face systems [20] , COTS outperformed others consistently for all time lapses. However, among three academic face systems, FaceNet and Arc-Face, respectively, achieved worst and best performances, because FaceNet uses softmax loss function which is known for not being capable of discriminating hard pairs [19] . Whereas, ArcFace is based on additive angular margin loss that simultaneously enhances the intra-class compactness and inter-class discrepancy. Similar observations can be seen in Table 3 for CADGMDV experiment. Besides, we can notice in Tables 2 and 3 that face mask decreases the performances of face systems. For example, the average verification rates at 0.1% FAR operating point using PFE for girls without and with mask, respectively, are 98.0% and 87.37%. Also, it is evident that face mask with aging leads to a greater performance degradation than only with mask, e.g., for boys without mask, the average verification rate with 0.01% FAR operating point at T1 is 92.38%, while it is 40.97% with mask at T6. Table 4 shows the results of joint gender when both probe and gallery samples are without mask ('Boys and Girls with Table 4 : Longitudinal verification rate (%) of face recognition systems on joint gender with and without mask. Model FAR T1 T2 T3 T4 T5 T6 Avg T1 T2 T3 T4 T5 T6 Avg T1 T2 T3 T4 T5 T6 Avg T1 T2 T3 T4 T5 T6 Avg T1 T2 T3 T4 T5 No-Mask') and when both probe and gallery samples are with mask ('Boys and Girls with Mask'). It can be seen in Table 4 that the performances of the systems are optimal when the acquisition time delay between probe and gallery is small (i.e., time lapse T1). Also, the face systems attained lower crossage verification performance when both probe and gallery samples are with mask than when both probe and gallery samples are without mask. Table 5 , we report the results of longitudinal closed-set identification rate of face systems on no-mask vs. no-mask, no-mask vs. mask and mask vs. no-mask scenarios. The mask vs. no-mask is representation of gallery vs. probe situation, where a missing child's photo is only with mask (i.e., gallery sample) and after some years it is being compared with no-mask probe sample. We can see in the Table 5 that, like in verification, the identification's performance decreases with increase in time lapses, but the rate of degradation is smaller. Moreover, the identification accuracy suffers from mask as well as compound effect of mask and aging, e.g., FaceNet at T1 attained 85.31% Rank-1 accuracy in nomask vs. no-mask, whereas it reached 28.11% at T6 in no-mask vs. mask. In Table 5 , in both no-mask vs. mask and mask vs. no-mask settings, performances varies based on the face algorithms. Moreover, the results show that on an average the mask vs. no-mask scenario achieved better accuracy than no-mask vs. mask setting. All in all, ArcFace and COTS achieved comparative performances in identification mode. The objective of this experiment is to study gender bias with eyeglasses and mask on verification. Out of 26, 258 images in ECLF dataset, only 1, 718 images from 396 boy and 346 girl subjects are with eyeglasses. For fairness, we selected 2, 22 subjects for each boy and girl group, where 88 subjects are with 2 images, 70 with 3 images, 46 with 4 images and 18 with 5 images. We can observe in Table 6 that even though boys and girls subjects are with eyeglasses but for girls the FRS achieved higher performances in both no-mask+eyeglass (i.e., both template and query are without mask) and mask+eyeglass (i.e., both template and query are with mask) setting. For example, COTS procured 99.60% and 100% (for no-mask) and 81.28% and 94.42% (for mask) at FAR 0.01% for boys and girls, respectively. It is also easy to see that mask+eyeglasses lessen the accuracies of the FRS, e.g., FaceNet accuracy diminished from 68.70% to 27.29% for boys. For eyeglass+(no-) mask, COTS performed better than ArcFace. Driven by the COVID-19 pandemic and subsequent face mask conformity, this paper, contrary to prior works, investigated the impact of aging with face mask on child subject recognition. Particularly, the empirical efficacy of four FRS was conducted under face masked children cross-age verification and identification scenarios. This study assembled longitudinal Indian children (i.e., boys and girls aged from 2 to 18) cohorts database with synthetic masks, and showed that face systems' performances are severely deteriorated by aging with masks. Moreover, the study found that accuracy of FRS is affected by mask with eyeglasses. Also, the identification levels of girls in the ECLF appear to be higher than boys. In future, we will work towards creating a longitudinal child database with real masks and different ethnicities, developing FRS that are inherently robust to face mask aging, and investigating face mask aging as a face presentation attack. The effect of wearing a mask on face recognition performance: an exploratory study Part6b: Face recognition accuracy with face masks using post-covid-19 algorithms Extended evaluation of the effect of real and simulated masks on face recognition performance Longitudinal study of child face recognition Facenet: A unified embedding for face recognition and clustering Face recognition algorithm bias: Performance differences on images of children and adults Labeled faces in the wild: A database for studying face recognition in unconstrained environments Maskedface-net -a dataset of correctly/incorrectly masked face images in the context of covid-19 Indian masked faces in the wild dataset Masked face recognition datasets and validation Child face age progression and regression using self-attention multi-scale patch gan Identifying missing children: Face age-progression via deep feature aging Longitudinal study of automatic face recognition Conditional perceptual adversarial variational autoencoder for age progression and regression on child face masked education? the benefits and burdens of wearing face masks in schools during the current corona pandemic A survey of face recognition techniques under occlusion Fairface challenge: Analyzing bias in face recognition Probabilistic face embeddings Arcface: Additive angular margin loss for deep face recognition Facevacsdbscanid 5.6.0