eswa.eps Expert Systems with Applications 36 (2009) 9719–9728 Contents lists available at ScienceDirect Expert Systems with Applications j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / e s w a Improving the interest operator for face recognition Yong Xu a,*, Lu Yao a, David Zhang b, Jing-Yu Yang c a Bio-Computing Research Center, Harbin Institute of Technology, Shenzhen Graduate School, Shenzhen, Guang dong 518055, China b Biometrics Research Center, The Hong Kong Polytechnic University, China c School of Computer Science and Engineering, Nanjing University of Science and Technology, China a r t i c l e i n f o Keywords: Face recognition Feature extraction Interest operator 0957-4174/$ - see front matter � 2009 Elsevier Ltd. A doi:10.1016/j.eswa.2009.02.032 * Corresponding author. Tel.: +86 752 26032458; fa E-mail address: laterfall2@yahoo.com.cn (Y. Xu). a b s t r a c t When the conventional interest operator is used as the feature extraction procedure of face recognition, it has the following two shortcomings: first, though the purpose of the conventional interest operator is to use the intensity variation between neighboring pixels to represent the image, it cannot obtain all vari- ation information between neighboring pixels. Second, under varying lighting conditions two images of the same face usually have different feature extraction results even though the face itself does not have obvious change. In this paper, we propose two new interest operators for face recognition, which are used to calculate the pixel intensity variation information of overlapping blocks produced from the original face image. The following two factors allow the new operators to perform better than the conventional interest operator: the first factor is that by taking the relative rather than absolute variation of the pixel intensity as the feature of an image block, the new operators can obtain robust block features. The second factor is that the scheme to partition an image into overlapping rather than non-overlapping blocks allows the proposed operators to produce more representation information for the face image. Experi- mental results show that the proposed operators offer significant accuracy improvement over the con- ventional interest operator. � 2009 Elsevier Ltd. All rights reserved. 1. Introduction Since the interest operator was first proposed, it has been wildly used to represent image or target feature. For example, Moravec (1981) used an interest operator to compute the intensity vari- ances in the horizontal, vertical and both diagonal directions for each pixel point in an image and selected the minimum of these values as the variance of that point. Nasrabadi and Choo (1994) used the interest operator in the stereo vision correspondence. In Nasrabadi and Choo (1994), the interest operator was applied to all image blocks and the point having the local maximal variance in each local block was selected as the so-called interesting point of that block. Căleanu, Huang, Gui, Tiponu, and Maranescu (2007), Căleanu (2000), Căleanu (2001), Zhao, Huang, and Sun (2004) and Zhao (2007) used the interest operator to extract fea- tures from face images. The interest operator was also exploited for target recognition (Haber & Modersitzki, 2004). Generally speaking, the interest operator for recognition can be viewed as a feature extraction algorithm that calculates variation information in different directions of the image pixel intensity. Note that when the interest operator is applied to face recognition, the first step is to divide each image into the same number of blocks with a fixed ll rights reserved. x: +86 755 26032461. size. Then feature extraction is performed for each block by using the interest operator. Finally, one can treat the feature extraction results of all the blocks of a face image as the representation of this image and can classify face images using the representation and a classifier. It should be pointed out that complex imaging conditions such as varying pose, facial expression and lighting conditions may re- strict the ability of the interest operator. Actually, complex imaging conditions degrade performances of most of face recognition tech- niques (Adini, Moses, & Ullman, 1997). In addition, the human face is a deformable object and a face can produce different deforma- tions at different times, resulting in various facial expressions. Con- sequently, there may be much difference between a block of one image of a face and the same block of another image of the same face, if the two images are associated with different poses or facial expressions. Face recognition using the conventional interest oper- ator, therefore, has the following shortcoming: the feature extrac- tion results of different images of the same face might have much difference. Another shortcoming of the conventional interest oper- ator stems from the algorithm itself. Under varying lighting condi- tions the algorithm usually also produces different feature extraction results for the same face, even if the face itself does not have any change in pose and expression. This is because vary- ing lighting conditions produce different intensity values for the image pixel. For example, high intensity lighting usually produces mailto:laterfall2@yahoo.com.cn http://www.sciencedirect.com/science/journal/09574174 http://www.elsevier.com/locate/eswa 9720 Y. Xu et al. / Expert Systems with Applications 36 (2009) 9719–9728 high intensity value for the pixel and consequently the variation between two neighboring pixels usually also appears to be high. On the other hand, low intensity lighting usually produces low intensity value for the pixel and consequently the variation be- tween two neighboring pixels is also low. As a result, when we compare the feature extraction results of two images that are ob- tained under the high and low intensity lighting conditions, a low similarity may be produced. The third shortcoming of the con- ventional interest operator is that it cannot obtain all variations be- tween neighboring pixels. Indeed, since the conventional interest operator is applied to only pixels within an image block, the con- ventional interest operator is not able to compute the variations between two neighboring pixels that are located in two blocks, respectively. This is illustrated by Fig. 1. In this paper, in order to overcome the shortcomings of the con- ventional interest operator, we propose to exploit new operators and the overlapping block partition scheme for face recognition. The rationale of the proposed approach is as follows: first, because the improved interest operator takes the relative rather than abso- lute variation of image pixel intensity as the features of the face image, this approach can produce more stable face features for the same face than the conventional interest operator, especially under varying lighting conditions. This is helpful for obtaining a satisfactory classification accuracy. Second, since in our approach different blocks overlap one another, our approach allows intensity variations between all pixels to be provided whereas the conven- tional interest operator cannot do so. Experimental results illus- trate that our approach can produce a higher accuracy in comparison with the conventional interest operator. Experiments also show that a linear feature extraction approach can be com- bined with the proposed interest operators to obtain further per- formance improvement. 2. The conventional interest operator As mentioned above, the conventional interest operator evalu- ates the variation of image pixels in the horizontal, vertical and both diagonal directions for each block of an image. The conven- tional interest operator can be described as follows: it calculates the mean l and the center variance r2 of a block using (1) and (2), respectively. It calculates r20; r 2 45; r 2 90 and r 2 135, which respec- tively stand for the intensity variations of block pixels in horizon- tal, diagonal 45�, vertical and diagonal 135� directions, using (3)– (6), respectively. Fig. 1. Illustration of the non-overlapping block partition scheme used for the conventional interest operator. This figure shows that an image is divided into nine non-overlapping blocks each consisting of a number of pixels. For a block, we denote the boundary pixels that adjoin another block by ‘�’ and we denote the other pixels by ‘‘�”. Note that for two ‘�’ pixels respectively located in two blocks, even if they are neighbors, the pixel intensity variation of the two pixels cannot be obtained by the conventional interest operator. u ¼ 1 P � Q XP x¼1 XQ y¼1 pðx; yÞ ð1Þ r2 ¼ 1 P � Q XP x¼1 XQ y¼1 ½pðx; yÞ� u�2 ð2Þ r20 ¼ 1 P � Q XP�1 x¼1 XQ y¼1 ½pðx þ 1; yÞ� pðx; yÞ�2 ð3Þ r245 ¼ 1 P � Q XP�1 x¼1 XQ�1 y¼1 ½pðx þ 1; yÞ� pðx; y þ 1Þ�2 ð4Þ r290 ¼ 1 P � Q XP x¼1 XQ�1 y¼1 ½pðx; y þ 1Þ� pðx; yÞ�2 ð5Þ r2135 ¼ 1 P � Q XP�1 x¼1 XQ�1 y¼1 ½pðx þ 1; y þ 1Þ� pðx; yÞ�2 ð6Þ Hereafter we suppose that the size of each block is P � Q and p(x, y)(1 6 x 6 P, 1 6 y 6 Q) represents the pixel intensity of the point (x, y) in a block. Fig. 2 shows the feature extraction results of a face image obtained using the conventional interest operator. From Fig. 2, we can clearly see that if two original images of the same face are obtained under varying conditions, their resultant images may have obvious difference. 3. Description of our approach Our approach consists of two components, an overlapping block partition scheme and an improved interest operator. The goal of our approach is to improve image presentation ability of the con- ventional interest operator. The basic idea for improving the inter- est operator is to take the relative variation of the gray intensity as features of the image block. Based on this idea, we develop two new interest operators as shown in the following. 3.1. Improved interest operator 1 Improved interest operator 1 is defined as follows: r2 ¼ 1 P � Q XP x¼1 XQ y¼1 ½pðx; yÞ� u�2 ðu þ c1Þ 2 ð7Þ r20 ¼ 1 P � Q XP�1 x¼1 XQ y¼1 ½pðx þ 1; yÞ� pðx; yÞ�2 ðu þ c1Þ 2 ð8Þ r245 ¼ 1 P � Q XP�1 x¼1 XQ�1 y¼1 ½pðx þ 1; yÞ� pðx; y þ 1Þ�2 ðu þ c1Þ 2 ð9Þ r290 ¼ 1 P � Q XP x¼1 XQ�1 y¼1 ½pðx; y þ 1Þ� pðx; yÞ�2 ðu þ c1Þ 2 ð10Þ r2135 ¼ 1 P � Q XP�1 x¼1 XQ�1 y¼1 ½pðx þ 1; y þ 1Þ� pðx; yÞ�2 ðu þ c1Þ 2 ð11Þ where u is also defined by (1) and c1 is a positive constant. Indeed, the new operator as defined in (7)–(11) produces the relative rather than absolute variation of the pixel intensity by dividing the result of the conventional interest operator by a quantity associated with the mean of the pixel intensity. The difference between improved interest operator 1 and the conventional interest operator is as fol- lows. As presented above, varying lighting condition is one typical factor that makes two corresponding pixels from two images of the same face have quite different intensities. As a result, under varying lighting condition, two corresponding image blocks from two images of the same face usually appear to have different inten- sity variations. The conventional interest operator, therefore, usu- Fig. 2. The original face and resultant images obtained using the conventional interest operator. Each row shows the face and resultant images. For example, (b), (c), (d), (e) and (f) respectively stand for the resultant images on r2; r20; r 2 45; r 2 90; r 2 135 of the face image shown in (a). The original face image is first divided into a number of non- overlapping 2 by 2 blocks and then the conventional interest operator is implemented for each image block. Y. Xu et al. / Expert Systems with Applications 36 (2009) 9719–9728 9721 ally does not perform well in producing stable features for the same face under varying lighting condition. However, by using the rela- tive variation of the pixel intensity as shown in (7)–(11) as the fea- ture of the face image, improved interest operator 1 is able to produce more stable features for the same face under varying light- ing condition. The use of constant c1 can prevent the operator from being unfeasible in the case where the mean of the pixel intensity is zero. 3.2. Improved interest operator 2 Improved interest operator 2 is designed as follows: r2 ¼ 1 P � Q XP x¼1 XQ y¼1 jpðx; yÞ� uj u þ c2 ð12Þ r20 ¼ 1 P � Q XP�1 x¼1 XQ y¼1 jpðx þ 1; yÞ� pðx; yÞj u þ c2 ð13Þ r245 ¼ 1 P � Q XP�1 x¼1 XQ�1 y¼1 jpðx þ 1; yÞ� pðx; y þ 1Þj u þ c2 ð14Þ r290 ¼ 1 P � Q XP x¼1 XQ�1 y¼1 jpðx; y þ 1Þ� pðx; yÞj u þ c2 ð15Þ r2135 ¼ 1 P � Q XP�1 x¼1 XQ�1 y¼1 jpðx þ 1; y þ 1Þ� pðx; yÞj u þ c2 ð16Þ where u is still defined by (1) and c2 is a positive constant. Differing from improved interest operator 1, the second improvement opera- tor takes as the feature of an image block the result of the conven- tional interest operator divided by the sum of the mean of the pixel intensity and c2. The use of c2 enables the operator to be workable in the case where the mean of the pixel intensity is zero. Improved interest operator 2 also has the similar advantages to improved interest operator 1. 3.3. Overlapping block partition scheme As mentioned above, though the conventional interest operator was developed to calculate the pixel intensity variation, not all var- iation information between neighboring pixels could be obtained. Indeed, the conventional interest operator cannot calculate the variations between two neighboring pixels respectively located in two blocks as shown in Fig. 1. In order to overcome this draw- back of the conventional interest operator, we propose to partition an image into a number of overlapping blocks and to extract fea- tures from each block by using either of the two interest operators respectively presented in Sections 3.1 and 3.2. Note that in our partition scheme two horizontally neighboring blocks overlap one another and so do two vertically neighboring blocks. Fig. 3 shows that how an original 6 � 6 image is partitioned into four overlapping image blocks each having the size of 4 � 4. Here each rectangle unit represents a pixel. The overlapping region is represented by the shadow, which shows that half of each block is overlapped by a neighboring block. Our approach implements improved interest operator 1 (or improved interest operator 2) for each image block produced by the overlapping block partition scheme. Advantages of our approach can be described as follows. First, because our approach takes the relative rather than absolute vari- ation of the pixel intensity as the feature of an image block, the fea- ture extraction result appears to be less affected by the lighting 2 2 Fig. 3. Illustration of the scheme to partition an image into overlapping image blocks. The overlapping region is represented by the shadow. Each image block is of size of 4 � 4. Half of every block is overlapped by a neighboring block. For example, half of the 4 � 4 image block located in the left top is overlapped by its right neighbor image block. For the same block, its down neighbor 4 � 4 block also overlaps half of its size. 9722 Y. Xu et al. / Expert Systems with Applications 36 (2009) 9719–9728 condition than the conventional interest operator. This helps our approach obtain more robust features than the conventional inter- est operator. Second, our approach allows information of the inten- sity variation of all neighboring pixels to be obtained due to the overlapping partition scheme, whereas the conventional interest operator cannot do so. Fig. 4 shows an original face image and the feature extraction result of this image obtained using improved interest operator 1. The overlapping block partition scheme can be shown more clearly by the following example: if the original image is repre- sented by an 80 � 80 matrix and we partition the original image Fig. 4. The original face image and the resultant images obtained using improved interes example, (b), (c), (d), (e) and (f) respectively stand for the resultant images on r2; r20; r 2 4 divided into overlapping 4 by 4 blocks and then improved interest operator 1 is implem image block. into a number of 4 � 4 overlapping blocks and half of each block is overlapped by a neighboring block, we will obtain 1521 overlap- ping image blocks. When we calculate each of the r2; r20; r 2 45; r 2 90; r2135 values for all the 1521 image blocks of the original image, the result that we obtain is a 1521-dimensional vector. The vector can be also shown as a 39 � 39 matrix as illustrated in Fig. 4. Since each block has its own five values r2; r20; r 2 45; r 2 90; r 2 135, the feature extraction result of the original image can be regarded as five 1521-dimensional vectors or five 39 � 39 matrices. Concatenating these one-dimensional vectors, we can obtain a 7605-dimensional vector. More generally, if the original image is an m � n matrix and we partition the original image into a number of s � t overlapping blocks and half of each block is overlapped by a neighboring block, (2m/s � 1) � (2n/t � 1) image blocks will be generated. As a result, for this original image, the feature extraction result produced by our approach will be a 5(2m/s � 1) � (2n/t � 1)-dimensional vector. Similar to the features obtained using the conventional interest operator, the features obtained using our approach are also usually very high-dimensional. However, as shown in Section 5, linear fea- ture extraction procedures such as 2DPCA (two-dimensional PCA) or 2DFLD (two-dimensional Fisher discriminant analysis) (Cho, Chang, Kim, & Lee, 2006; Kongsontana & Rangsanser, 2005; Mutelo, Khor, Woo, & Dlay, 2006; Nhat & Lee, 2005; Sanguansat, Asdornw- ised, Jitapunkul, & Marukatat, 2006; Wang, Wang, Zhang, & Feng, 2005; Xu, Zhang, Yang, & Yang, 2008; Yang, Zhang, Frangi, & Yang, 2004; Yang et al., 2004) can be used to transform the feature extraction results of the interest operator into lower-dimensional features. t operator 1 with c1 = 10. Each row shows a face image and the resultant images. For 5; r 2 90; r 2 135 of the face image shown in (a). Note that the original face image is first ented for each image block. Half of an image block is overlapped by a neighboring 0 2 4 6 8 10 12 14 16 18 20 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 2DPCA IO+2DPCA(2×2) IO+2DPCA(2×4) (a) 0 2 4 6 8 10 12 14 16 18 20 0.4 0.5 0.6 0.7 0.8 0.9 1 2DFLD IO+2DFLD(2×2) IO+2DFLD(2×4) (b) 110 120 130 140 150 160 170 180 190 200 0.74 0.75 0.76 0.77 0.78 0.79 0.8 0.81 PCA IO+PCA(2×2) IO+PCA(2×4) (c) Fig. 5. Face recognition results on the AR database obtained by different linear feature extraction procedures combined with or without the conventional interest operator. (a) Classification right rate associated with 2DPCA, (b) classification right rate associated with 2DFLD and (c) classification right rate associated with PCA. The horizontal coordinate represents the number of the transforming axes of the linear feature extraction procedure used for feature extraction and the vertical coordinate represents the classification accuracy. The horizontal and vertical coordinates of Figs. 6, 7, 9–11 are same as this figure. Table 1 Experimental result comparison of our approach and other approaches on the AR face database. Best accuracy(%) Mean of the accuracy(%) Size of the block 2DPCA 78.8 70.1 2DFLD 81.7 76.4 PCA 79.6 78.9 IO + 2DPCA 79.0 69.5 2 � 4 IO + 2DFLD 81.3 70.0 2 � 4 IO + PCA 78.8 76.0 2 � 4 OIIO1 + 2DPCA 81.7 75.7 2 � 4 OIIO1 + 2DFLD 84.2 78.0 2 � 4 OIIO1 + PCA 82.1 81.4 2 � 4 OIIO2 + 2DPCA 91.9 82.2 2 � 4 OIIO2 + 2DFLD 93.5 86.7 2 � 4 OIIO2 + PCA 91.9 90.1 2 � 4 Y. Xu et al. / Expert Systems with Applications 36 (2009) 9719–9728 9723 4. More discussion on improved and the conventional operators In this section we will compare improved interest operators with conventional interest operator and some other algorithms that exploit gradient information of the image. First, we analyze the relationship and difference between the conventional interest operator and the gradient operator. As presented above, the inter- est operator computes the pixel intensity variation in different directions. Gradient operators also evaluate the pixel intensity var- iation. The difference between the interest operator and the gradi- ent operator are as follows: a gradient operator produces a vector i.e. gradient vector that uses two components to indicate the gra- dient whereas the interest operator obtains only scalar values. The square-root of the squared sum of the two components of the gradient vector is usually used to denote the magnitude value of the gradient. The similarity between the interest operator and the gradient operator is as follows: for the conventional interest operator, r245 and r 2 135 as defined in (4) and (6) act as the mean of the squares of the two components of the Roberts gradient oper- ator (Gonzalez & Woods, 1987), respectively. Additionally, accord- ing to the definition of the gradient, r20 and r 2 90 as defined in (3) and (5) can also be respectively regarded as the mean of the squares of two components of a gradient operator. Consequently, we can conclude that the interest operator and the gradient oper- ator provide image gradient information in different ways. The interest operator aims at calculating ‘average’ gradient information in different directions of an image block, whereas the gradient operator provides gradient information of each image pixel in the form of a vector. Gradient operators and improved gradient opera- tors have been applied to recognition problems (Gao & Leung, 2002; Haber & Modersitzki, 2004; TakaÂcs, 1998; Wei & Lai, 2006) and image matching problems (Ando, 2000; Wolfson & Rigoutsos, 1997). Approaches exploiting image gradient information for recogni- tion can be classified into two classes. The first class of approaches exploits the image gradient information itself to perform recogni- tion. The approaches proposed in Haber and Modersitzki (2004) and Wei and Lai (2006) as well as the conventional interest oper- ator are examples of the first class. In Haber and Modersitzki (2004) and Wei and Lai (2006), ‘normalized gradient’ and ‘relative gradient’ were respectively used as image features. In Wei and Lai (2006), the magnitude of the conventional gradient of a pixel point was first divided by the sum of a constant and the maximum mag- nitude of the gradient in an image block. Then the division result was taken as ‘relative gradient’ of the pixel point. The second class of approaches firstly exploits the image gradient information to produce image edges and then employs image edges to perform recognition. The rationale of the second class of approaches is that edge information is a useful object representation feature (Gao & Leung, 2002). The approaches used in Gao and Leung (2002), Ta- 0 2 4 6 8 10 12 14 16 18 20 0.4 0.5 0.6 0.7 0.8 0.9 1 2DPCA IIO+2DPCA(2×2) IIO+2DPCA(2×4) OIIO(1/2)+2DPCA(2×2) OIIO(1/2)+2DPCA(2×4) (a) 0 2 4 6 8 10 12 14 16 18 20 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 2DFLD IIO+2DFLD(2×2) IIO+2DFLD(2×4) OIIO(1/2)+2DFLD(2×2) OIIO(1/2)+2DFLD(2×4) (b) 110 120 130 140 150 160 170 180 190 200 0.74 0.75 0.76 0.77 0.78 0.79 0.8 0.81 0.82 0.83 PCA IIO+PCA(2×2) IIO+PCA(2×4) OIIO+PCA(2×2) OIIO+PCA(2×4) (c) Fig. 6. Face recognition results on the AR database obtained by different linear feature extraction procedures combined with or without improved interest operator 1. (a) Classification right rate associated with 2DPCA, (b) classification right rate associated with 2DFLD and (c) classification right rate associated with PCA. 0 2 4 6 8 10 12 14 16 18 20 0.4 0.5 0.6 0.7 0.8 0.9 1 2DPCA IIO+2DPCA(2×2) IIO+2DPCA(2×4) OIIO(1/2)+2DPCA(2×2) OIIO(1/2)+2DPCA(2×4) (a) 0 2 4 6 8 10 12 14 16 18 20 0.4 0.5 0.6 0.7 0.8 0.9 1 2DFLD IIO+2DFLD(2×2) IIO+2DFLD(2×4) OIIO(1/2)+2DFLD(2×2) OIIO(1/2)+2DFLD(2×4) (b) 110 120 130 140 150 160 170 180 190 200 0.78 0.8 0.82 0.84 0.86 0.88 0.9 0.92 PCA IIO+PCA(2×2) IIO+PCA(2×4) OIIO(1/2)+PCA(2×2) OIIO(1/2)+PCA(2×4) (c) Fig. 7. Face recognition results on the AR database obtained by different linear feature extraction procedures combined with or without improved interest operator 2. (a): classification right rate associated with 2DPCA. (b): classification right rate associated with 2DFLD. (c): classification right rate associated with PCA. 9724 Y. Xu et al. / Expert Systems with Applications 36 (2009) 9719–9728 Fig. 8. Some face images of a same face. These images are obtained under different illumination conditions. (a) Face images obtained in the cases where the azimuth angle and the elevation angle of the light source with respective to the camera axis are small. (b) Face images obtained in the cases where the azimuth angle and the elevation angle of the light source with respective to the camera axis are quite large. 0 2 4 6 8 10 12 14 16 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 2DPCA IO+2DPCA(2×2) IO+2DPCA(2×4) (a) 0 2 4 6 8 10 12 14 16 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 2DFLD IO+2DFLD(2×2) IO+2DFLD(2×4) (b) 110 120 130 140 150 160 170 180 190 200 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 PCA IO+PCA(2×2) IO+PCA(2×4) (c) Fig. 9. Face recognition results on the YaleB database obtained by different linear feature extraction procedures combined with or without the conventional interest operator. (a): classification right rate associated with 2DPCA. (b): classification right rate associated with 2DFLD. (c): classification right rate associated with PCA. Y. Xu et al. / Expert Systems with Applications 36 (2009) 9719–9728 9725 kaÂcs (1998) and Wang et al. (1998) are three typical examples of the second class. In TakaÂcs (1998), the binary coding result of So- bel edge detection operator was used as face edge feature. In Gao and Leung (2002), the line edge map approach detected the edge feature and then classified faces using the edge feature. It was as- sumed that image edge detection using gradient operators was al- most not influenced by varying lighting condition (Gao & Leung, 2002); however, the approaches in Gao and Leung (2002), TakaÂcs (1998) and Wang et al. (1998) are all not able to obtain truly invari- ant facial features with respect to lighting condition. This is be- cause the pixel intensity value varies with the lighting condition and consequently the gradient value seems to be also variable with respect to the lighting condition. The interest operator appears to be also helpful for indicating salient image edge information in different directions as shown in Fig. 4. Actually, an interest operator obtains ‘average’ directional edge of an image block since it sums gradient information in a cer- tain direction. Improved interest operators have the following advantage: by improving the conventional interest operator, im- proved operators enable the obtained ‘average’ directional edge information of a face image block to be less variable with respect to the lighting condition. This is very beneficial to face recognition. In addition, since the average relative variation of the pixel inten- sity appears to in some extent be robust to facial expression and pose, improved interest operators may also produce more stable face feature than the conventional interest operator under the con- ditions of varying facial expression or pose. 5. Experimental result In this section we compare our approach and the conventional interest operator using the AR and YaleB face databases. We eval- uate the similarity between the feature extraction results of two face images by using sðx; yÞ¼ xT y jjxjj � jjyjj ; ð17Þ where x, y stand for two vectors respectively corresponding to the feature extraction results of the two face images. After the similar- ities between a testing sample and each of all training samples are computed, we select out the training sample that has the maximum similarity to the testing sample and classify the testing sample into the class that the selected training sample belongs to. 5.1. Experiments on the AR face database The AR face database includes more than 4000 face images showing faces with different facial expressions, in varying lighting 0 2 4 6 8 10 12 14 16 0.65 0.7 0.75 0.8 0.85 2DPCA IIO+2DPCA(2×2) IIO+2DPCA(2×4) OIIO(1/2)+2DPCA(2×2) OIIO(1/2)+2DPCA(2×4) (a) 0 2 4 6 8 10 12 14 16 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 2DFLD IIO+2DFLD(2×2) IIO+2DFLD(2×4) OIIO(1/2)+2DFLD(2×2) OIIO(1/2)+2DFLD(2×4) (b) 110 120 130 140 150 160 170 180 190 200 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 PCA IIO+PCA(2×2) IIO+PCA(2×4) OIIO(1/2)+PCA(2×2) OIIO(1/2)+PCA(2×4) (c) 9726 Y. Xu et al. / Expert Systems with Applications 36 (2009) 9719–9728 conditions and occluded in several ways (Yang et al., 2004)1. Each subject have 26 face images. For each subject, the filenames of the 26 images contain the numbers from 1 to 26, respectively. If the number contained in the filename is 1, we call the image the first image of the subject, and so on. We use the computer generates 13 random integers in the range of from 1 to 26. For every subject, we take the 13 images whose filenames contain the numbers from the random integer sequence as the training samples and consider the remaining samples as test samples. The generate random integer sequence is 2, 4, 7, 8, 10, 15, 16, 19, 21, 22, 24, 25, 26. Fig. 5 shows face recognition results on the AR database ob- tained by different linear feature extraction procedures combined with or without the conventional interest operator. Note that here- after by IO (Interest Operator) we denote the conventional interest operator. By IIO1 we denote improved interest operator 1 based on the non-overlapping partition scheme. By IIO2 we denote im- proved interest operator 2 based on the non-overlapping partition scheme. In addition, we use OIIO1 (Overlapped Improved Interest Operator 1) to represent improved interest operator 1 based on the overlapping partition scheme. We also use OIIO2 to represent improved interest operator 2 based on the overlapping partition scheme. If the interest operator is followed by a linear feature extraction procedure, we add some indicators to the denotation. For example, ‘IO + 2DPCA’ means that feature extraction is imple- mented by the conventional interest operator followed by 2DPCA. We use a bracket to show the size of each block or the overlapping region. For example, ‘IO + 2DPCA(2 � 2)’ means that the image is partitioned into a number of 2 � 2 non-overlapping image blocks; whereas ‘OIIO(1/2) + 2DPCA(2 � 2)’ means that the image is parti- tioned into a number of 2 � 2 overlapping image blocks and 1/2 re- gions of two neighboring blocks are overlapped each other. According to Fig. 5, the combination of PCA and the conventional interest operator is not able to obtain a higher accuracy than PCA. The combination of 2DPCA and the conventional interest operator may produce a higher or lower accuracy than 2DPCA. So does the combination of 2DFLD and the conventional interest operator. Table 1 shows experimental result comparison of our approach and other approaches on the AR face database. From Table 1 and Figs. 5–7, we can see that the combination of either of the two im- proved interest operators and 2DPCA or 2DFLD can produce a high- er accuracy than 2DPCA or 2DFLD. The combination of an improved interest operator and 2DPCA or 2DFLD also performs better than the combination of the conventional interest operator and 2DPCA or 2DFLD. Fox example, the combination of the conventional inter- est operator and 2DPCA obtains the mean of recognition accuracy of 69.5% and the best accuracy of 79.0%. When OIIO1 is combined with 2DPCA, the mean of the accuracy and the best accuracy are 75.7% and 81.7%, respectively. For the combination of OIIO2 and 2DPCA, the mean of the accuracy and the highest accuracy are 82.2% and 91.9%, respectively. This means that compared to the combination of the conventional interest operator and 2DPCA, the combination of OIIO2 and 2DPCA improves the mean accuracy and the highest accuracy 12.7% and 12.9%, respectively. 5.2. Experiments on the YaleB face database The images in the YaleB database2 are obtained with varying illuminations and unfixed poses and there exists a wide rang of illumination cases. To focus on face recognition with varying illu- minations, we select and use 45 face images with pose 00 of every person. We crop each of these images to obtain a 32 � 32 image Fig. 10. Face recognition results on the YaleB database obtained by different linear feature extraction procedures combined with or without improved interest operator 1. (a): classification right rate associated with 2DPCA. (b): classification right rate associated with 2DFLD. (c): classification right rate associated with PCA.1 h t t p :/ / c o b w e b . e cn . p u rd u e . e du / ~ a l e i x / a l e i x _ f a c e _ D B . h t m l ; h t t p : // c o b - web.ecn.purdue.edu/~aleix/ar.html. 2 http://cvc.yale.edu/projects/yalefacesB/yalefacesB.html. http://cobweb.ecn.purdue.edu/~aleix/aleix_face_DB.html http://cobweb.ecn.purdue.edu/~aleix/ar.html http://cobweb.ecn.purdue.edu/~aleix/ar.html http://cvc.yale.edu/projects/yalefacesB/yalefacesB.html 0 2 4 6 8 10 12 14 16 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 2DPCA IIO+2DPCA(2×2) IIO+2DPCA(2×4) OIIO(1/2)+2DPCA(2×2) OIIO(1/2)+2DPCA(2×4) (a) 0 2 4 6 8 10 12 14 16 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 2DFLD IIO+2DFLD(2×2) IIO+2DFLD(2×4) OIIO(1/2)+2DFLD(2×2) OIIO(1/2)+2DFLD(2×4) (b) 110 120 130 140 150 160 170 180 190 200 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 PCA IIO+PCA(2×2) IIO+PCA(2×4) OIIO(1/2)+PCA(2×2) OIIO(1/2)+PCA(2×4) (c) Fig. 11. Face recognition results on the YaleB database obtained by different linear feature extraction procedures combined with or without improved interest operator 2. (a): classification right rate associated with 2DPCA. (b): classification right rate associated with 2DFLD. (c): classification right rate associated with PCA. Table 2 Experimental result comparison of our approach and other approaches on the YaleB face database. Best accuracy(%) Mean of the accuracy(%) Size of the block 2DPCA 77.2 74.5 2DFLD 75.2 69.0 PCA 58.8 58.8 IO + 2DPCA 78.4 70.9 2 � 2 IO + 2DFLD 78.8 72.3 2 � 2 IO + PCA 81.6 79.7 2 � 2 OIIO1 + 2DPCA 82.8 75.5 2 � 4 OIIO1 + 2DFLD 92.8 84.1 2 � 2 OIIO1 + PCA 85.8 85.1 2 � 4 OIIO2 + 2DPCA 94.4 89.9 2 � 2 OIIO2 + 2DFLD 98.0 90.3 2 � 4 OIIO2 + PCA 95.6 92.1 2 � 2 Y. Xu et al. / Expert Systems with Applications 36 (2009) 9719–9728 9727 (Xu, Yang, Jin, & Zheng, 2006). We test our approach using the ob- tained images. Some face images of a same face are shown in Fig. 8. Fig. 9 shows face recognition results on the YaleB database ob- tained by different linear feature extraction procedures combined with or without the conventional interest operator. Fig. 10 shows face recognition results on the YaleB database obtained by differ- ent linear feature extraction procedures combined with or without improved interest operator 1. Fig. 11 shows face recognition results on the YaleB database obtained by different linear feature extrac- tion procedures combined with or without improved interest oper- ator 2. Table 2 shows experimental result comparison of our approach and other approaches on the YaleB face database. Table 2 and Figs. 9–11 show that the combination of improved interest operator 1 or 2 and a linear feature extraction procedure performs better than the combination of the conventional interest operator and the same linear feature extraction procedure. Fox example, the combination of the conventional interest operator and 2DFLD obtains the mean of recognition accuracy of 72.3% and the highest recognition accuracy of 78.8%. When OIIO1 is combined with 2DFLD, the mean of the accuracy and the highest accuracy are 84.1% and 92.8%, respectively. For the combination of OIIO2 and 2DFLD, the mean of the accuracy and the highest accuracy are 90.3% and 98.0%,, respectively. This means that compared to the combination of the conventional interest operator and 2DFLD, the combination of OIIO2 and 2DFLD improves the mean accuracy and the highest accuracy 18.0% and 19.2%, respectively. 5.3. More experimental analysis In this subsection, we further investigate our approach by ana- lyzing the similarity of different images of the same face for the AR face database. Analysis is performed for the first, second, seventh, eighth and eleventh face images of each subject. The used five images of one subject are shown in Fig. 12. When compared with image (a), images (b) has different expression, image (c) is ob- tained under different lighting condition, images (d) and (e) are two occluded face images. Note that after the conventional interest operator or the pro- posed interest algorithm is applied to all the five images of a face, we can also evaluate the similarities between the resultant images Fig. 12. Illustration of the five images of a face used for similarity analysis. Table 3 Means of the similarities between the resultant images of (a) and those of (b), (c), (d), (e). Each row shows the means of the similarities between the resultant images associated with an interest operator of image (a) and images (b), (c), (d), (e). IIO1 and IIO2 respectively denote improved interest operators 1 and 2 applied to the non- overlapping partition scheme. Similarity of (b) and (a) Similarity of (c) and (a) Similarity of (d) and (a) Similarity of (e) and (a) IO 0.916 0.861 0.848 0.816 IIO1 0.946 0.941 0.914 0.868 OIIO1 0.950 0.950 0.920 0.877 IIO2 0.966 0.959 0.924 0.901 OIIO2 0.968 0.964 0.938 0.911 9728 Y. Xu et al. / Expert Systems with Applications 36 (2009) 9719–9728 of image (a) and each of images (b), (c), (d), (e) by using s1ðx1; y1Þ¼ xT 1 y1 jjx1jj�jjy1jj , where x1, y1 represent the two one-dimen- sional vectors of the resultant images of the two original face images, respectively. Note that an interest operator produces five resultant images for each original image, so the corresponding one-dimensional vector used for similarity computation is ob- tained by treating the five resultant images of the original image as an integral image and by concatenating the rows or columns of the integral image. For each subject, we respectively compute the similarities be- tween the resultant images of image (a) and those of images (b), (c), (d), (e) produced by an interest operator. Then we calculate the means of the similarities based on all the subjects. The means obtained in the case where each image block is of the same size of 2 � 2 are shown in Table 3. In the overlapping block partition scheme, half of each block is overlapped by a neighboring block. The second to fifth columns of Table 3 respectively show the means of the similarities between the resultant images of image (a) and those of images (b), (c), (d), (e). From Table 3, we can see that, when improved interest opera- tors 1 and 2 are applied to non-overlapping image blocks, the ob- tained similarities between the resultant images are higher than the similarities of the resultant images obtained using the conven- tional interest operator. Moreover, OIIO1 and OIIO2 produce higher similarities respectively than IIO1 and IIO2. This implies that the combination of the improved interest operator and the overlapping block partition scheme is more helpful for obtaining high similar- ities than the combination of the improved interest operator and the non-overlapping partition scheme. Therefore, we can conclude that both the improved interest operator and the overlapping par- tition scheme are useful for resulting in higher similarities for the resultant images of image (a) and those of images (b), (c), (d), (e). This also means that our approach can effectively reduce the differ- ence between different images of the same face, which is very ben- eficial to face recognition. 6. Conclusion The new approach proposed in this paper differs from the fea- ture extraction approach using the conventional interest operator in the following two aspects. The first aspect is that the proposed approach takes the relative rather than absolute variation of the pixel intensity as the feature of an image block. This allows the proposed approach to obtain robust block features that are less af- fected by varying imaging conditions such as varying lighting and facial expression. The second aspect is that the proposed approach adopts the overlapping block partition scheme. This enables the proposed approach to fully reflect the pixel intensity variation information and to produce more representative feature extraction results. As a result, the proposed approach can do better in face recognition than the conventional interest operator. The analysis and experimental results illustrate the feasibility and the satisfactory performance of the proposed approach. Exper- imental results show that the use of the proposed approach can ob- tain more than 10 percent accuracy improvement. Acknowledgement We wish to thank 863 Program Project (No. 2006 AA01Z193), National Natural Science Foundation of China (Nos. 60602038, 60632050) and Natural Science Foundation of Guangdong prov- ince, China (No. 06300862) for supporting. References Adini, Y., Moses, Y., & Ullman, S. (1997). Face recognition: The problem of compensating for changes in illumination direction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 721–732. Ando, S. (2000). Image field categorization and edge/corner detection from gradient covariance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(2), 179–190. Cho, D. U., Chang, U. D., Kim, B. H., & Lee, S. H. (2006). 2D direct LDA algorithm for face recognition. In Proceedings of the fourth international conference on software engineering research, management and applications, WA, United States (pp. 245– 248). Căleanu, C. D. (2000). Facial recognition using committee of neural networks. In Proceedings of the fifth seminar on neural network applications in electrical engineering, NEUREL, Belgrade, Yugoslavia (pp. 97–100). Căleanu, C. D. (2001). Face recognition using parallel neural processing and interest operator method. Ph.D. Thesis, University POLITEHNICA Timisoara. Căleanu, C. D., Huang, D.-S., Gui, V., Tiponu, V., & Maranescu, V. (2007). Interest operator versus Gabor filtering for facial imagery classification. Pattern Recognition Letters, 28, 950–956. Gao, Y., & Leung, M. K. H. (2002). Face recognition using line edge map. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24, 764–779. Gonzalez, Rafael C., & Woods, Richard E. (1987). Digital image processing (2nd ed.). Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc.. Haber, E., & Modersitzki, J. (2004). Intensity gradient based registration and fusion of multi-modal images. Technical Report, Department of Mathematics & Computer Science, Emory University, Atlanta. Kongsontana, S. & Rangsanser, Y. (2005). Face recognition using 2DFLD algorithm. In Proceedings of the eighth international symposium on signal processing and its applications (vol. 2, pp. 675–67). Moravec, H. (1981). Robot rover visual navigation. Ann Arbor, MI: University of Michigan Research Press. Mutelo, R. M., Khor, L. C., Woo, W. L., & Dlay, S. S. (2006). A novel fisher discriminant for biometrics recognition: 2DPCA plus 2DFLD. ISCAS2006, IEEE, 4325–4328. Nasrabadi, N. M., & Choo, C. Y. (1994). Hopfield network for stereo vision correspondence. In M. Gupta & G. Knopf (Eds.). Neuro vision systems principles and applications (vol. 2, pp. 442–458). IEEE. Nhat, V. D. M., & Lee, S. (2005). Improvement on PCA and 2DPCA algorithms for face recognition. In Proceedings of the fourth international Conference on I Singapore Image and Video Retrieval (CIVR), Singapore (pp. 568–577). Sanguansat, P., Asdornwised, W., Jitapunkul, S., & Marukatat, S. (2006). Two- dimensional linear discriminant analysis of principle component vectors for face recognition. IEICE Transactions on Information and Systems, E89-D(7), 2164–2170. TakaÑcs, B. (1998). Comparing face images using the modified hausdorff distance. Pattern Recognition, 31, 1873–1881. Wang, L. C., Der, S. Z., & Nasrabadi, N. M. (1998). Automatic target recognition using a feature decomposition and data decomposition modular neural network. IEEE Transactions on Image Processing, 7, 1113–1121. Wang, L., Wang, X., Zhang, X., & Feng, J. (2005). The equivalence of two-dimensional PCA to line-based PCA. Pattern Recognition Letters, 26, 57–60. Wei, Shou-Der, & Lai, Shang-Hong (2006). Robust and efficient image alignment based on relative gradient matching. IEEE Transactions on Image Processing, 15, 2936–2943. Wolfson, H., & Rigoutsos, I. (1997). Geometric hashing: An overview. IEEE Computational Science & Engineering Magazine, 4(4), 10–21. Xu, Y., Yang, J.-Y., Jin, Z., Zheng, Y.-J. (2006). Local correlation classification and its application to face recognition across illumination. In The international conference of machine learning and cybernetics, Dalian (pp. 3277–3281). Xu, Y., Zhang, D., Yang, J., & Yang, J.-Y. (2008). An approach for directly extracting features from matrix data and its application in face recognition. Neurocomputing, 71, 1857–1865. Yang, J., Zhang, D., Frangi, A. F., & Yang, J. Y. (2004). Two dimensional PCA: A new approach to appearance-based face representation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24, 131–137. Zhao, T. (2007). Several approaches for feature extraction and selection for face recognition. M.S. Degree Thesis, University of Harbin Institute of Technology (in Chinese). Zhao, Z. Q., Huang, D. S., & Sun, B. Y. (2004). Human face recognition based on multi- features using neural networks committee. Pattern Recognition Letters, 25, 1351–1358. Improving the interest operator for face recognition Introduction The conventional interest operator Description of our approach Improved interest operator 1 Improved interest operator 2 Overlapping block partition scheme More discussion on improved and the conventional operators Experimental result Experiments on the AR face database Experiments on the YaleB face database More experimental analysis Conclusion Acknowledgement References