International Journal of Advanced Network, Monitoring and Controls Volume 04, No.04, 2019 DOI: 10.21307/ijanmc-2019-067 24 Detection of Blink State Based on Fatigued Driving Lei Chao School of computer science and engineering, Xi’an Technological University Xi’an, China E-mail:callofduty2015@163.com Wang Changyuan School of computer science and engineering, Xi’an Technological University Xi’an, China E-mail:cyw901@163.com Li Guang School of computer science and engineering, Xi’an Technological University Xi’an, China E-mail:865413666@qq.com Shi Lu School of computer science and engineering, Xi’an Technological University Xi’an, China E-mail:892178238@qq.com Abstract—In recent years, with the improvement of the national economy, the penetration rate of automobiles has been increasing, and traffic accidents have also increased. Fatigue driving is the main factor in many traffic accidents. Fatigue driving can cause the driver's inattention, slow response, and make wrong decisions on danger signals, which affect the driver's personal safety. In modern development, driving safety is developing towards intelligence and safety. Therefore, the detection of driver fatigue has become a generally accepted demand. This paper proposes a method to calculate the threshold of blinking, which can detect the blinking state of the driver in real time through video. During the driving process, when the driver is in the closed eye state for a long time, an early warning is issued to avoid the accident. This paper uses Python language to achieve the first, through the digital image technology call Dlib open source library to detect 68 feature points of the face, and then measure the aspect ratio between the length and width of the human eye, and finally through the Kmeans clustering algorithm to collect the ratio The analysis yields the blink threshold. The experimental results show that the recognition rate is 92.5% when the video frame rate is 30, and the recognition accuracy is 92.5%. The experimental results show that the method designed in this paper can quickly detect the fatigue characteristics of the human eye, has a higher recognition rate and accuracy for fatigue driving, and helps reduce the occurrence of traffic accidents. Keywords-Blinking Algorithm; Fatigue Detection; Digital Image Processing; Clustering Algorithm; Key Points Of Human Eyes I. INTRODUCTION With the improvement of people's material living standards, cars have become the main means of transportation for people, but the growing number of vehicles has led to more traffic accidents. According to statistics, fatigue driving is the main cause of traffic accidents[1,2].Under normal circumstances, the medical community believes that there are two reasons for fatigue driving, one is because the driver's attention is too concentrated, and the other is that the body does not rest well. Because of being in this state for a long time, the body will be fatigued, lose concentration, the driver will snoring, lose concentration, decrease the ability to judge dangerous situations, and cause traffic accidents. At present, there are relatively few applications of fatigue driving equipment in China's in-vehicle systems. Fatigue detection mainly through facial features, eye and mouth features, human electrical signal characteristics and convolutional neural network characteristics[3,4,5].The detection of facial features is generally based on the frequency of blinking eyes, the degree of mouth opening, and the frequency of head movements due to fatigue. The fatigue of human body electrical signals is generally the measurement of surface EMG signals, because human fatigue can be expressed by muscle physiological information. Surface EMG signals can reflect real-time physiological processes of muscle information and physiological signals on the skin surface. Convolutional neural networks generally extract facial features through image processing methods, and then extract the main features through convolutional layers, pooling layers, and fully connected layers to analyze and determine whether fatigue. Chen[6] uses the ASM algorithm to accurately locate the eyes and mouth area, calculates the eye's aspect ratio, mouth height value, and black and white pixel ratio near the mouth, and obtains the blink frequency and mouth opening degree. The degree of mouth opening is used as an input to the fuzzy International Journal of Advanced Network, Monitoring and Controls Volume 04, No.04, 2019 25 inference engine to obtain three types of fatigue levels to accurately quantify the degree of fatigue The method proposed in this paper is to judge the driver's fatigue driving according to the characteristics of the human eye. Because the digital image processing open source visual library OpenCV comes with a human face detection library, but the disadvantage is that the lighting requirements are very high, the lighting slightly changed, it will be difficult to locate or inaccurate positioning[7]. Therefore, this paper chooses Dlib open source library to detect human eye features. Firstly, the 68 face feature points provided by the Dlib open source library are used to accurately calibrate the position of the face and the human eye, and then the aspect ratio between the length and the width of the human eye is measured. Finally, the Kmeans clustering algorithm is used to analyze the collected ratio. The threshold of blinking. Figure 1 below, a is the 68 face feature points marked by Dlib, and b is the feature point on the face of the paper. (a) 68 feature points of a face annotation (b) Recognition of face images Figure 1. Facial feature points II. RELATED WORK A. Blink detection and threshold analysis methods This chapter mainly introduces the blink algorithm formula and blink threshold analysis method. The blink threshold analysis method uses the Kmeans clustering algorithm in machine learning. There are many methods for blink detection, such as support vector machine classification, eye movement sequence analysis, convolutional neural network feature extraction, eye feature point analysis, etc. This article uses the eye feature analysis method. Threshold analysis methods in machine learning usually use regression algorithms, decision tree methods, Bayesian methods, and clustering algorithms. This article uses the Kmeans clustering algorithm in machine learning. B. Blinking formula There are currently many methods in the field of blink detection. Andrej Fogelto et al[8] analyzed the relationship between the speed of blinking and the duration of time through the cyclic neural network (RNN), so as to better distinguish the state of blink and blink, through the comparison of one-way and two-way circulating neural networks. One-way neural networks work better in blink detection. However, neural networks have their own limitations. For example, network merging will lose a lot of information. In the field of face recognition, the relationship between part and whole is neglected, and each person's face features are different, so it takes a lot of time to train. parameter. RenAnhu et al[9] trained the blink classifier through the AdaBoost algorithm, but the AdaBoost algorithm is very sensitive to the discrete data of the blink. The detection of the eye part during image processing is easily affected by the speed of light and object movement, just as the blinking behavior is fast. the process of. ZengYouwen et al[10] used the correlation between computer signals and the number of blinks to determine the fatigue state, but the experiment required special equipment and equipment, and the operation was difficult and difficult to implement. Since the blinking algorithm proposed by Soukupová[11] has measured the blink threshold of the aspect ratio of the eye, using the support vector machine[12] method to analyze the collected threshold of the blink to finally obtain an EAR of 0.2, the algorithm of this paper is the In addition, by measuring the aspect ratio of the eye, the Key clustering algorithm in machine learning is used to obtain the blink threshold. As shown in Figure 2.1a, the absolute value of the longitudinal distance ab of the eye will become smaller when the eye is blinking. At this time, the ratio of cd to ab will suddenly become larger, so we analyze the threshold when the ratio International Journal of Advanced Network, Monitoring and Controls Volume 04, No.04, 2019 26 becomes larger when blinking. Figure 2a shows the distance between ab and cd, and b is the position of the human eye feature point marked by Dlib. This paper proposes the blink threshold formula as follows:  5362 412 sho pppp pp ldBlinkthre        (a)  (b) Figure 2. (a) The lateral distance is cd longitudinally ab; (b) dlib human eye calibration features C. Kmeans clustering algorithm The Kmeans algorithm is a relatively common algorithm in clustering algorithms. Its advantage is that it is easy to implement and understand, and the calculation speed is fast. The core idea is to calculate the distance between the sample point and the centroid of the cluster, and divide the calculated result into the same cluster as the sample point with the centroid of the cluster. The similarity between samples in K-means is determined by the distance between them. The closer the distance is, the higher the similarity is. The common distance calculation methods are Euclidean distance, Euclidean distance and Manhattan distance. European distance. In the cluster analysis, the formulas for two m-dimensional samples xi=(xi1,xi2,xi3...,xim) and xj=(xj1,xj2,xj3...,xjm) are as follows:     m k jkiked xxt 1 2 )(dis   The steps of the k-means algorithm are as follows: 1) First randomly select the centroids of K clusters. 2) Calculate the Euclidean distance from each sample point to each centroid, and classify it into the cluster with the smallest center of mass, and then calculate the centroid of each new cluster. 3) After all the sample points are divided, recalculate the position of the centroid of each cluster, and then iteratively calculate the distance from each sample point to the centroid of each cluster, and then re-divide the sample points. 4) Repeat steps 2 and 3 until after the iteration, the partitioning of all sample points remains unchanged, and K-means gets the optimal solution. The main problem of the calculation result is to ensure the convergence of the algorithm. Here, the square error is calculated by the following formula, which is used to illustrate that the clustering effect can minimize the sum of squares in each cluster.    2 1i )( )( ,    K ic i uxucJ    uc,j represents the sum of squares of the distance from each sample point to its cluster, )(u ic represents the centroid of the cluster to which the i-th sample belongs, and the smaller  uc,j , all the sample points and their clusters The smaller the distance, the better the quality of the division. The termination condition of the K-means algorithm is that  uc,j converges to a minimum. In order to achieve clustering, the maximum value of the objective function is obtained. Take a one-dimensional array as an example. J = ∑ ∑ (x(i) − uc(i)) 2 xj∈ui k i=1   Transform the above formula to get: 𝜕𝐽 𝜕𝑢𝑖 = 𝜕 𝜕𝑢𝑖 ∑ ∑ (𝑥𝑗 − 𝑢𝑖 ) 2 𝑥𝑗∈𝑢𝑖 𝑘 𝑖=1 International Journal of Advanced Network, Monitoring and Controls Volume 04, No.04, 2019 27 = ∑ ∑ (𝑥𝑗 − 𝑢𝑖 ) 2 𝑥𝑗∈𝑢𝑖 𝑘 𝑖=1 = (−2) ∗ ∑ (𝑥𝑗 − 𝑢𝑖 ) 𝑥𝑗∈𝑢𝑖 When    ij u ij ux x 0)(*2- ）（ ui = 1 |ci| ∑ xjxj∈ui The result of the optimization is to calculate the mean of the cluster. During the experiment, the algorithm may be too slow to achieve effective results because the data set is too large. Therefore, you can specify the maximum number of convergence times for the K-means algorithm or specify the cluster center transformation threshold. When the algorithm reaches the maximum number of times or When the cluster center rate of change is less than a certain threshold, the algorithm stops updating. K-means algorithm advantages: easy to understand, easy to implement, high operating efficiency, the disadvantage is that the greedy strategy is used to cluster the sample points, resulting in easy local convergence of the algorithm, slower data processing in big data, and outliers and The noise is very sensitive, and a small number of outliers and noise points can have a significant impact on the averaging of the algorithm. III. THE EXPERIMENT This article mainly introduces two aspects, one is to introduce the source of the experimental data set, and the other is to analyze the experimental data.Figure 3 shows the author's own data collection and analysis. From left to right, the first picture is the analysis of the blink value of the author's 5 blinks, and the middle figure is when the blink threshold is 5.1 when Kmeans = 2 The blink data graph, the right graph is the actual record of the author's experiment. Figure 4 shows the data of a random experimental sample in the public data set. From left to right, the first figure analyzes the blinking value of the aspect ratio of the 3 blinks in the experimental sample. The blink data graph at 5.1, the right graph is the actual record of the experimental sample.This article uses the blink data set provided by Zhejiang University[13]. All the data in the data set is collected under the natural light of the room. There is no special illumination. The collected equipment is a common camera that comes with the computer. There are 80 video clips and 20 people participate. , four segments per person, these four segments represent a. front view without glasses; b. front view with thin edge glasses; c. front view with black frame glasses; d. View. Participants spontaneously blink at the front of the camera at normal speed. The size of each video is 320*240, the frame rate of the video is 30fps, and the duration of recording is generally about 5 seconds. The experimenter usually blinks about 1 to 6 Times, a total of 255 blinks. The following are the results of data collection and analysis. The text person has 5 blink-eye aspect ratio data Kmeans=2, the blink data chart The text person image Figure 3. The following is the experimental data of the paper International Journal of Advanced Network, Monitoring and Controls Volume 04, No.04, 2019 28 Public data set 3 times blinking eye aspect ratio data Kmeans=2, blinking data chart Public data set sample Figure 4. Public data set sample The following table a is a comparison of the public data sets provided by Zhejiang University and the experimental results of the text person. Table b is a comparison of other methods with the method of this paper. RenAnhu[9] trained the classifiers of blinking and closed eyes through the Adaboosts algorithm. The person in the video is then tested for blinking. Zhang Wei[14] performed a correlation analysis of the blink of the eye by analyzing the left forehead EEG signals Attention and Meditation and Blink data. TABLE I. COMPARED WITH PUBLIC DATASETS Experimental sample 𝐁𝐥𝐢𝐧𝐤 𝐭𝐡𝐫𝐞𝐬𝐡𝐨𝐥𝐝 Blink times Number of recognition Recognition rate Text person 5.1 100 90 90% Public data set 5.1 255 236 92.5% TABLE II. COMPARED WITH OTHER LITERATURE literature Recognition rate 9 91.5% 14 83.7% This article 92.5% IV. CONCLUSION This paper overcomes the shortcomings of digital image processing and OpenCV vision open source library, and combines the existing open source Dlib machine learning library,The data between the vertical and horizontal ratio of blink is calculated by mathematical method, and the threshold value of the vertical and horizontal ratio of blink is analyzed by means of kmeans clustering algorithm in machine learning. According to the analysis of the public data set of Zhejiang University, when the threshold value of the vertical and horizontal ratio of blink is 5.1, the accurate recognition rate of blink is 92.5%. Through the experimental comparison, this algorithm can effectively detect the fatigue state of blink, which is more important This algorithm is fast, efficient and easy to transplant to various devices, and has great practical value in the field of fatigue driving. The shortcomings of the paper: for fatigue monitoring, not only eyes as a reference point, nose tip shaking, mouth opening and so on have an impact on face fatigue, so the fatigue detection algorithm in this paper needs to be improved. REFERENCES [1] M. Hülsmann,D. Donnermeyer,E. Schäfer. A critical appraisal of studies on cyclic fatigue resistance of engine‐driven endodontic instruments[J]. International Endodontic Journal,2019,52(10). [2] Pierre Thiffault,Jacques Bergeron. Monotony of road environment and driver fatigue: a simulator study[J]. Accident Analysis and Prevention,2003,35(3). International Journal of Advanced Network, Monitoring and Controls Volume 04, No.04, 2019 29 [3] Liu Longfei, Wu Shizhen, Xu Wangming. Real-time detection method of fatigue driving based on face feature point analysis[J]. Television Technology, 2018, 42(12): 27-30+55. [4] Yan Wang,Rui Huang,Lei Guo. Eye gaze pattern analysis for fatigue detection based on GP-BCNN with ESM[J]. Pattern Recognition Letters,2019,123. [5] Driver’s Fatigue Detection Based on Yawning Extraction[J]. Nawal Alioua,Aouatif Amine,Mohammed Rziza,Aboelmagd Noureldin.International Journal of Vehicular Technology. 2014 [6] Chen Xin, Li Weixiang, Li Wei, Zhang Wenqing, Zhu Yuan. Multi-feature fusion fatigue detection method based on improved ASM [J]. Computer Engineering and Design, 2019, 40 (11): 3269-3275. [7] Rafael C.Gonzalez,Richard E.Woods.Digital Image Processing,Third Edition[M],2017 [8] Andrej Fogelton,Wanda Benesova. Eye blink completeness detection[J]. Computer Vision and Image Understanding,2018. [9] Ren Anhu, Liu Bei. Face Recognition Blink Detection Based on Adaboost[J]. Computer and Digital Engineering, 2016, 44(03): 521-524. [10] Zeng Youwen, Feng Zhen, Zhu Yabing, Li Qi.Relationship between the number of blinks and fatigue based on EEG experiment[J].Journal of Changchun University of Science and Technology(Natural Science Edition),2017,40(01):123-126. [11] Tereza Soukupova ,́Jan Cˇ ech, Eye blink detection using facial landmarks[J]. 21st Computer Vision Winter Workshop(CVWW),2016 [12] J. Manikandan,B. Venkataramani. Study and evaluation of a multi-class SVM classifier using diminishing learning technique[J]. Neurocomputing,2009,73(10). [13] F.Song, X.Tan, X.Liu and S.Chen, Eyes Closeness Detection from Still Images with Multi-scale Histograms of Principal Oriented Gradients, Pattern Recognition, 2014. [14] Zhang Wei, He Jian, Zhang Yan, Zhou Ming. A wearable fatigue driving detection system based on EEG and blink frequency[J]. Computer Engineering, 2017, 43(02): 293-298+303. https://kns.cnki.net/kcms/detail/detail.aspx?filename=SJHD14081900000343&dbcode=SJHD