key: cord-0047213-mj7b5p45
authors: Takahashi, Ryo; Ogawa-Ochiai, Keiko; Tsumura, Norimichi
title: Visualizing Blood Flow of Palm in Different Muscle Tense State Using High-Speed Video Camera
date: 2020-06-05
journal: Image and Signal Processing
DOI: 10.1007/978-3-030-51935-3_20
sha: 7f4f39d0b1958948d5ad8844e3b2856258f5064a
doc_id: 47213
cord_uid: mj7b5p45

In this paper, we propose a method to visualize blood flow of palm in different muscle tense state using RGB high-speed camera. Recently, new modalities are needed to develop a more accurate system to non-contact multi-modal affect analysis. Then, we focus on muscle tense. The muscle tense is caused by stress. Hence, the muscle tense is one of the effective modalities for non-contact multi-modal affect analysis. However, it is very difficult to measure muscle tense in the real environment because it requires a contact-type sensor. Therefore, we use iPPG to visualize the pulse wave during muscle tense from the skin video taken with the RGB video camera. As a result of this experiment, we found that it was possible to recognize the difference in pulse wave during muscle tense from the video that visualized the pulse wave. From this result, the realization of non-contact measurement of muscle tense can be expected.

Non-contact methods for affect analysis have been developed in recent researches [1] [2] [3] . The methods use various modalities such as facial expression, voice, and heart rate. However, these modalities are less for stress assessment. We assume that muscle tense can be used as one of the modalities because the muscle tense is caused by stress. When a person to be state of nervous, it is known that it causes tense various parts of the body. By detecting this muscle tense state by any method, stress check and concentration measurement are possible. Therefore, measuring muscle tense is very important.

Currently, methods such as visual inspection and palpation by technicians are used to check the degree of muscle tone. However, differences occur for each technician in this method. Moreover, it is difficult to secure a technician and diagnosis is difficult because the method is based on experience of technician. In addition, since contact-type measurement requires a dedicated device, it is difficult to realize it in general applications such as stress monitoring and measurement of concentration. From such a background, non-contact measurement of muscle tense is required.

In this research, therefore, we propose a non-contact, non-invasive, and low restrictive method of measurement for muscle tense state. And we verify that it is possible to measure muscle tense using this method. In order to realize this non-contact measurement of muscle tense, we focus on the blood flow. It is known that blood flow stagnates because muscles press blood vessels in muscle tense state. In previous studies, we proposed visualization of blood flow using an RGB camera to facilitate examination of diseases such as arteriosclerosis. Hence we thought that it was possible to distinguish from blood flow if part of the body was in a state of muscle tone using visualization of blood flow. 

In the proposed method, pulse wave is acquired from RGB video and visualized the pulse wave is created [4] . Then, we verify that whether it is possible to distinguish between muscle tense and not muscle tense from the video. 

Pulse waves are blood pressure and volume changes in the peripheral vasculature associated with the beating of the heart. In general, a device called a contact type photo plethysmography is used to acquire a pulse wave. This device is attached to a fingertip and acquires reflected light by irradiating a fingertip with green light. Since hemoglobin in the blood absorbs this green light, it is possible to measure the pulse wave by measuring the intensity of the reflected light. However, in pulse wave acquisition by a photo plethysmography (PPG), it is necessary to contact the skin, and it is not possible to acquire spatial pulse wave distribution in the skin region.

A method called image-based photo plethysmography (iPPG) has been proposed as a method of acquiring this pulse wave without contact [5] . In this method, we focus on the fact that the G component of the video of the skin taken with the RGB camera is correlated with the blood volume. The method obtains pulse waves from time-series changes of G component. However, this method is not robust to fluctuations in illumination light and body movement, and the noise of acquired pulse waves is very large. Hence, many improved iPPG methods were proposed for robust measurement [6, 7] . In this paper, we use a new method of iPPG using a method called pigment component separation [8] . In this method, first, hemoglobin components are extracted from RGB video using pigment component separation. Since the blood volume of the skin changes in association with the heart beats, it is possible to estimate the pulse wave by calculating the average pixel value of the hemoglobin component video in the skin area of the face or palm [9] . Since the pulse wave estimated by this method contains much noise, we need to extract only the band of 45 to 180 [bpm], which is the human heart rate using the band pass filter. The pulse wave can be acquired with high accuracy by this process. Figure 1 shows procedure of acquiring pulse wave from RGB video.

In order to visualize muscle tone, it is necessary to visualize the blood flow state. At first, the skin video is divided into small regions such as 4 × 4 [px], and the hemoglobin component is extracted using pigment component separation to acquire the pulse wave. After that, pulse waves are acquired in the abovementioned procedure from each divided area. The area in video is painted with pulse wave extracted from each area. The area is painted red in the high area of the pulse wave and blue in the low area. Figure 2 shows the correspondence between the pulse wave amplitude and the color map. By above processing, it is possible to create a pulse wave visualized video. Figure 3 shows the result of analysis of the video of the face and the palm. In addition, Fig. 3(a) shows the time change corresponding to one beat of the face area, and Fig. 3(b) shows the same time change of the palm area. It can be seen from Fig. 3(a) that in the area of the face, the area corresponding to the peak of the pulse wave transitions from the chin to the forehead. It is known that blood vessels in the face extend from the neck to forehead. This structure is consistent with the experimental results. It is considered that the experimental results are reasonable. It can be confirmed from Fig. 3 (b) that the pulse wave is propagating around the blood vessel located at the shallow place also in the palm region. From the above results, it is thought that visualization of pulse wave propagation could be realized.

In the previous section, it is verified whether muscle tension can be identified by visualization of blood flow by using the pulse wave visualization method introduced. In this experiment, subjects do two patterns experiments to make muscle tone. The first is the action of biting teeth. The Subjects are asked to bite only one side tooth to make muscle tone. We take a video of the face at this time. The second is the action of applying power only in one hand. The muscle tone is made by applying power only in one hand. At this time, take a video of the hand. It is thought that the muscles of the jaw or palm that are putting in force contract and blood flow stagnates. We visualize the blood volume of the face and palms at this time, and check whether it is possible to distinguish the person who is applying power in the teeth or hand.

In the experiment, we use a high-speed camera in the dark room, and we take a video of non-muscle tone and muscle tone state on the face and palm region. The time of taking video is 4 [s]. At this experiment, we set the frame rate of the high-speed camera 1000 [fps], and the resolution 1280 × 1024. Subject's head and palms are fixed on a base and we take a video in a stationary state. Figure 4 shows the experimental environment. We use a high speed camera MEMRECAM Q1m (nac Image Technology Inc.). It can take a video up to 2000 [fps] and 1280 × 1080 [px]. We also use two sets of artificial sun-lights SOLAX 100W (SERIC LTD.) as illumination. Figure 5 shows the time change of the visualized pulse wave of the face. Figure 5 (a) shows a state in which the teeth are not bitten and muscle tension is not present. It can be shown in Fig. 5(a) that there is no difference in the propagation of the pulse wave in a state where muscle tone is not generated. And it was shown in Fig. 5 (b) that no difference was observed in the pulse wave of the face even in the muscle tension state. This is thought to be due to the difference in the position of the masseter muscle and the facial artery. Figure 6 shows the time change of the visualized pulse wave of the face. is a state in which only the left hand is muscle tone. It can be shown in Fig. 6 (a) that there is no difference in the propagation of the pulse wave in a state in which not muscle tone. And it was shown in Fig. 6 (b) that the phases of pulse waves in part of the palm are largely different in the muscle tone state. Figure 7 shows enlarged view of 0.4 [s] of Fig. 6 . It can be easily confirmed that the red and blue parts are mixed in the muscle tone state from Fig. 7 . Focusing on a portion where the phase is different, it can be seen that only the blood vessel portion is different in phase. On the other hand, the area of blue and red decreases during muscle tone because the amplitude of pulse wave decreases. This is thought to be because blood vessels were compressed by muscle tone. From above result, we found that it was effective to measure the phase difference of the pulse wave in the palm for the discrimination of the muscle tone state by visualizing the pulse wave propagation.

In this study, we estimate pulse wave by tracking the time change of hemoglobin component extracted using skin pigment separation. And the pulse wave propagation was visualized in a non-contact by painting the region of video according to the pulse wave value. In addition, it was confirmed that a phase difference is generated in the palm pulse wave in the muscle tone state by using visualization of pulse wave propagation. In the future, we will aim at the quantitative evaluation of muscle tone status using dispersion of phase difference of pulse wave. It will be expected that this makes it possible to determine chronic muscle tone and the like due to stress in the future.

Hybrid feature and decision level fusion of face and speech information for bimodal emotion recognition

Predicting Ad liking and purchase intent: large-scale analysis of facial responses to Ads

Advertisement effectiveness estimation based on crowdsourced multimodal affective responses

Spatial visualization of pulse wave propagation using RGB camera

Remote plethysmographic imaging using ambient light

Motion robust remote photoplethysmography in CIELab color space

Improvements in remote cardiopulmonary measurement using five band digital camera

Non contact video based estimation of heart rate variability spectrogram from hemoglobin composition

Image-based skin color and texture analysis/synthesis by extracting hemoglobin and melanin information in the skin