key: cord-0957988-d2ls41uk
authors: Park, Jinyoung; Chung, Seok Young; Park, Jung Hyun
title: Real-Time Exercise Feedback through a Convolutional Neural Network: A Machine Learning-Based Motion-Detecting Mobile Exercise Coaching Application
date: 2022-01-06
journal: Yonsei Med J
DOI: 10.3349/ymj.2022.63.s34
sha: 78302985b253b809701a203589f54d120faed7f3
doc_id: 957988
cord_uid: d2ls41uk

PURPOSE: Mobile applications are widely used in the healthcare market. This study aimed to determine whether exercise using a machine learning-based motion-detecting mobile exercise coaching application (MDMECA) is superior to video streaming-based exercise for improving quality of life and decreasing lower back pain. MATERIALS AND METHODS: The same 14-day daily workout program consisting of five exercises was performed by 104 participants using the MDMECA and another 72 participants using video streaming. The Medical Outcomes Study Short Form 36-Item Health Survey (SF-36) and lower back pain scores were assess as pre- and post-workout measurements. Scores for the treatment-satisfaction subscale of the visual analog scale (TS-VAS), intention to use a disease-oriented exercise program, intention to recommend the program to others, and available expenses for a disease-oriented exercise program were determined after the workout. RESULTS: The MDMECA group showed a higher increase in SF-36 score (MDMECA, 9.10; control, 1.09; p<0.01) and a greater reduction in lower back pain score (MDMECA, -0.96; control, -0.26; p<0.01). Scores for TS-VAS, intention to use a disease-oriented exercise program, and intention to recommend the program to others were all higher (p<0.01) in the MDMECA group. However, the available expenses for a disease-oriented program were not significantly different between the two groups. CONCLUSION: The MDMECA is more effective than video streaming-based exercise in increasing exercise adherence, improving QoL, and reducing lower back pain. MDMECAs could be promising tools of use to achieve better medical outcomes and higher treatment satisfaction.

utilized in video-streaming apps. Early fitness apps provided only exercise information, and later apps allowed individual exercise data to be digitally accumulated and managed. With the accumulation of enormous digitalized exercise data. Smarter apps that provide feedback on the basis of accumulated individual exercise records have been developed. However, such feedback is limited to information gleaned from moving distances, including number of steps, speeds, and exercise duration and frequency. Strengthening and stretching exercises tend to be relatively neglected in the mobile healthcare market. Meanwhile, several mobile apps that make one-to-one matches between users and remote exercise coaches have emerged in the online market. 4, 5 Individually matched remote coaches can encourage users to exercise and modulate their exercise and lifestyle by using the platform provided in the apps. However, these services are unable cover consumers who do not want to communicate directly with a coach either by contact in person or online. Also, the practical limit to the number of users a coach can cover by remote training is a major commercial limit.

Motion analysis techniques have been used in the medical field to analyze disease-specific gait patterns to determine the causal factors of pathological gait, monitor responses to medical interventions, plan medical interventions, and classify diseases. [6] [7] [8] [9] In the field of sports medicine, motion analysis has been used to monitor the angle and angular velocity of an athlete's limb motion to prevent sports injury or enhance performance. [10] [11] [12] [13] With the development of sensors and Bluetooth devices, obtaining human kinematic data has become more convenient, such that patients can provide their movement data, even data obtained outside the hospital, to physicians in a less cumbersome way. 14 However, to date, medical research on motion analysis has been mainly based on data obtained from medical institutes.

LikeFit (WeHealed, Inc., Seoul, Republic of Korea), a machine learning-based motion-detecting mobile exercise coaching application (MDMECA) used in this study, was developed with the aim of providing accurate exercise guides to users based on convolutional neural network (CNN) technology. This is meaningful in that it provides users access to motion analysis outside hospital-based medical services. To our knowledge, the present study is the first to examine the medical effectiveness of MDMECA. This study was conducted under the hypothesis that patients who exercise using the MDMECA would attain an improved quality of life (QoL), compared with those who exercise using an exercise guide video obtained from an online video-streaming platform.

This retrospective case-control study used an exercise dataset obtained from WeHealed who maintained data on users of Lik-eFit who agreed to provide materials to third parties for medical purposes. Data analysis was conducted by a physiatrist in rehabilitation medicine at a tertiary hospital. The study protocol was approved by an Institutional Review Board (3-2020-0369) and complied with the principles of the Declaration of World Medical Association and Good Clinical Practice. None of the researchers received funding from WeHealed.

In order to compare the clinical outcomes of MDMECA over existing streaming exercise guide videos, a control group was directed to use exercise videos uploaded to a YouTube channel. The participants in each group were recruited from among more than 5000 office workers for a week before the start of the exercise program through an in-house e-mail promotion. Among the 196 participants recruited to the MDMECA group, 104 completed the exercise program using MDMECA. In the control group, among the first 85 participants recruited, 72 answered the post-exercise survey (Fig. 1 ). The participants in the MDMECA group were instructed to install the LikeFit app on their smartphones and perform the exercise program for 14 consecutive days. The participants in the control group were instructed to exercise for 14 consecutive days according to the YouTube videos of the same exercises as those in the MDME-CA group. The participants in each group were instructed not to participate in the other exercise programs.

The key technology of LikeFit is based on the Microsoft Common Objects in Context (COCO) Keypoint Detection Task, which is based on deep learning technology, a CNN. 15 The open-source dataset of more than 250000 people from Microsoft COCO includes labeled human keypoints that are essential in analyzing posture. 15 Through this artificial-intelligence technology, the present MDMECA was developed to analyze realtime human posture by analyzing extracted frame images from sensed motions. The MDMECA uses only a user's built-in smartphone front camera as a motion sensor, without the use of any other health-related devices. The users were instructed to wear clothes that cling to the body and to stand in front of a simple background (Fig. 2) . No more than one person was allowed to be in the camera frame. Referencing the Microsoft COCO Human Keypoint set, 14 keypoints (top of the head, neck, right/left shoulder, elbow, wrist, hip, knee, and ankle) were detected using the built-in camera (Fig. 3A) .

The following metrics were measured in real time during the workout using the obtained inter-related data from the 14 skeleton keypoints (Fig. 3B-D) : 1) segmental length between each keypoint, 2) velocity of each keypoint, and 3) angular velocity of the joint.

The users in the MDMECA group were encouraged in several ways. Over the 14-day exercise challenge program, when a user did not complete a daily workout, the app triggered an encouraging alarm in the late evening for the user to perform the exercise. To provide an accurate exercise guide, audio and visual feedback were given to users, based on a real-time analysis of the user's exercise pattern, through the built-in smartphone speaker and screen ( Fig. 4 and Supplementary Video 1, only online). The exercise data were classified according to the following categories: 1) motion, no motion detected/motion detected; 2) speed, slow/adequate/fast; and 3) range of motion, small/adequate/large. For example, when a user made a correct movement with an adequate speed, a visual alarm signaled "great" with audio feedback (Fig. 4A ). When the user did not bend the knee sufficiently during lunging exercises, a visual alarm signaled "miss, " and audio feedback prompted the user to "bend knee more" in real time (Fig. 4B ). In case of movement different from the movement in the guide video, a visual feedback of "miss" was signaled on the screen (Fig. 4C ). When the user raised their arm too quickly during arm-raising exercises, encouraging audio feedback of "make arm movement slower" was given with the visual feedback "miss. " However, when a user performed the exercise correctly, visual and audio complimentary feedback were provided. In case an exercise set was nearing its end, encouraging audio commentary was given to complete the exercise set.

The analyzed dataset consisted of data from pre-and postworkout surveys and workout performance. The surveys and the workout proceeded as follows: 1) Pre-workout survey. Before beginning the workout regimen, the participants in each group were asked to complete a pre-workout online survey using Google Forms sent via e-mail. The survey items included age, sex, history of surgery, daily sitting time, daily home exercise duration, visual analog scale (VAS) score for low-back pain due to any etiology, and QoL by the Medical Outcomes Study Short Form 36-Item Health Survey (SF-36). [16] [17] [18] [19] 2) Workout process. The participants in the MDMECA and control groups were instructed to perform the same exercise program, which consisted of five different exercises ( Table 1 ). The participants in the MDMECA group followed a 14-day exercise program. After being informed of a brief schematic exercise sequence, the participants were provided with a precise instructional video set in the app. Then, the users exercised while viewing the exercise guide video provided on the smartphone screen and checking their simultaneously filmed exercise motion in one corner of the smartphone screen. The participants in the control group exercised using the same exercise guide video after receiving the instructional video as a YouTube clip. 3) Postworkout survey. After the workout, the participants complete a questionnaire in a post-workout online survey using Google Form. The survey included the following items: exercise days out of 14, daily exercise achievement rate (the percent of exercise achieved to the total daily exercise amount), treatment satisfaction, intention to use a disease-oriented program, intention to recommend the program to others, available expenses for a disease-oriented program, and the QoL for SF-36 and VAS for low-back pain.

The primary outcome was the users' QoL surveyed using SF-36, which was scored on a scale from 0 to 100, with 100 representing the highest level of functioning possible, and the co-primary outcome was VAS score for lower back pain. The secondary outcomes were exercise adherence and treatment satisfac-tion. Thus, the following survey items were used in statistical analysis: daily home exercise duration, treatment satisfaction, intention to use for disease-oriented programs, intention to recommend the program to others, and available expenses for a disease-oriented program. Exercise performance rate was calculated on the basis of post-workout surveys and was calculated by multiplying the actual exercise days out of the total 14 days by the daily exercise achievement rate, which was calculated as the percentage of the workout performed per day for a given amount of exercise: Exercise performance rate=exercise days/14×daily exercise achievement rate.

The basic characteristics of the two groups were compared using an independent t test (age, low-back pain, sitting time, and SF-36 score) and a chi-square test (sex, history of spinal surgery, and daily home exercise duration). The mean SF-36 scores for lower back pain were analyzed using a paired t test within each group, and group differences were compared using a t test by calculating each score change within each group. The scores for treatment satisfaction, intention to use a disease-oriented exercise program, and intention to recommend the program to others were compared between the groups using a t test. The available expenses for a disease-oriented exercise program were compared using the chi-square test. Correlations between the exercise performance rate and outcomes were analyzed using the Pearson correlation analysis.

The baseline characteristics of the MDMECA and control groups are listed in Table 2 . Age, history of spinal surgery, lower back-pain score, daily sitting time, and daily home exercise duration did not show group differences. A significant difference in sex composition was found between the groups (p< 0.01). SF-36 scores were higher in the control group (p<0.01). The SF-36 scores for emotional well-being, social functioning, pain, and general health were higher in the control group.

After the 14-day regimen, the mean SF-36 score significantly increased in the MDMECA group (9.10±10.96, p<0.01), but showed no significant increase in the control group (1.09± 10.14, p=0.37). The change in SF-36 score from before to after workout (ΔSF-36) showed a significant difference between the groups (p<0.01). In sub-item analysis, increases in pre-and post-workout scores for energy/fatigue, emotional well-being, social functioning, pain, and general health were higher in the MDMECA group (Table 3) .

The mean lower back pain score decreased after the workouts in both groups. The mean difference in lower back pain score between pre-and post-workout (ΔVAS) was greater in the MD-MECA group (-0.96±1.82, p<0.01) than in the control group (-0.26±0.71, p<0.01; p<0.01 between the groups) ( Table 3 ).

The survey results for treatment satisfaction are listed in Table 4 . The score for the treatment satisfaction visual analogue scale The proportion of participants who were willing to pay for a disease-oriented exercise program seemed to be higher in the MDMECA group (73.1%) than in the control group (62.5%); however, the difference was not statistically significant. The surveyed available expenses for a disease-oriented exercise program did not differ between the groups (p=0.59).

Only two of the 85 participants (2.4%) in the control group showed a 100% performance rate, but 104 of the 196 participants (53.1%) in the MDMECA group completed 14 days of exercise. Assuming that people with follow-up loss in each group did not exercise at all, the exercise performance rate was clearly higher in the MDMECA group [MDMECA, 53.1% (53.0); control, 31.0% (26.8); p<0.01]. Among all participants from both groups who responded to the post-workout survey, exercise performance rate was negatively correlated with ΔVAS score (r=-0.35, p<0.01) and positively correlated with ΔSF-36 score (r=0.38, p<0.01).

In the control group, the relationship between exercise performance rate and outcomes could be confirmed. Pearson correlation analysis revealed a significant correlation between exercise performance rate and ΔVAS score (r=-0.33, p<0.01), while there was no significant correlation between exercise performance rate and ΔSF-36 score (r=0.14, p=0.23). However, in the sub-item analysis of SF-36, score gaps for energy/fatigue (r= 0.25, p=0.04) and general health (r=0.24, p=0.047) were statistically correlated with exercise performance rate. Further, the exercise performance rate showed positive correlations with treatment satisfaction (r=0.48, p<0.01) , intention to use a disease-oriented program (r=0.39, p<0.01), and intention to recommend the program to others (r=0.43, p<0.01). 

The MDMECA was more effective than video streaming-based exercise in relieving lower back pain and improving QoL. Although whether the exercise performance rate and ΔVAS or ΔSF-36 scores are related could not be confirmed within the MDMECA group, we found that the higher the exercise performance rate, the greater the reduction in lower back pain and the better the QoL in all of the participants from both groups who responded to the post-workout survey. The greater reduction in lower back pain scores and improvement in the MD-MECA group may have resulted from a higher exercise performance rate, in addition to the machine-learning-based motiondetecting technique and feedback on making correct posture and motion. Furthermore, the higher exercise performance rate in the MDMECA group may have resulted from the individualized feedback system.

The scores for TS-VAS, intention to use a disease-oriented exercise program, and intention to recommend the program to others were higher in the MDMECA group than in the control group. This suggests that MDMECA holds the potential to spur an influx of new users or the use of new programs by existing users than video streaming services. However, the available expenses for a disease-oriented exercise program were surveyed and showed no significant difference between the two groups. However, this may be due to the cost ranges being too wide or an insufficient number of users for cost estimation. This suggests that service providers may have to consider options other than direct app-based service sales to generate revenue and to attain price competitiveness over other mobile exercise services.

The pre-workout baseline SF-36 scores of the participants in both groups showed relatively higher scores for the following sub-items than other sub-items: physical functioning, role limitations due to physical health, and role limitations due to emotional problems. This implies that those who applied to use the digital exercise service were characterized by little difficulties in physical or emotional functioning in daily life and social relationships. On the other hand, the participants in both groups showed relatively lower scores for the following sub-items: energy/fatigue, emotional well-being, social functioning, pain, and general health. These scores were lower in the MDMECA group. After workout, the subscores improved in both groups. This may be simply the result of service use, but it may also be an indicator that the users attained their goal of service use. However, because the purpose of using the service has not been investigated in advance, the actual goal of the users could not be evaluated.

The strength of the MDMECA lies on the use of motion-detecting technology based on machine learning. This enables the ability to provide individualized, real-time feedback to users by audio and visual cues. Machine learning-based motion-detecting technology is thought to cover both the limitations of oneto-one matching from the perspective of service providers, which increases the price of service production, and the lack of individualized feedback in video streaming services from the perspective of service users. Users can perform the exercise at a convenient time of the day without being constrained by an appointed time and place. Although this technology cannot completely replace in-person feedback, by providing customized feedback through motion detection, the service provider can reduce labor costs. Therefore, the service can be used at a relatively low cost.

Similar to other home-based exercise apps, the MDMECA has the advantage in that it allows the performance of exercises in a personal space. When public fitness facilities become unavailable, the use of this app can be more efficient, especially for those who need feedback and encouragement during exercise. This app is distinct in that it enables individual feedback during exercise and has an alarm function to increase exercise adherence.

This study has a limitation in that the number of participants in the MDMECA group and the control group did not match 1:1. In the MDMECA group, since the post-workout surveyed data were collected from the participants who completed the 14-day exercise program, the relationship between exercise performance rate and other outcomes could not be analyzed. The surveyed available cost ranges for the MDMECA services may be too wide to estimate the exact cost. In addition, the survey lacked an analysis of whether the aim of the service user and the actual gait matched.

In conclusion, the MDMECA is more effective than video streaming-based exercise in increasing exercise adherence, improving QoL, and reducing lower back pain. 

A home-based exercise program for nondisabled older adults

Motivating patients with shoulder and back pain to self-care: can a videotape of exercise support physiotherapy?

The effect of a home based exercise intervention package on outcome in ankylosing spondylitis: a randomized controlled trial

Pennington L. Feasibility of parent communication training with remote coaching using smartphone apps

Effectiveness of a home-based rehabilitation program after total hip arthroplasty driven by a tablet app and remote coaching: nonrandomized controlled trial combining a single-arm intervention cohort with historical controls

Using gait analysis' parameters to classify Parkinsonism: a data mining approach

Gait analysis alters decision-making in cerebral palsy

Gait disturbances in patients with stroke

Evaluation of gait in Duchenne Muscular Dystrophy: relation of 3D gait analysis to clinical assessment

Biomechanical analysis of the "running" vs. "conventional" diagonal stride uphill techniques as performed by elite cross-country skiers

Footfall patterns of a runner with an Achilles tendon rupture

Baseball pitching kinematics, joint loads, and injury prevention

Determine an effective golf swing by swing speed and impact precision tests

Wristbands containing accelerometers for objective arm swing analysis in patients with Parkinson's disease

Microsoft coco: common objects in contex

Validating the SF-36 health survey questionnaire: new outcome measure for primary care

Standardized rehabilitation and hospital length of stay among patients with acute respiratory failure: a randomized clinical trial

Effect of a home-based exercise intervention of wearable technology and telephone coaching on walking performance in peripheral artery disease: the HONOR randomized clinical trial

Self-reported physical activity is associated with lupus nephritis in systemic lupus erythematosus: data from KORean lupus network (KORNET) registry