key: cord-1054396-9eclqa5w
authors: Yanyan, Fan; Zhuoxin, Wang; Shanshan, Deng; Hekai, Lv; Fuzhi, Wang
title: The function and quality of individual epidemic prevention and control apps during the COVID-19 pandemic: A systematic review of Chinese apps
date: 2022-01-31
journal: Int J Med Inform
DOI: 10.1016/j.ijmedinf.2022.104694
sha: 718a10536cfb558779d29488af6158e1eeb35246
doc_id: 1054396
cord_uid: 9eclqa5w

BACKGROUND: Coronavirus disease 2019 (COVID-19) has caused a global pandemic that has driven the widespread use of applications (apps) for outbreak management in China, but the characteristics and quality of these apps are currently unknown. OBJECTIVE: The first objective of this study was to investigate the functional characteristics of individual epidemic prevention and control apps in China, and the second objective was to evaluate the quality of these apps. METHODS: We searched the Qimai(TM) mobile application data analysis platform and the Aladdin(TM) WeChat applet data analysis platform with keywords and quantified the search results based on the search index, relevance, and the Aladdin index to identify apps with high public popularity. The quality of the apps was rated by 2 independent raters using the Mobile App Rating Scale (MARS). The intraclass correlation coefficient (ICC) between raters was used as a measure of interrater reliability. RESULTS: All 20 of the included apps had acceptable quality. Functionality had the highest score, followed by information quality, aesthetics, and engagement. There were no significant differences between the independent apps and WeChat applets in app quality (t=1.907, p=0.073) and subjective quality (t=0.899, p=0.381). These apps were related to COVID-19 individual prevention and control, and the functional features that contributed to the quality of the apps were grouped into six categories, i.e., health self-checking and reporting, news about COVID-19, scientific publicity and education, telemedicine services, personal travel inquiries, and digital contact tracing. CONCLUSIONS: Individual COVID-19 prevention and control apps in China were developed by adding epidemic prevention and control functions to existing social apps rather than independently developing apps. The overall quality of such apps was acceptable, but scores in the engagement section were generally low, especially for WeChat applets.

currently available COVID-19 apps. For example, the overall quality of these apps related to COVID-19 prevention and control in India, the USA, and the UK was inconsistent, with high functionality scores, low engagement scores, and app downloads unrelated to quality [19] . When evaluating COVID-19 contact tracking apps in Europe, it was found that these apps scored highly in terms of functionality, aesthetics, and information and that there was little difference among them.

In addition, there is a direct link between app quality and adoption rates, which in turn were associated with lower infection rates [26] .

During the COVID-19 outbreak, many emergency apps for prevention and control were developed in China. However, there is still a lack of empirical research to evaluate the quality of these apps. Therefore, the first objective of this study was to investigate the functional characteristics of individual epidemic prevention and control apps in China, and the second objective was to evaluate the quality of these apps.

Given the current situation of epidemic prevention and control app usage in China, a systematic search of the Qimai TM app data platform and the Aladdin TM WeChat applet data platform from 21 to 25 January 2021 was conducted, following the PRISMA systematic literature review guidelines, and a checklist is provided in Multimedia Appendix A [27] . The apps were obtained as follows:

a. App Data Analysis Platform: The well-known domestic mobile app data analysis platform Qimai TM was used to retrieve apps related to COVID-19. The platform provides data related to all apps in the App Store, Google Play, and 9 major domestic Android markets (Huawei, Xiaomi, OPPO, Vivo, Meizu, Baidu, 360, App Store, and Pea Pod); the platform also provides tools for keyword expansion, app store optimization (ASO), and intelligent matching of association terms [28] , which assisted in obtaining data in this study. The keyword expansion tool uses big data analysis to find additional keywords with relevance based on the keywords entered, and the platform also provides relevance ranking and search index values. The greater the relevance, the better the results of the keyword selection. Regarding the keyword search index, also known as search popularity, the larger the search index was, the more people searched for that particular term, and the greater the search popularity of a certain topic. In this study, we selected apps that met the requirements by selecting keywords with more than 50% relevance to the keyword "xin guan" (new crown) and combining them with a search index ≥ 4605 (usually, keywords with a search index ≥ 4605 have a large search volume). The search date was January 25, 2021.

b. WeChat Applet Data Analysis Platform: The WeChat applet is a mobile application developed by Tencent based on the WeChat platform that does not require downloading, installation, or registration; the applet is unique because of its development simplicity, ease of use, and direct accessibility to the public, becoming the "new favourite" of people affected by fast-paced, large public emergencies and security events [29] . We retrieved WeChat applets based on the Aladdin Index, which is an index developed by Beijing Aladdin Future Technology Co., Ltd., that specifically evaluates the influence of WeChat applets. This influence is calculated by daily activity, some shares, length of stay, depth of visit, and other WeChat applet indicators [30] . In this study, we applied the keywords "Yiqing" (epidemic), "Fangyi" (epidemic prevention), "Kangyi" (antiepidemic), "Zhanyi" (war on epidemic), "Yikuang" (epidemic status), "Yifang" (epidemic prevention), "health punch card", "isolation and prevention and control", "joint prevention and control", "new crown", "new pneumonia", "new coronavirus", and "new prevention and control".

We searched all WeChat applets related to COVID-19 in the Aladdin applet list. The search date was January 21, 2021. c. Supplementary Approach: Using snowball sampling, 20 undergraduates from a university and 10 employees from a public institution were selected as seeds, and their school or work unit asked them what kind of COVID-19 apps they had used in the past year. They were invited to recruit others through WeChat or QQ workgroups and their interpersonal network to identify COVID-19 apps that were widely used in their organization but were not identified by the above two search strategies. Feedback information was provided through WeChat and SMS messaging. The feedback period was from December 24, 2020, to January 24, 2021.

The app exclusion criteria were as follows: ① no health self-checking and reporting or tracking function; ② non-simplified Chinese language; ③ availability to only internal personnel of the organization (the internal account must be provided to assess the relevant function); and ④ advertising and product sales as the main purpose. The app screening process is shown in Figure 1 (see Multimedia Appendix B for screening details). 

This study was conducted to rate the quality of apps using the MARS, which consists of four parts. The first part (app classification) is used to collect descriptive and technical information in the App Store. The second part (app quality ratings) measures app quality considering four dimensions: section A (engagement), section B (functionality), section C (aesthetics), and section D (information), with a total of 19 items. The third part (subjective app quality) has 4 items in total.

Finally, the app-specific portion includes a total of 6 items. These items are rated on a Likert scale Apps screened by accessing and using the apps after they were downloaded and installed (n=208) (1=inadequate, 2=poor, 3=acceptable, 4=good, 5=excellent), and an average MARS score ≥ 3 points (out of 5) is considered to be of "acceptable" quality [31] .

The review process consisted of three steps. First, descriptive and technical information on the 20 apps was collected from the app stores (Apple Store & Android Market) and the WeChat applet platform (see Multimedia Appendix C). Second, a pre-assessment was conducted by three reviewers.

Before conducting the assessment, three reviewers watched an instructional training video provided by the MARS developers and discussed and reached a consensus on the content of the questions [32] .

Then, three apps and three WeChat applets were randomly selected for a review exercise and score discussion; the apps were reassessed by a third reviewer if any disagreements arose. Third, the quality of the 20 included apps was evaluated using MARS. All eligible apps were installed on smartphones (Android device: HUAWEI P30 Pro model VOG-AL00, version 10; iPhone device:

iPhone 6S model A1700, version: 11.41). Then, to ensure consistency of the evaluation results, the evaluation of the sub-dimensions was completed simultaneously for all apps. The sub-dimensions of Section A for all apps were assessed first, followed by the sub-dimensions of Section B. The evaluation process was conducted in parallel, which means that the two reviewers independently rated and evaluated the same apps at the same time, and interrater reliability was measured.

Data were analysed using descriptive and analytical statistics. Quantitative variables are described using means and standard deviations (SDs). Categorical variables are described using frequencies and percentages. The intraclass correlation coefficient (ICC) between raters was used as a measure of interrater reliability for the MARS. The two-way mixed, average measures model with absolute agreement was used to estimate the reliability of the average measure between two raters [17] . Data collection and collation were completed using Excel 2016 (Microsoft Inc., Washington DC, USA), and data analysis was completed using IBM SPSS Statistics version 26 (IBM Inc., New York, USA).

Among the 20 eligible apps, 11 independent apps were obtained from the App Store, and 9

applets were obtained from the WeChat platform. The characteristics of the apps are shown in Table 1 . Eleven independent apps were designed for both the Android and iOS platforms and could be downloaded for free, with the number of downloads and ratings provided by users. Nine of these independent apps had been downloaded more than 10 million times. However, 54.54% of these independent apps had a user rating of less than 3 stars, and only 3 apps in the Apple Store had a user rating of 4 stars or more. In addition, 11 of the included apps were developed by Chinese government departments, and the remaining 9 were developed by commercial organizations. The main target groups for these apps were the general population (70%), the student population (20%), and employees of enterprises and institutions (10%). Nineteen of the apps had been updated within the past year. 

In this study, we comprehensively reviewed the features of these 20 apps related to COVID-19 personal prevention and control, and the functional features that contributed to the quality of the apps were grouped into six categories, namely, health self-checking and reporting, news about COVID-19, scientific publicity and education, telemedicine services, personal travel inquiries, and digital contact tracing ( Table 2 ). 

All 20 apps included in this study had acceptable quality (defined by a MARS score ≥3) (described in Tables 3 and 4 ). In the four dimensions of app quality ratings (Section A, Section B, Section C, and Section D), the highest scores for all 3 platforms (iOS, Android, and WeChat applets)

were in the functional dimension, followed by information quality, aesthetics, and engagement.

Only 2 independent apps (WeChat and Alipay) achieved a score of 4 or more on both the overall quality and subjective scales, indicating high or excellent quality [31] . There were no significant differences in the 4 dimensions (Section A, B, C, D) of the app quality portion of the MARS and subjective quality scores between the iOS and Android platforms of the independent apps. In addition, there was no significant difference between the independent apps and WeChat applets regarding app quality scores (t=1.240, p=0.231) or subjective quality scores (t=0.636, p=0.533) ( Table 5 ). The interrater reliability of the engagement section of the subscales, the app quality total scores, and the subjective quality scores were all excellent (Table 6 ). Detailed MARS scoring data are available in Multimedia Appendix D. 

In this study, the functions and features of 20 personal prevention and management apps related to COVID-19 in China were classified and described in detail, and their quality was rated by the MARS. Health self-checking and reporting (17, 85%) and personal travel inquiry functions (14, 70%) were the main purposes of developing these apps to prevent the rapid spread of the novel coronavirus.

Mobile apps that support contact tracking are now considered an assistive technological tool that many governments around the world are utilizing to manage COVID-19, as these apps turn smartphones into tracking tools. In this study, 13 (65%) of the apps allowed the ability to access and apply health QR codes. During the early stages of the epidemic, the Chinese government used health QR codes as "electronic health cards" for Chinese residents for daily travel and as an effective epidemic prevention and control tool [34] [35] [36] . In China, different regional (provinces/municipalities) governments have already launched various apps with similar functions to prevent and control epidemics, enabling dynamic authentication of local residents' health status, with different colours (red, yellow, or green) indicating different levels of restrictions or regulations. This largely eliminates cumbersome reporting procedures, improves the efficiency of epidemic surveillance, and minimizes the possibility of transmission. However, there are some problems, as many provinces and cities have their own online health QR codes, and the health QR codes among the provinces/cities are not standardized; therefore, cross-provincial travel usually requires users to

repeatedly register and enter their information on multiple platforms and reapply to obtain a regional health QR code. Some users who register to receive a local health QR code still need to isolate for 14 days, causing great hindrance to normal travel by the public.

In this study, there were 10 (50%) apps, including 5 independent apps ("National Government Service Platform", "Wanshitong", "DingTalk", "WeChat", "Alipay") and 5 WeChat applets ("Xi'an Citizen One-Code Pass", "Guizhou health code", "Shen I Nin-Zizhu Shenbao", "Wuhan Zhanyi", "Sichuan Tianfu health code") that provided various forms of healthcare services, such as online disease consultation, virus test result reporting, vaccination information, and nucleic acid testing facility information. There were 8 apps (40%), including the 5 independent apps and 3 applets ("Xi'an Citizen One-Code Pass", "Shen I Nin-Zizhu Shenbao", and "Wuhan Zhanyi") mentioned above, that also provided news about COVID-19. These "noncontact" online services help users maintain social distancing and home isolation, relieving the pressure on a large number of offline hospitals and alleviating people's fear of uncertainty. In addition, it is worth noting that 4 of the apps and 4 applets had multiple functions, including consultation, education (epidemic status information and science education), contact tracking, and local epidemic risk information. Through the quality review of MARS, it was found that the above independent apps/WeChat applets, which can provide a wide range of functional services and links to external extensions, tended to be excellent in app quality and subjective quality, with 80% (8/10) scoring 4 and above. Therefore, we suggest that when developing and designing COVID-19-related apps, the above-mentioned features should be considered to help improve the quality of the app, thereby increasing the user acceptance and adoption rate.

Privacy protection is another key topic concerning COVID-19 prevention and control apps.

The widespread use of digital contact tracking technology has aroused public concern about travel restrictions and the privacy and security of personal data. China has a top-down policy whereby managers of public places, such as communities or supermarkets, have the right to deny access to individuals who cannot provide a valid health QR code. In other words, individuals have little freedom of choice, even if they are unwilling to participate in digital contact tracking. A similar problem exists in some East Asian countries that use digital tracking technology [37] . 

Currently, several tools have been developed to evaluate an app's quality, such as the tool for evaluating drug use management apps [38] and the tool for evaluating the quality of healthcarerelated apps [39] . In this study, we chose MARS as the app quality assessment tool, mainly for the following reasons. First, in the process of app screening, we found that most Chinese epidemic prevention and control apps were developed by adding epidemic prevention and control functions to existing social apps, such as Alipay or WeChat, rather than independently developing apps.

MARS evaluation items have wider applicability [17] ; thus, it is appropriate to use MARS to evaluate an app's quality. Second, the MARS has been widely used in many studies with good reliability and validity [18, 19, 26] . In addition, video instructions are provided for the MARS on how to use the tool, which provided technical support for our study.

Based on the MARS rating results, in the 4 dimensions of app quality, 3 platforms (iOS, Android and WeChat applet) had the worst engagement scores. In particular, the engagement score of WeChat applet platforms was the lowest among them (mean = 2.66, SD = 0.72). The total quality (mean = 3.48, SD = 0.34), engagement, aesthetics (mean = 3.30, SD = 0.60) and subjective quality (mean = 3.64, SD = 1.16) of WeChat applets were all lower than those of independent apps.

Additionally, the 4 independent apps and 7 WeChat applets developed by the Chinese government scored only 2.65 (SD=0.67) in terms of engagement, indicating that even the government-developed COVID-19 apps had low user engagement. The majority of prevention-focused WeChat applets (77.78%) in this study were developed by local governments and built on a digital platform with high "traffic" (WeChat). Studies have shown that Chinese users encounter few operational barriers when using WeChat or Alipay, yet the installed epidemic prevention applets have become the main online tools due to their small size, low cost, ease of sharing, accessibility, and direct reach [40] .

Through analysis of the scores of each item on the MARS subscale, we found that independent apps lacked some additional components or interesting videos or images to make them more attractive, which was the main reason for the low engagement score. WeChat applets cannot be customized according to user preferences (e.g., basic settings, such as sounds, notifications, and content), and the allowance of user input and provision of feedback functions are the main reasons for the low engagement scores. A study has shown that user engagement is very important, and there is a significant correlation between user engagement and increased app adoption [26] . Therefore, we suggest that developers should not only focus on meeting users' needs in all aspects of functionality, multi-dimensionality, and depth, as well as provide high-quality information on epidemic prevention and control, but also pay attention to app design in terms of entertainment, interest, interactivity, customization and other engagement features, which will help enhance user retention and achieve wider promotion.

This study has certain limitations. First, the data collection period for this study was only from December 24, 2020, to January 25, 2021, which means that the results of this study provide only a cross-sectional view of a rapidly evolving app market. Second, some apps were excluded for reasons such as the requirement of an organization's internal registration code or inability to be opened due to the relatively old iOS platform app installation device (iphone 6 s). Third, Chinese is a profound, vast, and complicated language, and a word or phrase often has multiple meanings. An identical phrase can represent opposite meanings in different contexts. There are many Chinese words related to COVID-19, so the search terms we used were not exhaustive. In addition, our searches were limited to the Apple Store, Android Market, and WeChat app in China, which may have caused selection bias in the study sample. country). However, the growing usage of digital tracking tools raises the potential risk of personal data leaks. Therefore, we recommend that relevant departments not only urge software developers to standardize privacy protection policies for current epidemic prevention and control software, but also strictly manage the large amount of personal health data collected in the software, and regulate the trace management of the data use process to ensure the traceability of any data leakage incidents and risks.

COVID-19 Dashboard by the

NIH).Department of Health and Human Services

The Most Important Trends to Know

Global digital population

Assessing the Use of

Mobile Health Technology by Patients: An Observational Study in Primary Care Clinics

Digital contact tracing for COVID-19

COVID-19 Mobile Positioning Data Contact Tracing and Patient Privacy Regulations: Exploratory Search of Global Response Strategies and the Use of Digital Tools in Nigeria

Awareness, acceptance, and adoption of the national digital contact tracing tool post COVID-19 lockdown among visitors to a public hospital in Singapore.Clinical microbiology and infection : the official publication of the European Society of Clinical Microbiology and Infectious Diseases

Early Perceptions of COVID-19 Contact Tracing Apps in German-Speaking Countries: Comparative Mixed Methods Study

Covid-19: Researchers launch app to track spread of symptoms in the UK

Using Information Technology to Manage the COVID-19 Pandemic: Development of a Technical Framework Based on Practical Experience in China

Decentralized privacy-preserving proximity tracing: Fraunhofer HHI

Mobile Apps for Individuals With Rheumatoid Arthritis: A Systematic Review

A Content Analysis of Mobile Phone Applications for the Diagnosis, Treatment, and Prevention of Urinary Tract Infections, and Their Compliance with European Association of Urology Guidelines on Urological Infections

Expert Involvement and Adherence to Medical Evidence in Medical Mobile Phone Apps: A Systematic Review

Security Recommendations for mHealth Apps: Elaboration of a Developer's Guide

Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps

A review and content analysis of national apps for COVID-19 management using Mobile Application Rating Scale (MARS)

A Systematic Review of Smartphone Applications Available for Corona Virus Disease 2019 (COVID19) and the Assessment of their Quality Using the Mobile Application Rating Scale (MARS)

Evaluation of Mobile Apps Targeted at Patients With Spondyloarthritis for Disease Monitoring: Systematic App Search

Assessing Apps for Patients with Genitourinary Tumors Using the Mobile Application Rating Scale (MARS): Systematic Search in App Stores and Content Analysis

Mobile Phone Apps for Food Allergies or Intolerances in App Stores: Systematic Search and Quality Assessment Using the Mobile App Rating Scale (MARS).JMIR mHealth and uHealth

Smartphone Applications to Support Sleep Self-Management: Review and Evaluation

Measuring the Quality of Mobile Apps for the Management of Pain: Systematic Search and Evaluation Using the Mobile App Rating Scale

Mobile Apps for Blood Pressure Monitoring: Systematic Search in App Stores and Content Analysis

Quality and Adoption of COVID-19 Tracing Apps and Recommendations for Development: Systematic Interdisciplinary Review of European Apps

The PRISMA 2020 statement: an updated guideline for reporting systematic reviews

eHealth Literacy: Essential Skills for Consumer Health in a Networked World

Aladdin Future Technology Co.Aladdin Data Statistical Platform.Date

Free smoking cessation mobile apps available in Australia: a quality review and content analysis

Covid Contact tracing apps are a complicated mess: what you need to know.Date

Personal information protection regulation in the application of health code

Hangzhou health code: the improvement of local innovation and expansion of risk governance

Health Code: Automated Rating and Utilization of Personal Epidemic Risk

The US is rolling out digital contact tracing. How has it been working in other countries?Date

Quality Assessment of Medical Apps that Target Medication-Related Problems

Development and Evaluation of an Evaluation Tool for Healthcare Smartphone Applications

Decoding China's COVID-19 'virus exceptionalism': Community-based digital contact tracing in Wuhan

 The COVID-19 apps in China were mainly built on social media platforms

 The COVID-19 apps lack health education functions related to novel coronaviruses

 The overall quality of the WeChat applets for COVID-19 was relatively low

Lv Hekai: Checked the accuracy of extracted data and performed statistical analysis. Wang Zhuoxin: Participated in the software screening, selection, and data extraction. All authors read and approved the final version of the manuscript. No conflict of interest exits in the submission of this manuscript, and manuscript is approved by all authors for publication. I would like to declare on behalf of my co-authors that the work described was original research that has not been published previously, and not under consideration for publication elsewhere

The authors thank American Journal Experts (AJE) for performing English language editing. 

Wang Fuzhi: Conceptualized the study, designed the methods, and supervised the study. Fan Yanyan: Constructed the search strategies; performed the software screening, study selection, data extraction process, and statistical analysis; and wrote the original draft. Deng Shanshan: Participated in app screening, selection, and data extraction. Lv Hekai: Checked the accuracy of the extracted data and performed statistical analysis. Wang Zhuoxin: Participated in the software screening, selection, and data extraction. All authors read and approved the final version of the manuscript.

☒ The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.☐The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: