key: cord-0781967-r2d6n2d8 authors: nan title: Design and Feasibility Study of the Mobile Application StopTheSpread date: 2020-09-08 journal: IEEE Access DOI: 10.1109/access.2020.3022740 sha: 8e836e60ea3de444bc9b554612d55c3aef424674 doc_id: 781967 cord_uid: r2d6n2d8 The emergence of recent disease outbreaks calls for the design of new educational games aimed at increasing awareness in disease prevention. This article presents StopTheSpread, an educational mobile application that seeks to improve awareness about the best practices to prevent the spreading of seasonal flu in the general public. StopTheSpread integrates concepts in network science and epidemiology, within a freely available mobile application that provides a unique learning experience for free-choice learners about flu prevention. StopTheSpread teaches users basic concepts about flu prevention, within a series of games of increasing difficulty that maintain user engagement and offers a user-friendly design. StopTheSpread provides a summary of the best practices to prevent flu spreading according to the guidelines of the Centers for Disease Control and Prevention, and the World Health Organization, while connecting users to citizen science projects aimed at worldwide flu tracking. Through Facebook, Twitter, and email we reached volunteers during the COVID-19 confinement, to conduct an online feasibility study, toward assessing learning outcome in playing with our mobile application. Our results indicate that the use of StopTheSpread increased by 20% the awareness about the spreading mechanism of flu, compared with the baseline population. Improving scientific literacy is key to increase the awareness in disease prevention [1] - [3] . Given the wide portion of the population that could benefit from the design of new educational approaches, informal science learning activities have recently found fertile terrain that make them preferable to structured formal learning programs in many circumstances [4] . Informal learning takes place outside the school environment, for example in museums, field trips, and online [5] , [6] . It is voluntary, unstructured, unsequenced, learner-led, and unplanned [4] , [7] . Its spontaneous and interactive nature could be central to improving scientific literacy [8] - [10] . Entertaining and challenging educational games The associate editor coordinating the review of this manuscript and approving it for publication was Francisco J. Garcia-Penalvo . are a class of informal learning approaches that is particularly effective for educating the general public [11] , [12] . Previous studies provide evidence that educational games have a positive effect on learning outcomes [13] and recommend their use as instructional methods [14] . Within the existing literature, there are several examples of successful educational games that offer alternative types of learning. For instance, in [15] , a gamelike computer-modeling environment, ''StarLogo,'' offers an interactive and graphical platform to create a complex system, where agents exchange information to one another in a non-trivial way. The game is able to ease the comprehension of complex systems in youth. In [16] , an online science education game, ''Uncommon Scents,'' provides a virtual framework where users can perform simple biological experiments, such as comparing the behavior of mice when exposed VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ to air or toxic gas. Such a virtual framework facilitates learning of the biological consequences of exposure to toxic chemicals. Likewise, a mobile application, ''simufish,'' creates a user-friendly environment where users can interact with virtual fish, toward promoting learning of fish behavior [17] . The game ''Conectado'' provides a platform where users are placed in the role of victims of cyber-bullying, making them reflect on its problems. This game is able to raise empathy with victims and increase bullying awareness [18] . Relevant examples of successful educational games can also be found within the context of disease prevention [19] - [23] . ''Epidemic Containment Game'' [20] is a powerful theoretical tool to study voluntary vaccination enacted before the spreading of diseases. In [21] , participants playing a board game gained knowledge about risk-factors associated with the development of heart diseases and cancer. In [22] , a question-and-answer game, ''ZIG-ZAIDS,'' increased the awareness in AIDS prevention. Particularly interesting is the educational game ''Vax,'' which was originally designed to teach elements of disease spreading to the general public. Vax is a web-based platform developed in 2014 around the idea of networks [24] , [25] , where nodes (circles or dots) represent individuals and links (edges or segments) represent interactions between pairs of individuals. Vax builds upon the technical literature [26] to explain how diseases spread in a population from infected individuals to healthy ones via the available interactions. It demonstrates that the knowledge of the individuals' interactions slows down the spreading of the disease. Specifically, Vax teaches that vaccination and quarantine are viable prevention mechanisms to slow down, and eventually stop, the contagion process. The teaching capabilities of Vax, however, have yet to be fully assessed. Here, we build on the idea proposed by the web-based game Vax and introduce StopTheSpread [27], [28] , a mobile application to teach the best practices to prevent flu spreading. Similar to Vax, StopTheSpread uses networks to represent how diseases spread in a population. Different from Vax, StopTheSpread provides unique features that facilitate users' engagement and learning and we briefly describe them in the following. It targets the seasonal flu disease, which causes thousands of deaths per year (e.g. 64, 000 in the US during the 2017-2018 season [29] ). It is a mobile application, suitable to entertain and educate the general public [30] , [31] . It connects users of the mobile application to current citizen science projects aimed at tracking seasonal flu, such as, InfluenzaNet [32] and Flutracking [33] . In this way, users of StopTheSpread can actively contribute to tracking how flu spreads. Also, StopTheSpread explains the best practices to prevent flu spreading according to the World Health Organization [34] and the Centers for Disease Control and Prevention [35] , through useful summaries of such practices and links to official websites of these organizations. Can StopTheSpread increase the awareness of flu prevention? In order to answer this question, we performed a feasibility study during the worldwide COVID-19 confinement. Due to the impossibility of carrying in-person studies, we car-ried it online using Facebook, Twitter, and email. In the study, we randomly divided volunteers in an experimental and a control group. Volunteers in the experimental group installed and used StopTheSpread before answering survey questions. Volunteers in the control group performed the activities in the opposite order, so that their ability to answer the survey questions could be considered as a baseline on which to test the learning value of StopTheSpread. Specifically, by comparing the scores of the two groups, we expected to gain insight into the potential of StopTheSpread to teach basic concepts about flu prevention. The rest of the manuscript is organized as follows. In Section II, we describe StopTheSpread and its goals. In Section III, we explain the experimental design and present our results. In Section IV, we discuss our findings, propose possible future disease-related educational activities, and provide conclusive remarks. The mobile application ''StopTheSpread'' is built using Expo [36] v34.0.0, a platform that provides a set of tools and services to develop and deploy mobile applications on the Apple [27] and Google Play [28] stores. The core functionality of the application is a game where players have to halt the spreading of an epidemic in the least number of days using a set of pre-defined preventive actions. In Fig. 1 , we illustrate the conceptual map of StopTheSpread, composed of six main pages, whose functionalities are listed below. • The ''Tutorials'' page instructs the users on how to play the game and lists the main learning outcomes. • The ''Play'' page proposes five games of increasing difficulty, four based on real social networks derived from the SocioPatterns project [37] , and one based on the Barabási-Albert generative model of artificial heterogeneous networks [38] . where key information about flu disease is provided to the user. • The ''Global Flu Tracking'' page points users to citizen science projects aimed at worldwide flu tracking. • The ''References'' page provides hyperlinks for users interested in learning more about flu prevention [34] , [35] , Network Science [24] , [25] , and the SocioPatterns project [37] . • The ''Credits'' page acknowledges support from international organizations and lists the main contributors to the development of the mobile application. In the following, we explain the content of the six pages in the application (''Tutorials,'' ''Play,'' ''About Flu,'' ''Global Flu Tracking,'' ''References,'' and ''Credits''), providing a rationale for our choices. Then, we present a detailed explanation on how the considered networked systems are generated. Finally, we describe the flu-like epidemic model implemented, and we list the main variables used in the Play page. This page comprises four subpages, illustrated in Fig. 1 , top right. The first three subpages are dedicated to the main learning outcomes of the mobile application and they illustrate the current state of the art in flu prevention [34] , [35] and Network Science [24] , [25] . The last subpage explains to users the role of each button in the game. We focus our discussion on the three informative subpages. Scientists represent a set of interacting individuals with a network, where humans are represented as dots (nodes) and their interactions with lines (links) [24] , [25] . In this subpage of the Tutorials page, we put forward a comparison between reality and modeling, as shown in Fig. 2 . In reality, individuals interact with others when they are close enough, while scientists model this phenomenon using a line to connect the two dots that correspond to the individuals under observation. In the mobile application, we also illustrate an aggregated version of all interactions registered in a given day, which corresponds to the ''Full day'' representation at the bottom of Fig. 2 . For simplicity, we do not contemplate the representation of multiple interactions. Flu spreads in a population through the set of available interactions, from infected individuals to healthy ones. In StopTheSpread, we depict individuals and their interactions in the form of a network and consider that individuals can be in one of three possible states: healthy, exposed, and infected. Healthy individuals can contract the disease from FIGURE 2. Screenshot of a page in the tutorial ''How do scientists model interactions between humans?'' On the left, we schematize interactions through physical proximity, enclosing in ellipses people who are close enough to interact throughout a day. On the right, we illustrate how scientists model interactions through networks, in different times of the day. Alongside observations during a day, we show a daily aggregated view of interactions with the corresponding network model. infected individuals. Exposed individuals are infected but cannot infect anyone. Similar to the seasonal flu, they spontaneously become infected after a transition phase (incubation). Infected individuals are infected and can infect others. The two panels in Fig. 3 illustrate a typical case of disease progression and transmission. In Fig. 3(a) , a time snapshot of the network is illustrated, where three infected individuals are present. Fig. 3 (b) represents a successive snapshot, where the individual in the central position of the network has recovered from the infection and has infected two individuals who interacted with him. These individuals are now exposed and they will eventually transition to infected. The network visualization helps the general public to focus on important epidemiological concepts. For instance, users may observe that some individuals may create more interactions than others; that individuals in central positions in the network have a different role from those occupying peripheral positions; that the risk associated with an interaction depends on the state of health of both peers; and that no spreading can occur if a healthy individual interacts only with other healthy individuals. According to the World Health Organization [34] and the Centers for Disease Control and Prevention [35] , several actions can be taken daily to slow down the flu spreading. In our mobile application, users can experience the effect of three different actions: isolation, behavioral changes, and vaccination. Behaviors corresponding to these actions are listed below. -''Isolation:'' avoid close contact with people who are sick; or stay home when you are sick. -''Behavioral Change:'' wash your hands often; avoid touching your eyes, nose, or mouth; and practice good health habits (get plenty of sleep, eat nutritious food, stay hydrated, etc.). -''Vaccination:'' take a flu shot. In StopTheSpread, some individuals cannot be vaccinated; this feature models the practical inability of vaccinating the entire population, due for example to medical conditions of a minority of individuals [39] . An example of the use of the three actions (isolation, behavioral changes, and vaccination) is illustrated in Fig. 4 . In panel (a), the population includes two infected individuals and two individuals who cannot be vaccinated. The user decides to change the behavior of an individual, remove an interaction between an infected individual and a healthy one simulating isolation, and vaccinate another individual. As a result, the disease is stopped and no more infected individuals are present in the system, as illustrated in Fig. 4 (b). . Two screenshots that summarize the function of the users' available actions (isolation, behavioral changes, and vaccination) described in the tutorial ''What can people do to slow down the flu spreading?''. In the displayed population, two individuals are infected (red dots) and two cannot be vaccinated (black squares), as shown in panel (a). As an illustration, we implement the following actions: (i) we change the behavior of an individual (blue dot) interacting with one infected; (ii) we remove the interaction (blue line) between a healthy individual and an infected one; and (iii) we vaccinate the individual (green dot) in contact with both infected ones. These actions are effective in stopping the spreading and the population is now disease-free, as depicted in panel (b). StopTheSpread is designed in the guise of an educational game to increase users' engagement [11] and boost their learning process [12] . To this end, our game is designed to present increasing levels of difficulty, obtained by properly tuning the epidemic model parameters, as explained in Section II-C and Appendix A. The goal of the game is to halt the spreading in the least number of rounds, which correspond to days; the faster the disease propagates, the harder it is to halt it in the game. The flu spreading is halted when there are no individuals in either the exposed or the infected state. At the beginning of the game, a number of individuals are randomly selected and assigned to the infected state; a possible initial configuration is shown in Fig. 3(a) . Then, the disease starts spreading from infected individuals toward healthy ones, as in Fig. 3(b) , and users can use the three available actions (isolation, behavioral changes, and vaccination) to slow down, and eventually stop, the spreading, as exemplified in Fig. 4 . Real-world systems are modeled through the use of real-world networks inferred from experimental campaigns on human proximity interactions (Hospital [40] , Workplace [41] , High School [42] , and Primary School [43] ) and are chosen to be close to most of the users' daily life situations. This choice suggests to users that networked systems are pervasive and ubiquitous. A further game level is based on the Barabási-Albert generative model for complex networks [38] and is useful to highlight the role of heterogeneity in the interaction patterns of real social networks. This page is in the form of Questions&Answers, as it deals with practical notions. This page is divided into three subpages: (i) information about the flu disease; (ii) actions to prevent flu spreading; and (iii) advantages and risks of taking a flu shot as illustrated in Fig. 1 . Users willing to help scientists to tracking the spreading of flu worldwide can do so by contributing to the Global Flu Tracking page, referred to in the middle left area of Fig. 1 . Therein, we provide hyperlinks to be accessed by users currently in Australia and New Zealand (Flutracking [33] ), or in Europe (InfluenzaNet [32]). Depending on the area considered, the online recruitment process may vary. Full information about how to contribute to flu tracking is available in the linked websites. Users who would like to learn more about disease prevention, Network Science, or to access the datasets used in the game find information in this page. We provide hyperlinks in the Reference page, referred to in the bottom right area of Fig. 1 . Users find here information about the main agencies and institutions supporting the mobile application, as well as the main contributors to it (Fig. 1, bottom left) . We consider four networks inferred from real time-resolved interaction networks and a synthetic network. The four real networks are provided by the SocioPatterns project [37] and are obtained by recording face-to-face interactions using proximity sensors [44] . These sensors are able to detect time-resolved interactions with a resolution of 20 seconds, thereby offering the possibility to record time variations in temporal patterns. Furthermore, these data from the SocioPatterns project contain additional information about the individuals who participated in the study. For instance, we know that individuals are: nurses or doctors in the Hospital system [40] , working in one of the departments in the Workplace system [41] , students of one of the classes in the School systems [42] , [43] . Here, we manipulated the original four real datasets to render their presentation more appealing to potential users of the mobile application. To this end, we considered only a subset of the individuals in each of four real datasets, based on their number of connections. Specifically, we retained: pairs of nurses interacting at least 100 times in the Hospital system [40] , pairs of workers in the 'DSE'' department interacting at least 27 times in the Workplace system [41] , pairs of students in the ''PSI'' class interacting at least 35 times in the High School system [42] , and pairs of students in the 1A class interacting at least 55 times in the Primary School system [43] . These real networks look entangled due to the high presence of loops, that is, three or more individuals forming a close-ended chain of interactions. On the contrary, the synthetic network was created using the Barabási-Albert generative model [38] , which reproduces a tree-like structure where no loops are present. Such a synthetic network should be better suited to render the heterogeneity of many real systems [45] , where a few individuals generate most of the interactions and the others only a small fraction of them. The number of individuals considered in the system is N = 40 and the number of initially connected individuals is m 0 = 2. Then, new individuals (one by one) are added to the system, such that the actual number of individuals in the system is n = m 0 + 1, m 0 + 2, . . . , N . Each new individual generates an interaction toward one of the other n − 1 individuals in the system. We indicate with i the generic individual who can receive such interaction and associate with this event a probability where k i represents the number of connections of individual i and C corresponds to the total number of connections in the system. New individuals are added until we reach a total number of individuals N = 40. Equation (1) represents a preferential attachment rule [45] , according to which a newly added individual is more likely to interact with another one having a high number of connections. Users are allowed to modify the network structures by removing links to prevent epidemic spreading. Removed links spontaneously reappear according to a stochastic rule, which mimics the end of the isolation period. VOLUME 8, 2020 C. EPIDEMIC MODEL Depending on the considered disease, a proper epidemic model should be adopted [46] . As the StopTheSpread mobile application focuses on the transmission of the seasonal flu, we selected an adapted version of the well-known Susceptible-Infected-Susceptible model [47] to contemplate incubation and possible reinfections during the observation time. The progression and transmission of the epidemic model is schematized in Fig. 5 and its parameters varied to design games of increasing difficulty, as explained in Appendix A. . Both panels indicate with circles all possible states: S for susceptible, V for vaccinated, B for behavior-changed, E for exposed, and I for infected. Arrows represent possible transitions from one state to another. Solid arrows are accompanied by a symbol indicating the probability per unit time that the transition occurs. A star close to a symbol indicates that the transition may occur only upon interaction with an infected individual. Dashed arrows represent transitions enabled by the user's behavior, who has the opportunity to change the behavior and vaccinate a limited amount of individuals every day. Once an individual is infected with the flu, it may take a few days before the individual develops symptoms and is able to spread the pathogen to others [48] . This period is known as incubation time, and the related state is known as exposed. After a certain delay, exposed individuals become infectious and can infect others. In this vein, the exposed state was conveniently added to the model [49] . We contemplated behavioral changes enacted to comply with the good practices to prevent flu spreading [35] . Our choice is grounded in recent scientific discoveries that pinpoint the impact of behavioral changes on disease spreading [50] , [51] . In the model used in our mobile application, healthy individuals may change their behavior to enact self-protective strategies. This phenomenon is paralleled by the introduction of a further ''B'' state in our model. Individuals in this behavior-changed state, upon interacting with infectious individuals, transition to the exposed state with a lower probability than susceptible individuals who have not enacted self-protective behavioral changes (''S'' state) [35] . These healthy individuals are less likely to be infected. Furthermore, individuals with their behavior-changed may follow the good habits to prevent flu spreading for a limited period; for instance, they may forget to wash their hands for a series of days, thereby transitioning back to the susceptible ''S'' state. We also included a vaccination mechanism. Vaccination is a determining factor to prevent disease spreading [35] , supported by several experimental and theoretical studies [52] - [54] . In our mobile application, healthy individuals, as well as individuals who enacted self-protective behavior, can be vaccinated. Once vaccinated, these individuals cannot become infected for a long-but-limited amount of time (in the real world, due to the seasonal mutation in the pathogen strain, the effectiveness of the flu vaccine is limited to a season or even less) [55] . In addition, we assumed that some individuals should not be vaccinated [39] , due to their medical condition, and we represented such individuals in a squared shape. Overall, our model includes five possible states: susceptible (or healthy), vaccinated, behavior-changed, exposed, or infected. The set of possible states is denoted as X = {S, V , B, E, I }, where S represents susceptible individuals, V vaccinated individuals, B individuals who enacted behavioral changes (that is, behavior-changed state), E exposed individuals, and I infected individuals. Individuals in the healthy state, vaccinated state, and behavior-changed state are all healthy. The possible transitions from one state to the other are represented in Fig. 5 . They are tagged with symbols indicating the corresponding probability. Possible transitions can be divided into three main categories. A spontaneous transition is a transition that does not depend on any of the interactions. As shown in Fig. 5 , the spontaneous transitions are: from the vaccinated state to the susceptible one (with probability β); from the behavior-changed to the susceptible state (with probability α); from the exposed to the infected state (with probability ω); and from the infected to the susceptible state (with probability µ). Furthermore, a removed interaction may spontaneously reappear (with probability δ); this transition is not shown in Fig. 5 , as the state transition graph does not contemplate network formation phenomena. These transitions occur only if an infected individual interacts with either a susceptible individual or one with their behavior-changed. As depicted in Fig. 5 , if the interaction under exam is between an infected individual and a susceptible one, then the susceptible one becomes exposed with probability λ. If an infected individual interacts with another in the behavior-changed state, the probability that it becomes exposed is reduced to ηλ, where η < 1 is a behavioral parameter that makes less likely for the individual to become exposed to the flu disease (and eventually infected) [50] , [51] . Furthermore, users may decide to use the isolation action, thereby removing an interaction between two individuals. If the removed interaction destroyed is between an infected individual and a healthy (either susceptible or with the behavior-changed) one, then the healthy individual cannot be infected. Users may perform three different actions: isolation, behavioral changes, and vaccination. As previously described, isolation only removes an interaction between two individuals. Differently, individuals who change their behavior voluntarily transition from the S to the B state, as illustrated in Fig. 5 . Similarly, applying the vaccination action to an individual in either the susceptible or the behavior-changed states makes the individual to transition to the vaccinated state, as in Fig. 5(a) . Vaccination can only be applied to individuals who can be vaccinated. In order to explore the effectiveness of StopTheSpread in increasing awareness of flu prevention, we performed an online feasibility study during the worldwide COVID-19 confinement in April and May 2020. Such a study has been promoted and conducted using Facebook, Twitter, and email. Our study reached a total of 68, 841 people, 56 of whom completed the online activity. Our results indicate that StopTheSpread can increase awareness about the flu disease. Specifically, users are found to score 20% better on questions addressing how flu spreads, after the interaction with StopTheSpread. Here, we explain the process of the data collection and analysis and we present our main results. We advertised our online activity through the official accounts of the Dynamical Systems Laboratory. Specifically, we used Twitter (@DynamicalSyste2), Facebook (Dynamical Systems Laboratory), and email (dsl.nyu@gmail.com). Our Facebook post reached 27, 348 people and 632 clicked the link that directed them to our study, while 41, 493 people were reached via our Twitter post and 33 clicked the link to the study. We are not able to anticipate the exact number of people reached via email, as well as how many people have clicked the link to our study. Our online study started on the 2nd of April 2020 and ended on the 19th of May 2020. We spent $150 to promote the Facebook post and other $150 to promote the Twitter post. Overall, 141 volunteers gave us consent to collect data, 57 volunteers answered survey questions, and 56 volunteers (44 in the control group and 12 in the experimental group) correctly completed the entire activity. To the best of our knowledge, none of the volunteers had prior experiences with StopTheSpread, whereby they were randomly reached through electronic means. Participation rate is consistent with previous studies [56] - [58] , where only a small fraction of the people reached actively participate in the proposed activity. In Fig. 6 , we report the available statistics on the data collection. The online activity consisted of completing a consent form, filling a short anonymized survey, and interacting with our mobile application StopTheSpread. After completing the consent form, volunteers were randomly divided into control and experimental groups. Volunteers in the control group first filled the survey, and then interacted with our mobile application. Volunteers in the experimental group first interacted with our mobile application, then filled the survey. We reckon that, by design, volunteers in the control group are more likely to answer survey questions than volunteers in the experimental group. Consistent with this claim, we registered The complete list of questions along with their corresponding codes is presented in Table 1 . All other information collected from the consent and survey forms are available in Appendices B-A and B-B, respectively. Data manipulations were performed using the programming language Python 3, with the Jupyter notebook. We operated a preliminary data cleaning phase, where volunteers' emails were used to match the answers in the consent form with those in the survey to ensure volunteers gave us consent to use the data. We considered two emails as equivalent if they had at most one character different from one another to account for potential typos by the users. After matching answers in the consent form with the ones in the survey form, we anonymized the data by removing the email field. Furthermore, we homogenized the answers to the question ''In what country are you currently in?'' as follows. The answers ''Italy,'' ''italy,'' ''Italia,'' ''Italia,'' ''italia,'' or ''Iraly'' are converted to ''Italy;'' ''UK'' or ''Uk'' are converted to ''UK;'' and ''USA,'' ''Usa,'' or ''United States'' are converted to ''USA''. In the answers to the question ''What is, in your opinion, the best method we have to prevent flu spreading?'', we removed all subjects, verbs, and conjunctions, such as ''we,'' ''be,'', or ''or''. We also converted the words ''vaccino,'' ''vacinazione,'' ''vaccines,'' ''vaccin,'' ''vacine,'' and ''vaccination,'' which all become ''vaccine''. In order to more easily interpret the demographic information collected in Table 1 (a), we considered answers to questions D4, D5, D6, D7, and D8 to be either affirmative (numerically represented with a one) or negative (numerically represented by a minus one). We treated ''Yes'' as an affirmative answer and, ''Maybe,'' ''I do not have children,'' and ''No'' as negative answers. To quantify the learning outcome of volunteers, we assigned numerical values to the volunteers' answers of the second section of the survey questions (''Q'' questions). In the questions requiring a unique answer (Q1, Q2, Q3, Q4, Q5, and Q6), we assigned a score equal to one when volunteers correctly answer the question, and zero otherwise. In the question requiring multiple answers (Q7 only), we assigned a score equal to the number of correct answers minus the number of wrong answers. We did not convert the answers in Q8 to a numerical format, as the purpose of this question was to create a wordcloud that would show the general sentiment of the users to the best method available to prevent flu spreading. We analyzed volunteers' score corresponding to the cleaned, anonymized, and numerical data. We focused on volunteers' score in each of the seven questions (from Q1 to Q7), as well as on the volunteers' score obtained by summing multiple answers. For instance, the score Q1Q2Q3 indicates the sum of the scores gained in questions Q1, Q2, and Q3, while the total score is the sum of the scores gained in all questions. When separating volunteers in groups based on their answers to the demographic information collected in Table 1 (a), we used a two-tailed, non-parametric statistical test to compare the volunteers' score. Specifically, we considered the Kruskal-Wallis H-test for comparing the median scores of multiple groups [59] , the Mann-Whitney rank test to compare the scores of two independent groups [60] , and the Wilcoxon signed-rank test to compare the scores of two dependent groups [61] . We reported as ''W'' the test statistic and as ''p'' its corresponding p-value. When comparing groups through the Wilcoxon signed-rank test, we also studied effect size through Cliff's delta, or ''d'' [62] . This quantity is a practical complement to hypothesis testing, whereby it quantifies how often the values in a group are larger than the values in the other group. We set the significance level equal to 0.05. A total of 56 volunteers correctly completed the online activity. The average age of these volunteers is 37.2 ± 16.3 (mean ± one standard deviation). The highest educational level of three volunteers is Middle School, of 15 is High School, of 23 is Bachelor degree, of eight Master degree, and of seven is a PhD degree or higher. A total of 49 volunteers were interested in flu prevention, while seven volunteers were not. Also, 13 volunteers took a vaccine in the last 12 months prior to the completion of the survey, while 43 volunteers did not. The answers of all general demographic questions can be found in Table 8 in Appendix B-C. A qualitative representation of our results is illustrated in the wordcloud in Fig. 7 , where we observe the predominance of the word ''vaccine'' as the best method available to prevent flu spreading. We performed two different quantitative analyses. First, we explored whether volunteers' background, collected via the demographic information in Table 1 (a), did explain the score volunteers' achieved in answering questions about disease prevention, in Table 1(b) . 1 Second, we tested the hypothesis that StopTheSpread teaches the best practices to prevent flu spreading. To this end, we compared the score gained by volunteers in the control and experimental groups. We study whether volunteers' background was related with either a high or low score. We divided the volunteers in groups sharing similar demographic information from their answers to the questions in Table 1 (a). We compared the score that these different groups reached while answering the questions in Table 1 (b), using the appropriate statistical test. For instance, we compared the score of men versus women (D3 in Table 1(b) ) and, the score of those who took the flu vaccine versus those who did not (D7 in Table 1(b) ). Using the Mann-Whitney rank test, we registered that volunteers interested in flu prevention scored higher than volunteers not interested in it (W = 267.5, p < 0.05), in Fig. 8(a) . We also discovered that vaccinated volunteers demonstrated a better knowledge of flu-related information than non vaccinated ones, thereby scoring higher in questions Q1, Q2, 1 We exclude the demographic information D2, ''In what country are you currently in?'', from this analysis as our sample size is too small. Table 1 (a). We consider question D6 about their interest in flu prevention, in panel (a), and question D7 about whether they took the flu vaccine in the last 12 months, in panel (b). For each boxplot, the red line is the median, the box delimits the first and third quartiles, and the whiskers identify the minimum and maximum values. The asterisk indicates that the two groups have significantly different scores (p < 0.05). and Q3 in Table 1 (b) that are related with the administration of the flu-shot (W= 390.0, p < 0.05), in Fig. 8(b) . Finally, we considered volunteers' score as a function of their educational level. In order to compare groups of volunteers having similar size, we considered three different educational levels: (i) volunteers who completed Middle or High School (18 individuals), (ii) volunteers who obtained a Bachelor degree (23 individuals), and volunteers who earned Master or PhD degree (15 individuals). Using the Kruskal-Wallis H-test, we found that the educational level significantly affected the total score reached in our questionnaire (W = 17.2, p < 0.001), in Fig. 9 . By applying the Mann-Whitney rank test, we documented that volunteers with Master or PhD degrees scored significantly higher than volunteers who completed Middle or High School (W = 223.5, p < 0.01). Furthermore, volunteers with a Bachelor degree VOLUME 8, 2020 FIGURE 9. Boxplots of volunteers' score depending on their educational level, corresponding to question D6 in Table 1 (a). For each boxplot, the red line is the median, the box delimits the first and third quartiles, and the whiskers identify the minimum and maximum values. The asterisks indicate that groups have significantly different scores (p < 0.001) . scored significantly higher than volunteers who completed Middle or High School (W = 352.5, p < 0.001). Boxplots containing the total score as a function of the educational level, aggregated as shown in Fig. 9 , along with significant p-values from the Mann-Whitney rank test are displayed in Table 2 . We did not detect any other significant relationship between volunteers' background and score using the Mann-Whitney rank test. Specifically, males' score was comparable to females' (W = 379.5, p = 0.97); volunteers of different age scored similarly (W = 457.0, p = 0.28); volunteers with a family member working as scientist or health professional scored comparable to volunteers who had no relatives in these fields (W = 379.5, p = 0.28); and the willingness to vaccinate children did not impact the volunteers' score (W = 246.5, p = 0.18). We then examined whether StopTheSpread taught the best practices to prevent flu spreading. To this end, we randomly divided volunteers in a control group and a experimental group. The sampled populations had similar demographic traits. By applying the Mann-Whitney rank test, we found that volunteers in the control and experimental groups has a Table 9 in Appendix B-C, we report volunteers' demographic information by separating control and experimental groups. Our expectation was that volunteers in the experimental group scored higher than volunteers in the control group. We found that all volunteers in the experimental group correctly answered Q4, about how flu spreads in the population, with a 20% increase in the average score with respect to volunteers in the control group, whose average score is 0.84, as shown in Fig. 10 . However, we did not detect any significant difference between the experimental and control group using the Mann-Whitney rank test (W = 306.0, p = 0.14), FIGURE 10 . Average score of volunteers in the control (gray) and experimental (black) groups. The ''All'' columns represent the average score for the entire activity. The average score in the Q7 column is normalized to one by dividing it by six (maximum score for that question). Similarly, the average score in the All column is normalized to one by dividing it by twelve (maximum score for the entire activity). because the control group had a high average score, close to one. This ceiling effect may be due to the media coverage about COVID-19, which, as discussed in Appendix C, increased the awareness of some flu-related concepts. Non-significant differences were found using the Mann-Whitney rank test when comparing the score of control and experimental groups for other flu-related questions, as seen in Fig In this article, we introduce StopTheSpread, a mobile application that aims at teaching the best practices to prevent flu spreading. StopTheSpread proposes games of increasing difficulty to maintain the user engaged [11] and boost their learning process [12] . Users are likely to be familiar to the environments represented in these games (Hospital, Workplace, and School), which introduce a social component in StopTheSpread that, in turn, favors the users' learning outcomes [63] , [64] . Also, StopTheSpread translates in the form of a game the guidelines issued by the Centers for Disease Control and Prevention and the World Health Organization, thereby facilitating users' understanding of their effectiveness in a real-world environment [65] . In order to explore the effectiveness of StopTheSpread in increasing awareness of flu prevention, we performed a feasibility study during the worldwide COVID-19 confinement. We recruited a total of 56 volunteers, randomly divided into a control and a experimental group. Volunteers in the control group constitute the baseline population on which to test the learning value of StopTheSpread, whereby they replied to survey questions without prior use of the app, based on their own knowledge of flu prevention. Volunteers in the experimental group were the only ones to interact with StopTheSpread prior to answering to the survey questions. By analyzing the collected data, we found that volunteers are more aware of flu-related concepts if they have a high educational level (Bachelor degree or higher), or are interested in flu prevention. The relationship between volunteers' knowledge and education as well as the relationship between knowledge and interest are already established in the field of disease prevention [66] - [70] . Also, we determined that vaccinated volunteers demonstrate a better knowledge about the flu shot. Our result extends the findings of a previous study [71] , which identified a similar relationship but focused on health care workers only. In contrast with a previous study [67] , we did not detect any score difference due to volunteers' gender and age groups. The reasons for our disagreement with the previous results may be due to differences in the studies' design. In [67] , the influenza A (H1N1) was considered, while we focused on flu. Furthermore, in [67] , the study was mostly carried out in India, while our population was mainly composed of volunteers in Italy and United States. The comparative analysis between the score of volunteers in the control and experimental group only partially answers our research question (that is, can StopTheSpread increase the awareness of flu prevention?). We found that StopTheSpread positively influenced the knowledge of how flu spreads in the studied population because volunteers in the experimental group always answered that question correctly, with a score 20% higher than volunteers in the control group. This positive learning outcome was likely achieved by representing the flu spreading using networks, which are a powerful tool to provide entertaining and explanatory visualizations [72] , [73] . The knowledge of other flu-related concepts, however, was similar between volunteers in the experimental and control groups. A possible reason for our inconclusive results is that the teaching capabilities of StopTheSpread were assessed only online, which poses greater challenges than in person learning [74] , [75] . Another factor that may partially hide the teaching capabilities of StopTheSpread is the massive media coverage about COVID-19 done during our data collection. It is well known that media coverage influences public opinions [76] , [77] : here, we offer evidence that the COVID-19 coverage positively influenced the volunteers' knowledge of how flu spreads, how to become immune from flu, and what is the incubation period. The main limitation of the study is the number of participants, which is only 56 (44 in the control group and 12 experimental group) out of approximately 70,000 people reached through electronic means (Facebook, Twitter, and emails). The modest rate of conversion from individuals reached to volunteers in the study of 0.1% may be due to the overload of information regarding epidemic diseases during the worldwide confinement period, when this study was performed, along with the prioritization of their online time allocation toward learning about specific COVID-19-related topics. The results of our feasibility study should be regarded as a stepping stone, upon which to deploy over a wider population. We recommend such a deployment to occur when the threat posed by the pandemic will be limited. Not only it is presently difficult to advertise any learning activity on epidemics online, but also it is challenging to capture people interest in general flu-related topics. Although the groups were homogeneous with respect to a wide range of demographics traits, the control group was much larger than the experimental group. The lower participation in the experimental group can be explained by observing that volunteers in the experimental group first interacted with StopTheSpread, then answered survey questions, while volunteers in the control group performed the activities in the opposite order. Thus, volunteers in the control group might have decided to not install the mobile application after having answered the survey questions. Experimental biases are common in the scientific literature [78] - [80] and an in-person assessment of StopTheSpread could have favored a balanced participation between volunteers in the control and experimental groups because all participants would download the mobile application. Overall, our results suggest that the awareness of the best practices to prevent flu spreading depends on the volunteers' background, in particular on their educational level, interest in flu prevention, and prior vaccination history. Also, we found that all volunteers interacting with StopTheSpread improved their awareness about the spreading mechanism of flu, scoring 20% higher than the baseline population. The experimental protocol was approved by the institutional review board (IRB) at New York University (IRB-FY2019-3328). The source code for the Android and iOS versions are available upon request. The anonymized survey data and the code used to analyze them are available at [81] . Although parameters used in the mobile application StopTheSpread are inspired by empirical studies, their exact value is set for entertainment purposes only. Specifically, games have increasing difficulty to maintain user engagement [11] and to boost their learning process [12] . In harder games, users should find it more difficult to halt the spreading. In fact, in the Hospital system (the easiest game) users can halt the disease transmission using a maximum of six preventing actions per day (that is, two vaccinations), while in the Artificial system (the hardest game) users have only a maximum of three actions. Also, the probability that an individual changes its state varies in different games. For instance, infected individuals spontaneously become susceptible (healthy) again with probability µ = 0.45 in the Hospital system, while this event is less likely in the Artificial System, where this probability is µ = 0.2. • # changing the behavior actions per day: 1. • # vaccination actions per day: 1. • Transitions due to an interaction: λ = 0.45, and η = 0.1. In Table 3 , we list the information collected from the consent forms. Here, volunteers cannot skip any question because all information collected have critical importance in the study. In particular, when volunteers see the symbol ''***'' on top of the symbol ''???'', they are assigned to the experimental group, while volunteers in the control group see the symbols in the reversed order. In Tables 4, 5 , 6, and 7, we list all the questions in our study. General demographic information is collected through the questions in Tables 4 and 6 , while the awareness of the best practices to prevent flu spreading is assessed through the questions in Tables 5 and 7. Volunteers give an open answer the questions in Tables 4 and 5, while they decide amongst possible choices for the questions in Tables 6 and 7 . The email address is the only mandatory information in the survey questions. We have to match the answers of the volunteers between the consent form and the survey questions to properly assign volunteers in either the control or experimental group. Answering the other questions is optional and their order randomized. In Table 8 , we report the answers to all general demographic questions. We remind that we treat ''Yes'' as a affirmative answer and, ''Maybe,'' ''I do not have children,'' and ''No'' VOLUME 8, 2020 as negative answers. In Table 9 , we list the answers of the volunteers by separating control and the experimental groups. In order to provide an explanation on why some flu-related concepts have a higher score than others, as shown in Fig. 10 , we analyzed Google Trends data [82] . We considered the worldwide ''search interest'' parameter (that is, the worldwide popularity of the search) over the past 52 weeks. We found different peaks in the popularity of selected keywords, as shown in Fig. 11 . The keywords ''Flu shot'' and ''Flu shot available'' (related with Q1, Q2, and Q3 in Table 1 (b)) have a peak at the beginning of October 2019, in correspondence of the beginning of the flu season. The keywords ''Flu immune,'' ''Flu spreading,'' and ''Flu incubation period'' have a peak at the beginning of March 2020, in correspondence of the worldwide quarantine (related with Q4, Q5, and Q6 in Table 1 (b)). Volunteers taking the survey in April-May 2020 were likely to have forgotten the information collected in October 2019, while they still remembered what they searched in March 2020. Therefore, it is tenable that volunteers correctly answered Q4, Q5, and Q6, more than questions Q1, Q2, and Q3. By applying the Cliff's delta and Wilcoxon signed-rank test, we confirmed this claim by comparing the volunteers' score in answering to the two set of questions, finding an agreement with our expectation (d = 0.51, W = 66.5, p < 0.001; Fig. 12 ). MATTHIEU NADINI received the B.Sc. degree (cum laude) in physics from the University of Modena and Reggio Emilia, in 2015, the M2R degree in physics of complex systems from Paris Diderot University, in 2017, the M.Sc. degree (cum laude) in physics of complex systems from the Polytechnic University of Turin, in 2017, and the Ph.D. degree from the New York University Tandon School of Engineering, in 2020, working under the supervision of Prof. Porfiri and Prof. Rizzo. He has been a Postdoctoral Research Associate with the City, University of London and The Alan Turing Institute, since July 2020. His research interests include network science, epidemiology, computational social science, and human behavior. VOLUME 8, 2020 SAMUEL RICHMOND received the degree in computer science from the New York University Tandon School of Engineering, in January 2020. He started his work as a Research Assistant with the Dynamical Systems Laboratory, in March 2018, and has since worked on projects involving mobile and web development, citizen science, rehabilitation, human-computer interaction, and virtual reality. He worked for Manifold Robotics, from September 2019 to December 2019, building computer vision for autonomous boats and currently works at Google as a Software Engineer for YouTube Search. Professor with the New York University Tandon School of Engineering, with appointments in the Department of Mechanical and Aerospace Engineering, the Department of Biomedical Engineering, and the Department of Civil and Urban Engineering. He is engaged in conducting and supervising research on dynamical systems theory, multiphysics modeling, and underwater robotics. He is the author of more than 300 journal publications. He is a Fellow of the American Society of Mechanical Engineers (ASME). He was a recipient of the National Science Foundation CAREER Award. He has been included in the ''Brilliant 10'' list of Popular Science in 2010 and his research featured in all the major media outlets, including CNN, NPR, Scientific American, and Discovery Channel. His other significant recognitions include invitations to the Frontiers of Engineering Symposium and the Japan-America Frontiers of Engineering Symposium organized by the National Academy of Engineering; the Outstanding Young Alumnus Award by the College of Engineering, Virginia Tech; the ASME Gary Anderson Early Achievement Award; the ASME DSCD Young Investigator Award; and the ASME C. D. Mote, Jr., Early Career Award. He has served on the Editorial Board for the Journal of Dynamic Systems, Measurement and Control (ASME), the Journal of Vibration and Acoustics (ASME), the IEEE CONTROL SYSTEMS LETTERS, the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS-I: REGULAR PAPERS, and Mechatronics. Beyond misinformation: Understanding and coping with the 'Post-Truth' era Formal and informal learning in science: The role of the interactive science centres Bridging the gap between formal and informal science learning Formal, Non-Formal and Informal Learning in the Science Scientific literacy: A conceptual overview Learning science in informal environments: People, places, and pursuits Policy statement of the 'informal science education' ad hoc committee The 95 percent solution: school is not where most Americans learn most of their science Documenting and Assessing Learning in Informal and Media-Rich Environments Gameplay engagement and learning in game-based learning: A systematic review Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning Educational games-are they worth the effort? A literature survey of the effectiveness of serious games To Game or Not to Game? Developing games and simulations for today and tomorrow's tech savvy youth The impact of a science education game on students' learning and perception of inhalants as body pollutants SimUfish: An interactive application to teach K-12 students about zebrafish behavior Applicability of a cyberbullying videogame as a teacher tool: Comparing teachers and educational sciences students Using video games as educational tools in healthcare An extended exploration to the epidemic containment game Promoting health in schools through a board game Evaluation of the ZIG-ZAIDS game: An entertaining educational tool for HIV/Aids prevention Networks: An Introduction Analysis and control of epidemics: A survey of spreading processes on complex networks StopTheSpread: Google Play Store CDC: Estimated Influenza Illness Student preferences for mobile app usage Evaluation of learning outcomes using an educational iPhone game vs. Traditional game WHO: Global Influenza Strategy CDC: Flu Prevention Emergence of scaling in random networks CDC: Who Should not Get the Vaccine SocioPatterns Project: Hospital Dataset SocioPatterns Project: Workplace Dataset SocioPatterns Project: High School Dataset SocioPatterns Project: Primary School Dataset Dynamics of Person-to-Person interactions from distributed RFID sensor networks Internet: diameter of the World-Wide Web A thousand and one epidemic models,'' in Frontiers in Mathematical Biology Modeling Infectious Diseases in Humans and Animals CDC: Incubation Period Analysis of an SEIRS epidemic model with two delays Epidemic spreading in temporal and adaptive networks with static backbone Contagion processes over temporal networks with time-varying backbones Effective vaccination strategies for realistic social networks Social networks and vaccination decisions Imitation dynamics of vaccination behaviour on social networks Online recruitment methods for Web-based and mobile health studies: A review of the literature Recruitment of mental health survey participants using Internet advertising: Content, characteristics and cost effectiveness Web-based recruiting for health research using a social networking site: An exploratory study Use of ranks in one-criterion variance analysis On a test of whether one of two random variables is stochastically larger than the other Individual Comparisons by Ranking Methods,'' in Breakthroughs in Statistics Dominance statistics: Ordinal analyses to answer ordinal questions Gender and social comparison effects in computer-based problem solving On the effectiveness of game-like and social approaches in learning: Comparing educational gaming, gamification & social networking How Computer Games Help Children Learn What do young adults know about the HIV/AIDS epidemic? Findings from a population based study in Karachi Public knowledge, attitude and behavioural changes in an indian population during the influenza a (H1N1) outbreak The power of prevention and what it requires Differences in public emotions, interest, sense of knowledge and compliance between the affected area and the nationwide general population during the first phase of a bird flu outbreak in israel The role of topic interest in survey participation decisions Correlation between healthcare Workers' knowledge of influenza vaccine and vaccine receipt Explanation through network visualization Visualizing social network concepts Student barriers to online learning: A factor analytic study Study looks at online learning vs. traditional instruction Mass-media coverage, its influence on public awareness of climate-change issues, and implications for Japan's national campaign to reduce greenhouse gas emissions Agenda setting and international news: Media influence on public perceptions of foreign nations Self-disclosure and the volunteer: A source of bias in laboratory experiments Volunteer bias in research on friendship among emerging adults What's in a trial? On the importance of distinguishing between experimental lab studies and randomized controlled trials: The case of cognitive bias modification and alcohol use disorders Online Data Collection: Flu Awareness The authors would like to thank members of Dynamical Systems Laboratory at New York University Tandon School of Engineering for useful discussions. They also would like to express their gratitude to Jalil Hasanyan for helping in the data collection process. Finally, they would like to acknowledge the help of New York University in deploying the mobile application ''StopTheSpread'' in the Apple [27] and Google Play [28] stores.