key: cord-0802966-4sebxnqa authors: Chen, M.; Abdul-Rahman, A.; Archambault, D.; Dykes, J.; Ritsos, P.D.; Slingsby, A.; Torsney-Weir, T.; Turkay, C.; Bach, B.; Borgo, R.; Brett, A.; Fang, H.; Jianu, R.; Khan, S.; Laramee, R.S.; Matthews, L.; Nguyen, P.H.; Reeve, R.; Roberts, J.C.; Vidal, F.P.; Wang, Q.; Wood, J.; Xu, K. title: RAMPVIS: Answering the challenges of building visualisation capabilities for large-scale emergency responses date: 2022-04-28 journal: Epidemics DOI: 10.1016/j.epidem.2022.100569 sha: f456f38bda6c118814e0426f48cd181c31737609 doc_id: 802966 cord_uid: 4sebxnqa The effort for combating the COVID-19 pandemic around the world has resulted in a huge amount of data, e.g., from testing, contact tracing, modelling, treatment, vaccine trials, and more. In addition to numerous challenges in epidemiology, healthcare, biosciences, and social sciences, there has been an urgent need to develop and provide visualisation and visual analytics (VIS) capacities to support emergency responses under difficult operational conditions. In this paper, we report the experience of a group of VIS volunteers who have been working in a large research and development consortium and providing VIS support to various observational, analytical, model-developmental, and disseminative tasks. In particular, we describe our approaches to the challenges that we have encountered in requirements analysis, data acquisition, visual design, software design, system development, team organisation, and resource planning. By reflecting on our experience, we propose a set of recommendations as the first step towards a methodology for developing and providing rapid VIS capacities to support emergency responses. Visualisation and visual analytics (abbreviated as VIS) has been used extensively in many mission-critical applications and healthcare applications. Since the emergence of COVID-19, data visualisation has been widely visible in traditional and online media for disseminating information related to COVID- 19 . Meanwhile what has not been obvious to the public is the fact that VIS techniques can and should be used to help healthcare scientists and experts in combating COVID-19. In particular, since epidemiologists and modelling scientists encounter a huge amount of collected data and simulation data on a daily basis [84] , it is not difficult ORCID(s): 0000-0001-5320-5729 (M. Chen); 0000-0002-6257-876X (A. Abdul-Rahman); 0000-0003-4978-8479 (D. Archambault); 0000-0002-8096-5763 (J. Dykes); 0000-0001-9308-3885 (P.D. Ritsos); 0000-0003-3941-553X (A. Slingsby); 0000-0001-6788-251X (C. Turkay); 0000-0001-9365-7420 (H. Fang); 0000-0002-5834-2658 (R. Jianu); 0000-0002-6796-5670 (S. Khan); 0000-0002-3874-6145 (R.S. Laramee); 0000-0001-5643-0585 (P.H. Nguyen); 0000-0003-2589-8091 (R. Reeve); 0000-0001-7718-3181 (J.C. Roberts); 0000-0002-2768-4524 (F.P. Vidal); 0000-0001-9270-247X (J. Wood); 0000-0003-2242-5440 (K. Xu) to infer that we need to provide epidemiological modelling workflows with as much VIS support as possible. However, there have been some challenges for many epidemiologists and modelling scientists to receive adequate VIS support. These challenges include: a. Epidemiologists and epidemiological modelling scientists are not accustomed to receiving VIS support systematically. In some disciplines, modelling scientists have received VIS support systematically. For example, in visualisation journals and conferences, there are hundreds of research papers on VIS support for computational fluid dynamics, including many surveys (e.g., [64] ). For the past a few years, the topic of providing machine learning workflows with VIS support has been growing rapidly [82] . In contrast, VIS papers on supporting epidemiological modelling are very rare. This suggests that scientists in epidemiology may not be used to the notions that they could visualise their data at their fingertips, could have visualisation experts to design visual representations J o u r n a l P r e -p r o o f Journal Pre-proof RAMPVIS: Visualisation Capabilities for Large-scale Emergency Responses specifically for their models, and could monitor and analyse the behaviours of their models and parameters dynamically. b. Visualisation is widely mistaken only for information or knowledge dissemination. In many modelling applications, VIS techniques are commonly used for scientific and public dissemination, but seriously underused in all other stages of a modelling workflow, which typically consists of a set of iterative processes, such as (a) data collection and observation; (b) hypothesis formulation and causality analysis; (c) model development, testing, validation, and comparison; and (d) model deployment, monitoring, and improvement. Ideally, modelling scientists and epidemiologists could have a quick glance of dynamic data anytime when there is a need (cf. stock brokers observing stock market data), access effective overviews of spatiotemporal patterns of the disease development and control (cf., meteorologists observing satellite images, contour maps, etc.), be provided with external memorisation of data to stimulate hypotheses and contemplate various decisions (cf. a general pacing around in a war room in front of many maps), and receive advice from an ensemble of analytical algorithms and visualisations about similarity, anomalies, clusters, correlation, causality, and association hidden in the data (cf. a CEO consulting specialists). c. There are not enough visualisation researchers around to support epidemiologists and modelling scientists. Mathematically, deriving an optimal model to forecast the contagion patterns of COVID-19 in different conditions (e.g., geographical, social, seasonal variation; different human intervention; etc.) is an intractable problem. It is an absolutely vital strategy to involve many modelling scientists and epidemiologists to develop different models because probabilistically, many developed models can produce sensible forecasts under some conditions. The more modelling scientists and epidemiologists can observe real-world data, examine model behaviours in different conditions, and compare the quality of different models, the more likely they can gain a better understanding and improve model performance in varying conditions. In 2020, there were some 100 university teams in the UK working on different epidemiological models. In comparison, there are only around a dozen of VIS teams in the UK. It is not feasible to pair a VIS team and a modelling team individually. d. There is a lack of a VIS infrastructure that can quickly be adapted to support epidemiologists and modelling scientists. The most cost-effective way to deliver VIS support to many modelling scientists and epidemiologists would be to have a technical infrastructure, which would host many applicable VIS techniques, and enable modelling scientists and epidemiologists to visualise any relevant data and analytical recommendations at their fingertips. Such an infrastructure could potentially support many other operations for combating COVID-19. Of course, it is understandable that we did not have such a VIS infrastructure ready in anticipating the COVID-19 pandemic. Ideally one could clone and re-purpose an existing VIS infrastructure, and adapt existing VIS tools for analysing and visualising epidemiological data. However, partly because of (b), there has not been adequate investment in the past for developing such a VIS infrastructure for any application. Consequently, during the emergency, there was not an existing infrastructure to clone, repurpose, or adapt. Nevertheless, "complaining does not work as a strategy" [75] , and "every challenge [we] face today makes [us] stronger tomorrow" [5] . RAMP VIS [78] is a group of 22 VIS volunteers, who answered a call in May 2020 to support the modelling scientists and epidemiologists in the Scottish COVID-19 Response Consortium (SCRC) [94] , which was part of the rapid responses organised by the Royal Society (UK) [93] . A few more volunteers joined the RAMP VIS group in the summer and autumn of 2020. As a volunteering operation in an emergency context, the VIS volunteers encountered many challenges. For example, the time urgency demanded rapid development of usable VIS tools, the travel restriction and the domain experts' heavy workload hampered in-depth requirements analysis, and parallel developments of pandemic models and data infrastructure entailed delays in accessing data to be visualised. In addition, there was a shortage of skilled developers for designing and engineering a VIS system, and a fair amount of uncertainty in organising and scheduling volunteering resources. The VIS volunteers made time urgency as their top priority, and were grouped into seven teams according to the available VIS expertise as well as different VIS needs in the SCRC modelling workflow. The grouping also enabled each team to progress independently in terms of requirements analysis, visual design, and system engineering. A few volunteers took part in different teams, facilitating knowledge sharing and collaboration among the teams. To our best knowledge, the experience of RAMP VIS volunteers is unprecedented in the VIS literature and epidemiology literature. In this paper, we describe our effort during 2020 to address the challenges in providing VIS support to epidemiologists and modelling scientists. By reporting and reflecting on our experience, we highlight the need for developing and providing VIS capacities to support a data-intensive emergency response. In this section, we review the application of VIS to emergency response and healthcare and discuss the existing methodologies that may be used to develop VIS in such applications. J o u r n a l P r e -p r o o f [25] . Kwan and Lee [54] incorporate geospatial visualisation into a realtime 3D emergency response system to support quick response to terrorist attacks. Chittaro et al. [16] introduce VU-Flow, a 3D visual environment that provides navigation guidance to users during emergency simulations. Based on the data collected from large community disaster events (e.g., 9/11 and Hurricane Katrina), Campbell et al. [12] use visualisation-based interactive simulation for training emergency response teams. Natarajan and Ganz [72] introduce distributed visual analytics for managing emergency response between geographically dispersed users. Waser et al. [101] incorporate visual designs into simulation-based investigation of flood disasters to recommend appropriate response strategies. Maciejewski et al. [61] introduce PanViz, a VIS toolkit providing decision support for simulated pandemic scenarios. Ribicic et al. [79] develop a VIS interface to provide flood simulations to non-expert users. Konev et al. [52] incorporate interactive visual designs into a simulationbased approach for flood protection planning. Gelernter et al. [33] provide visualisations to guide first responders at a crisis scene. Visualising social media data is commonly used to support emergency responses for situational awareness [60] , resource allocation [48] , critical infrastructure management [97] and post-disaster analytics [45, 66, 73] . Whitlock et al. [103] integrate VIS tools with mobile and immersive technologies to support critical operations during emergency response. VIS plays a critical role in mission-critical applications such as space missions. Abramyan et al. [2] develop an immersive visualisation environment for controlling space robots remotely on the Earth. Edell and Wortman [26] introduce advanced visualisations to assist the diagnosis of operational problems and failures for Van Allen Probes, a NASA space mission. Furthermore, the IEEE Conference on Visual Analytics Science and Technology (VAST) host an international challenge workshop annually since 2006. Competition entries demonstrated novel VIS solutions for epidemic spread [40, 38] , illicit activities [39, 22, 23] , security streaming data [19, 20, 21] and natural disasters [18] . Unlike most of the prior work generally carried out in preparation for a future emergency, this work was conducted during the period of an emergency response to the COVID-19 pandemic. VIS for Healthcare. The healthcare industry benefits from the adoption of Visualisation and Visual Analytics. Rind et al. [80] review VIS tools developed for the exploration of electronic health records. Carroll et al. [13] review 88 articles on VIS tools for infectious disease. Gotz and Borland [36] discuss challenges unique to the healthcare industry and the critical role that VIS plays in the domain. McNabb and Laramee [65] conduct an extensive survey of surveys including the adoption of VIS in the healthcare sector. Preim and Lawonn [77] survey the use of VIS for supporting decision making in the public health sector. The VIS techniques used in healthcare often incorporate analytical techniques. For example, Event Sequence Simplification is used to reduce visual complexity of sequential clinical events [105, 70, 37, 42] , and support a more efficient decision support process [4] . Natural Language Processing is used to extract textual data from raw clinical datasets [109, 34, 89] . Machine Learning help automating the processing of clinical data and providing guidance to clinicians and researchers, including Active Learning [6] , Support Vector Machine [99] , Topic Modelling [35] and Recurrent Neural Networks [55] . The main difference between this work and the prior work is that we had to develop multiple VIS capacities rapidly to address different VIS needs concurrently. Methodologies for VIS Applications. Design study methodology [83] builds on the nested model of design and validation [71] to provide guidelines, pitfalls, and a process that help visualisation researchers design systems in applied contexts. This methodology uses the metaphor of "the Trenches", which is where we found ourselves, and so, just as others have adapted its concepts and processes across varied settings [56, 91] , this paper reports our approach that adopts, adapts and sometimes contradicts established guidance in the context of rapid emergency response. Workshops [49, 51] can speed up requirements gathering. The five-design sheets methodology [81] can structure the sketching process. Collaborative design methodologies [59] can address issues due to travel restrictions, and assist rapid visualisation design processes [24] . Some prior work in VIS advocates different software engineering methodologies [11, 46] . The agile approach (Kanban boards, SCRUM, etc.) is particularly suitable for developing VIS systems in applications with changing characteristics of data, users, and tasks. A recently-proposed method based on the cost-benefit analysis can potentially be used to discover shortcomings in a VIS workflow and explore potential solution systematically [14] . We have drawn inspiration from these methodologies in our work. The Scottish COVID-19 Response Consortium (SCRC) [94] was established in April 2020 by researchers in three Scottish organisations in response to a call from the Royal Society for Rapid Assistance in Modelling the Pandemic (RAMP) [93] . The goal of the Consortium was to develop a more robust and clearer understanding of potential mediumand long-term strategies for controlling the COVID-19 epidemic in Scotland and in the UK. The Consortium currently has over 150 members from 36 organisations. On 14 May, Dr. Richard Reeve, the SCRC modelling coordinator, first met a VIS scientist. They discussed the SCRC's overall requirements for visualisation. As shown in the first sketch in Figure 1 The symbols "V" in pink circles indicate the needs for VIS. The discussion helped establish the need for visualisation capacities not only for dissemination of modelling results but also, perhaps more importantly, for many processes during model development and improvement. the widespread perception of visualisation as a tool only for information or knowledge dissemination. The VIS scientist described how different VIS techniques could enable domain experts to observe data quickly, analyse data with the aid of data mining algorithms, and improve their models through, e.g., visualisation of ensemble data, parameter space, and results from sensitivity or uncertainty analysis. Dr. Reeve embraced the idea of integrating VIS techniques throughout the modelling workflow, and revised the original sketch soon after (Figure 1(b) ). The VIS scientist (referred to as VIS coordinator hereinafter) indicated the need to enlist the help from many VIS experts. The following day (15 May), the VIS coordinator sent an email call for VIS volunteers to many VIS scientists, researchers, and developers in the UK, some of whom forwarded the call to others. By June 1, 22 VIS volunteers (including the VIS coordinator) answered the call. There are 19 faculty members, two industrial researchers, and one academic research officer. Among them, 14 indicated being able to prototype VIS software, and seven indicated willingness to engineer VIS systems. By June 2, the coordinator held meetings with all 21 volunteers individually or in small groups. The VIS volunteers had been using a diverse range of programming platforms. The most common denominator is D3.js [9] . Five VIS volunteers had experience of coding in D3.js (one became unavailable a few weeks later). At that time, several teams in the SCRC were working on six different epidemiological models and one team on inference and model validation, while substantial effort was devoted to the development of a data infrastructure for storing modelling results as well as captured data related to COVID-19 spread in Scotland. For the VIS volunteers, there were many unknowns, such as what data might be available, how it may be retrieved, what were the requirements of individual domain experts and individual models, and so on. While the agile methodology in software engineering [57] and the nested model [71] in visualisation advocate the necessity of iterative requirements analysis and software evaluation, they do not prescribe a full requirements analysis and software evaluation within a single iteration. Otherwise, they would be similar to the waterfall methodology. In the VIS literature, many application papers indicate that it usually takes many months to acquire a meaningful set of requirements (e.g., six months in [1, 30] and 12 or more months in [58, 27] ). To support an emergency response, a lengthy delay due to requirements analysis would not be acceptable. Hence we had to complement user-centred requirements analysis with the existing knowledge documented in the VIS literature, and commenced the development as soon as we had understood a partial set of requirements. As reviewed in Section 2, many papers in the literature reported VIS techniques and tools for supporting healthcare applications, model development, and mission-critical operations. If one can identify the data types, user tasks, and user knowledge in an application, one can relate them to the requirements in previously reported applications that featured similar characteristics of data, tasks, and users. During the two weeks when we were recruiting VIS volunteers, we gained our understanding of: • Datatypes -Based on several briefs from the SCRC modelling teams, we quickly learned that there would be a huge amount of time series data, and some geographical data (e.g., maps), network data (e.g., contact tracing), and multivariate data (e.g., demographic data). Building on our knowledge of VIS literature, we anticipated that some other types of data that might result from analytical algorithms, such as similarity matrices. • User tasks -Building on our knowledge of other VIS applications, we quickly established that there would J o u r n a l P r e -p r o o f Journal Pre-proof be a need for viewing time series in different ways for observation and comparison, in order to evaluate a model run against captured data, other runs, and other models. We anticipated that some analytical tasks would benefit from data mining algorithms, and at a later stage, domain experts would become interested in ensemble data visualisation and parameter optimisation. • User knowledge -Building on our experience working with other domain experts, we anticipated that (i) domain experts were highly knowledgeable about their own models, but could not avoid frequent observation of captured data and model results; and (ii) they were familiar with the major geographical locations in Scotland, but would need to incorporate map-based visualisation for smaller regions in Scotland and other UK regions. Meanwhile, we also consulted the abstracted theories and methodologies in the VIS literature. We observed that there was a need for all four levels of visualisation [15] . Figure 2 is an abstract representation of the workflow in Figure 1 . Although disseminative visualisation was the initial requirement, from Figure 2 , we can easily anticipate the needs for performing other types of visualisation tasks: • Observational Visualisation. During an epidemiological emergency, many epidemiologists and modelling scientists need to observe a huge amount of data routinely, such as viewing daily updates in different regions, simulation results of different models or different versions of a model, and previously captured or simulated data (to aid memory recollection). For example, one of the commonly encountered data types is time series. Given a time series with data values, observing these values in a time series plot is much quicker than reading values. Hence, for domain experts to observe time series routinely, enabling them to perform observational tasks at their fingertips can collectively save a huge amount of time at the scale of . Our initial requirement analysis did not identify the existence of any efficient and systematic support for such observational tasks. It would mean that domain experts either had to spend a lot of valuable time and cognitive resource to fiddle with spreadsheets or other tools to create visualisation plots or had difficulties to see enough data or view data frequently enough. • Analytical Visualisation. Many visualisation tasks are about identifying or determining the relations among the data objects being depicted without explicit visual confirmation of such relations. As the number of possible relations or grouping patterns is a combinatorial function of the number of data objects, such analytical tasks are usually more complex and time consuming when there are many data objects in visualisation imagery. Figure 2 : Visualisation tasks can be categorised into four levels according to the complexity of their search spaces. The workflows for epidemiological modelling, especially during an emergency, can benefit significantly from observational, analytical, and model-developmental visualisation in addition to disseminative visualisation. While visualisation can significantly reduce the time required for information acquisition, it can also enable experts to reason with the data in conjunction with their knowledge about, for instance, related facts and events in the reality, shortcomings in data collection and processing, and scientific understanding about the models concerned. data points) representing situations in regions, visually determining the similarity among these time series or grouping them according to their similarity is more complex than observing each time series independently. In terms of the space complexity in computer science, the search space has a polynomial growth in relation to . Hence, when is a big value, it is highly desirable to use algorithms to reduce the search space by for instance, ranking the similarity or recommending clusters. Such algorithmic suggestions can be conveyed to the domain experts through various types of visual illustration, such as connecting, ordering, highlighting, etc. Meanwhile, it is necessary for the domain experts to be aware that different algorithms may yield different algorithmic suggestions and algorithms may not have considered all variables necessary for correct J o u r n a l P r e -p r o o f Journal Pre-proof suggestions. In an emergency scenario, it is typically uncertain about the best algorithm or missing variables. Hence, VIS tools should try to offer different algorithms and empower domain experts to judge the quality of algorithmic suggestions. In general, a reasonable algorithm is expected to reduce the search space significantly even if it may not result in an absolutely correct suggestion. • Model-Developmental Visualisation. Developing epidemiological models is a mission-critical operation in combating COVID-19 [62] . There is a diverse range of models in terms of their epidemiological conceptualisation, mathematical specification and computational structure. Each model typically has many parameters, and different parameter combinations effectuate different model behaviours. Hence, the search space for an optimal model is usually intractable. For example, given parameters, each with possible values, there are combinations of parameter values. Some models have over 100 parameters, and many parameters are real numbers. Hence, it is unlikely that one can explore all combinations. It is crucial to empower model developers to use their knowledge to explore the search space intelligently, effectively, and efficiently. In some disciplines, such as computational fluid dynamics and machine learning, VIS researchers commonly design and develop model-specific VIS tools to enable model-developers to explore their search space effectively and efficiently (e.g., [64, 82] We therefore concluded that although disseminative visualisation was the initial requirement, the above considerations about data, tasks and users confirmed that the priority should be given to observational, analytical, and modeldevelopmental visualisation [15] . We also anticipated that the increasing complexity from observational to analytical and to model-developmental visualisation compels increasing depth of collaboration, time needed for requirement analysis, creativity in visual designs, and effort for iterative design evaluation and optimisation. Meanwhile, we could not and should not delay the infrastructural development, which demanded rather-scarce skills and experience of designing and engineering deployable VIS systems. The VIS volunteers were thus organised into several teams, including a generic supporting team (focusing on observational visualisation), an analytical support team, four modelling support teams, and a disseminative visualisation team. We placed all D3.js developers into the generic support team, and distributed other VIS volunteers according to their expertise and time capacity. In the following four sections, we report the activities of these teams, including further requirement analysis conducted by each team. Infrastructure Setup. The generic support team consists of mainly VIS volunteers who can program in D3.js and have developed and deployed VIS systems. Our requirement analysis indicated that domain experts were not able to observe data regularly. Using the recently-proposed method for optimising VIS workflows [14] , we quickly identified that this was caused by the cost of reading and visualising data using spreadsheets, and a solution is to develop a VIS infrastructure closely coupled with the SCRC data infrastructure that was being developed. The goal of the team was to enable observational visualisation for every piece of data held by the data infrastructure. The development of the SCRC data infrastructure started several weeks before VIS volunteers joined SCRC. A group of professional research software engineers (also volunteers) have carried out the design and implementation since. The goal is to capture the provenance of models and their results, enabling all contributing elements traceable from results to models and the conclusions drawn. Transparency is thus a key principle. All models and core software components are open-source [95] . Hence the VIS infrastructure has also been developed in the open. (STFC) provides the data and VIS infrastructure featuring virtual machines on the STFC cloud service, including a chat platform for collaboration and a data registry for web applications. The readiness of STFC for emergency responses enabled the hardware for the VIS infrastructure to be available within 24 hours after our request. As emergency responses, the SCRC data infrastructure, VIS infrastructure, and six epidemiological models were developed in parallel. While the generic support team was waiting for the measured data, we had access to some Scotland data in three spreadsheets, which contained over 300 time series and a few data tables. We anticipated that there would be at least thousands of time series when data from other regions and model runs became available. Such scale would be a challenge to the domain experts as well as the VIS developers. Users would need to assess the relevant plots quickly, while developers would need to adapt each visualisation program (referred to as a VIS function) to other applicable data with minimal development effort. The architecture of the RAMP VIS server, featuring its ontology, agents, and services, and their relationships with the UI, web pages (dashboards and plots), VIS functions (programs and test data), fetched and derived data streams, and SCRC data infrastructure. • A personal portal for storing frequently-used visualisations; • Because each visualisation is given a unique URL, users can also tag frequently-used visualisations on a web browser; • A collection of dashboards, each providing links to other dashboards and visualisation plots. On the RAMP VIS server [74] , there are broadly two types of visualisations, dashboards and plots. Each dashboard is designed to show key indicators and/or summary plots that some domain experts need to view frequently. For example, it may show the daily data of a region or a summary of a model run (Figure 3(b) ). Some information may serve as overviews while others serve as detailed views. The data objects, visual widgets, or summary plots on a dashboard are all clickable, providing a gateway to fuller or more detailed visualisations or other dashboards when required. In addition to a set of pre-defined dashboards, the generic support team provides a service to SCRC domain experts for constructing new dashboard whenever needed. The team have developed a variety of visualisation plots. As shown in Figure 3 (c), some plots feature interactive capabilities, some compare multiple data streams, some convey the analytical results produced by data mining algorithms, and some display modelling results with estimated uncertainty. Architecture and Ontology. To address the aforementioned second need for propagating each VIS function (either a dashboard or a plot) to other applicable data, the team has designed and developed an ontology-and agent-based architecture for the RAMP VIS server [50] . When a VIS developer fetches a task of writing a VIS function in D3.js, an infrastructure manager creates a program template with one or more example data streams. The binding of the VIS function and the given data streams results in a unique web page. Once the development completes, the VIS function can be reused by replacing the sample data stream with other applicable data streams. Through a simple UI, the infrastructure manager queries a Search Service to find all suitable candidate data streams, finalises a collection of data streams for propagation, and calls a Propagation Agent to create new bindings (and web pages) for these data streams automatically. As illustrated in Figure 4 , in the VIS infrastructure, an ontology provides the vital support to the search facilities that enable users to find desired dashboards and plots and the infrastructure manager to find applicable data streams for propagation. The ontology is a graph data structure that stores the relationships among all VIS functions, all data streams, and all data-VIS bindings (and the resulting web pages). Because we modelled the ontology using a document data model [17] , we implemented the ontology using three MongoDB database collections, which are: • OntoVis for defining and keeping the records of all VIS functions and their metadata; • OntoData for storing the records of all data streams and their metadata; • OntoPage for maintaining the binding points between VIS functions and data streams, their metadata and the URLs of the corresponding web pages. As shown in Figure 4 , in addition to the aforementioned Search Service and Propagation Agent, there are: • A Downloader Agent for fetching data from dynamicallychanging data automatically from the SCRC data infrastructure; • A set of Analytical Agents, each applies an analytical function or data mining algorithm to a predefined collection of data streams and generates derived data to be rendered, e.g., a similarity matrix of a collection of time series (see also Section 5); • An Indexing Agent that periodically scans the database operation logs and updates various textual descriptions in the ontology, which may be used in search or displayed by VIS functions; • A Thumbnail Agent for creating and updating the thumbnails of dashboards and plots that may change due to the dynamic change of the underlying data. • A Bookmark Service for managing bookmarks in users' portals; • An Authentication Service for approving a user's login action; • An Authorization Service for distinguishing ordinary or administrative users. The RAMP VIS server was implemented with backend microservices using two state-of-the-art REST-API frameworks: JavaScript-based NodeJS and Python-based Flask. The I/O-intensive operations (e.g., database or file-system access) are performed asynchronously and are implemented with a NodeJS microservice. The CPU intensive operations (e.g., running Analytics Agents) are implemented with a Flask microservice. The Flask framework also provides Analytics Agents with some off-the-shelf analytical libraries, e.g., NumPy, SciPy, scikit-learn. The SCRC data infrastructure provides the Downloader Agent with Python APIs to fetch H5 data. As exemplified by numerous visual analytics papers, VIS applications with a large data repository are expected to employ both data analysis algorithms and visualisation techniques. For example, analytical tasks have been critical for assessing model performance and uncertainty (e.g., comparing predictions to observations and comparing multiple model runs) [53] , and for exploring epidemiological data (e.g., identifying areas, time periods, or demographic groups exhibiting similar trends in outbreak progression) [7] . With the huge number of time series to be hosted by SCRC data infrastructure, we anticipated that simply relying on observational visualisation might not be efficient or effective. We thus grouped several VIS volunteers with strong data mining experience into the analytical support team. After several attempts to acquire detailed requirements for analytical visualisation capacity, we learned a high-level requirement from some modelling scientists, i.e., "crossmodel validation" would be needed at some stage. With their expertise and initiatives, the team members anticipated that comparing time series would be an unavoidable analytical need because of the sheer volume of dynamically expanding time-series data. The initial data available to the team contained hundreds of time series for different regions of Scotland, different indicators (e.g., test, case, hospitalised, and fatality), different genders and age groups, and so on. While waiting for modelling data to be prepared for comparative analysis, the team started to develop visual analytics techniques for summarising, simplifying, and comparing time series and for searching and visualising patterns and structures in the data. Building on the VIS literature on timeseries [3] , we anticipated the following analytical tasks: (i) discovering recurring trends, (ii) looking for outliers, (iii) identifying clusters, and (iv) measuring similarities according to one or more characteristics (e.g., scale, gradient, time lag, etc.). The team's anticipation was confirmed when more data and specific requirements arrived several months later. Guided primarily by these tasks, we experimented with a number of analytical visualisations in conjunction with an analytical toolbox that consists of different metrics and algorithms for quantifying the characteristics of individual time series, computing pairwise similarities, and transforming the time series to feature spaces that enable their similarities and clustering to be visualised. Analytical Toolbox. We started developing the analytical toolbox by creating a library of low-level analytical filters that treat time series as 1-D signals. Since time series may contain noise, various types of filters (e.g., flat, Hanning, Hamming, Bartlett, and Blackman windows) can be used to smooth time series when required. We then added a comprehensive library of analytical metrics for measuring the distance, difference, similarity, or error between two time series (e.g. mean square error and many of its variations, Pearson correlation coefficient, structural similarity index measure, mutual entropy, Spearman correlation coefficient, Kendall's tau, peak signal-to-noise ratio, -test, and so on). We further included algorithms such as dynamic time warping (DTW) and dimensionality reduction methods such as principal component analysis (PCA) and multi-dimensional scaling (MDS). Analytical Visualisations. We also experimented with a number of visual representations, focusing on comparing time series with data points. If one needs to determine any group of segments of time series that may be similar, the number of possible groups to be observed would be at the level of ( ), hence using analytical algorithms to narrow down the search space is highly beneficial [15] . However, relying on metrics alone is not sufficient since time series could be similar/dissimilar due to factors not encoded in the data (e.g., differences in terms of demography and intervention). In that respect visualisation provides ways for domain experts to incorporate their knowledge when analysing and comparing different time-series. Figure 5 shows one set of our experiments for analysing the time series associated with the 14 regional health boards in Scotland. • A time-lag visualisation compares two time series by registering them using a cross-correlation that computes the displacement of one time series relative to the other. A viewer can foresee what the future may look like in one board if it follows the same trends as another board, but with a delay. • A heatmap matrix shows pairwise similarity scores among all time series reported by different health boards. Row or column headers can be accompanied by time-series profiles for detailed observations. This is especially important when the similarity/difference measures are difficult for viewers to interpret. • A force-directed graph produces a layout where the nodes represent time series, and the length of each edge encodes the similarity/difference (short/long) between a pair of nodes. This visual representation is particularly useful for users to discover clusters of similar time series and outliers. • A chord diagram, which is shown in Figure 3 , places time series as segments/nodes along a circle, and uses the thickness or color of each chord to encode the similarity/difference measures. • A scatter plot compares time series in their feature space. Typically, two most important features (e.g., principal components computed using PCA) are selected as the axes of a 2-D space, and each time series is positioned as glyphs in the space according its feature coordinates. When the analytical support team examined these experiment results with the domain experts, one domain expert commented "These give us a lot to think about. It is not that we do not require these. We were just overwhelmed by what visualisation can do." Infrastructural Support. As shown on the right of Figure 5 , following the experiments, the analytical support team selects analytical algorithms and visual representations to be integrated into the VIS infrastructure maintained by the generic support team. Each analytical algorithm becomes an analytical agent, while each visual representation becomes a plot. Because many data streams in the infrastructure are updated dynamically, each analytical agent is scheduled to recompute various measures and generate derived data automatically. In this way, when an analytical plot is called, it always displays the analytical results based on the latest data. Mathematically, finding an optimal model to forecast the contagion patterns of COVID-19 in different conditions (e.g., geographical, social, seasonal variations, different human interventions, etc.) is an intractable problem. Nevertheless the effort to develop better models and improve existing ones is both necessary and desirable [62] . When VIS volunteers were first gathered together, we anticipated that supporting the model development in SCRC would be the most challenging undertaking, because VIS would have to support the search for better models in an NP space (i.e., the EXPSPACE class) [15] . We therefore organised some 10 VIS volunteers into four modelling support teams (referred to as Teams , , , and below), providing opportunities for each team to focus on supporting one or two SCRC modelling teams through close collaboration. Model Simulation.jl [43] simulates the spread of COVID-19 based on spatial proximity and its effect on the local population according to its demographic structure, over time. The inputs are population counts in 10-year age-bands in 1×1km 2 to 10×10km 2 grid cells across Scotland. Given an initial set of "seed" locations on day 0, the model outputs the number of people in different COVID-affected categories for each subsequent day by age group in the same grid cell. Figure 6 shows a set of simulation results. We met the domain experts as the first version of the model was being created. There was an urgent need to visually inspect the relative proportions of COVID-affected individuals in different categories over time and space. When discussing the high-resolution model outputs and strategies one might use to summarise them to validate model outputs and (later) to compare different modelling scenarios, the need to freely explore these prior to establishing fixed tabular summaries became apparent. As this need was so urgent, we quickly established two VIS requirements: to enable (i) studying the relative proportions of COVID-affected individuals in different categories over space and time, and (ii) exploring the results at different scales from Scotlandwide to 1-km 2 neighbourhoods. Our solution to these challenges was to use interactive "tilemaps" [87] (also known as "glyphmaps" [104] and "embedded plots" [41] ) with (a) glyphs representing multiple aspects of the modelled output together and (b) on-the-fly interactive gridding of the output at a suitable resolution in response to zoom/pan user interaction. The technical challenge was to make interaction, with this very high-resolution data quick and responsive enough to facilitate, rather than impede, exploration. On the left of Figure 6 , data at a coarse spatial resolution is superimposed on a map of Scotland. The glyphs in the top, left image show the aggregated temporal trend as the population moves through the COVID-affected categories, with significantly lower proportions of affected populations in the lower populated areas of NW Scotland. The image below this is a snapshot on day 32, showing the different rates at which the virus is affecting the population. The right column is a more detailed view of SE Scotland, as a result of zooming/panning. The snapshot on day 32 (bottom right) shows that the virus is affecting different age groups differently. This is largely due to differences in resident population structure. Domain experts with knowledge of population density can see that low-density areas seem to act as "firebreaks". Although this is the inner working of the spatial spreading algorithm, its appearance in the visualisation started a debate among the domain experts, influencing the next stage of model development -to investigate the importance of population density on speed of disease spread. The interactive tilemaps with glyphs have provided a basis for the ongoing work for evaluating the relative importance of different factors in the modelling and comparison of different lockdown scenarios. The Simple Network Simulation (SNS) [28] is a disease state progression model that computes numbers of people in a series of states for specified age bands, for different geographic units, over time. The inputs are population counts in each of the age bands, spatial units, social contact data J o u r n a l P r e -p r o o f capturing interactions between age groups and a spatial interaction matrix that determines the extent of the likely flows of people, and thus transmission, between each pair of spatial units. The model calculates the number of people in different states of the disease for each age group for each spatial unit on a daily basis. These outputs vary according to a number of model parameter settings. When we connected with the SNS team, they were considering generating a lockdown spatial interaction matrix by grouping daily travel-to-work flows for those in different employment sectors according to the likely effects on their jobs of the "work from home" edict of April 2020. Their graphical approach used origin-destination matrices of data recorded in the UK Census of Population. However, flow quantities, group differences, and any geographic variations or effects of scale were difficult to see (Figure 7 ). The effects of this or other model inputs on models outputs were unknown. We collectively identified the opportunity to apply established VIS principles to ongoing efforts to visually explore two specific domain questions: (a) which types of workers should we include in the input network (and what difference does this make to outputs)? (b) what do model outputs look like (and how do they vary over time and by age-group)? These gave plenty of scope for using judicious visual design and interactive methods for filtering, highlighting and selection to develop elegant answers to challenging questions. Further questions that we hoped to address at a higher level were: (c) how do the answers vary with scale and geography? (d) is visualisation effective in answering these questions (in this context)? Our solutions were developed as data sketches [58] embedded in structured documents for discourse around data and design [107] and engaged in regular video conferences and online discussion in a series of tight redesign loops [63] . The discourse resulted in some preliminary answers to these questions. For instance, for questions (a, c), the nature of the input network has an effect on outputs, with distinct spatial variation and greater effect at smaller scales. We redesigned graphics, allowing us to identify the areas most affected by the employment sector selection (our naive modelling of lockdown), areas with high out-of-area epidemiological importance, and the sources of those who visit them ( Figure 7 ). These views were informative but do not dictate the scale or nature of the network we should use. We began by looking at NHS boards, but increased resolution to local authorities and then the higher resolution 'Middle Layer Super Output Areas' (MSOA) for selected areas as our answers, our knowledge of the kinds and scales of effect and their likely geographies developed. The application of VIS principles was informative and effective, and included guidelines and designs known to, used by, and sometimes developed by, the team, for example: • GridMaps [29, 67] -to add spatial information to origin-destination matrices and address the occlusion that occurs when flows are shown between variously sized geographic units using standard projections. • OD Maps [106] -to show the geographic variation of the effects of the different input networks. • Animated transitions [88] between alternative layouts -to relate spatial and semi-spatial geographic projections. • Interactive selection and filtering -to vary view parameters and support exploration. Asymmetric link curves [31, 108] to clearly show bi-directionality and asymmetry in the spatial interactions. For (b) and (c), initial model outputs were not particularly varied by space, time or attribute, but we do have plausible candidate methods that will allow such variation to be detected and assessed when the SNS models are finalized. Importantly, the visualization work resulted in better knowledge of the input network and its lockdown characteristics, and an emphasis on smaller scale "data zones". The collaboration therefore enables us to be more confident about injecting visualisation into the modelling process (d), to shine light on the models as they are developed, tested, and parameterised, in ways that had not been considered by the modellers (Figure 1) , and perhaps at fine scale. By the time VIS volunteers joined the SCRC, the contact tracing model [69] was already in development. The model simulates the spread of COVID-19 through a dynamic network that encodes the potential contacts among millions of individuals. These simulations result in some very large temporal networks [44] . We immediately established bi-weekly meetings with the modelling scientists. It quickly became clear that the domain experts did not have access to bespoke network visualisation tools, and were primarily relying on standard plots to view simulation results as disease progression curves and some summary metrics such as . They were also comparing different intervention policies through such plots. We noticed that the temporal networks, which were used to derive the summary information, were never visualised. Hence, the first urgent requirement for VIS was to enable domain experts to observe such networks in order to gain an intuitive understanding about the temporal and topological behaviours of the model. The visualisation would also assist domain experts in communicating modelling results and informing policy making. We addressed the requirement by using existing network visualisation tools to minimise the delay due to software development. This allowed us to build familiarity with the model and the data, while providing example visualisations to stimulate our discussions with the domain experts. We then progressed to more advanced VIS techniques, e.g., employing scalable graph-drawing techniques [85, 86] , geographic-inspired metaphors [32] , and graph-theoretic analysis for derived metrics to complement network visualisation ( Figure 8) . With bi-weekly collaboration meetings providing continuous feedback and ideas, we improved our prototypes iteratively through a web-based "project diary" and an open software repository [100] . As the collaboration matured over the period, we observed a trend that both domain experts and VIS volunteers actively contributed to the discussions on model building and visual design together. It became difficult to label whether a discussion was about visualisation or the model itself, and there were more discussions on generating insight than on producing software. One group of SCRC modelling scientists have been focused on quantifying epidemic characteristics, effect of intervention, model performance, result uncertainty, and parameter sensitivity [90] . From the onset, the VIS volunteers for supporting these modelling activities anticipated the use of ensemble and uncertainty visualisation. Uncertainty is a common property of epidemiological models. The basic uncertainty visualization typically depicts the main computational results (e.g., an aggregated time series) in conjunction with other visual information for conveying the uncertainty (e.g., some amorphous pattern behind the main time series, or additional time series representing the 25 and 75 percentiles of the ensemble). Such visual representation are commonly used in modelling. While the generic support team was developing these basic plots (e.g., top-right image in Figure 3 ), this team focuses on sensitivity analysis that helps modelling scientists to discover the reasons behind uncertainty. During an initial collaboration meeting, domain experts confirmed the need for visualising the sensitivity of model parameters. At that time, the domain experts were working on their models in parallel, thus VIS volunteers were not able to obtain multi-run simulation data in the early months of the collaboration. VIS Team took initiatives to study two COVID-19 models in the public domain, attempting to generate multi-run simulation data. Before this attempt could yield useful output, one SCRC model, ABC-smc [76] produced multi-run simulation data for uncertainty visualisation and parameter space analysis. By attending modelling scientists' meetings, the team were able to observe the interactions between the perspectives of modelling and uncertainty quantification, and established a set of requirements. Similar to most multi-run simulation problems, VIS needed to support the analysis of many sets of model parameters and outputs. In this case, each dataset consists of some 200 time series and their corresponding parameter sets. One obvious requirement is to visualise the uncertainty featured in the set of time series. As this is a common requirement for all models with time series outputs, we passed the requirement to the generic support team (see Figure 3 ). The team focused on the more complex tasks, i.e., (i) to identify input/output relationships, (ii) to determine key curve features such as maximum or largest slope, and (iii) to compare outputs from a number of different model runs. Immediately after the requirements analysis, we started to develop a VIS system iteratively, with increasing facilities for analytics, visualisation, and interaction. Figure 9 shows the current prototype after several iterations. We used the design approach of coordinated multiple views [10] , the visual analytics approach for computing a set of curve features [92] , a parallel coordinate plot [47] for viewing and filtering multi-dimensional parameter sets and curve features, and aggregated curves summarising the outputs of the selected parameter sets. We are in the process of introducing new facilities, such as slicing the multidimensional parameter space [98] , using functional box plots [102, 68] to summarize many curves, and the contributionto-the sample-mean plot [8] to show the sensitivity of outputs to input parameters. While most VIS volunteers were distributed among the aforementioned six teams, we also created a small team for disseminative visualisation since public information dissemination was the original overall requirement. One epidemiology researcher with interest in data visualisation also joined the team. The team explored the prevailing approaches, in the UK and internationally, in public-facing visualisations related to the pandemic. This ranged from those produced by a number of governments (e.g., the four home nations in the UK), organisations (e.g., WHO, UK ONS), universities (e.g., Johns Hopkins dashboards), media outlets (e.g., FT Coronavirus tracker), and non-commercial web services (e.g., Worldometers). The team concluded that we should complement, but not duplicate, the existing effort, and defined our goal as to inform the public about activities of SCRC through storytelling visualisation. We identified the following requirements: (i) to maintain scientific rigour, (ii) to retain scientific language, and (iii) to abstract visualisation output. For example, one of our initial designs used football result prediction and weather forecasting metaphors. Our rigorous consideration indicated that the former might be associated with gambling, whereas the latter could be perceived as inaccurate, and they could have negative connotations to how epidemiological modelling would be perceived. Figure 10 illustrates our process for creating storytelling visualisation. It started with an ideation phase where we elaborated preliminary concepts and ideas. This was done using a combination of the five-design sheets [81] methodology ( Figure 10(a) ), animated PowerPoint mock-ups, and web-based prototyping. These sparked off other explanatory forms, e.g., infographics and slide packs, such as in Figure 10(b) , which were used to describe each epidemiological model and the corresponding visualisations. As it would be difficult to apply a unified narrative to every model, we opted for allowing each story to be developed independently. Current storytelling visualisations have been implemented as: (a) web-based presentations using the Reveal.js framework, with SVG-based animations and the potential for directly feeding into them visualisations created by other teams, (b) as video outputs of animated presentations, and (c) as infographics, created using graphics editors and creative design tools. We are in the process of creating a public web server for hosting these storytelling visualisations. In this section, we reflect on our experience of developing VIS capacities for emergency response, and translate our reflections to a set of recommendations as a step towards a new methodology. Reflection on general perception of VIS as a dissemination tool. Expert users often see visualisation as "for informing others" rather than "for helping myself". This can be a big stumbling block during requirement analysis. As illustrated in Figure 1 , Dr. Reeve's response during the first meeting helped overcome this stumbling block, shortening the delay in requirements analysis by months. Meanwhile, to the disseminative visualisation team, creating such visualisation has not been an easy journey, especially without the advice from an expert on public engagement. Reflection on requirements analysis. During the pandemic, domain experts were extremely busy. Different teams did not follow the same formula for requirement analysis. and followed the recommended method for user-centred requirement analysis, and benefited from frequent engagement. The generic support team and team identified urgent requirements quickly and began their development without much delay. The analytical support team, team , and disseminative visualisation team had to use their knowledge to anticipate and analyse the potential requirements. In emergency responses, all of these are valid methods. Several teams had positive experience in using quickly-produced visualisations to stimulate requirement analysis. Team made good use of several communication mechanisms, including searchable threaded chat streams, structured feedback, and design exposition. A few VIS volunteers also found it rewarding to attend domain experts' meetings that at first appeared irrelevant. Reflection on team organisation. The categorisation of visualisation tasks based on the complexity of the search space of the possible solutions is relatively new [15] . It informs us that model optimisation is an NP process in general, and it requires all three levels of visualisation, i.e., observational, analytical, and model-developmental visualisation, which correspond to solution spaces of complexity ( ), ( ), and EXPSPACE (an NP class). The complexity is likely to impinge on the effort for identifying VIS requirements. Hence, having a VIS team working with each modelling team was necessary for establishing such understanding. All teams quickly identified and addressed the observational requirements related to individual models, and some have started to address the requirements for model analysis and model optimisation. Meanwhile, the generic support team progressed to the development stage quickly because of not only the necessity but also the less complex search space. Reflection on VIS resources. Using volunteer effort is not an ideal solution for emergency responses. It would be more efficient if we could utilise an existing technical and knowledge infrastructure for such an emergency response, if such an infrastructure had existed for other operations and had an advanced VIS server and a team of VIS developers who were knowledgeable about different levels of visualisation tasks. Our volunteering effort was a make-shift solution, which benefited strongly from the academic knowledge infrastructure in the UK. Its progress could be more rapid if there were more development resources. The organisation of VIS volunteers partly reflects the need to concentrate most development resources in the generic support team. Nevertheless, the outcomes delivered by the VIS volunteers between June and December 2020 without any funding are unprecedented. This demonstrates the importance of VIS as well as volunteering effort in emergency responses. Reflection on VIS in epidemiological modelling. From the perspective of epidemiological modelling, the RAM-PVIS effort allowed us to appreciate how VIS may be used in many aspects of modelling workflows, in addition to disseminating modelling results. Developing epidemiological models during an epidemic or pandemic is a not a new phenomenon. To have epidemiology teams develop multiple models is not uncommon, but activities by SCRC and RAMPVIS have demonstrated that the scale, complexity and dynamic nature of the many datasets required to support COVID-19 modelling benefits enormously from infrastructural support featuring data management, analysis, and visualization capabilities. Success in an emergency response, such as combating COVID-19, rarely hinges on a single correct model or a critical visual design. More likely, it will benefit from the collective effort of continuously developing, evaluating, and improving models, and from the capabilities for enabling rapid observation of numerous pieces of data and analytical results throughout the modelling workflows. The RAMPVIS effort showed that collective VIS support is necessary and feasible, and such capabilities should be available to epidemiological modellers from the very beginning in future emergency responses. Recommendation. Our approaches, experience, and reflections may be translated to the following recommendations for future VIS applications in emergency responses: • In December 2014, US President Barack Obama spoke to the National Institutes of Health (USA): "There may and likely will come a time in which we have both an airborne disease that is deadly. And in order for us to deal with that effectively, we have to put in place an infrastructure" [96] . Shadbolt et al. outlined the future need for a data ecosystem [84] . VIS should be part of any data ecosystem, and be closely coupled with or integrated into data infrastructures. The "readiness" of VIS technical and knowledge infrastructures will make a difference. While it may not be feasible to build an infrastructure for every type of potential emergency, we can benefit significantly to have a few VIS infrastructures that are ready to be cloned, re-purposed, and adapted for different emergency responses. • We need to make a serious effort to redress the common misconception that visualisation is only a dissemination tool. In particular, we need to help modelling scientists to become accustomed to use VIS techniques throughout their workflows. Perhaps the best way to broaden the uses of VIS in modelling workflows is to enable more collaborative research between VIS scientists and modelling scientists in different disciplines. • While the agile principle [71] fits well with VIS development for supporting emergency responses, one should be open-minded about different approaches. The diverse approaches taken by different VIS teams in the RAMP VIS effort indicated that standard practice might not always be applicable. VIS development in emergency responses can benefit tremendously from the existing VIS knowledge, in the form of theories, methodologies, literature, and personal experience. The VIS community should improve its "readiness" by advancing abstract VIS knowledge in the form of theories and methodologies. In this paper, we have reported the work carried out by a group of VIS volunteers to support modelling scientists and epidemiologists in combating COVID-19. Our approaches to the challenges that we have encountered are rare and valuable contributions to the first step towards a methodology for developing and providing VIS capacity to support emergency response. In November 2020, the UK Research and Innovation awarded funding to the group, transforming the volunteering effort to a more structured VIS operation J o u r n a l P r e -p r o o f Journal Pre-proof RAMPVIS: Visualisation Capabilities for Large-scale Emergency Responses in 2021. This allows us to develop a VIS infrastructure that can be deployed to support some ongoing modelling effort as well as be served as a major example to influence VIS infrastructure. Meanwhile, we continue to strengthen the collaboration between VIS researchers and epidemiological experts, developing more domain-specific VIS techniques for supporting epidemiological modelling workflows. Rulebased visual mappings -with a case study on poetry visualization Stage: Controlling space robots from a CAVE on Earth Visualization of Time-Oriented Data Temporal reasoning for decision support in medicine The Light in the Heart A visual active learning system for the assessment of patient wellbeing in prostate cancer research Time series regression studies in environmental epidemiology Contribution to the sample mean plot for graphical and numerical sensitivity analysis D 3 data-driven documents A coordination model for exploratory multiview visualization The mythical man-month Emergency Response Planning and Training through Interactive Simulation and Visualization with Decision Support Visualization and analytics tools for infectious disease epidemiology: A systematic review An ontological framework for supporting the design and evaluation of visual analytics systems What may visualization processes optimize? VU-Flow: A visualization tool for analyzing navigation in virtual environments MongoDB: the definitive guide: powerful and scalable data storage VAST Challenge VAST Challenge VAST Challenge VAST Challenge VASTChallenge2018/challenges/Mini-Challenge2 VAST Challenge VASTChallenge2018/challenges/Mini-Challenge3 Rapid development of visualization dashboards to enhance situation awareness of COVID-19 telehealth initiatives at a multihospital healthcare system Information visualization for emergency management: A systematic mapping study Visualization of a Spacecraft Mission Software System QualDash: Adaptable generation of RAMPVIS: Visualisation Capabilities for Large-scale Emergency Responses visualisation dashboards for healthcare quality improvement Simple Network Simulation Improved grid map layout by point set matching Categorical colormap optimization with visualization case studies Interactive poster: Overlaying graph links on treemaps GMap: drawing graphs as maps Visualization and communication tool for emergency response PhenoStacks: Cross-sectional cohort phenotype comparison visualizations PhenoLines: Phenotype comparison visualizations for disease subtyping via topic models Data-driven healthcare: Challenges and opportunities for interactive visualization DecisionFlow: Visual analytics for high-dimensional temporal event sequence data VAST Challenge 2011 MC1 -Characterization of an Epidemic Spread VAST Challenge 2011 MC3 -Investigation into Terrorist Activity VAST Challenge 2010 MC2 -Characterization of Pandemic Spread Visualizing complex data with embedded plots Visual progression analysis of event sequence data Temporal networks UofC-Bayes: A Bayesian approach to visualizing uncertainty in radiation data The pragmatic programmer Parallel coordinates: A tool for visualizing multi-dimensional geometry RescueMark: Visual analytics of social media data for guiding emergency response in disaster situations: Award for skillful integration of language model A framework for creative visualization-opportunities workshops RAMP VIS REST-ful API Extending recommendations for creative visualization-opportunities workshops Run watchers: Automatic simulationbased decision support in flood management Interactive visual analysis of families of function graphs Emergency response after 9/11: the potential of real-time 3D GIS for quick emergency response in micro-spatial environments RetainVis: Visual analytics with interpretable and interactive recurrent neural networks on electronic medical records Bridging from goals to tasks with design study analysis reports Agile and Iterative Development: A Manager's Guide Human-centered approaches in geovisualization design: Investigating multiple methods through a longterm case study Distributed synchronous visualization design: Challenges and strategies SensePlace2: GeoTwitter analytics support for situational awareness A pandemic influenza modeling and visualization tool Modelling: Understanding pandemics and how to control them. The Newton Institute Preprint Action design research and visualization design Over two decades of integration-based, geometric flow visualization Survey of Surveys (SoS) -Mapping the landscape of survey papers in information visualization RAMPVIS: Visualisation Capabilities for Large-scale Emergency Responses Using a multi-level and multi-resolution visual analytics software to understand the aftermath of a catastrophe Small multiples with gaps Curve boxplot: Generalization of boxplot for ensembles of curves Contact-Tracing-Model Temporal event sequence simplification A nested model for visualization design and validation Distributed visual analytics for collaborative emergency response management EQSA: Earthquake situational analytics from social media RAMP VIS User Interface The Last Lecture. Two Roads A survey of visual analytics for public health Visualization and visual analytics in support of rapid assistance in modelling the pandemic (ramp Sketching uncertainty into simulations Interactive information visualization to explore and query electronic health records Sketching designs using the five design-sheet methodology VIS4ML: An ontology for visual analytics assisted machine learning Design study methodology: Reflections from the trenches and the stacks The challenges of data in future pandemics. The Newton Institute Preprint Drawing dynamic graphs without timeslices Event-based dynamic graph visualisation Tilemaps for summarising multivariate geographical variation Configuring hierarchical layouts to address research questions Doccurate: A curation-based approach for clinical text visualization Challenges in estimation, uncertainty quantification and elicitation for pandemic modelling. The Newton Institute Preprint Design study "lite" methodology: Expediting design studies and enabling the synergy of visualization pedagogy and social good Visualization of time-series data in parameter space for understanding facial dynamics Rapid assistance in modelling the pandemic: Ramp The Scottish COVID-19 Response Consortium The Scottish COVID-19 Response Consortium, 2020b. Repositories Remarks by the president on research for potential ebola vaccines Can twitter save lives? A broadscale study on visual social media analytics for public safety Sliceplorer: 1D slices for multi-dimensional continuous functions NLPReViz: An interactive tool for natural language processing on clinical text Contact tracing visualizations Nodes on Ropes: A comprehensive data and control flow for steering ensemble simulations Contour boxplots: A method for characterizing uncertainty in feature sets from simulation ensembles Designing for mobile and immersive visual analytics in the field Glyphmaps for visually exploring temporal patterns in climate data and models LifeFlow: Visualizing an overview of event sequences Visualisation of origins, destinations and flows with od maps Design exposition with literate visualization RAMPVIS: Visualisation Capabilities for Large-scale Emergency Responses Visualizing the dynamics of london's bicycle-hire scheme AnamneVis: A framework for the visualization of patient history and medical diagnostics chains RAMPVIS: Answering the Challenges of Building Visualisaton Capabilites for Large-scale Emergency Responses This statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) statement is signed by all the authors This statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) statement is signed by all the authors This statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author This statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) Author's name (typed) statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) statement is signed by all the authors (a photocopy of this form may be used if there are more than 10 authors): Author's name (typed) statement is signed by all the authors The authors would like to express their gratitude to the SCRC management team for its leadership, all SCRC members for their selfless efforts, the UK Science and Technology Facilities Council (STFC) for providing hardware to SCRC and RAMP VIS, and the STFC for providing and managing the SCRC hardware platform.In particular, we would like to thank all other VIS volunteers for their contributions to various VIS activities during 2020, including Professor Nigel W. John, Dr. Helen C. Purchase, Dr. Stella Mazeri, Dr. William Teahan, Benjamin Nash, Tianci Wen, Dr. Dylan Rees, Elif E. Firat, and James Scott-Brown. In addition, we acknowledge that some VIS activities have received support from Dr. Alessio Arleo, Dr. Peter Butcher, Dr. Cameron Gray, Jeewan Hyongju, and Edison Mataj.We would like to express our appreciation for Andrew Lahiff for setting up and managing the RAMP VIS VMs, Sonia Michell for offering innumerable pieces of advice on data products, and Dr. Iain McKendrick for leading the discussion on authorship guidelines.Last but not least, our big thanks go to the modelling scientists and epidemiologists who worked with the VIS teams closely and provided valuable expert advice as well as datasets to be visualised. They include Dr. Jess Enright, Dr. Claire Harris, Dr. Glenn Marion, Dr. Sibylle Mohr, and Dr. Thibaud Porphyre. All persons who have made substantial contributions to the work reported in the manuscript (e.g., technical help, writing and editing assistance, general support), but who do not meet the criteria for authorship, are named in the Acknowledgements and have given us their written permission to be named. If we have not included an Acknowledgements, then that indicates that we have not received substantial contributions from non-authors.  Enhancing epidemiological modelling workfows with a range of visualizaton and visual analytcs (VIS) support;  Supportng an emergency response with a large-scale VIS volunteering efort that is unprecedented in the feld of VIS or previous emergency responses;  Demonstratng the feasibility for developing a VIS infrastructure as well as VIS sofware for supportng four-levels of visualizaton;  Evidencing that VIS capabilites should be available to epidemiological modellers from the very beginning in future emergency responses. All persons who meet authorship criteria are listed as authors, and all authors certify that they have participated sufficiently in the work to take public responsibility for the content, including participation in the concept, design, analysis, writing, or revision of the manuscript. Furthermore, each author certifies that this material or similar material has not been and will not be submitted to or published in any other publication before its appearance in the Epidemics (Elsevier). Please indicate the specific contributions made by each author (list the authors' initials followed by their surnames, e.g., Y.L. Cheung). The name of each author must appear at least once in each of the three categories below. Authors would like to confirm that they are not aware of any potential conflict of interest.