key: cord-0040096-75dzc62b authors: Lindgaard, Gitte; Dudek, Cathy; Tsuji, Bruce; Noonan, Patrick; Sumegi, Livia; Stojmenovic, Milica title: Emergency Response in Simulated Terrorist Attacks: Usability Lessons Learned date: 2012-05-17 journal: Usability in Government Systems DOI: 10.1016/b978-0-12-391063-9.00039-0 sha: ec2a6dc011a998f80c14c773c8c4f8795f99b170 doc_id: 40096 cord_uid: 75dzc62b nan Television shows such as CSI Miami (Crime Scene Investigation) have done much to raise public interest in, and awareness of, the advanced science, technology, detective work, and human knowhow that go into solving serious crimes. They have, however, also created some misconceptions concerning the time it takes to conduct these investigations and the level of resources that firstresponder agencies have at their disposal. In reality, there is room for improvement in all aspects of support for managing criminal events, including in the range of existing database systems and technologies. This chapter outlines some of the most important lessons we have learned from our involvement in the definition, UI design, and usability evaluation of applications intended to support the spot, and identifying potential offending explosive or chemical agents, even before they really know what they are dealing with. There is no way that a team of HCI researchers would be allowed to observe the activities of first responders in a real major CBRNE event, the kind of which has, thankfully, not been experienced in Canada as yet. Instead, simulated events are staged around the country, but as these are extremely time-consuming and expensive to arrange, they occur relatively rarely. In the course of obtaining the necessary data for our HCI tasks, we have had several opportunities to observe, record, and analyze such simulations. Simulated events offer experienced CBRNE teams an opportunity to practice managing these as well as offering additional training for responder teams with limited experience. It is not surprising that few of our standard UCD methods (see e.g., Preece, Rogers, & Sharp, 2011; Shneiderman, Plaisant, Cohen, & Jabobs, 2009 ) apply comfortably in those environments. Still, our team has been able to contribute to the generation of user requirements and UI design, and the UI evaluation of a suite of such applications over the past 10 years. The rest of the chapter presents what we have learned in the design, development, and evaluation of three projects in which we have been involved. It ends with a summary of the takeaway messages that we think may be useful to user experience practitioners who face similar uncertain, often unpredictable conditions "in the wild" (Hutchins, 1995) . The area in which a CBRNE event has taken place is designated the "hot zone" into which only first responders are allowed. The hot zone is surrounded by the "warm zone" in which, in a large event with mass casualties, a team of operations specialists representing the various responder agencies are located. This is also where first responders and casualties are decontaminated before exiting to the "cold zone" that surrounds the "warm zone." One task of police "generalists" is to seal off all three zones to prevent unauthorized people from entering into the entire controlled area and risk contamination or injury. For safety reasons, the event management team located in the command post is placed well away from the crime scene, often even beyond the designated "cold zone." In large events, the operations specialists are responsible for directing the actions of the first responders in the hot zone and for relaying relevant information to the management team in the command post. Each member of the command post team is a team leader for their group of responders. The team as a whole is responsible for the strategic management of the event and for the overall action plan and amendments to the same as the event unfolds. In smaller events in which no operations specialists are required, command post team members communicate directly with the first-responder teams in the hot zone. A CBRNE crime scene involves the presence of numerous civilian agencies such as police, fire, medical, and veterinary responders. Each agency has its own roles and responsibilities, its own procedures, language, reporting formats, and so on, but the management of a large adverse event requires all involved to work together as a team. The command post team typically comprises officers representing Emergency Medical Services (EMS), specialist police officers, and hazardous materials ("hazmat") technicians, who belong to a special branch of the firefighting services, as well as an event manager. Depending on the nature and scope of the event, the event manager may be a police officer, a hazmat technician, or, in the case of a major rapidly spreading infectious disease outbreak such as the Severe Acute Respiratory Syndrome (SARS) in 2003, a public health authority. The EMS team leader is responsible for the first-responder paramedics, whose role is to perform triage on casualties before they are decontaminated, discharged, or transported to a hospital. In a large-scale event involving mass casualties and possible deaths, a senior public health officer may be called in to direct the flow of patients to different hospitals. That person is also responsible for keeping the media and the Minister of Health informed of the unfolding event. If the event escalates, spreading further and going beyond a certain state/province, federal public health authorities are called in to lead the investigation in collaboration with their state/provincial and local counterparts. If the threat goes beyond a country's border, they are also responsible for reporting it to the World Health Organization (WHO). The police officers include "generalists" who seal off the area surrounding the cold zone, re-directing traffic around it. A trained bomb technician is responsible for identifying the explosive materials and for preventing further bombs from detonating. He is likely to work closely with the hazmat technician when a "dirty bomb" (Dingle, 2005; Petroff, 2003; Zimmerman & Loeb, 2004 ) that may involve nuclear materials or other dangerous chemicals as well as explosives is suspected. If, for example, a suspicious package or backpack with unknown contents has been detected in a public area, the bomb technician and hazmat technician may send in a robot to help in the investigation before allowing people into the hot zone. These robots are equipped with remotely controllable equipment, including mobile video cameras and tools with which to take samples of any chemicals to help identify the offending agent without endangering first responders. The Forensic Identification Specialist (called FIS or "Ident") sends his staff into the hot zone once the area has been rendered safe, to collect evidence for a possible subsequent court case. As the interval between the occurrence of an event and the case going to court is often several years, all items found in the hot zone must be given an ID, be photographed, and packaged in several layers of sealed plastic bags for storage. The FIS officer signs each bag as he seals it. If for any reason the item is removed before the court case, it is resealed and the relevant officer again signs each bag. A radiofrequency ID (RFID) tag attached to each exhibit enables the police to retrieve all evidence associated with a given event once the court case goes ahead. The hazmat team's responsibilities include taking samples of the offending agent in the hot zone and relaying important information to the command post team leader. This may include the color and texture of the agent, any smells or fumes exuding from it, or any other physical or behavioral signs they are able to discern. Together with information about the nature and extent of injuries of people in the hot zone, which the EMS first responders report to their team leader, this information is passed to other members of the hazmat team, located in special fire trucks equipped with a plethora of online and printed resources. Their role is to identify the agent as quickly as possible and communicate their findings to the command post team leader. The team leader decides on the level of Personal Protective Equipment (PPE) that everyone in the hot/warm zones must wear and on the decontamination agent to be used. PPE may involve impervious full body suits, gas masks, heavy boots, and gloves. The team leader also controls the entry into and exit from the hot zone to ensure that only authorized personnel is present. The firefighters decontaminate everyone exiting from the hot zone. Before entering the hot zone, the EMS team records every first responder's vital signs (pre-vitals)radial pulse, blood pressure, respiration rate -and notes the precise time of entry. For safety reasons, a first responder is allowed to remain in the hot zone only for a strictly controlled period of time. Once his time is up and after being decontaminated, his post-vitals are recorded to ensure that he has not been unduly affected or even injured while in the hot zone. Communication between the first responders and their operational team or, in smaller events, with their command post team leaders, is maintained via radio. Each team uses a unique frequency to ensure clear communication between the responders and their team leader. Frequencies are agreed upon at the outset of the response. In addition, some command post team leaders rely on laptops and on cell phones for other communication during the event. MedPost presents, in one central place, an aggregate view of the essential information about casualties (number of people affected, who they are, where they are, their condition, etc.). It enables integration of communications between on-scene responders, hospital staff, and the response community involved in managing a CBRNE event. By capturing medical information about casualties and allowing different levels of responders to view it, it maintains, or improves, the situational awareness (Artman & Waern, 1999; Garbis & Altman, 2004; Riley, Endsley, Bolstad, & Cuevas, 2006) of the event among the specialists managing the crisis. Figure 7 .1 shows the communication flows and the data sources for people who may be linked to MedPost in the event of a major outbreak. As Figure 7 .1 shows, MedPost can communicate with users in several hospitals and temporary treatment centers. The Rapid Triage Management Workbench, a first-responder triage application that shares critical information among people responsible for first response, casualty care, command and control, and public communication, is used to track, manage, and assign casualties to the proper point of care. It facilitates the triaging of casualties from the event scene through to transport at the hospital by capturing initial medical data in the field (Lindgaard et al., 2006) . Finally, there is communication with individual clinics, the command post, and with the public health authorities. To generate user requirements for each target user population, our team had prepared a fictitious but plausible scenario (Rosson & Carroll, 2001) for each type of CBRNE event. The explosion scenario, for example, read as follows: A few minutes ago on the 10AM CBC radio1, you heard news of a terrifying accident in Montre´al's Central station. The magnitude of the disaster is not yet known. It is unclear if a highly toxic gas has been accumulating in the city's ageing sewerage system that runs underneath the railway net, or if it is an act of terrorism. Members of the public have apparently been complaining about extremely foul smells in the streets of Old Montre´al around that area for over two weeks. However, because garbage handlers have been on strike for the past four weeks, the accumulating garbage in the streets was believed to be the source of the smells. According to the report, a series of at least five deafening explosions occurred in rapid succession around 9:52AM. At least two buses are on fire; the station is apparently in ruins, and it is not known how many passenger trains may also have been hit by the blast. No information is as yet available about the number of casualties. Jeremy Hall, the Mayor of Montre´al, is on the line requesting your immediate assistance. Having recently provided funding to install the MedPost system in two of the city-owned hospitals, Mr. Hall is aware of your involvement in its development as well as of your expertise at handling major disasters in Canada and abroad. MedPost has not been fully implemented yet, but Mr. Hall believes it may be possible to use a limited version of it to help monitor the situation. This limited version, which is intended for training medical staff, is capable of tracking up to five types of data selectable from a wider range, but no one in Montre´al knows which data types will be most valuable for managing the situation efficiently, perhaps using a mixture of electronic and manual data tracking. They are also unsure as to what can, and what must be, reported to whom, when, and how often. What would you advise? While our team prefers to work with the rough hand-drawn paper prototypes (Arnowitz, Arent, & Berger, 2007; Beaudouin-Lafon & Mackay, 2003 ) that we know encourage lively and informative discussion in participatory design sessions (Schuler & Namioca, 1993), our client was uncomfortable presenting such rough, "unprofessional" looking drawings to future users. Therefore, they had produced quite pretty-looking Hypertext Markup Language (HTML) pages even though none of us knew much about the roles of the different players before the day-long session. Yet, no sooner had the users -in this case, experts who had collaboratively managed the SARS event in 2003 -read the above scenario than they began to act out their respective roles with such an intense urgency and sense of realism that they seemed to have forgotten the artificiality of the situation. No attention whatsoever was paid to the HTML pages displayed on the wall. Perhaps the most important lesson we learned was the kinds of information to which different users must not have access. To keep the Minister of Health and the media informed on progress and prevent accidental information leaks about individual casualties, their present whereabouts, their condition, or any names of people affected, these users must not have access to such detailed information. They see aggregate data only, such as the total number of cases, the number of new cases since the last briefing, the number of suspected and confirmed cases, and the number of deaths believed to be attributable to the event. The coroners' role includes distributing casualties among the surrounding hospitals. The number of patients with severe burns, for example, turned out to be an extremely important item of information for them because there are only eight burns beds in Montréal. Before these are filled, information must therefore be obtained about vacant burns beds in hospitals in other cities. Our very first, very crude mock-up HTML draft of the summary screen, which was subsequently shown to the domain experts for comment, is presented in Figure 7 .2. As new information becomes available, the data is automatically updated in MedPost. Two updates are shown, with the most recent report shown next to the relevant field header. Thus the total number of cases (casualties) increased from 17 to 25 in the period 10:20-10:57. Eight new cases had been identified in the intervening period. The user can see when either the minister or the press was last notified, making it easy to focus on the most important data-presented first-namely the total number of cases and the number of new cases since the last report was received. Because the users were able to act out their roles in such detail, the summary screen hardly changed as the design of the application matured. Save for the look and feel of the final application, the content of this crude first prototype for this user population of which the screen in Figure 7 .1 is a part remained virtually unchanged. Teleconferences were subsequently held with the client's epidemiologist and the infection control nurses. Scenarios again informed the discussion identifying their specific requirements. In contrast to the event managers, the epidemiologist needs access to information about suspected and confirmed cases in her geographical region's hospitals, nursing homes, and other institutions. However, unlike the infection control nurse, who also needs access to patients' names and home addresses, the epidemiologist cannot see those specific details. Figure 7 .3 shows the epidemiologist's view in a more advanced version of the MedPost prototype. Note the blank fields that are filled only for the infection control nurses' view. The conspicuous red button labeled "HIDE" in the upper right corner of the screen was added for security reasons to prevent unauthorized access to the system. Pressing the "HIDE" button logs the user out, requiring him to login again upon his return. MedPost automatically saves the (presumably) half-finished task and takes the user directly back to where he left off rather than forcing him to start from the opening screen. This enables him quickly to continue the task he was working on when he was interrupted. The PROBE project combines existing software applications to create an integrated and expandable CBRNE crime scene support tool allowing the various first-responder agencies to communicate and share CBRNE event data and resources in real time. We were able to observe several CBRNE simulations to gather user requirements and in a user-based prototype test. The scenarios given to responders upon arrival at the scene were always prepared by the event organizers, police, or fire chiefs. Researchers thus had no control over the nature, scope, or content of the scenario, and nor was the user test in any way under our control. The first simulation included five command post team leaders: two EMS officers, one bomb technician, one FIS, and one hazmat technician as well as several teams of first responders representing all three agencies, and over 20 stadium employees acting as casualties. Five HCI researchers, each equipped with a digital video camera, a still camera, an audio recorder, and note paper, observed and recorded activities in the command post throughout the event in which we derived user requirements for PROBE. The simulation took place in a large sports stadium in the center of Toronto with a seating capacity of over 55,000 and in which a baseball game was said to be ongoing at the time. The scenario stated only that an unknown object or objects had been thrown from above into a section of lower-level seats and that an unknown number of people would need medical attention and decontamination as quickly as safety would permit. The command post team's task was to manage the situation by determining how to proceed, allocating resources, distributing manpower in the area while providing rapid medical triage to the casualties, conducting accurate recordkeeping, and minimizing the risk of contamination of response team members. When the command post team members arrived, EMS and police first responders who were already inside the stadium provided first-hand assessments of the situation from the area that would eventually be declared the hot zone. All team members had their own radios, and each agency communicated on a different frequency channel. The EMS and hazmat team leader kept in contact with their dispatch in case an actual CBRNE call would come through, and the hazmat team leader also communicated with specialist staff in the hazmat vehicle. Three CBRNE calls were made during the simulation, which is highly unusual. This forced the teams to reconfigure because the number of fully trained CBRNE teams is limited. Three researchers each observed the team leader(s) in the command post (EMS, police, hazmat) throughout the simulation. One captured activities in the command post as a whole, and the fifth researcher recorded first-responder activities in the hot zone. In spite of Williams' (2006) warning that it can be a poor strategy to record everything, all researchers recorded as many activities as possible of the person(s) they were observing. With five channels of simultaneous data collection, it was impossible to keep track of all interactions and thus to identify weaknesses and communication breakdowns on the spot. The event took roughly 3 hours, followed by a 1-hour briefing session in which all responders, team leaders, our client, and our team participated. For the data analysis, all video and audio recordings were transcribed ad verbum, adding any notes taken during the exercise. These were merged into a single file and arranged in a minute-by-minute fashion as recommended in the cognitive ethnographic framework, in an effort to reconstruct the entire event. Table 7 .1 shows an extract of the merged data file. The time is shown in the leftmost column, followed by the source of the original data (video, audio, notes), and then the data obtained from each agency. An ID was assigned to each media file, together with its duration in minutes, both shown in bold in the table. This transcript enabled us to reconstruct the entire event, which was useful for identifying communication breakdowns, and from those, infer user requirements that were then given to the developers and expert responders for discussion and verification. For each of the 47 user requirements thus identified, the times and verbatim sources of the underlying rationale were indicated, enabling the experts directly to access the context of our inference from the transcript. Since we are not domain experts in any of the agencies' fields, this was very valuable; several misunderstandings on our part were identified that had led to misinterpretation, and hence to an unjustified user requirement. These were corrected before coding commenced on the advanced prototype. In the end, however, we only needed data obtained in the first half of the event, when responders and team leaders were making sense of the situation. Once they had identified the chemical agent they were dealing with, the number of casualties and types of injuries had been determined, and decontamination was proceeding, the management process settled into a routine that did not require much system support. In subsequent observations, only two researchers were present, but recording equipment was placed next to each command post team leader to capture both his statements and incoming radio information. The commercial eXplosives Identification Tool (XIT) is expected eventually to network with a database of improvised explosive devices yet to be designed, and into Socius, an explosives incident database, which is currently being implemented in 27 languages throughout the European Union. All three form part of the suite of CBRNE-related applications that our client is developing. By the time our team became involved with the ongoing design of XIT, system and user requirements had already been determined. Our task is therefore to provide prototypes to assist with the UI design and to run usability evaluations. The user requirements stretch over many pages of very detailed items. Two important lessons have been learned so far in this project. First, the concept of a "prototype" varied considerably among members of the project team, and constraints included the need to work with specific development tools that impose a particular UI style or look and feel. Therefore, even our early prototypes were assessed against the very specific requirements, leaving very little room for "thinking outside the UI design box". The second important lesson was that, for this user population, vocabulary, images, and content of the prototypes must precisely match reality and hence the users' rather specific expectations even of an initial low-fidelity prototype. Thus, when turning up to run an extensive, very early prototype pen/paper-based usability test in which we had expected up to 40 participants at a symposium held for first responders from around the country, we were unable to run the test. This was due to concerns with the inaccuracy especially of the images of bomb materials that we had found on the Internet. One screen displaying some of those images is shown in Figure 7 .4. This, we were told, would turn the users right off and thus negatively affect eventual user acceptance of the eventual application. Consistent with UCD methodology, we had focused on testing the navigation model and the advanced search module that allows text and image searches using an approximate approach to the content. Unfortunately, that was unacceptable. Thus, even when attempting to test a low-fidelity, high-level pen/paper prototype presented in Balsamiq (balsamiq.com/products/mockups) the prototype was expected to be a true representation of the eventual system, leaving no room whatsoever for pretending. Evidently, the seriousness of the context in which the application will be used assumed a far higher level of importance than we had expected. Instead, we should have presented the "images" at a higher level of abstraction -that is, without pictures -as we did in the next version of the prototype (Figure 7 .5). Space limitations have allowed us to present only some of what we consider the most interesting and compelling lessons we have learned over the past 10 years in the context of assisting with the generation of user requirements for surveillance and threat monitoring, and for its UI design and evaluation. • Users may be very sensitive to the authenticity of materials presented in prototypes. Therefore, be aware of, and respect, your target users' expectations and the level of seriousness they place on even high-level early prototypes. • This authenticity concerns text, terminology, and images. If you cannot obtain authentic highquality images, use high-level abstractions to represent these instead. • People are better at acting out their roles than at talking about them. Therefore, when writing usage scenarios, make sure you represent your users' reality as closely as possible, enabling them to act out their roles early in the requirements capture stage. Make sure also that you adjust the usage scenarios to each user population. • Occasionally, certain users should not be able to access particular items of information. This is especially important if you are designing a time/safety/mission critical application, as sensitive information could inadvertently be leaked by people who should not know details that are irrelevant to their roles and responsibilities. • Record and analyze data only in the first half of a CBRNE or other crisis simulation. By far, most activity takes place at the beginning of a response to a crisis, when the responders are trying to piece together a coherent picture of the type and magnitude of an unfolding event. Once they know what they are dealing with, they go into a routine response mode in which they can rely on their knowledge and experience. • Reconstruct minute-to-minute parts of an adverse event from data transcriptions originating in several files. By collating all data from different media and possibly from different researchers into one master data file, everyone will have the same picture. When extracting user requirements from that file, keep a reference to the point in time at which the observation was made that led you to articulate the requirement. This greatly facilitates the user requirements reviews with all stakeholders and helps to correct any misunderstandings. • Select the episodes in the event that are worthy of detailed analysis. It is easier to select the best data from smaller sections. Therefore, divide the event data into smaller sections and then perform a quick tally of the relative amount of relevant verbal activity in each section to help select the best data to analyze. When an event stretches over many hours, it may not be necessary to analyze all verbal episodes to generate useful requirements. • Eliminate all unrelated conversations and statements from the data file, but calculate the percentage of such conversations. An indication of the amount of irrelevant conversation among the event response management team can provide information about the level of stress in the team and hence about the difficulty of their task. The more taxing the task is, the fewer irrelevant conversational episodes are likely to occur. In our attempts to deliver high-quality products, we work very closely with the client's personnel, and we are constantly trying to help specialists from different disciplines who are involved in the projects to understand the user perspective and how to design for it. In this chapter, we have tried to share what we consider some of the most important lessons we learned while working on prototypes for a suite of applications designed to support the management of crisis situations such as a large CBRNE terrorist attack with mass casualties. We selected episodes that stood out to us as requiring a somewhat different approach to that we would typically adopt when designing UIs for people whose daily work does not involve serious threats to human safety. In MedPost, we were surprised to discover the power of the scenario for revealing details of the experts' roles and responsibilities that rendered the UI design a very easy task. In particular, it was interesting to note the importance of knowing which items of information should not be accessible to some users. In PROBE, we had underestimated the value of linking each user requirement to particular points in the minuteby-minute verbalizations captured in the master data file of the event. In the XIT prototype, we were quite taken aback by the importance to users of including authentic images to illustrate how an imagebased search might work. Although these lessons were learned in the specific context of designing UIs for supporting the management of CBRNE events, we believe that they are generalizable to other types of projects. Effective prototyping for software makers Distributed cognition in an emergency co-ordination center The human-computer interaction handbook Dirty bombs: Real threat? Team situation awareness as communicative practices Cognition in the wild User needs analysis and requirements engineering: Theory and practice Responding to "dirty bombs Interaction design: Beyond human-computer interaction Collaborative planning and situation awareness in army command and control Usability engineering: Scenario-based development of human computer interaction Participatory design: Principles and practices Designing the user interface: Strategies for effective human-computer interaction Using cognitive ethnography to study instruction Dirty bombs: The threat revisited. Defense Horizons Coordination in rapidly evolving disaster response systems Cognitive psychology and team training: Shared mental models in complex systems Situation awareness Towards a theory of situation awareness in dynamic systems Theoretical underpinnings of situation awareness: A critical review Individual differences in pilot situation awareness Measuring team situation awareness in decentralized command and control environments Assessing team performance in the operating room: Development and use of a "black box" recorder and other tools for the intraoperative environment Supporting disaster response and recovery through improved situation awareness Cutting the Gordian Knot of misguided performance measurement Group situation awareness and distributed decision making: From military to civilian applications Medical knowledge overload: A disturbing trend for physicians Sensory and information inputs overload: Behavioral effects Distributed cognition DiCoT: A methodology for applying distributed cognition to the design of team working systems Dynamic decision making: Human control of complex systems Team decision making in complex environments We thank all first-responder participants for allowing us to observe their activities, ask questions, and record data in sometimes stressful situations. We also thank our client, AMITA Corporation, and its personnel for their persistent patience, help, feedback, and good humor. This research was sponsored by CRTI Grant #06-0255TA-Medical and Casualty Command Post (MedPost), CRTI Grant # 06-0317TD PROBE -Crime Scene Support Tool for Police, Hazmat, & EMS, and CRTI Grant #08-0131TD Commercial eXplosives Identification Tool (XIT).