key: cord-0919412-bplelgpr authors: Rosella, Laura C.; Wilson, Kumanan; Crowcroft, Natasha S.; Chu, Anna; Upshur, Ross; Willison, Donald; Deeks, Shelley L.; Schwartz, Brian; Tustin, Jordan; Sider, Doug; Goel, Vivek title: Pandemic H1N1 in Canada and the use of evidence in developing public health policies – A policy analysis date: 2013-02-13 journal: Soc Sci Med DOI: 10.1016/j.socscimed.2013.02.009 sha: 59e70474f7eae42943ba63363f887cdaa91e29f9 doc_id: 919412 cord_uid: bplelgpr When responding to a novel infectious disease outbreak, policies are set under time constraints and uncertainty which can limit the ability to control the outbreak and result in unintended consequences including lack of public confidence. The H1N1 pandemic highlighted challenges in public health decision-making during a public health emergency. Understanding this process to identify barriers and modifiable influences is important to improve the response to future emergencies. The purpose of this study is to examine the H1N1 pandemic decision-making process in Canada with an emphasis on the use of evidence for public health decisions. Using semi-structured key informant interviews conducted after the pandemic (July–November 2010) and a document analysis, we examined four highly debated pandemic policies: use of adjuvanted vaccine by pregnant women, vaccine priority groups and sequencing, school closures and personal protective equipment. Data were analysed for thematic content guided by Lomas' policy decision-making framework as well as indicative coding using iterative methods. We interviewed 40 public health officials and scientific advisors across Canada and reviewed 76 pandemic policy documents. Our analysis revealed that pandemic pre-planning resulted in strong beliefs, which defined the decision-making process. Existing ideological perspectives of evidence strongly influenced how information was used such that the same evidentiary sources were interpreted differently according to the ideological perspective. Participants recognized that current models for public health decision-making failed to make explicit the roles of scientific evidence in relation to contextual factors. Conflict avoidance theory explained policy decisions that went against the prevailing evidence. Clarification of roles and responsibilities within the public health system would reduce duplication and maintain credibility. A more transparent and iterative approach to incorporating evidence into public health decision-making that reflects the realities of the external pressures present during a public health emergency is needed. An effective public health response to a novel infectious disease will reduce the population impact. However, time constraints and uncertainty can compromise the response. This was seen with the 2003 Severe Acute Respiratory Syndrome (SARS) outbreak where policies were developed under scientific uncertainty due to lack of published evidence and limited experience with the virus (MacDougall, 2007) . Some have suggested this uncertainty contributed to the failure to optimally control the outbreak in a timely manner in certain highly affected locations and resulted in unintended consequences such as lack of public confidence (National Advisory Commitee on SARS and Public Health, 2003) . Given the history of serious pandemic influenza outbreaks and the emergence of the H5N1 avian influenza virus, concerns about a human influenza pandemic are paramount. As a result several governments and institutions have implemented extensive pandemic plans to be better prepared and minimize uncertainty. On June 11, 2009 the World Health Organization (WHO) raised the pandemic alert to its highest phase in response to a novel strain of H1N1 influenza (Eurosurveillance editorial team, 2009). Despite extensive planning, the use of evidence to develop policies was complicated by several factors. Public health officials had to make decisions quickly, often when information was still being gathered. Coordination was necessary across multiple levels of public health (international, federal, provincial, local) and sectors (hospitals, schools, workplaces) . Finally, pandemic policymaking has been shown to be influenced by political factors (Garoon & Duggan, 2008) and can be strained by antiviral and vaccine shortages, overwhelming demands on public health systems and intense public scrutiny (Timpka et al., 2009) . This study sought to examine the Canadian public health system response to the 2009 H1N1 pandemic (pH1N1). Our objective was to analyse the public health decision-making process and identify the factors that influenced the uptake and application of evidence for public health policy decisions. This policy analysis used a multiple case study design to examine policies at the federal, provincial and local levels of public health in Canada. Four policies were studied: (a) vaccine priority, (b) use of adjuvanted vaccine among pregnant women, (c) school closures as a containment strategy, and (d) recommendations on N95 respirators as a form of personal protective equipment (PPE) ( Table 1 and Supplementary Table 1) . Study participants were from five Canadian provinces (British Columbia, Alberta, Ontario, Quebec and Nova Scotia). An initial list of 41 key informants was developed by identifying officials in federal, provincial or municipal governments, provincial or local medical officers of health, members of a pH1N1 scientific advisory group, and clinical experts in infectious diseases and/or public health. A snowball sampling method was then used to identify an additional 23 participants. We initiated recruitment by email between July and September 2010 and all interviews were complete by November 2010. All consenting participants were given the choice on the policy of focus for the interview based on their role/ responsibility during the pandemic. Many participants were involved in multiple policies and some spoke to more than one during their interview. Ethics approval for this study was obtained from the Health Sciences Research Ethics Board at the University of Toronto, Canada. All participants provided written informed consent. The semi-structured interview guide was developed to capture the decision-making process with a focus on the use of evidence. All interviews were conducted by telephone by the principal investigator (LC) or research assistant (JT), recorded and transcribed verbatim. A second analyst (AC), present for all interviews, took additional notes and independently verified all transcriptions. We sought public documents that detailed a pH1N1 policy recommendation related to the four policies. Primary documents included: guidance documents, statements of recommendations, interim or final government, advisory or scientific group reports and pandemic plans. Secondary documents included: information sheets, surveillance bulletins, journal publications and references from primary documents. Documents outlining recommendations were limited to those released between April 1, 2009 and May 31, 2010 while other documents were restricted to those released up to December 31, 2010. These date restrictions were in place to keep the analyses focused on pH1N1 and were determined by those responsible for monitoring these documents during the pandemic. Documents were identified through a search of an archive of published and grey literature related to pH1N1. Librarians developed this archive by performing daily local, national and international scans (www.ophla.ca). In addition, a direct search of selected public health and governmental websites was performed. Data coding was used to abstract and synthesize data from transcribed interviews and policy documents using descriptive and iterative methods (Miles & Huberman, 1994; Yin, 1994) . We used Lomas' model for decision-making (Lomas, 2000) to guide the analysis because it captures the interactive and multidirectional knowledge transfer that occurs in the decision-making process focussing on both evidence or knowledge and its transfer within a Research studies had yet to show the significance of airborne (in addition to droplet) transmission for pandemic influenza Studies emerged that raised doubts on the ability of N95 respirators to prevent transmission of influenza among HCWs N95 respirators were recommended pandemic plans policy subsystem ( Supplementary Fig. 1 ). It also emphasizes the importance of context-based decision-making incorporating three domains that influence the process: values, institutions and information. We also considered Sabatier's advocacy coalition framework (ACF) as it similarly emphasizes beliefs and ideologies (Sabatier, 2007) . However, this framework is less effective in describing the role of institutions. Dobrow's context and evidence utilization framework (Dobrow, Goel, & Upshur, 2004) is well-suited for examining contextual factors influencing the uptake of evidence, but has limited application to the policy and institutional subsystems where decisions take place, which was critical for pH1N1. We also allowed thematic content to evolve independently of Lomas' framework. The principal investigator (the primary rater) developed the coding guide based on the framework and adaptations from its use in a previous analysis (Wilson et al., 2007) . The interviews were distributed between the primary and secondary rater for independent coding. A subset of transcriptions was coded by both raters to evaluate inter-rater differences. In addition, the primary rater reviewed all transcripts and coding a final time to ensure consistency. A similar process was followed for the document analysis. We reviewed coded data for themes guided by the research objectives and domains in Lomas' framework as well as themes that emerged that were not part of the framework. NVivo 9 was used to assist with data collection, organization and analysis. We conducted 40 interviews, including six pilot interviews, and included all in the analysis. The primary roles of participants during pH1N1 are summarized in Supplementary Table 2 . Seventy-six relevant documents were identified with 12 covering more than one policy (Supplementary Table 3 ). We present our findings according to the Lomas framework and generalize themes overall, providing specific examples within policies in Table 2 . Three competing ideologies emerged with respect to the use of evidence: (1) the evidence-based ideology: science should be the first and foremost consideration in developing policies, and decisions made in the absence of good quality evidence can result in suboptimal policies; (2) the policy-based ideology: science is a minor component of the policymaking process and evidence is meant to inform policy rather than drive policy; and, (3) a hybrid or pragmatist ideology. Most participants associated the term evidence with the traditional research-based form of scientific evidence, which were primarily epidemiological in nature. With a perspective derived from evidence-based medicine (EBM), participants with an evidence-based ideology placed considerable value on the peer-review process associated with scientific publications and conventional epidemiological study designs. Without a peer-review process, this group felt uneasy with proceeding to a policy decision. Furthermore, participants expressed frustration when science was re-evaluated in the policy arena, believing this compromises an evidence-based approach. Their key view is that evidence should be central and at the basis of any good public health policy decision, and that good evidence leads to good policy. "[Information on public health measures] are published in grey literature or as evaluations or reports that were implemented. It's hard to trust in the same way because it's not peer-reviewed and not subjected to the same rigor." (participant 25, scientific advisor) This ideology ran into a major barrier when particular standards of evidence were deemed necessary to proceed. For example, the use of adjuvanted vaccine had not been shown to be harmful in pregnant women, but advisory groups had difficulty offering conclusive advice without evidence from randomized controlled trials (RCT) conducted in pregnant women. "The largest and most glaring gap was data on vaccine in pregnancy. That's a major issue that we still haven't solved. we end up using vaccines and other things in pregnant women yet we exclude them from the clinical trials where they would provide the data on their safety and effectiveness." (participant 40, scientific advisor) In contrast, participants with the policy-based ideology shared the perspective that evidence is meant to only inform policy and not dictate policy. Under certain circumstances, they believed it was justified to give less priority to science in favour of other contextual issues. Participants with this ideology felt that scientists often lack expertise in policy-making. This ideology acknowledges that personal belief and ability to sway individuals also play important roles. Furthermore, the group stressed that the purely science-based approach over-simplifies the reality of developing policies since the determining factors were often contextual. "in a way you generate knowledge that's no use to the field because it's inapplicable even though the science is very appropriate. you're out of touch with the evidence mapping the policy." (participant 69, public health official) "In my view scientific advice into that [N95] decision would not have changed anything." (participant 02, scientific advisor) The pragmatist ideology can be seen as the centre of the continuum between evidence and policy-based ideologies. The participants who adopted this ideology felt science should be a primary consideration, but other factors including expert advice, clinical factors, logistics and ethics can be equally important and should be "respected." This group was more likely to have an openview of what constitutes evidence and tended not to make broad brush statements about either evidence or policy considerations, accepting that science can trump other considerations if compelling enough and vice versa. This group felt comfortable acting on the best available evidence and recognized that reaching consensus with respect to the science is often not possible. ".science is a tool. We can debate the science for years but we have to make the best decision given the circumstance." (participant 13, public health official) "The decision-makers were all fully aware of the evidence and I think in a perfect environment would have chosen not to recommend the N95 masks. But they were working in the face of political and legal commitments as per the collective agreement. The evidence was available. It was considered but other factors ultimately drove the decision." (participant 45, scientific advisor) "Implementation considerations really are an incredibly important. confidence and public communication considerations are also really important considerations. They need to be respected every bit as much as the scientific and evidentiary considerations." (participant 04, public health official) Having not agreed upon definition of evidence results in further challenges in determining what counts as an evidence-based decision. While many participants support the pragmatic thinking, they recognize that current decision-making bodies lack the models to guide this in practice in a defensible way. In the current model, science evaluation and advisory recommendations are independent of decision-making. This separation has been recognized as essential to maintaining scientific integrity (Black, 2001) and is consistent with the evidence-based ideology. In situations where there was a lack of evidence, such as for the adjuvanted vaccine, it was difficult to proceed using an evidence-based ideology; therefore, other values and influences, such as the precautionary principle became of greater importance. "It's another great example of the corner you get yourself into when you start making decisions that are not evidence-based. I mean the original decision to suggest that pregnant women should receive non-adjuvant vaccine, there was no evidentiary basis for that at all. It was one of these sort of theoretical precautionary principle sorts of things." (participant 47, public health official) "you can always say we'll take the precautionary principle and let's just play it absolutely safe and just go with the non-adjuvant, even though it may be less effective and it was not available." (participant 19, clinical expert) In a pandemic there is uncertainty at the outset. As a result, prior beliefs on epidemiological patterns were very influential. Beliefs were well entrenched because pandemic planning has been an ongoing public health effort for several years. Beliefs were most strong with respect to disease severity, health care burden and transmission. Severity was expected to be moderate to severe, having the epidemiological characteristics of previous pandemics, particularly the 1968 Asian flu pandemic or seasonal influenza (Public Health Agency of Canada (PHAC), 2008). Participants also referenced the 1918e19 pandemic, avian influenza and SARS, indicating that historical memory played an important role in influencing beliefs. The health care burden was assumed to be high, having major resource implications and occupational safety concerns for health care workers. Transmission was assumed to occur through droplet spread with the possibility of airborne transmission. During pH1N1, emerging evidence that did not support these beliefs was more difficult to accept or incorporate into decision-making. "One of the key considerations related to the N95 issue was the fact that we already had a pre-existing policy in place. that we would recommend that health care workers use N95 or the equivalent for protection. So you're not going to suddenly mid-gear change your policy." (participant 04, public health official) "trying to change the way you respond during a time of crisis is very difficult." (participant 51 , public health practitioner) Many participants felt that as pH1N1 progressed, attention was not paid to the severity of the disease soon enough and the subsequent scaling down of events to match the severity was delayed. "we operated the same, whether or not it was a severe pandemic or not." (participant 57, public health practitioner) "We shouldn't have a war plan and stick with it right through to the bitter end, which I think is what we did this time. I actually don't know how you titre up or titre down but probably it's packaged with how we're communicating so that there is some room to move back if the threat is less severe than you initially worried about." (participant 38, scientific advisor) Beliefs were also entrenched as a result of the SARS experience, despite SARS being a hospital-outbreak versus the pH1N1 community outbreak. The impact of SARS on health care workers influenced beliefs about the severity of H1N1 and how we should prepare and respond. "Maybe over a long period of time and as things like SARS become more distant in the memories of people working in health care that might change things." (participant 43, clinical expert) Several competing interests factored into the use of evidence. Health care unions were interested in protecting their workers. Professional medical societies were interested in the population they serve (e.g. Canadian Pediatric Society (CPS) for children, Society of Obstetricians and Gynaecologists of Canada (SOGC) for pregnant women and educational boards needed to consider both parents and school employees). Scientific advisors were tasked with evaluating the science and providing an unbiased recommendation while staying true to the quality of the science and the needs of the overall population versus specific subpopulations. Policymakers in governmental institutions were faced with balancing these and several other interests. At different jurisdictional levels in Canada, provincial and local level decision-makers needed to balance science with the overall needs and contextual factors in their community. "When we were looking at the dosage and the number of vaccines that needed to be given to children. both the pediatricians and the public health position were based on defensible science. But the same science was interpreted differently by the two groups depending on whether their major concern was for the one individual child in front of them or whether it was in terms of how many kids could we vaccinate in how short a period of time to give a population level of protection." (participant 32, public health official) "we compromised on the N95s because we felt okay, so it's better to have masks used potentially unnecessarily but actually have people willing to come to work rather than risk people refusing to come to work." (participant 32, public health official) The complexity of Canada's executive, legislative and bureaucratic structures of public health complicated the policy process. This complexity has been noted previously to influence public health (Frank & Di Ruggiero, 2003) . With multiple levels of action, participants reported signs of disconnect between those responsible for reviewing the evidence and those exercising decisions in their respective jurisdictions. In addition, participants stated that recommendations made at the international or national level often did not consider local issues. Internationally, the WHO and US Centres for Disease Control (CDC) were highly influential. "the vaccine decision-making process was very complex because it involved multiple levels of public health from WHO, to PHAC, to the province and then down to the local health unit." (participant 45, scientific advisor) Informal institutional structures were also influential. Pandemic H1N1 was the focus of intense interest and pressure from multiple citizen and advocacy groups, particularly in relation to vaccination. Pandemic plans, recommendations from the WHO and decisions made in other countries also affected how evidence was used in Canada. "Full consideration has been given to international activities and declarations and notices issued by the World Health Organization." (Health Canada, 9 A.D.) "The fact that some other jurisdictions, such as Mexico and the US, implemented measures such as school closures put more pressure on Ontario to adopt similar measures." (Ontario Ministry of Health and Long-Term Care, 2010) Information producers included scientists/researchers, advisory groups and committees/working groups. Participants frequently expressed uncertainty about who was responsible for answering what questions. This resulted in confusion on where to look for information, particularly when differing recommendations were based on the same evidence. Information purveyors included: government, professional organizations, media and advisory organizations with and without decision-making capability at local, provincial, federal and international levels. A common theme was that the credibility of information purveyors was paramount and needed to be preestablished. Scientific advisory groups formed during pH1N1 did not have the opportunity to establish their credibility, making their recommendations difficult to accept. For example, the National Advisory Committee on Immunization (NACI) is responsible for scientific guidance on immunizations; however, another vaccine advisory group was created specifically for the pandemic and NACI not consulted until later on. "It's no good having a scientific decision-making body that only exists in the setting of emergencies or one that is sufficiently far from government that they're not looking at that advice and using that advice on a regular basis." (participant 02, scientific advisor) Credibility also posed challenges at times because recommendations coming from an established and credible organization were difficult to challenge or modify (i.e. WHO). Along with credibility, themes emerged around the challenges in achieving consensus on scientific evidence. Participants were particularly concerned about the significant amount of time spent trying to achieve consensus, commenting that long debates over scientific merits under time constraints is not helpful. It was recognized that consensus is often not feasible and therefore, different models of consensus should be considered. "a cabinet approach, which is that you may disagree with the final decision but you go with it and you live with it." (participant 18, public health official) When contradictory or unclear messages were given, the ensuing confusion resulted in decreased public confidence and increased time invested to rationalize the differences. "[WHO] made a recommendation that pregnant women should be offered non-adjuvant vaccine preferentially because there wasn't much data on the use of adjuvant, or so they claimed, in pregnant women, which set the tone for Canada. we had already ordered adjuvant vaccine for our population. So that was a bit of a challenge for us." (participant 04, public health official) "The untimely and ambiguous recommendations for dosing in pregnant women and young children from national and international sources put Alberta in a vulnerable position where credibility of its public health decision-makers was jeopardized because of issues beyond their control." (Alberta Health and Wellness, 2010) One of the most influential information purveyors, the media, was looked at in an overall favourable light. Unlike previous public health events, such as SARS where the media reaction was sensationalized and disruptive at times (Lewison, 2008; Washer, 2004) , participants felt that most of the time the media took a balanced and responsible approach in their reporting by seeking appropriate advice from credible experts. "I don't think the media drove any of our policies, other than forcing us to try and be as clear in our communications as possible." (participant 32 , public health official) However, at certain points during the pandemic, such as around the immunization campaign, it was perceived that media coverage did have a strong influence. "initially [the media] influenced the public perception of the vaccine. Originally it was in a negative way so that there were questions to the safety and the need for the vaccine. But then subsequently after the death of [child], it was for demand of the vaccine. Then all of the media attention on the lines and the long waiting times just added public pressure to go beyond the priority groups that were determined by the policy." (participant 41, scientific advisor) Academic journals were also cited as an influential purveyor. Study participants acknowledged that journals made an attempt to speed up their review and publishing of pandemic studies and made articles freely available. Nonetheless, the process was critiqued by some who commented that at times it disrupted the flow of information. "there wasn't an awful lot of science being published about pandemic H1N1 in a timely enough fashion to affect decision-making. . So it would be really good to have a fast track peel-off kind of system of public access where people can review and it doesn't necessarily preclude subsequent publication .. because scientists need to be able to do that." (participant 32, public health official) Views on the sufficiency of information were divergent. Many participants expressed the desire for more information/evidence despite an acknowledgement that decision-makers may not be equipped to handle new or emerging information, even if the scientific quality is high. Several additional themes emerged that were not explicitly captured by the Lomas framework. Comments on the use of evidence revealed concerns around transparency (i.e. operating in such a way that it is easy for others to see what actions are performed) and explicitness (policies fully and clearly expressed, leaving nothing implied). Specifically, because criteria for the inclusion or exclusion of evidence were unclear, decision-making structures themselves were described by participants as being a mystery at times. Some participants not at the decision-making table were not confident in their ability to accept and implement recommendations because they were not privy to the decision-making structure. Scientific advisors wanted to know how scientific recommendations provided were ultimately used and balanced with other factors. If advice was not explicitly referenced, those providing the scientific advice assumed it was not considered. Some felt that decision-makers often do not cite other factors, such as logistical, political and social issues as rationales because those issues are perceived as less justifiable compared to science. This lack of explicitness can result in the unintended consequence of decisions appearing to be made without appropriate justification. "There wasn't a lot of transparency in those decisions or at least feedback to the people who were making the recommendations. It's hard to know whether they also considered the evidence. or whether there were some other factors that were forcing them to go against the recommendation." (participant 25, scientific advisor) A clear documentation of the process would serve to make the process more explicit and satisfy concerns raised. "the bottom line is a decision has to be made. You have to sometimes go right down to consensus and expert opinion or extrapolation from other types of evidence or other categories of evidence to make your recommendation. I think there has to be rigor that goes along with that and documentation and clarity on the reasoning." (participant 45, scientific advisor) Several participants commented on the multiple parallel processes which took place. Not only were there parallel processes for decision-making at multiple levels, but the generation/evaluation of evidence also happened at multiple levels and sometimes even within the same jurisdiction or institution. Such duplication contributed to inefficiencies and confusion in the recommendations. "all of us were scrambling across all of our different domains to understand as best as we could what could be learned at any given moment. In retrospect it was so fundamentally inefficient." (participant 16, scientific advisor) Participants noted that First Nations groups were at increased risk of mortality and morbidity and warranted vaccine priority, but overall did not speculate on the broader causes of the disproportionate mortality and morbidity. This issue was not the focus of the study or the policies examined, which could be an explanation for why this was not discussed in more detail by the participants. In contrast to previous public health events such as SARS (National Advisory Commitee on SARS and Public Health, 2003) , the CreutzfeldteJakob disease tainted blood scandal (Wilson et al., 2001) and the Walkerton Escherichia coli outbreak (Schuster et al., 2005) , a unique aspect of pH1N1 was the advance planning. Substantial investments had been made in pandemic planning since SARS, and the identification of H1N1 in Mexico in early 2009 initiated further preparations for its anticipated spread. Advance planning did minimize uncertainty, but it also generated assumptions that created conflict when handling emerging evidence. Furthermore, evidence was interpreted differently according to the three ideologies of evidence. According to Fishbein and Ajzen's theory of reasoned action, pandemic plans are manifestations of the intentions for a pandemic, a conscious action plan which functions as a major determinant of future behaviour (Fishbein & Ajzen, 1975) . Pandemic plans are a complex collection of assumptions based on various types of evidence. This creates a box in which policymaking begins and sometimes stays. As the outbreak unfolded it became apparent that data on severity did not match the models and underlying assumptions of most plans. Decision-making, however, continued to be framed by these models. Pandemic plans are intended to provide practical guidance and not indoctrinate ideology; however, this analysis shows that the latter prevailed. This finding is consistent with the theory of cognitive dissonance (Festinger, 1957) which explains the difficulties in adjusting when there is a discrepancy between prior held beliefs and new information based on the existing situation. Other decision-making theories can offer insight into policy choices made despite the evidence. Challenging the content of the pandemic plans, for example on the use of N95 respirators, was avoided in certain jurisdictions given the significant conflict that would arise. This is consistent with conflict avoidance as described by the conflict model of decision-making (Janis & Mann, 1977) . A decision that involves a potential conflict provokes a degree of stress, the excess or absence of which is in turn a major determinant of the final decision. The N95 decision in Ontario was a result of "defensive avoidance" such that the risks to labour disruption would not be worth challenging a recommendation from the pandemic plan. The school closure policy went against a priori guidance, which largely endorsed school closure as a containment strategy. According to the conflict model, the (largely societal) risks associated with school closures outweighed the risks of staying with the decision. These two policies contrast in how differences in antecedent conditions, available evidence and resulting decisionalconflict, guided the policy actions. It is possible that the nature of the situation, and not theory, explained the N95 decision as the political and non-evidentiary considerations were considered legitimate. Compared with N95, very little epidemiological evidence existed for school closures with support mostly coming from modelling studies. However, the potentially large societal costs of closing schools supported a change in direction. The vaccine sequencing decisions also largely agreed with pandemic plans although the manifestation of the policy was extremely heterogeneous across Canada and within provinces. With the adjuvanted vaccine for pregnant women, little to no a priori information existed and therefore cognitive dissonance did not apply. Instead, the precautionary principle was largely used to justify decisions. There are some limitations to this study. As interviews were conducted a few months after the pandemic was declared over, recall bias is a possibility. Verification of our findings with the document analysis is not susceptible to this recall bias and supported similar themes around the role of beliefs, information purveyors and ideologies. Nonetheless we acknowledge that hindsight may influence responses. All participants work within the public health or health care system. As a result, the views of individuals outside the public health system were not represented. While this is appropriate given our research question, these findings represent only one particular stream of perspectives. One of the limitations of using the Lomas framework in this analysis is the unidirectional flow between information domains and institutional structures for decision-making. During pH1N1, this exchange was multidirectional and several individuals were both producers of information and decision-makers within the formal structures, making it difficult to conceptually separate their unique influences. Lomas' framework also does not capture Canada's hierarchical, jurisdictional and institutional levels of decision-making and their interactions. Finally, the placement of evidence ideology in the values component of the model can be questioned as it is reflective of a normative stance versus a value trait. Rationalism demands that decision-makers choose the best alternative from a set of options (Betsch, Haberstroh, & Hohle, 2002) . However, under the congruence of multiple pre-planning and entrenched assumptions, the precautionary principle and evidence-based paradigms, a complex situation emerges whereby the decision-making process cannot be easily tracked or justified according to a straightforward set of criteria. Pre-planned approaches have many benefits including clarifying accountability and strategy during uncertainty. However, caution needs to be exercised when pre-planned approaches are in place to ensure policy options are not constrained by plans. We suggest that pandemic plans need to be written and used in a way that accommodates dynamic responses to fit an emerging situation. This includes minimizing scenarios based on single assumptions and conducting a range of sensitivity analyses. An emergent public health situation that affects a large proportion of the population will necessarily require a balance between various factors. Both evidential and policy considerations are important and improvements in addressing both effectively and explicitly within decision-making processes are needed. We recommend an iterative scientific review that re-considers science/ evidence based on the policy landscape. The challenge with this approach is that in an EBM paradigm this can be thought of as compromising evidence. We have noted throughout our analysis that this is already happening; therefore, being more explicit about the process will allow for more critical investigation. Functionally, public health needs a model that can support these types of decision constructs, but at the same time ensure rigor. One modifiable construct that would be beneficial to amend going forward is to reconsider what constitutes evidence. This analysis suggests a more comprehensive and inclusive concept of evidence that recognizes the variety of contexts in which evidence is generated and used (Upshur, VanDenKerkhof, & Goel, 2001) , and that is more appropriate during an emerging public health event. This can be facilitated by increased collaboration and discussion among researchers and policymakers to ensure the relevant evidence is generated. Such collaborations would also facilitate greater credibility of information purveyors, which is particularly important when evidence does not fit prior expectations. Credibility, transparency and explicitness in the decisionmaking process will improve understanding and uptake of policy decisions. In particular, our analyses revealed that when and why evidence/advice is or is not considered needs to be clearly articulated. This includes outlining steps, logic, key assumptions, limitations and trade-offs considered, evidentiary or otherwise. The role of ideological perspectives and prior beliefs should also be made explicit given how influential they are in defining decisions from the outset. We acknowledge that detailed recording of evidencebased or consensus-derived decisions to ensure transparency may be idealistic in the face of time and other pressures during a crisis. One possible recommendation could be a more rigorous use of Incident Management System (IMS) processes which would serve not only to improve processes but also to ensure better documentation. At the minimum, critical post-outbreak analyses of detailed real-time records of the meetings and transcripts should occur to improve future public health actions. Finally, with many different groups addressing the same questions, inefficiencies still exist. The public health system needs to leverage existing relationships rather than create new structures during a pandemic. In addition, better clarity on who is responsible for answering what questions is needed to avoid duplication and mixed messages. Addressing inefficiencies will require trust and collaboration, supporting the themes identified around credibility. This analysis reflects the tension that occurs during emergency public health decision-making and highlights several important considerations for public health planning. Going forward, we must ensure that our decision-making structures evolve in light of the challenges experienced during pH1N1. This study was supported by the Institute of Population and Public Health and the Knowledge Synthesis and Exchange Branch of the Canadian Institutes of Health Research (CIHR). LCR was supported by a CIHR postdoctoral fellowship in public health policy. KW is supported by a Canada Research Chair in public health policy. The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by the Ontario Agency for Health Protection and Promotion is intended or should be inferred. Review of Alberta's response to the 2009 H1N1 influenza pandemic Explaining routinized decision making e a review of theories and models Evidence based policy: proceed with care Pandemic alert level 6: scientific criteria for an influenza pandemic fulfilled A theory of cognitive dissonance Belief, attitude, intention, and behaviorIn An Introduction to theory and research Public health in Canada: what are the real issues? Discourses of disease, discourses of disadvantage: a critical analysis of National Pandemic Influenza Preparedness Plans Notice of decision from influenza A (H1N1) 2009 pandemic monovalent vaccine (without adjuvant). Ottawa, ON: Her Majesty the Queen in Right of Canada Decision-making: A psychological analysis of conflict and commitment The reporting of the risks from severe acute respiratory syndrome (SARS) in the news media Connecting research and policy Toronto's health department in action: influenza in 1918 and SARS in 2003 Pandemic (H1N1) 2009: A review of Ontario's response The Canadian pandemic influenza plan for the health sector Theories of the policy process Infectious disease outbreaks related to drinking water in Canada Population-based simulations of influenza pandemics: validity and significance for public health policy Meaning and measurement: an inclusive model of evidence in health care Representations of SARS in the British newspapers Variant CreutzfeldteJakob disease and the Canadian blood system after the tainted blood tragedy A policy analysis of major decisions relating to CreutzfeldteJakob disease and the blood supply We wish to thank Beata Pach, librarian at Public Health Ontario, for assistance with searching and retrieval for the document analysis. Supplementary data related to this article can be found at http:// dx.doi.org/10.1016/j.socscimed.2013.02.009.