key: cord-0727855-hfybvav6 authors: Tokazhanov, Galym; Tleuken, Aidana; Durdyev, Serdar; Otesh, Nurlan; Guney, Mert; Turkyilmaz, Ali; Karaca, Ferhat title: Stakeholder based weights of new sustainability indicators providing pandemic resilience for residential buildings date: 2021-08-26 journal: Sustain Cities Soc DOI: 10.1016/j.scs.2021.103300 sha: 3ec6e72f4cfc03db4c0643c18cee8ff2badae6bb doc_id: 727855 cord_uid: hfybvav6 During COVID-19, the building and service characteristics of residential buildings turned out to be more critical due to lockdowns. The present research assesses the importance of new sustainability indicators for residential buildings in three categories (e.g., Health and Safety, Environmental Resources Consumption, and Comfort) that provide resilience for pandemic periods. The opinions of stakeholders on the identified indicators were collected and then analyzed. ‘Health and Safety’ category is found to be the most critical among the others. The prevention of virus propagation, mental health, and building air quality are three crucial indicators playing essential roles in the health and safety category. In more detail, innovative smart technologies, including touchless technologies, are identified as a priority in preventing virus propagation. Outdoor spaces and safe indoor places for socialization are weighted as essential in supporting the well-being and mental health of the resident. Finally, air filtration and segregation of medical waste indicators are considered critical in preventing the spread of viruses. There was a consensus among the local and international experts since they did not significantly report differing opinions for the majority of the indicators. However, there was a shift in experts’ opinions towards pandemic-oriented indicators compared to conventional sustainability indicators. The COVID-19 outbreak, caused by severe acute respiratory syndrome-related coronavirus 2 (SARS-CoV-2), was first reported in December 2019 in China (Nghiem et al., 2020) . The disease quickly spread globally, and the WHO (World Health Organization) characterized it as a pandemic in March 2020 WHO (2020) . To prevent the spread of the virus, people around the globe have been forced to self-isolate at home (Dietz et al., 2020) , leading to considerable lifestyle changes, e.g., switching to teleworking and remote studying. A year after the outbreak, people are eager to enter the post-pandemic world with the start of vaccination BBC (2020). Nevertheless, current mutations in the existing virus as well as the expected appearance of new zoonotic diseases in the future make preparation for a possible pandemic world crucial (Hui et al., 2020; Morris, 2020) . Global quarantines have led to a vast increase in the amount of time spent at home. This experience has led to a rethinking of residential housing units, not only as a living place but also as working, studying, fitness, and leisure spaces. Homes have become the only space for safe pastimes (Allam & Jones, 2020) . The COVID-19 experience is already changing requirements for sustainable and resilient buildings in terms of health and safety, environmental resource consumption, and personal comfort (Tokazhanov et al., 2020) . Therefore, the perception of sustainable homes is being reassessed in different studies (Cuerdo-vilches et al., 2020; Tleuken et al., 2021) . It is important to clearly define the difference between resilience and sustainability. According to Hassler & Kohler (2014) , sustainability focuses on future stability, while resilience represents readiness for the potential disasters of the dynamic and unpredictable future. In this sense, the present study is related to resilience as the ability of A manuscript submitted to Sustainable Cities and SocietyMay, 2021 residential buildings to withstand future pandemics' social, economic, and health-related challenges. Thus, the indicators used in the study are referred to as 'resilience indicators for a pandemic.' Moreover, a resilient building built for a pandemic decreases the negative outcomes of potential pandemics, contributing to a more stable and thus sustainable future. Assessing the sustainability of buildings is a challenging and evolutionary process that is continually enhanced to conform to up-to-date requirements. The primary purpose of assessment tools is to guide decision-making processes (Mateus & Bragança, 2011) . The sustainability of buildings is usually evaluated by focusing on environmental, economic, and social factors (Bragança et al., 2010; Waer & Sibley, 2005) . Globally, numerous tools have been developed for either sustainability or green building assessment. The most prominent examples include BREEAM (Building Research Establishment Environmental Assessment Method), LEED (Leadership in Energy and Environmental Design), and CASBEE (Comprehensive Assessment System for Built Environment Efficiency), which have been widely used both in industry and academia (Bernardi et al., 2017) . Sustainability assessment tools consist of several categories (or criteria) with multiple indicators (or factors) that define sustainability within a framework (Castro et al., 2017; Mahmoud et al., 2019) . These indicators can demonstrate a tool's scope. For example, the majority of indicators of WELL assessment are dedicated to social comfort, while the LEED criteria are more focused on energy conservation (Tleuken et al., 2021) . Each indicator is assigned special weights, although the weighting systems themselves might vary from simple aggregation to more complex formulae. These weights also indicate the importance of specific indicators UN-Habitat (2017) . The definition of indicators and weighting systems usually complies with the regulations of the country of origin UN-Habitat (2017) . Nevertheless, the involvement of stakeholders can also benefit the development of evaluation procedures in terms of ecological, functional, visual, and economic feasibility (Bal et al., 2013; . With their diverse expertise and varying interests, stakeholders' opinions can be invaluable in the development process. The specific advantages of stakeholder assessment have been highlighted in several studies Mathur et al., 2008) . The present study aims to analyze the opinions of various stakeholders regarding the weights of pandemic-resilient sustainability indicators that contribute to the sustainability of residential buildings during pandemics. More specifically, the main research aims are (1) to obtain the weights of indicators based on stakeholders' opinions and evaluate the agreement level among stakeholders and (2) to analyze the indicators, subcategories, and categories based on the opinions of stakeholders on these indicators according to their professional backgrounds (e.g., academic vs. industry; industry vs. medical) and geographic backgrounds (local vs. international). In addition, the present research aims to identify whether there is a paradigm shift in defining residential buildings' sustainability and the criteria regarding which opinions have significantly shifted. Although the pandemic can be classified initially as a health crisis, the novel coronavirus (COVID-19) continues to create challenges in all sectors of society (Tokazhanov et al., 2020) . It has been widely reported that the pandemic will cause some social and economic reshaping and the restructuring of global socioeconomic systems (Debata et al., 2020; Nicola et al., 2020; Sannigrahi et al., 2020) . In addition, a variety of studies are being conducted on the importance of urban structure and built environment conditions during the pandemic. For example, Viezzer & Biondi (2021) investigated the influence of urbanization, socioeconomic conditions, and the amount of vegetation on the number of COVID-19 cases in Brazil. It was concluded that a higher urbanization level, especially a high density and population, and lack of vegetation and trees were the most critical factors contributing to a higher number of confirmed cases of and deaths related to COVID-19 Viezzer & Biondi (2021) . The results of the study highlight the need to reconsider the current urban structure and include more natural vegetation and greenery in cities. Hu et al. (2021) illustrated that the built environment, including the quality of housing and living conditions, is strongly correlated with COVID-19 death cases (Hu et al., 2021) . Furthermore, Wang (2021) reviewed China's vision regarding future urban design, highlighting the importance of the quality of the urban living environment, public health security, and the prevention of disasters such as the COVID-19 pandemic Wang (2021) . Leng et al. (2020) suggested a new sustainable design for the courtyard environment, which would consider the distribution of airborne diseases and pollutants, risk of infection, and drought sensation (Leng et al., 2020) . Moreover, indoor built environment conditions can be controlled and maintained to prevent the transmission of the virus since higher temperatures and humidity decrease the risk of virus transmission (V et al., 2020) . Water and wastewater management, one of the important aspects of urban areas, is also greatly influenced by the COVID-19 pandemic, requiring careful management and special attention to prevent the spread of the virus Gude & Muire (2021) . These issues are a clear indication of the need for future changes in urban design and the conditions of the built environment. The construction sector is no exception to the need for restructuring (Tokazhanov et al., 2020) , as it is one of the leading sectors regarding its linkages with other sectors and the services it offers to society Serdar & Syuhaida (2016) . However, this sector, as one of the major causes of the depletion of natural resources (López Ruiz et al., 2020) , has not been operating sustainably, even prior to the pandemic . During the pandemic, as total lockdowns were implemented in several countries, the role of the construction industry in society gained further importance. Quarantines that were put in place, leading to social isolation, significantly exacerbated the adverse psychological effects of the pandemic (Brooks et al., 2020) . Thus, whether residential buildings offer a comfortable, healthier, and safer environment for their dwellers has been questioned Pinheiro & Luís (2020) . For example, a recent analysis on the effects of greenery on dwellers' mental health revealed the importance of green areas provided by dwellings since greater exposure to greenery leads to better mental health (Dzhambov et al., 2020) . Moreover, it was proven that the ventilation of the building is a crucial factor preventing the propagation of the novel coronavirus Sun & Zhai (2020) . The assessment of the sustainability performance of buildings has been widely studied (AboulNaga & Elsheshtawy, 2001; Kaatz et al., 2006; Mateus & Bragança, 2011; Tokbolat et al., 2018) . Various assessment rating tools (Haapio & Viitaniemi, 2008) , methods AlWaer & Kirk (2012), and frameworks (Akhanova et al., 2020) have been proposed for different types of buildings. Hence, depending on the type of project (e.g., residential, commercial), several assessment criteria and indicators have been reported for determining that the project is truly sustainable (Mahmoud et al., 2019) . For example, Park, Yoon, and Kim (Park et al., 2017) analyzed construction material-related indicators that are embedded in sustainability assessment tools. Lee and Burnett (Lee & Burnett, 2008 ) evaluated three sustainability tools in terms of their energy use criteria, whereas Wei et al. (2015) reported the results of indoor air quality assessment using 31 tools. The review of the context (sustainability assessment of buildings) suggests that the reported findings are limited to a specific project or assessment type (e.g., energy, water, indoor air quality). Thus, these sustainability assessment approaches are not sufficient, particularly in responding to a pandemic. Therefore, there is a need for a pandemic-specific sustainability assessment to ensure that buildings can provide a virus-preventive environment while keeping communities healthy and socially active. However, to date, no study has focused on the development of sustainability indicators that would address the requirements for residential buildings under pandemic conditions. Considering that the discussion of sustainable buildings indeed begins as an extension of green buildings (GBs) and green building certification systems have already gained worldwide recognition, in this phase of research, the green building context has been previously reviewed, and the indicators that contribute to residential buildings' response to pandemics were identified in our previous study (Tleuken et al., 2021) . Table 1 presents the indicators previously identified by Tleuken et al. (2021) (Tleuken et al., 2021) . The present study determined the weights of pandemic-resilient indicators, which were developed in our previous study, by gathering stakeholder opinions. This study contributes to the development of the sustainability of residential buildings and green building certification systems (GBCSs) by suggesting that new sustainability indicators that address pandemic conditions be considered. To the best of our knowledge, no study has investigated stakeholders' opinions on pandemic-resilient sustainability indicators. The indicators in Table 1 are divided into three major categories, which are further divided into subcategories. Each subcategory is directly related to the category to which it belongs (i.e., the mental health subcategory is directly related to the health & safety category). However, subcategories belonging to the same category do not necessarily relate to each other (i.e., the mental health subcategory is not related to the prevention of virus propagation subcategory). The relevance of the indicators to the pandemic and their importance have been previously explained in detail in our earlier study (Tleuken et al., 2021) . Stakeholders were involved in our qualitative research, in which data were collected through multiple focus groups, and the study assessed stakeholders' views toward key sustainability indicators. Stakeholders' opinions on the importance of the indicators relate to their conceptions of how things are with their specific characteristics or conditions, and they are understood in terms of the satisfaction of their expectations and demands Wolf (2018) . The methodology consists of several steps, including the recruitment of experts, the identification of indicators, the determination of techniques for survey development (dotmocracy), and the selection of statistical tools for data analysis, as described in detail in the following subsections. Stakeholders can be defined as people who have interests in research projects with the potential to influence or be influenced. When stakeholders are involved in an activity or event, they have particular requirements and interests, and they can criticize or support the idea of the research (Kassam et al., 2019) . Narayanan & Sharma (2019) stated that the involvement of stakeholders in the development process of an assessment methodology is key for achieving solutions that are environmentally, functionally, aesthetically, and economically viable for all involved. Nonetheless, it is not always clear what a stakeholder constitutes and what their involvement in research processes means, either for themselves or for the utilization of the research findings. The involvement of different stakeholder groups in sustainability research serves the expected aims since it incorporates stakeholders' references to individuals or groups that have claims, rights, or interests that might be influenced or violated. In particular, the potential for coproductive research can be achieved to increase the research impact. There are multidimensional benefits of stakeholder engagement in research, including (1) developing better management techniques, (2) satisfying ethical requirements, and (3) developing a forum for dialog to facilitate mutual social learning (Kassam et al., 2019) . Thus, the opinions of stakeholders could be deemed critical to the proper assessment and analysis of requirements. Experts were invited to participate in the study via e-mail. The e-mail list of experts was prepared based on the classification of their expertise. Experts from three main fields-academia (professors, lecturers, researchers, etc.), industry (industry engineers), and medicine (doctors, LS1. Availability of selfdependent services in the residential complexes LS2. Urban/community farming public health, medical practitioners, and students)-were invited to participate in the study. Stakeholders were classified to minimize selection bias; thus, different views were taken into account. A total of 111 participants from 17 different countries agreed to take part in the study. The majority of participants (n = 85, ~77% of all participants) were from Kazakhstan; six were from Turkey; three were from the US; two each were from Saudi Arabia, the UK, South Korea, and Belgium; and one each was from several other countries. Foreign experts were invited to compare the opinions of local stakeholders with those of foreign stakeholders. Since the pandemic is a global issue, pandemic-resilient indicators are relatable to all countries and can be evaluated independently of geographic proximity. Participants were divided into three main groups, namely, academia (Acd) experts (53 participants), industry (Ind) experts (37 participants), and medical (Med) experts (21 participants) ( Table 2 ). The academic expert group included professors, graduate students, and research assistants with expertise in related fields, such as environment, construction, energy, sustainability, construction materials, layout design, and architecture. Industry experts were people who work for different engineering companies and have expertise in civil engineering, mining and geosciences, climate change, urban planning, chemical engineering, human-centered design, architecture, and interior design. Medical experts were employed in the medical sphere and included medical students and practicing medical experts (doctors, public health professionals, sanitary professionals, epidemiologists, scientific researchers, and nurses). The reason for this classification of respondents was that the current pandemic challenges living space conditions for sustainability during quarantine. To address the sustainability issues of residential buildings under pandemic conditions, there is a need for multidisciplinary and transdisciplinary studies that include experts with both technical and health backgrounds (D'alessandro et al., 2020) . Previous studies including stakeholders used expert panels of 14 (Arditi & Gunaydin, 1999; Olawumi & Chan, 2018 , 2019 , 12 (Gunhan & Arditi, 2005) , and 120 members (Ahmad & Thaheem, 2017) . Hence, an expert panel size of 111 was considered sufficient to provide reliable responses. The indicators used in the present study were recently suggested by Tleuken et al. (2021) (Tleuken et al., 2021) to evaluate the sustainability of residential buildings under pandemic conditions (Table 1) . It should be noted that some indicators were the same as those used in other green building certification systems; however, their application as an indicator of pandemic resilience is considered new. The explanation of the relation of each of the indicators to pandemics in detail can be found in our previous work (Tleuken et al., 2021) , which is addressed in the methodology section of this paper (Table 1 ). The identification of new indicators and their further division into subcategories and categories were implemented via two steps: (1) a literature review of recent publications, blogs, news, and reports related to COVID-19 and the sustainability of residential buildings and (2) a roundtable discussion with brainstorming activity involving experts. Because of the primary scope of improving residents' health and safety during a pandemic, the range of the indicators was wider than that of conventional sustainability indicators. The three main categories-health & safety (H&S), environmental resource consumption (ERC), and comfort-were further divided into subcategories, which are represented using corresponding abbreviations (e.g., PVP, prevention of virus propagation). According to the WHO's constitution, "health is a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity" WHO (1946) . The present study assessed the health of residents during a pandemic by focusing primarily on mental health (MH) and physical health (e.g., prevention of virus propagation (PVP) and indoor air quality (AQ)-associated health problems that might occur during lockdowns). However, social well-being was linked to a limited number of subindicators (e.g., MH2, MH3, PC3, LS1, LS2) and was not subcategorized together under the (H&S) category. Finally, the indicators included different numbers of items represented by a number added to the abbreviation of the subcategory (e.g., PVP1 involved the use of new smart/innovative technologies). A more detailed description of indicator identification, selection, and categorization methodology can be found in our previous study (Tleuken et al., 2021) . In the current study, the methodology for the identification of the weights of the indicators and subcategories involved stakeholder participation. The stakeholders were separated into subgroups based on their experience and professional backgrounds (e.g., medical experts, construction experts, and academic experts). A survey with dot-voting (dotmocracy) features was employed to satisfy the requirements of the activity by collecting experts' opinions on the importance (weights) of the categories, subcategories, and indicators. Dotmocracy is a technique for collaborative decision-making where groups' expertise and wisdom about priorities on specific items are used for voting. In dotmocracy, participants are given a certain number of votes (usually dot stickers) to prioritize the items from the highest to the lowest ranking by distributing votes among a total number of items (in particular groups) Hidalgo (2018) . The dotmocracy technique has been used in various areas, such as community disaster resilience planning (Bowles et al., 2016) , citizen science (Senabre et al., 2018) , management Hidalgo (2018) , and risk communication (Adams, 2019) . This approach is beneficial for group decision-making and finding a consensus in collaborative research planning and management Hidalgo (2018) . The freedom to distribute the votes as participants wish provides flexibility in voting, allowing us to explicitly highlight the importance of one item compared to that of others. Participants are not restricted to a specific range of the score (e.g., a 5-point scale or 10-point scale). Hence, the dotmocracy technique was chosen to be used for prioritizing and evaluating the indicators and their weights. Because of the limited person-to-person communication during the pandemic conditions, the dotmocracy technique was transformed from the 'physical distribution of votes among items' into an electronic version. While the process of voting and its principles remained unchanged, physical votes were replaced by electronic votes that could be seen on participants' screens (i.e., on an Excel sheet), and the participants could allocate the available votes among the indicators. The stakeholders were then asked to evaluate the indicators of each subcategory separately since the total number of indicators was too large to compare them together, which would make the voting process cumbersome. The number of available dots for indicators of each subcategory was five times the number of indicators in that subcategory. For example, the PVP subcategory had six indicators, which means that 30 votes (6 × 5) were available to be distributed among the indicators of that subcategory (Fig. 1) . The same principle was applied to the indicators of other subcategories. The Excel file containing the list of categories, subcategories, and indicators was sent to the experts. The experts were asked to type the number of votes they wanted to assign to each item in the cell next to the item to rank the importance of indicators, subcategories, and categories. The numbers of available votes were indicated in the same Excel sheet. Personal information, including workplace, position, expertise, and living place, was also collected during the survey. The statistic interrater agreement (IRA) was proven to be a reliable option in similar previous studies involving stakeholders (Olawumi & Chan, 2018 , 2019 Olawumi & Chan, 2018; Shi et al., 2008) . Another advantage of the technique is that it is not difficult to calculate the agreement and understand it. Moreover, it can be implemented on various types of data, including ordinary, nominal, interval, and ratio-level data Goodwin (2001) . The criteria deduced by LeBreton & Senter (2007) were used as an interpretation of IRA results: 0.00-0.30 ("lack of agreement"), 0.31-0.50 ("weak agreement"), 0.51-0.70 ("moderate agreement"), 0.71-0.90 ("strong agreement"), and 0.91-1.00 ("very strong agreement"). Eq. (2) was used to identify the IRA of each factor: where SD is the standard deviation, A is the maximum scale value, B is the minimum scale value, M is the mean value, and n is the number of stakeholders. Since the IRA is not valid when the mean value is equal to scale boundary values (e.g., 1 and 5, for a 5-point Likert scale), Eqs. (3) and (4) were used to calculate the upper and lower mean limits. Since there is no maximum scale value in the dotmocracy technique, the maximum scale value was set as 10. The average available votes for one item was 5. However, if an item received a score higher than 10, then the score was set as the maximum scale value. Since the indicators were not compared together but were compared within a subcategory, it was necessary to compare and evaluate subcategories together to compensate for a possible unequal evaluation of indicator weights. Hence, stakeholders were then asked to evaluate the importance of the subcategories by comparing them to each other and assigning them scores in the same way as for the indicators as well as for the categories. It is important to note that the real weights of the indicators compared among all indicators were different and can be calculated based on the score of the subcategory and category to which they belong. Eq. (5) was used to calculate the total score of each indicator: where T is the total score of the indicator, I is the indicator's score, S is the score of the corresponding subcategory, and C is the score of the corresponding category. In this way, all indicators could be compared to identify the most important indicators with the highest total scores. For example, among academic experts, the indicator score of PVP4 was 4.40 and that of WM1 was 5.46. However, since the scores of their subcategories (5.90 for PVP and 4.68 for WM) and categories (6.09 for H&S and 4.64 for ERC) were different and were also taken into account, the total score of PVP4 was higher than the total score of WM1. The IRA was analyzed to check the agreement of experts on each item separately (Table 4) . Acd experts had the highest number of responses (n = 53) among all groups, and there was a higher level of agreement among Acd experts than among the other groups of experts. In summary, the majority of the items in the Acd group showed strong agreement or very strong agreement, and four indicators showed moderate agreement (PVP1, PVP5, AQ1, WWM2). A low IRA value indicates that the respondents gave different scores, resulting in high variance. For example, natural light is not a well-established technique against virus propagation, leading to PVP5 indicators showing moderate agreement. Additionally, 'use of smart/innovative technologies' is a vague statement, which could lead to different interpretations by different experts, leading to moderate agreement regarding the PVP1 indicator. Moreover, 'availability of separate toilets for the infected' is a new idea and has never been implemented previously, resulting in different opinions on the indicator. The reason that most of the items showed strong agreement was the large number of respondents. The IRA is simply computed using actual variance and maximum possible variance. A higher number of respondents leads to a higher maximum possible variance. The IRA of industry experts (37 responses) showed less agreement than that of the Acd experts; eight indicators showed moderate agreement, and the other items showed strong/very strong agreement. The average SD and IRA results for the Acd group and Ind group were similar (SD: 1.51 and 1.58; IRA: 0.82 and 0.80, respectively), meaning that a lower IRA was directly related to the number of respondents. A lower number of respondents led to a lower maximum possible variance, resulting in a lower IRA value. The results for medical experts (21 responses) showed several disagreements; two items showing 'lack of agreement' and four items showing 'weak agreement' were identified. The Acd and Ind groups comprised mostly engineering backgrounds, while the Med group included only individuals with medical backgrounds. In addition, the terminology and the concepts of the factors and indicators used in the study could be considered more well-known engineering/sustainability concepts than medical terms, with a self-explanation potential that may not be as obvious as that of medical experts. Consequently, the average SD (1.98) and IRA (0.69) indicated that the lack of proper sustainability/ engineering background of medical experts led the experts to have scattered opinions. For example, PVP1, which showed moderate agreement among Acd experts, lacked agreement among medical experts because of the minimal perceptions and knowledge of these individuals regarding new innovative/smart technologies against pathogen propagation. The fact that the PVP1 indicator showed moderate agreement among Acd experts and lack of agreement among Med experts indicates that it needed to be more specifically described and delivered to the experts. Additionally, the fact that the AQ1 indicator showed moderate agreement among Acd and Ind experts and a lack of agreement among Med experts indicates that it needed to be clearly described to improve the understanding of the indicator. Additionally, the number of Ind and Med experts needed to be increased to achieve better agreement. In addition, even though the IRA results were directly dependent on the number of respondents in the stakeholder group, the IRA is a useful tool to identify indicators with relatively low agreement within a stakeholder group. The technique was successfully used to identify the indicators that were the most controversial within stakeholder groups. Table 4 summarizes the results of the survey, and the mean value, standard deviation (SD), and maximum/minimum scores are given. Since the voting procedure for the indicators of different subcategories and categories was separate, the discussion of the results is also handled separately. The results for all three expert groups (Acd, Ind, Med) indicated that stakeholders agreed that H&S was the most important category (average scores of 6.09, 6.03, and 6.62 for academia, industry, and medical experts, respectively) (Fig. 2) . According to the Acd and Ind groups, the ERC category was the second most important category, with scores of 4.64 and 4.57, respectively. Med experts assigned higher scores to the comfort category (4.17) than to the ERC category (3.69). Even though the comfort category collected more votes from Acd (4.25) and Ind (4.30) experts than from Med (4.17) experts, the Acd and Ind experts were still more concerned about the consumption of environmental resources, whereas the Med group indicated that the comfort of the residents was of higher importance than environmental resource consumption. Fig. 3 represents the mean scores of subcategories by different expert groups. Among the subcategories of the H&S category, the PVP subcategory was assigned the highest score among all three groups of stakeholders (Acd: 5.90; Ind: 6.27; Med: 5.95). The second highest score was given to the AQ subcategory by Ind (5.62) experts, to the PC subcategory by Acd (5.74) experts and to the MH subcategory by Med (5.90) experts. It is important to note that the PVP and MH subcategories received almost equal scores from Med experts (5.95 and 5.90, respectively), indicating that medical experts might argue that mental health is as important as preventing virus spread. The WWM subcategory had the lowest score among all three expert groups (Acd: 3.74; Ind: 3.05; Med: 3.19). These results come along with the risk of infection, which is higher via the air than via water or wastewater. Finally, the results highlighted the high opinion of stakeholders on the importance of mental health during long periods of being indoors, which has been experienced at straining levels during lockdowns. Within the ERC category, Acd experts gave almost equal scores to the EU (5.15) and WC (5.17) subcategories and the lowest score to the WM (4.68) subcategory. Ind experts also gave the lowest score to the WM (4.38) subcategory; however, the EU (5.49) subcategory was given higher scores than the WC (4.86) subcategory. Moreover, the perspective of Med experts significantly differed, as they assigned the highest score to the WC (5.29) subcategory followed by the WM (5.10) subcategory and the lowest score to the EU (4.62) subcategory. From a medical point of view, the WM subcategory, including its indicators such as 'proper segregation of medical waste', has higher importance, and the EU subcategory has the least importance. Acd and Ind experts gave the highest scores to the EU subcategory. Subcategories of the comfort category were given similar scores by all three groups of experts, showing a commonality of opinions among different stakeholders. The personal comfort (PC) subcategory (Acd: 5.74; Ind: 5.43; Med: 5.71) received higher scores than the local services (LC) subcategory (Acd: 4.26; Ind: 4.49; Med: 4.29). Personal comfort, including ICT infrastructure, personal space, and noise insulation, was considered more important than the availability of independent local services and urban farming. Fig. 4 represents the total scores of the indicators (calculated using Eq. (1)) for the three different expert groups, from the highest to the lowest average score. The use of touchless technology (PVP2) is the most important indicator in the prevention of virus propagation according to all three groups of experts (Acd: 5.66; Ind: 5.86; Med: 7.19 ). The fact that the Med experts gave this indicator the highest score indicates the crucial role of touchless technologies in preventing the spread of the virus via direct contact with contaminated surfaces. The PVP1 indicator received almost the same score as PVP2 by Ind experts (5.78), and it was the second most important indicator for Med experts (6.29). However, the Acd experts gave lower scores to PVP1 (4.79) than to PVP5 (4.85), which received the lowest scores from Ind (3.68) and Med (3.10) experts. In general, the Med experts' opinions differed from the Acd and Ind experts' opinions according to the importance given to the PVP1 and PVP2 indicators, while the Acd and Ind experts distributed their votes relatively evenly among indicators. Additionally, one of the Acd experts argued that PVP5 should be ranked higher than PVP6, and another Acd expert highlighted that natural light, temperature, and ventilation were the most crucial aspects when being locked down during quarantine. The MH1 (Acd: 5.36; Ind: 5.32; Med: 5.19) and MH3 (Acd: 5.21; Ind: 5.51; Med: 5.57) indicators of the mental health subcategory were given similar scores by all three expert groups. The MH4 (household-level activity/sport spaces) indicator was given the lowest score by the Acd (4.08) and Ind (4.27) experts, while the Med experts gave the lowest score to the MH2 (outdoor space) indicator (4.52). Interestingly, the Ind and Med experts highlighted the importance of MH1 (greenery and gardens) and MH3 (common building space), giving fewer points to MH2 and MH4, while the Acd experts gave relatively similar scores to MH1-3, giving lower points to the MH4 indicator. In the air quality subcategory, all three expert groups highlighted the importance of the AQ1 indicator, giving it the highest scores (Acd: 5.96; Ind: 5.57; Med: 5.95), while giving the lowest scores to the AQ3 indicator (Acd: 3.71; Ind: 4.32; Med: 3.33). This finding indicates the importance of air filtration in the prevention of pathogen spread. The AQ4 indicator received higher scores than the AQ2 indicator from the Acd experts (AQ4: 5.68; AQ2: 4.65) and Ind experts (AQ4: 5.19; AQ2: 4.76), highlighting the importance of natural ventilation, while the Med experts gave higher scores to AQ2 (5.52) than AQ4 (5.10), giving greater priority to the control and monitoring of indoor air pollution. For all three expert groups, in the WWM subcategory, WWM1 (specific measures to limit virus propagation at the household level) was the most important indicator (Acd: 6.11; Ind: 6.62; Med: 6.62), while the WWM3 indicator (separation of graywater) received the lowest average score (Acd: 4.05; Ind: 3.86; Med: 3.05). The WWM2 indicator received higher scores from the Med experts than from the other two groups of experts (Acd: 4.84; Ind: 4.19; Med: 5.29 ). In the 'energy use' subcategory, the EU1 indicator received the lowest average score from all three groups of stakeholders. The Ind and Med experts gave slightly higher scores to EU2 than to EU3. Conversely, the academic stakeholders gave a higher average score to the EU3 (use of energy-efficient appliances) indicator. In general, the EU indicators were similarly evaluated by all three groups of experts. The WM and WC indicators received similar scores, as all three groups' opinions correlated. Subsequently, the WM1 indicator received the highest scores (Acd: 5.46; Ind: 5.30; Med: 6.24) from all three groups of experts, whereas WM2 received the lowest scores (Acd: 4.61; Ind: 4.25; Med: 3.71). Furthermore, all respondents rated WC2 (Acd: 5.46; Ind: 5.70; Med: 5.38) as a more important indicator than WC1. Among all stakeholders' opinions, PC3 (noise insulation) was considered the most preferable to the other indicators. The other three indicators, PC1 (ICT), PC2 (indoor space adjustability), and PC4 (personal space), had close average values. Industrial experts preferred PC1 (5.22) to PC2 (4.49) and PC5 (4.70). Medical stakeholders denoted PC2 (4.71) as more valuable than the PC1 (4.24) and PC4 (4.29) indicators. This finding shows the importance of personal space during a pandemic. All three groups of experts prioritized the LS1 indicator (Acd: 5.41; Ind: 5.59; Med: 6.57) over LS2 (Acd: 4.59; Ind: 4.19; Med: 3.19 ). It should be noted that the difference between the scores for LS1 and LS2 given by the medical experts was much higher than that between the scores given to these indicators by the other groups of experts. Table 5 provides a summary of the results grouped by local (Kaz) experts and all other international (Int) experts, along with the differences in mean values. Both the Kaz and Int experts assigned scores to the categories in the same order of importance (H&S > ERC > Comfort). However, the Int experts assigned higher scores to the H&S category (Kaz: 6.04; Int: 6.62), whereas the Kaz experts assigned higher scores to the ERC and Comfort categories. A possible reason for this is that Kazakhstani citizens experienced extensive discomfort during the lockdowns since many families tend to be in one-room apartments, which may be drastically insufficient to provide proper comfort for a group with 3-5 family members, and this type of accommodation may not be common in the other countries where the Int experts were from. Among subcategories, the largest differences were found in the PVP (Kaz: 5.81; Int: 6.79) and WQ (Kaz: 5.00; Int: 4.12) subcategories. The higher score for the WQ subcategory may have been related to the fact that tap water in Central Kazakhstan and West Kazakhstan is not potable in many regions, including the capital of the country, Nur-Sultan city. Both groups of experts gave the highest score to PVP2 and the lowest score to PVP5. The PVP2, PVP3, and PVP6 indicators showed the largest differences in score. International experts gave higher scores to PVP2 and PVP3, highlighting the importance of touchless technologies, while PVP6 was rated higher by local experts. The MH3 indicator showed the largest difference (0.87) among the MH indicators, possibly due to less comfortable apartments, as mentioned above. AQ indicators had the same order of importance, with the largest difference being seen for the AQ3 indicator (Kaz: 4.03; Int: 3.23). Among WWM indicators, WWM1 was highlighted by local experts, while WWM2 was highlighted by international experts. Other indicators received relatively similar scores from both groups of experts. The above results indicate that sustainability assessment tools, including indicators, require country-specific adaptation. Moreover, Kazakhstan has four climatic zones, which might create a need for a climate-specific adaptation of sustainability assessment tools within the country (Akhanova et al., 2020) . For example, water quality might be a more severe problem for Central Asian countries than for the USA or European countries since there is a problem with potable water in Central Asia (Zhupankhan et al., 2018) . Another example is PVP6, which was more important for local experts. For Kazakhstani residents, the temperature and humidity of the house during different seasons are important for both health and comfort. In wintertime, when the outside temperature is very low, it is important to keep a normal temperature inside. Additionally, heaters that are hot all the time decrease the humidity of the indoor air. In a previous study (Tleuken et al., 2021) , the readiness of green building certification systems (GBCSs) for pandemics was analyzed using the same indicators. It should be emphasized that PVP4, PVP6, AQ2, and AQ4 are the only indicators that are also addressed by four different well-known GBCSs (i.e., BREEAM, LEED, WELL, and CASBEE) and could be considered 'conventional' indicators. The new indicators, which are obviously related to the current pandemic, are compared to the 'conventional' indicators to examine the difference in experts' opinions toward these indicators. The comparison indicates the prioritization of the indicators, showing that some indicators are a priority over others, which is further indicated by a 'shift' in the experts' opinion. The findings of the present study suggest that there is a significant shift in experts' opinions toward pandemic resilience. In detail, PVP4 was the least important indicator according to the Acd experts, and PVP6 received a lower score than PVP2 and PVP3. These results demonstrate a shift in the opinions of Acd experts in favor of pandemic-resilient indicators. Additionally, PVP1, PVP2, and PVP3 received higher scores than PVP4 and PVP6 from both Ind and Med experts, further underlining the shift in stakeholders' opinions. Finally, the AQ1 indicator received a higher score than AQ2 and AQ4 from all three groups of experts. Since AQ1 is directly related to the propagation of pathogens, it is another indication of the shift in stakeholders' opinions under pandemic conditions. All expert groups (Acd, Ind, and Med experts) assigned the health and safety category the highest score, agreeing that the health and safety of the residents are much more important than comfort or environmental resource consumption under pandemic conditions. Under the health and safety category, the prevention of virus propagation (PVP), mental health (MH), and air quality (AQ) subcategories again received the highest scores from all three expert groups. This result highlighted a consensus on the importance of the prevention of virus spread during a pandemic and the crucial role of air quality in the health of residents as well as their psychological state during lockdowns. As agreed upon by all experts, the use of touchless technologies was found to be the most promising way to prevent the spread of viruses, along with the use of other smart technologies and self-cleaning spaces. According to common stakeholder opinion, access to common building spaces with adequate safety precautions for the health of residents, as well as access to outdoor space, play important roles in providing the proper conditions for the healthy mental state of the residents. Since virus spread mainly occurs via airborne droplets, air filtration is crucial for the health of residents. The proper segregation of medical waste attracted the attention of all expert groups, as it received the highest score in the waste management (WM) subcategory. Among categories, the major difference is that the Med experts prioritized the comfort category, while the Ind and Acd experts prioritized the ERC category over the comfort category. Moreover, the PVP, PC, and MH subcategories were highlighted by Ind, Acd, and Med experts, respectively. This finding clearly shows the importance of virus spread prevention, comfort, and the mental health of residents during a pandemic. In general, Med experts' opinions were different from Acd and Ind experts' opinions, highlighting the considerable difference in the perspectives of experts with technical and health backgrounds. This finding illustrates the importance of including experts from different backgrounds when developing pandemic-resilient indicators. A shift in experts' opinions toward pandemic resilience indicators was also observed, as certain conventional indicators (such as adjustability of indoor temperature and humidity and monitoring and control of indoor air pollution) were given low scores, while new indicators (such as efficiency of air filtration systems for pathogen propagation and use of touchless technologies) that are directly pandemic related were assigned greater importance. This finding indicates that there is a need to revise existing sustainability assessment methods because of their inability to evaluate buildings' sustainability under pandemic conditions, which may be critical in the future. Finally, a comparison of the opinions of local experts with those of international experts showed a consensus on the importance of indicators and similarities in terms of the ranking of the indicators along with small differences in scores for a few individual indicators. However, the differences in a few indicators indicate that the sustainability of residential buildings should be strictly country-or climate-oriented because of the different needs of residents of different climatic zones and/or countries. The key contribution of the present study to the literature is that it questions whether a conventional understanding of sustainability matches the pandemic reality. The study evaluates the importance of the recently introduced pandemic resilience indicators, suggesting that there is a need for new sustainability requirements for residential buildings. The importance of some of these new sustainability indicators has been highlighted by the analysis of stakeholders' opinions. Overall, the study shows how to efficiently react to the pandemic by rapidly learning from the current situation and then designing a more sustainable future with better pandemic-resilient residential buildings. One limitation of the present study is that the dotmocracy technique did not allow us to conduct Cronbach's alpha reliability test, which is used to identify the internal consistency of the data. This is because the variance of the observed total scores was zero since the total number of votes was fixed at the beginning of the survey. Another limitation is that we were unable to compare the stakeholders' opinions regarding the same indicators in a time with no pandemic to provide a more precise elucidation of the shift in experts' priorities. Such a comparison is a future research need that can be completed after the pandemic ends. This study did not investigate indicators and factors related to social well-being in detail. Other research focusing on how residential buildings respond better to the social well-being of individuals staying for more extended periods of time in self-isolation is needed. The limited number of stakeholders and the large number of those from academia may also have been a limitation and may have created bias. The COVID-19 pandemic itself also limited the study due to the virus's nature, as the results of the study were related to opinions about residential buildings in the presence of a disease that is transmitted via air or through contact with contaminated surfaces. As COVID-19 is a novel disease, it cannot be concluded whether the paradigm shift is temporary or permanent; the nature of the shift depends on the different ways the disease will be approached in the future. In a future study, the same people can be asked to share whether their opinions have changed permanently after the pandemic is over. First Nations Boil Water Advisories: New Methods and New Approaches for Risk Communication Environmental sustainability assessment of buildings in hot climates: The case of the UAE Developing a residential building-related social sustainability assessment framework and its implications for BIM. Sustainable Cities and Society A multi-criteria decisionmaking framework for building sustainability assessment in Kazakhstan On the coronavirus (COVID-19) outbreak and the smart city network: Universal data sharing standards coupled with artificial intelligence (AI) to benefit urban health monitoring and management. Tackling Coronavirus (COVID-19) Contributing to a Global Effort Building sustainability assessment methods Perceptions of process quality in building projects Stakeholder engagement: Achieving sustainability in the construction sector Covid-19: First vaccine given in US as roll-out begins Interrater agreement of classification of photoparoxysmal electroencephalographic response An analysis of the most adopted rating systems for assessing the environmental impact of buildings Building resilient communities: A facilitated discussion Building sustainability assessment The psychological impact of quarantine and how to reduce it: Rapid review of the evidence Interrater agreement reconsidered: An alternative to the rwg indices Healthcare building sustainability assessment tool -sustainable effective design criteria in the Portuguese context A mixed approach on resilience of Spanish dwellings and households during COVID-19 lockdown COVID-19 and living space challenge. Wellbeing and public health recommendations for a healthy, safe, and sustainable housing COVID-19 pandemic! It's impact on people, economy, and environment 2019 novel coronavirus (COVID-19) pandemic: Built environment considerations to reduce transmission. MSystems Does greenery experienced indoors and outdoors provide an escape and support mental health during the COVID-19 quarantine? Environmental Research Interrater agreement and reliability Preparing for outbreaks -implications for resilient water utility operations and services Factors affecting international construction A critical review of building environmental assessment tools Resilience in the built environment Dotmocracy and planning poker for uncertainty management in collaborative research The role of built and social environmental factors in COVID-19 transmission: A look at America's capital city The continuing 2019-nCoV epidemic threat of novel coronaviruses to global health -the latest 2019 novel coronavirus outbreak in Wuhan Advancing key outcomes of sustainability building assessment Indicator rating methodology for rapid sustainability assessment method (RSAM) for existing residential buildings using opinions of residents A new stakeholder opinion-based rapid sustainability assessment method (RSAM) for existing residential buildings Key stakeholder opinions for a national learner education handover Answers to 20 questions about interrater reliability and interrater agreement Benchmarking energy use assessment of HK-BEAM Sustainable design of courtyard environment: From the perspectives of airborne diseases control and human health The circular economy in the construction and demolition waste sector -a review and an integrative model approach Development of sustainability assessment tool for existing buildings Sustainability assessment and rating of buildings: Developing the methodology SBToolPT-H Conceptualizing stakeholder engagement in the context of sustainability and its assessment Coronavirus outbreak is part of worldwide increase in disease spread Research on stakeholder opinions and financial performance in reliance industry The COVID-19 pandemic : Considerations for the waste and wastewater services sector The socio-economic implications of the coronavirus pandemic (COVID-19): A review Identifying and prioritizing the benefits of integrating BIM and sustainability practices in construction projects: A Delphi survey of international experts Critical success factors for implementing building information modeling and sustainability practices in construction projects: A Delphi survey Critical review of the material criteria of building sustainability assessment tools COVID-19 could leverage a sustainable built environment Examining the association between socio-demographic composition and COVID-19 fatalities in the European region using spatial regression approach Participatory design of citizen science experiments On-site construction productivity in Malaysian infrastructure projects Work in progress -improving interrater agreement used to measure learning outcomes The efficacy of social distance and ventilation effectiveness in preventing COVID-19 transmission Readiness assessment of green building certification systems for residential buildings during pandemics Construction professionals' perspectives on drivers and barriers of sustainable construction Assessment of green practices in residential buildings: A survey-based empirical study of residents in Kazakhstan Building sustainability assessment and benchmarking The contribution of dry indoor built environment on the spread of coronavirus: Data from various Indian states The influence of urban, socio-economic, and ecoenvironmental aspects on COVID-19 cases, deaths and mortality: A multi-city case in the Atlantic Forest Building sustainability assessment methods: indicators, applications, limitations and development trends Vision of China's future urban construction reform: In the perspective of comprehensive prevention and control for multi disasters Indoor air quality requirements in green building certifications WHO Timeline -COVID-19 Stakeholders' opinions of quality in Norwegian kindergartens Water in Kazakhstan, a key in Central Asian water management The authors acknowledge the financial support from Nazarbayev University Faculty Development Competitive Research Grant Program (FDCRGP; Funder Project Reference: 280720FD1904). The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. We would like to extend our special thanks and acknowledge the time and effort devoted by reviewers to improve this paper's quality.The authors acknowledge and thank Nurly Kuzdikbay for her help with the Graphical Abstract of this study.