key: cord-0840080-styfwuej
authors: Zheng, Han; Pee, L. G.; Zhang, Dan
title: Societal impact of research: a text mining study of impact types
date: 2021-07-10
journal: Scientometrics
DOI: 10.1007/s11192-021-04096-6
sha: 344066ad33b6e0efd8c66db2ad2acb6a0788096c
doc_id: 840080
cord_uid: styfwuej

In addition to academic impact, researchers are increasingly concerned with understanding and demonstrating the practical impact of research outside academia. Several frameworks capturing key impact types have been developed based on project experiences, expert opinions, and surveys. This empirical study seeks to contribute to this development by identifying impact types documented in 6,882 case studies submitted to impact evaluation groups in Australia (Engagement and Impact Assessment) and the United Kingdom (Research Excellence Framework). The results of text mining indicate three emerging impact types that extend existing frameworks in terms of the recognition of new opportunities, the length of use, and experience improvement, thereby allowing a variety of researchers, not just those who address popular, short-term, and instrumental issues, to understand and demonstrate their practice impact.

For decades, scientometrics researchers have examined scholarly impacts in terms of bibliometrics, webometrics, citation patterns, altmetrics, and authorship networks (e.g., Chi and Glänzel 2018) . Along with scholarly impact, researchers are increasingly seeking to understand how their work makes a difference in the real world (Glänzel and Chi 2020; Pee and Kankanhalli 2009) . The practice impact of research refers to "the observable benefit of research on relevant stakeholder groups beyond academia, such as individuals, organizations, communities, industries, or economies, generated through interactions with them" (Pan and Pee 2020, p. 4 ). In the emerging responsible research movement, many even consider societal needs as functional requirements for the design and development of new research projects (Asveld and van Dam-Mieras 2017) . On the other hand, researchers also face mounting pressure from taxpayers and funding agencies to demonstrate the return on investment in research (Wiek et al. 2014) . Many funding agencies now require a pathwayto-impact statement in grant applications; some agencies have begun to conduct impact evaluations regularly, such as Australia's Engagement and Impact Assessment, Italy's Research Quality Evaluation, the Netherlands's Standard Evaluation Protocol, and the United Kingdom's Research Excellence Framework (REF) .

To help researchers demonstrate how their work ultimately benefits those beyond academia, educate the public on the value of research, and convince funders of the necessity of continuously investing in research, frameworks that identify and organize different types of impact have been developed. They draw upon the experiences of researchers, research projects, and experts and take different perspectives of practice impacts. Some of them focus on the impact fields (e.g., social, technological, economic, and cultural; Moed and Halevi 2015) , while others focus on the impact dimensions (e.g., importance, value; Morrow et al. 2017 ) and the specific nature of the impact as research is utilized in practice (e.g., improvement in awareness, capacity, behavior; Morton 2015) . In this study, our focus is on the latter because they are more multidisciplinary and offer a more actionable perspective of achieving practice impact.

This study seeks to contribute to the development of frameworks on the nature of impacts by empirically identifying types of impact from 6,882 impact case studies submitted for impact evaluations in Australia and the UK (Australian Research Council 2018; Research Excellence Framework 2012). An impact case is a narrative that describes how research resulted in a change or had an effect on or benefited stakeholders outside academia. This dataset is unique in that it focuses specifically on how research has impacted practice. It permits a more data-driven approach to identifying different types of impact and complements prior approaches driven by existing concepts, such as surveys and literature reviews. The rich dataset documents the impact of various disciplines and offers a rare opportunity for a multidisciplinary understanding of impact types. Findings from a large dataset are also potentially more representative and generalizable than those from a small sample.

To account for existing frameworks while allowing new types of impact to emerge from the dataset, we used the abductive approach to analyze and interpret findings. The corpus of case studies was first analyzed with topic modeling. Each topic was then examined further by inspecting representative impact cases to identify themes. Finally, the themes were compared with those in existing frameworks to highlight opportunities for further development.

Frameworks delineating the specific nature of the impact of research utilized in practice were first reviewed to understand the state of development. Studies that analyzed impact case studies submitted to impact evaluation bodies were also reviewed to identify remaining research gaps.

Most of the existing frameworks were identified through case studies of selected research projects and interpretive reviews of publications documenting expert opinions and experiences (see Table 1 ). In comparison, there have been fewer quantitative analyses (e.g., surveys) and a lack of studies analyzing impact cases submitted for impact evaluations. Text mining of a large number of impact cases covering multiple disciplines is expected to complement existing approaches by providing an indication of the extent to which existing frameworks cover the types of impact observed as research is utilized in practice.

Most frameworks have identified four to six types of practice impacts related to research utilization. They range from usable research products to the application of research in practice to the benefits generated from research utilization (see Table 1 ). Collectively, these frameworks show that translating research results into forms that can be readily used in practice and making their availability known to potential users generates an impact in terms of awareness and affordance. The adoption of research products enhances users' capacity to make behavioral choices or strategic decisions. The utilization of research is also expected to have observable benefits on outcomes that matter in practice. The most recent framework by Pan and Pee (2020) covers all the common types of impact while distinguishing between efficiency and effectiveness as valuable outcomes of research utilization (see Table 2 ).

Most of the prior studies focused on specific disciplines such as marketing and education (see Table 1 ). While discipline-specific frameworks capture the unique ways each discipline generates impact and help to ensure that no discipline is disadvantaged (Sousa and Brennan 2014) , the demand for multidisciplinary frameworks is growing, as impactful research often spans different disciplines (Bornmann and Marx 2014) . Multidisciplinary frameworks also orientate researchers toward an epistemic culture, which makes visible the complex texture of knowledge as practiced in the social spaces of modern institutions by expanding the space of knowledge in action rather than simply observing disciplines or specialties as organizing structures (Pee and Chua 2016; Sousa and Brennan 2014) .

We also reviewed studies that analyzed impact cases submitted to impact evaluation programs, as these cases constitute our dataset (see Table 3 ). The objectives of prior studies ranged from understanding interpretations of practice impact by different disciplines and institutions (Terämä et al. 2016) to understanding the impact of specific disciplines (e.g., Kelly et al. 2016) and to developing text-mining approaches (e.g., Terämä et al. 2016) . This shows that impact cases constitute a very rich dataset for understanding the various aspects of practice impact, including types of impact. Notably, although some studies used text mining techniques to analyze REF impact cases and identify the fields of impact (e.g., clinical applications, education; Terämä et al. 2016) , there is still a lack of studies on the nature of impact due to research utilization. This study seeks to contribute to the stream of research on impact cases by addressing this gap.

Our dataset constitutes impact cases submitted to impact evaluation agencies in Australia and the UK (Australian Research Council 2018; Research Excellence Framework 2012). All 245 impact cases publicly accessible from the Australian agency's website (https:// datap ortal. arc. gov. au/ EI/ Web/ Impact/ Impac tStud ies; see Fig. 1 ) and 6,637 nonredacted cases available on the UK agency's website (https:// impact. ref. ac. uk/ cases tudies/) were 2. Attention generated (e.g., social media mentions) 3. Depth of use (e.g., number of adopters) 4. Breadth of use (e.g., types of use) 5. Improvement in efficiency (e.g., cost reduction) 6. Improvement in effectiveness (e.g., revenue) (2015) To extract common themes and messages that will form evidence of the broad impact of higher education research on wider society Text mining and content analysis 6,679 REF case studies A list of topics and keywords summarizing impact cases of different disciplines, impact pathways, and beneficiaries Greenhalgh and Fahy (2015) To understand the impact of research on community-based health sciences

Most case studies focused on direct and short-term impacts. Relatively low emphasis on types such as productive interactions and reduced mortality Engaging with policy and practice is a two-way process and can in turn lead to academics being asked to participate in the process of development of policy and critique the practice of governance, which in turn provides other challenges *Studies are listed in reverse chronological order retrieved for analysis. The cases span multiple disciplines, including health and life sciences, engineering, information technology, physical science, social sciences, arts, and humanities. Both agencies require submissions to describe practice impact in a "Details of the Impact" section and the contributing research in a separate section. To identify impact types, we focused on analyzing the "Details of the Impact" section. The final corpus containing a total of 9,870,261 words was analyzed in four steps. First, the large corpus was initially processed with word co-occurrence network analysis to identify popular phrases that might suggest prevalent types of impact (Jacobi et al. 2016 ).

Second, topic modeling was conducted to analyze the corpus in greater detail by identifying key topics and underlying keywords (Blei et al. 2003) . Third, each topic was manually labeled to indicate its focus. Finally, the labeled topics were further analyzed by comparing them with the six types of impact identified in existing frameworks, with the goal of identifying 1) any type that emerged in actual impact cases but not in existing frameworks and 2) any type that were in the frameworks but not prevalent in impact cases. The four steps are discussed in further detail in the following section.

The documents were preprocessed and analyzed using the Python 3 programming language in Jupyter Notebook, an open-source application for data analysis. In data preprocessing, we first tokenized the documents into a list of words. Punctuations and stop words were removed (i.e., grammatical words such as "the," "a," and "and," which do not add meaning to the text and very frequent words such as "impact," "research," "new," "page," "case," "study," "date," and "ref"). Bigram and trigram language models using Gensim's Phrases model (available at: https:// radim rehur ek. com/ gensim/ models/ phras es. html) were also used to extract two-or three-word phrases frequently occurring together in the document (e.g., "mental_health" and "climate_change"). Lemmatization was then performed to reduce inflected words to their dictionary form.

The preprocessed data were initially examined with word co-occurrence network analysis to identify popular phrases that might suggest prevalent types of impact (Jacobi et al. 2016) . When a keyword is paired with other keywords more frequently, that given keyword will build more links in the network and is then assumed to be a popular term. There were a total of 51,690,211 pairs of words in the corpus. Figure 2 shows 500 pairs (links) with the highest co-occurrence. Words such as "national," "international," "public," "development," and "support" frequently co-occurred with other words. This suggests that the breadth of impact and the extent to which research outputs contribute to improvement are likely to be the key types of practice impacts. The corpus was further examined with Latent Dirichlet Allocation topic modeling to identify more specific impact types.

Topic modeling is an unsupervised machine learning technique useful for uncovering hidden thematic structures or topics that occur in a collection of documents (Blei 2012) . A topic consists of a cluster of words or phrases that show similar patterns of occurrence; documents may relate to more than one topic, and topic modeling calculates a weight with which each topic relates to a particular document. Each topic is then manually labeled by interpreting the cluster of words and most representative documents. In the context of this study, "documents" analyzed are the impact cases retrieved from EI and REF public databases. The Latent Dirichlet Allocation (LDA) topic modeling package in MALLET (McCallum 2002) was employed.

To identify the most coherent model, we first computed the topic coherence score (Newman et al., 2010) for models of 10, 20, 30, …, and 100 topics (in steps of 10). It was found that the model with 60 topics scored highest (see Fig. 3 ). The model was then manually examined and compared with the 40-topic model and 70-topic model. All authors evaluated the results separately and agreed that the 60-topic model covered more relevant topics than did the 40-topic model and had fewer uninterpretable topics than the 70-topic model. Therefore, we selected the 60-topic model for further analysis.

Each of the 60 topics was manually labeled to capture its focus. We first identified the 20 words most representative of a topic based on their beta values. A beta value refers to the probability of a word being generated from a given topic (Chuang et al. 2013) . To further understand the meaning and context of the words, we examined the impact cases most representative of each topic (i.e., top 5%), which were impact cases that had the highest probabilities of including the topic, as identified by LDA topic modeling (Piepenbrink and Gaur 2017) . Figure 4 illustrates the top 9 topics (i.e., topics containing the highest number of case studies) with their corresponding words presented in word clouds. The font size of words in each word cloud was based on the beta value. For example, for topic 4, the most representative words included "patient," "test," "clinical," and "diagnostic," and the most representative cases discussed how research findings had contributed to increasing diagnostic accuracy. Therefore, the topic was labeled accordingly. 

The topics identified from impact cases submitted to EI and REF were compared with the types of impact in existing frameworks of research utilization (Pan and Pee 2020). The comparison was performed independently by the three researchers. The initial inter-rater agreement was 97 percent. The differences were then successfully resolved through a follow-up discussion. Table 4 shows the comparison with the most recent model (Pan and Pee 2020) , which covers all six types of impact (see Table 2 ). We observed that all six types in existing frameworks were present in impact cases. In addition, three new types emerged from the impact cases: recognition of new opportunities among potential users, length of use, and experience improvement for users. The top three impact types mentioned in the impact case studies were effectiveness improvement for users (36.99%), experience improvement for users (19.83%), and effort to translate research findings for users (15.49%) (Fig. 5) . The rest of this section elaborates on the three new impact types discovered.

"Recognition of new opportunities among potential users" refers to the extent to which research outputs contribute to debates and discussions around potential problems and solutions in practice. This type of practice impact is different from attention or awareness in that potential users consciously deliberate and evaluate research products' potential usefulness for intended as well as unintended contexts, leading to a deeper understanding of research products' affordances as well as constraints and an even clearer definition or redefinition of problems. For instance, philosophy researchers at the University of Southampton shared their research findings with over three million members of several different publics through campaigns and achieved "an array of cultural impacts, including bringing lay audiences to ask themselves new questions and reassess familiar problems; stimulating debate with respect to those questions and problems; and encouraging non-philosophers to explore material they would not otherwise have encountered" (University of Southampton 2014). Likewise, researchers at the University of Manchester explored how to increase citizen engagement, such as the donation of goods. They promoted research findings to various policymakers through a number of events, such as private meetings and public events. This served to stimulate policy debate on localism and the "Big Society," providing a foundation for demonstrating the practical impact of research (University of Manchester, 2014). "Length of use" refers to the duration research outputs have been put into practical use. This type of impact complements the depth of use/capacity and breadth of use/network effects by taking into account the temporality of research utilization and focusing on the sustained use of research outputs. For instance, the Million Women Study coordinated by the Cancer Epidemiology Unit at Oxford showed the relationship between hormone replacement therapy (HRT) and the development of breast, endometrial and ovarian cancers. The REF impact case demonstrated "the continuing impact of this research on behavior in terms of continued reduced HRT use" throughout the REF assessment period of 2008 -2013 . (University of Oxford 2014 . Similarly, other impact cases of this type demonstrated an impact in terms of increasing uptake over time, extended use, or persistent practices/policies. For example, the research project conducted by the University of Leicester had a significant impact on the development of continuing professional development for science educators in primary schools. It addressed the problem that many teachers lacked confidence and competence in science teaching. The project achieved a sustained impact on teachers' practice and students' learning and engagement (University of Leicester 2014).

"Experience improvement for users" focuses on people's sensory and emotional states. This type of impact complements effectiveness improvement and efficiency improvement by going beyond the utilitarian impacts of research products and recognizing emotional, symbolic, cultural, or social values as important aspects that people seek to improve in practice. For instance, researchers at the University of Ulster integrated and implemented various visual effects into augmented reality to enable a high-end cinematic experience. In practice, the research "expanded the aesthetic and genre of the narrative, sharing user experience and thus reaching significant new user demographics" (University of Ulster 2014). Healthcare research has improved the experience of patients, in addition to efficiency and effectiveness. For instance, research conducted at the University of Nottingham was the basis of a Healthy Living Pharmacies initiative. The initiative led to a more cost-effective delivery of public health services as well as an increase in service quality as perceived by the public. Specifically, 98% of users surveyed agreed that they would recommend the service, and 81% rated the quality of service as "excellent" (University of Nottingham 2014).

We set out to identify types of impact generated from research utilized in practice by analyzing a large dataset of impact cases. In addition to the six already identified in existing frameworks of research utilization, we observed three emerging types of practice impact, as summarized in Table 5 .

The findings should be interpreted in light of several limitations, which also indicate opportunities for further research. First, the impact cases in our sample mainly documented the impact of research conducted by institutions in Australia and the UK. The research and practical concerns in these countries might differ from those in other geographical regions. To further improve the representativeness of the findings, future research can include impact case studies from other regions or countries when they become publicly available. Second, impact cases describe impacts realized in the past, by definition. Therefore, our findings might not capture those arising from the latest technological advancements and novel phenomena (e.g., artificial intelligence). To keep the frameworks of research utilization updated, it is necessary to regularly analyze new impact cases. For example, future research can analyze the impact cases submitted to the upcoming REF 2021 and update the findings of this study as necessary. Third, there remains a possibility that new disciplines can arise in the long term for which the types we found are not applicable. This, again, points towards the need to collect and analyze new impact cases routinely to account for the latest developments.

The extended framework of research utilization with nine types of practice impacts is more representative of the range of impacts in that it is based on a large sample of 6882 impact cases. It offers a comprehensive yet concise overview of possible types of impact that can be used as a basis for further conceptual development. For research seeking to clarify the nature of impacts (i.e., what constitute practice impact?), the nine impact types can serve as the building blocks on which a more inclusive definition can be abstracted. For research examining the process of achieving practice impact (i.e., how to achieve practice impact?), understanding the types of impact provides a basis for charting out the pathway from research products to observable practice impact. For example, we found that the recognition of new opportunities among potential users is an impact often observed in impact cases. This impact is potentially achievable as soon as research findings are translated and communicated to users in a way that stimulates deliberation. This suggests a pathway that realizes practice impacts progressively and cumulatively, rather than only at the end of a lengthy process.

The extended framework is also more multidisciplinary, as our dataset has wide coverage, including arts, engineering, humanities, medicine, and sciences. In line with this, most, if not all, of the themes computationally extracted in topic modeling were not specific to a discipline (see Table 4 ). Indeed, practice impact should be viewed from the beneficiaries' perspective, and real-world challenges often transcend disciplinary boundaries (Bornmann 2013) . For research focusing on the evaluation of practice impact (i.e., how to measure practice impact), our extended framework identifies a set of impact types that can be used to assess specific disciplines as well as multidisciplinary projects. This also helps to promote interdisciplinary collaboration, as having a common set of impact types serves to bind different disciplines together. For scientometrics research, the extended framework can be applied in studies that analyze academic impact vis-à-vis practice impact. For example, the impact types can be used to rate impact cases quantitatively so that practice impact can be analyzed along with quantitative academic impact types such as the citation count of underlying research. More importantly, the extended framework can serve as a basis for developing a more balanced evaluation of impacts -one that promotes research serving the needs of science as well as society rather than "closed science and overemphasis on elite, English-only publishing practices," as the COVID-19 crisis manifests (Zhang et al. 2020 ).

The three emerging types of impact identified in this study increase the coverage of existing frameworks of research utilization while maintaining their parsimony for pragmatic application. They allow a variety of researchers, not just those who address short-term, popular, and instrumental issues, to demonstrate their practice impact. For example, "recognition of new opportunities," which emphasizes the discussions and debates sparked by research outputs, is especially suitable for research on philosophical, future-oriented issues such as information and ideas generated by artificial intelligence; "length of use" allows researchers working on niche yet consequential issues, such as digital preservation, to demonstrate their impact even when the absolute number of users adopting their research outputs is typically low; "experience improvement" is more relevant for research on personally meaningful issues such as information experience, compared to efficiency and effectiveness types.

The extended framework is also more realistic to the extent that it is based on impact cases documenting observed rather than expected impacts. It complements existing frameworks that have mostly been developed based on expert opinions, expectations, intuitions, and personal experiences. It is also operational in that the types of impact identified can be and have been demonstrated in practice. For researchers seeking to demonstrate their practice impact, the large number of possible metrics is often cited as a barrier and source of confusion (e.g., Given et al. 2015; Pee et al. 2008 ). The extended framework offers a starting point for quickly determining suitable types before delving into more specific metrics. Furthermore, the impact types can be combined with existing types of scholarly impact, such as citation count, to demonstrate the spectrum of one's research impact more clearly.

Latent Dirichlet allocation

What is societal impact of research and how can it be assessed? A literature survey

How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons

Comparison of citation and usage indicators in research assessment in scientific disciplines and journals

Topic Model Diagnostics: Assessing Domain Relevance Via Topical Alignment

Bracing for Impact: The Role of Information Science in Supporting Societal Research Impact

The big challenge of scientometrics 2.0: Exploring the broader impact of scientific research in public health

The Nature, scale and beneficiaries of research impact: An initial analysis of research excellence framework (REF) 2014 Impact Case Studies. HEFCE-Higher Education Funding Council for England

Research impact in the community-based health sciences: An analysis of 162 case studies from the 2014 UK research excellence framework

Achieving wider impact in business and management: Analysing the case studies from REF 2014

Quantitative analysis of large amounts of journalistic texts using topic modelling

Beyond the academy: applying anthropological research, a case study of demonstrating impact in the UK 2014 REF

Impact case studies submitted to REF 2014: The hidden impact of nursing research

Utilization of Social science research knowledge in Canada

Beyond REF 2014: The impact of impact assessment on the future of information research

MALLET: A Machine Learning for Language Toolkit

Multidimensional assessment of scholarly research impact

Exploring research impact in the assessment of leadership governance and management research

Progressing research impact assessment: A 'contributions' approach

Assessing the societal impact of research: The relational engagement approach

Usable, in-use, and useful research: A 3u framework for demonstrating practice impact

Duration, frequency, and diversity of knowledge contribution: differential effects of job characteristics

Knowledge Management Capability: A Resource-Based Comparison of Public and Private Organizations

Turnover in information systems development projects-managing forgetting

Topic models as a novel approach to identify themes in content analysis

The UK research excellence framework and the transformation of research production

Beyond academia-Interrogating research impact in the research excellence framework

Experiments to Stimulate Civic Behaviour

Uoa02-05: Hormone Replacement Therapy and Cancer Risk: The Million Women Study

REF impact case study: schopenhauer and nietzsche on the value of existence

Fairy Magic: Enabling Cinematic Experiences on Mobile Devices in Real-Time

Toward a methodological scheme for capturing societal effects of participatory sustainability research

How Scientific research reacts to international public health emergencies: A global analysis of response patterns