key: cord-0066759-xx9vflxw
authors: Liu, Jun; Ma, Yue; Sun, Xue; Zhu, Ziqi; Xu, Yanhua
title: A Systematic Review of Higher-Order Thinking by Visualizing its Structure Through HistCite and CiteSpace Software
date: 2021-08-18
journal: Asia-Pacific Edu Res
DOI: 10.1007/s40299-021-00614-5
sha: 81540d4ea655e585d864b50fa681dd53db759f1e
doc_id: 66759
cord_uid: xx9vflxw

With the continuous reform of the education curriculum, the focus of the curriculum is to provide personal and social needs-related knowledge. In the teaching process for this type of knowledge, the cultivation of students' higher-order thinking has aroused widespread concern in the education field. In this paper, CiteSpace and HistCite bibliometric methods are utilized to analyse the higher-order thinking literature in the core collection of Web of Science. Through an analysis of the number of articles, the research topic words and the countries from which the papers were sent, the overall characteristics of international higher-order thinking research are summarized. The primary finding of this research is that the field of higher-order thinking is on the rise. Secondary findings are the focus of higher-order thinking research, which can be divided into four aspects: higher-order thinking ability, instruction of higher-order thinking, the curriculum and sections of higher-order thinking, and higher-order thinking learning. In short, this research helps researchers understand the development of higher-order thinking, provides a theoretical foundation and an entry point for research directions, and not only provides meaningful and valuable references for follow-up related research but also promotes the development of higher-order thinking practice.

In the early part of the 20th century, education focused on the acquisition of basic literacy skills: reading, writing, and calculating. Complex understanding was thought to occur only by the accumulation of basic, prerequisite learning. Therefore, most schools did not teach students to think and read critically or to solve complex problems. Textbooks were filled with facts that students were expected to memorize, and most tests assessed students' ability to remember these facts. The main role of teachers was perceived as transmitting information to students (Bransford et al., 2000) . Learning objectives were sequenced to progress from simple, lower-order cognitive tasks to more complex ones. With the unprecedented growth of information and knowledge, the meaning of ''knowing'' is changing from the ability to remember and repeat information to the ability to discover and use information effectively. People not only learn simple and low-level knowledge but also develop a non-algorithmic, complex thinking mode, which is higher-order thinking. Such thinking involves uncertainty, the application of multiple criteria, reflection, and self-regulation (Resnick, 1987) . Creativity and innovation are important keys to success in any field in this era of rapid development. The generation of ideas involves activities for higher-order thinking skills that require high-level creative thinking and action. However, it is difficult to generate good ideas. Idea generation occurs in the human brain through cognitive, metacognitive, chemical and biological processes. Hence, complex thinking skills such as problem solving, creating, analysing, and evaluating are needed to process collected information to generate an idea. Furthermore, higher-order thinking challenges us to interpret, analyse or manipulate information. With higher-order thinking, an individual can utilize new information or prior knowledge and manipulate information to obtain a reasonable response to new situations. Consequently, creative ideas can only be generated through higher-order thinking rather than through low-level thinking through the application of knowledge learned in daily life. Therefore, in the changing world, it is becoming increasingly important to develop people's higher-order thinking. With the continuous development of education, the goal of cultivating learners' higher-order thinking has been emphasized in most educational practices. The focus of teaching has gradually shifted from traditional textbookbased rote learning to inquiry learning in the real world to better develop students' higher-order thinking.

Currently, there are many studies on the theory and practice of higher-order thinking. A large number of studies apply the development of higher-order thinking to the classroom and assess the development of students' higher-order thinking through testing. For example, Zohar and Peled (2008) tested the learning status of low achievers in high-level teaching and learning processes through four different experiments. Miri et al., (2007) applied the longitudinal case study method to explore the relationship between purposefully promoting the teaching of higherorder thinking skills and improving students' critical thinking ability. Mcloughlin and Mynard (2009) conducted research on online forums to promote higher-order thinking tools. Nonetheless, in the context of teaching higher-order thinking, the classic conceptual distinction made in the literature between pedagogical content knowledge and general pedagogical knowledge is fuzzy and unclear. It is difficult for teachers to combine their knowledge of teaching about thinking with the commonly applied concepts in the literature. This issue is related to scholars' debate about whether thinking strategies are general or specific. According to the method of inculcation, the teaching of thinking is combined with the teaching of specific content; thinking skills have general elements, while other elements are related to specific content. Thus, a systematic review of higher-order thinking education is important and appropriate to examine current applications and trends in this field.

This study applies an innovative approach to conduct the literature review through a systematic review with the support of HistCite and CiteSpace software. The research tools used in this study are HistCite and CiteSpace software because they are able to link and visualize the citation history and citation structure of articles in graphical form (Chaomei et al., 2012; Chen, 2006; Garfield, 2009; Lucio-Arias & Leydesdorff, 2008) . The combined application and support of this software can increase time efficiency and supplement the judgement and understanding demands involved in systematic review analysis. HistCite is free software. It can generate a chronological map of bibliography collections generated by topic, author, institution or source journal searches of the ISI science network. Web of Science (WoS) export files are created in which all cited references for each source document are captured. This provides an evidence map from listed journals that can guide and reinforce researchers' decisions and interpretations. CiteSpace software divides the network into clusters by identifying rapidly growing subject areas and discovering citation hotspots in the field of publications. CiteSpace software then simplifies WoS data for understanding and interpreting the chronological structure and connection with past research patterns and automatically marks clusters using citation terms, geospatial collaboration patterns and unique international collaboration fields. This free software with Java applications supports a unique type of co-citation network analysis-progressive network analysis-based on a time slicing strategy. A series of individual network snapshots defined on continuous time slice nodes are synthesized. These nodes play a key role in the evolution of the network and are turning points of knowledge. A systematic review is an orderly way to review and summarize research. The Cochrane Collaboration noted that systematic review provides a high-level overview of preliminary research on a specific research problem. This study attempts to identify, select, synthesize and evaluate all relevant high-quality research evidence to answer this question. Tsai (2013) noted that the purpose of the assessment of the education system is to obtain a clearer understanding of the recent situation. Furthermore, systematic analysis procedures are important in identifying associations between instructional design and the theoretical characteristics and best practices of research study. However, systematic evaluation in education has received little attention. This is due to the labour-intensive nature of scientific education methods and the need for researchers to judge and understand the potential value of research quality and results. Therefore, this study aims to utilize established procedures, identify connected structures, and address research gaps or incompleteness in this area for future studies. It is appropriate and worthwhile to use HistCite and CiteSpace software to systematically review research on higher-order thinking.

This study presents a systematic review of the literature related to higher-order thinking. In this way, it reveals the evolutionary path, research focus and research frontier of higher-order thinking research and provides a reference for research in related fields of higher-order thinking. Scholars reveal the development of practice from different theoretical perspectives and systematically pay attention to these issues when returning to these documents: What is the current research status of higher-order thinking? What is its evolution process? What research highlights does it include? When will higher-order thinking fully develop? In view of this, this paper employs a bibliometric method to analyse the development trends of higher-order thinking publications for the period of 1984-2020. Through an indepth discussion of the number of papers and references about higher-order thinking, this paper can provide the basis for research on higher-order thinking and education.

This study focuses on previous research studies on higherorder thinking, and the main source of journal exploration is the WoS database. In the WoS database, ''higher-order thinking'' or ''higher order thinking'' was used as the subject search term to retrieve all articles about higherorder thinking from 1984 to 2020. With the support of HistCite and CiteSpace software, the research process was divided into two main parts. First, HistCite analysis was conducted to identify the universe of articles and their citation links to central studies. Second, higher-order thinking studies were identified and isolated through HistCite and CiteSpace analysis.

In the Wos database, 1349 articles from 1984 to 2020 were retrieved by using ''higher-order thinking'' or ''higher order thinking'' as the subject search term. After confirming the publication year from 1984 to 2020 and deleting the literature with uncertain publication years, a total of 1320 articles were selected to meet the research requirements. The structure of the studies and relationships among the 1,320 papers identified in the WoS can be analysed by HistCite software (Fig. 1) . HistCite software was applied to the journal lists to generate chronological historiographs (i.e., a time-based network diagram) based on the relationship of the cited works, which is the number of citations of a paper within the collection. This was followed by a HistCite analysis of higher-order thinking to identify the citation patterns and select related studies for further indepth document analysis. General information about the results is provided on the first line of the first page ( Fig. 1 ) of the HistCite file that includes higher-order thinking. The abbreviation LCS stands for Local Citation Score and is the number of times a paper is cited by other papers in the local collection; GCS stands for Global Citation Score and shows the Citation frequency based on the total count in the Web of Science; LCR stands for Local Cited References and shows the number of references citing local papers; and CR stands for the number of cited references and shows the number of all citing references (for the use of the HistCite methods for algorithmic historiography, see (Garfield, 2004; Garfield et al., 2003) ). Due to the subscription history of the library and the coverage of WoS, the collection spans from 1984 to 2020. The timeline for the growth of higher-order thinking is exhibited by a HistCite presentation of the ranked citation index of 1,320 research articles within 706 journals by 3,042 authors and with 36,868 cited references. Figure 2 shows the HistCite graphmaker display of the historiograph of higher-order thinking that was generated based on the LCS. Figure 2 shows the complete citation relationship of the top 30 most valuable articles from 1984 to 2020, and the time span of these 30 articles is from 1990 to 2017. This includes the relationship of cited works with circle diameters proportional to the LCS as well as an arrow exhibiting the citation direction. The citation direction arrows show that 20 articles were either cited by others or cited other works within the WoS collection and illustrate the relationship of citations between papers. The larger the diameter of the circle, the more often this literature has been cited and the more important this article is in the field of higher order thinking. The number next to the circle indicates the serial number of the article among the 1320 articles. Figure 2 shows that the larger diameter of the circles corresponding to the 91st, 143rd, and 8th papers means that these three papers have been cited the most. The 91st is Higher Order Thinking Skills and Low-Achieving Students: Are They Mutually Exclusive? published by Anat Zohar et al. in 2003 . The 143rd is Purposely Teaching for the Promotion of Higher-order Thinking Skills: A Case of Critical Thinking published by Barak Miri et al. in 2007. The 8th is higher-order thinking in teaching social studies: a rationale for the assessment of classroom thoughtfulness published by Fred M. Newmann in 1990. Of these, the arrow pointing to the 91st paper has the highest number of citations, representing the fact that this paper has been cited most often by other papers in the same field, indicating that this paper has significant scholarly influence in the field of higher-order thinking.

The identified higher-order thinking database was submitted to CiteSpace analysis to report the cluster terms and search for other important articles through the cited references or bibliographic collections. The purpose of this study was to search for important higher-order thinking articles that are frequently cited within the community. The CiteSpace analysis data originated from 1320 articles published between 1984 and 2020. Figure 3 presents a timeline visualization of the clusters with automatically created labels (only highly cited papers in major clusters are shown). The cluster labels shown are useful for understanding the research scope or direction of higherorder thinking because these terms are frequently applied within the community. Furthermore, these terms are very useful for conducting research related to higher-order thinking development, the results, discussion, conclusions, and suggestions. Next, a narrative was generated for the analysis of the largest cluster. The automatically chosen cluster labels of the 14 largest clusters along with their size, identity number and silhouette value in brackets are presented. The largest cluster of higher-order thinking (#0) was educational robotics, which was active from 1992 to 2020. The second largest cluster (#1) was teachers' belief, which was active from 1993 to 2020. The third largest cluster was socio-scientific issues (#2). The fourth to tenth largest clusters were project management learning (#3), deep learning (#4), executive functioning (#5), digital learning game (#6), web-based hypermedia resource (#7), social competence (#8) and processing deficit (#9).

As shown in Fig. 4 , the research literature on higher-order thinking in the WoS core database is counted by time distribution. From the perspective of the total amount and increment of documents, international research on higherorder thinking presents two trends. First, the number of research papers in the field of higher-order thinking remained stable from 1984 to 2007. With no more than 20 papers published each year, international research on higher-order thinking is in the exploratory stage. The number of papers published in 2008 was 21, which marks the turning point in higher-order thinking research. After 2008, the number of higher-order thinking research papers increased year by year. Since 2014, the number of papers has increased sharply, reaching a peak of 180 in 2018. Research on higher-order thinking has entered the outbreak period, showing a vigorous development trend and the new vitality of international higher-order thinking research. Less research on higher-order thinking was published in 2019 than in 2018. It is presumed that the reason for its decline may be influenced by COVID-19. Some experiments and teaching practices in higher-order thinking research do not have a corresponding environment. One possible reason for the decline in the number of papers in 2020 is due to the limited speed of updates to the WoS Fig. 3 Cluster labels and terms generated from 1984 to 2020 A Systematic Review of Higher-Order Thinking by Visualizing its Structure Through HistCite and… database, and some papers published in 2020 are still not included in the database. Another may be due to the bottleneck in higher-order thinking after research has gone through theoretical exploration, coupled with the impact of the epidemic and the inability to conduct experiments. Therefore, this paper studies the number of published papers combined with high-order thinking, draws the development trend of different periods in the form of arrows, and predicts the trend of the number of published papers in 2020 with the dotted line.

An analysis of higher-order thinking through CiteSpace software was conducted to study the keyword co-word network knowledge map, as shown in Fig. 5 . As shown in the figure, ''critical thinking'' was the most influential core of the network. Terms such as skills, instruction, achievement and metacognition have great influence, reflecting hot subjects in higher-order thinking research. The top 30 highfrequency keywords according to the summary of the software analysis results are shown. These keywords represent important research content in the current study of higher-order thinking and reflect the research focus in the field to some extent. Among these keywords, the frequency of ''critical thinking'' is 55, so ''critical thinking'' is the core of the research field. In addition, skills, instruction and achievement appear relatively frequently, which indicates that they are hotspots of higher-order thinking research. Overall, combined with the high-frequency keywords of higher-order thinking research and the content of the classical literature, hot topics in higher-order thinking research in recent years include higher-order thinking ability, the instruction of higher-order thinking, the curriculum of higher-order thinking, and higher-order thinking learning.

As shown in Table 1 , the first aspect concerns research on higher-order thinking ability. High-frequency keywords included in the research hotspots are critical thinking (55), skill (54), higher-order thinking skill (29), metacognition (26), higher-order thinking skill (17), perception (14), literacy (14) and problem solving (13). For instance, Hwang et al. (2017) consider higher-order thinking skills to include problem solving, critical thinking, teamwork, communication, and creative thinking. Apino and Retnawati (2017) consider skills that involve students analysing, synthesising and evaluating and creating. In addition, critical, systematic and creative thinking (Miri et al., 2007) and metacognitive processes (Hmelo & Ferrari, 1997 ) are all part of higher-order thinking.

The second aspect involves research on instruction in higher-order thinking. Higher-order thinking requires good teaching practices and exercise. High-frequency keywords contained in the topic include instruction (31), assessment (31), performance (30), achievement (30), design (28), strategy (22), technology (21), blooms taxonomy (21), model (20), impact (18), pedagogy (17), feedback (16), argumentation (16), and environment (15). For example, Zohar and Dori (2003) , Tony (2008) and Mahoney and Harris-Reeves (2019) argued that higher-order thinking stems from Bloom's categories, in which analysis, synthesis and evaluation are higher-order thinking. For another example, Hwang et al. (2019) thought that the development of students' higher-order thinking skills can be facilitated through innovative flipped learning strategies and support systems. In addition, approaches such as game design thinking-based instruction (Mohamad et al., 2019) and peer assessment (Hadzhikoleva et al., 2019) are conducive to developing learners' higher-order thinking in teaching and Fig. 4 Annual distribution of the number of higher-order thinking research documents learning. As another example, Chiong and Lim (2020) explored the important role of family in the development of higher-order thinking skills under the influence of social policies and found that family factors have a significant role in influencing the development of learners' higherorder thinking.

The third aspect involves research on the curriculum and sections of higher-order thinking. High-frequency keywords contained in this research hotspot include science (47), curriculum (21), higher education (20), mathematics (16), children (15), science education (14) and elementary school (13). For instance, Nicely (1985) explored higherorder thinking as it is addressed in mathematics materials in primary and secondary schools. Miri et al. (2007) discussed whether students' critical thinking can be enhanced through purposeful teaching aimed at improving higherorder thinking skills within the framework of science education. FitzPatrick and Schulz (2015) fand that the importance of higher-order thinking is widely recognized in science education, where higher-order thinking is required.

The fourth aspect involves research on higher-order thinking learning. High-frequency keywords contained in this research hotspot include attitude (14) and active learning (14). For example, Lee and Choi's (2017) study confirmed that learners' attitudes towards technology use are related to higher-order thinking. Wu et al. (2019) found that learners' learning styles and motivation are related to the development of higher-order thinking; Saud et al. (2017) believed active learning can develop students' higher-order thinking skills. Figure 6 shows the hot research countries/regions generated by CiteSpace software. This figure sets ''topnperslice'' to 50; that is, 50 countries/regions with the highest reference frequency are selected every year to obtain a hot research country/region chart composed of 84 nodes and 138 connections. The statistical table of the top 10 countries/regions is selected, as shown in Table 2 . As seen in Fig. 6 and Table 2 , the hot spots of higher-order thinking research show a regional imbalance: researchers mainly concentrated in North America, represented by the USA and Canada. This is followed by Asia, represented by Malaysia, Taiwan Province of China, Israel, Turkey and Indonesia. In the third place is Oceania represented by Australia. Last, Europe is represented by England and Spain. There are 138 connections between countries/regions, which shows that there is considerable exchange and cooperation between countries/regions in research on higher-order thinking. It is necessary to further strengthen cross-regional academic exchanges and cooperation to promote the sustainable development of higher-order thinking and to promote the enrichment and development of innovative teaching modes to adapt to the information society. 

All findings in this study are derived from the results of the above analysis and from previous research. The primary finding of this research is that in terms of research progress, international higher-order thinking research has generally shown an upward trend, especially after 2014, when research related to higher-order thinking entered a period of rapid development. This may be because higher-order thinking has long been identified as a key predictor of success (Lee & Choi, 2017) . In the knowledge economy, higher-order thinking is even considered a fundamental competency requirement for the 21st century (Permatasari et al., 2018; Wu et al., 2019; Yao, 2012) . This requires educational reforms in many countries or regions (Fensham & Bellocchi, 2013; Singh et al., 2018) and education to focus more on the development and development of students' higher-order competencies (Kurniawati, 2019) . Therefore, influenced by educational policies and developments, an increasing number of scholars have launched research on higher-order thinking. Furthermore, the hotspot regions for research on higher-order thinking are mostly in countries such as the Americas, Asia and Europe.

The secondary finding of this study is that the hotspots of higher-order thinking research can be summarized in four areas: higher-order thinking ability, instruction of higher-order thinking, the curriculum and sections of higher-order thinking, and higher-order thinking learning. One explanation for this may be that higher-order thinking, as opposed to lower-order thinking (Abdullah et al., 2017) , requires a high level of thinking that encompasses multiple complex abilities (Apino & Retnawati, 2017; Hwang et al., 2017; Miri et al., 2007; Zohar, 2013) . Another possible explanation is that researchers believe that education is the main driving force promoting the development of learners' higher-order thinking, which requires specific methods, strategies, and media to be cultivated (Rubini et al., 2019) , as technology plays an important role in developing learners' higher-order thinking (Duan & Ieee, 2012; Lee & Choi, 2017) . Moreover, studies have found that the development of higher-order thinking is influenced by students' own factors (Lee & Choi, 2017; Wu et al., 2019) .

The significance of this study is manifested in the following two aspects. First, this study adopts new bibliometric methods and tools (HistCite and CiteSpace) and provides a more comprehensive tracking and visualization of research results in the field of higher-order thinking with more accurate data support and a more diversified analysis perspective, breaking through the limitations of traditional methods of literature content analysis. Second, this study uses the visualization method to quickly locate the key research results in the field of higher-order thinking, clarify the development history of higher-order thinking research, and more accurately identify the most active research frontiers and development trends, which provides a global knowledge map and literature basis for subsequent researchers to carry out in-depth research on higher-order thinking and has certain significance for the research and practice of higher-order thinking.

This study has some limitations. First, the literature data in the field of higher-order thinking are from WOS core journal papers, which are highly representative but still have some articles not included in time due to the limitation of the accession time, and not all data are available for the year 2020. In future studies, we will continue to track the literature of 2020 and expand the number and types of literature, such as adding conference papers, education policies, and other types of literature. Second, we cannot rule out the case that individual literature was not retrieved because it did not use a uniform formulation of higherorder thinking, but it does not affect the results of this study too much.

Author Contributions JL and YX designed the research; YM performed the literature research and data analysis; JL, YX, YM, and XS wrote the paper; ZZ reviewed and edited the paper.

Conflict of interest The authors declare no conflict of interest.

Mathematics teachers' level of knowledge and practice on the implementation of higher-order thinking skills (HOTS)

Developing instructional design to improve mathematical higher order thinking skills of students

How people learn: brain, mind, experience, and school

Emerging trends in regenerative medicine: A scientometric analysis in CiteSpace

CiteSpace II: Detecting and visualizing emerging trends and transient patterns in scientific literature

Seeing families as policy actors: Exploring higher-order thinking reforms in Singapore through low-income families' perspectives

Research about technology enhanced higher-order thinking

Higher order thinking in chemistry curriculum and its assessment

Do curriculum outcomes and assessment activities in science encourage higher order thinking?

Historiographic mapping of knowledge domains literature

From the science of science to scientometrics visualizing the history of science with HistCite software

Why do we need algorithmic historiography

Using peer assessment to enhance higher order thinking skills

The problem-based learning tutorial: Cultivating higher order thinking skills

A long-term experiment to investigate the relationships between high school students' perceptions of mobile learning and peer interaction and higher-order thinking tendencies

The era of flipped learning: Promoting active learning and higher order thinking with innovative flipped learning strategies and supporting systems

Improving students' higher order-thinking skills through problem-based learning in introduction to microeconomics course

What affects learner's higher-order thinking in technology-enhanced learning environments? The effects of learner factors

Main-path analysis and path-dependent transitions in HistCite TM -based historiograms

The effects of collaborative testing on higher order thinking: Do the bright get brighter? Active Learning in Higher Education

An analysis of higher order thinking in online discussions

Purposely teaching for the promotion of higher-order thinking skills: A case of critical thinking

Gamebased learning to teach higher order thinking in rural schools: Case studies in Sarawak Borneo

Higher-order thinking skills in mathematics textbooks

Identification of students difficulties in terms of the higher order thinking skills on the subject of work and energy

International conference on science and applied science

Education and learning to think

Enhancing critical thinking through the science learning on using interactive problem based module

A conceptual model of scenario based learning for developing higher order thinking skills in engineering education

A review of research on the use of higher order thinking skills to teach writing

Mathematics teachers' interpretation of higher-order thinking in Bloom's taxonomy

Integrating english for specific purposes courseware into task-based learning in a context of preparing for international trade fairs

The effects of learner factors on higher-order thinking in the smart classroom environment

Using modern educational technology promote learners' higher-order thinking skill

Challenges in wide scale implementation efforts to foster higher order thinking (HOT) in science education across a whole school system

Higher order thinking skills and lowachieving students: Are they mutually exclusive

The effects of explicit teaching of metastrategic knowledge on low-and high-achieving students