key: cord-0829706-pnf3ugbj authors: Pu, Shi; Xu, Hao title: Examining Changing Assessment Practices in Online Teaching: A Multiple-Case Study of EFL School Teachers in China date: 2021-07-30 journal: Asia-Pacific Edu Res DOI: 10.1007/s40299-021-00605-6 sha: e0e4102e1229b0765d277d1f1d4bcd422ab9c2e4 doc_id: 829706 cord_uid: pnf3ugbj This paper reports on the results from a multiple-case study of how EFL school teachers in China coped with online assessment for 10 weeks caused by the Covid-19 pandemic. We looked into the online assessment practices of seven teachers from seven regions of China, through individual interview, teachers’ journals, instructional designs for online teaching, and online classroom observation. The findings reveal that the participants went through a transition from neglecting assessment to overdoing assessment in their online teaching process. Whilst they used a range of online assessment methods, they did not systematically incorporate online assessment into their online curricula. Analysis shows that this issue is most likely to result from teachers’ lack of understanding of the nature of online assessment as an intrinsic component of curriculum and a major means of enhancing students’ learning. Implications will be discussed. Assessment refers to the activity of measuring learners' 'achievement and progress in a learning process' (Gikandi et al., 2011 (Gikandi et al., , p. 2336 . It is a core component of all kinds of teaching and learning system in formal education (Benson, 2003) . In an online learning environment, few would contest that effective assessment is essential to teaching and learning (Gaytan & McEwen, 2007) . Online learning is a form of distance learning enabled by the internet and/or web-based technologies without requiring teachers and learners to meet at the same place (Dabbagh & Bannan-Ritland, 2005; Gikandi et al., 2011) . The increased physical distance makes it difficult for online instructors to monitor students' learning progress and to provide timely response (Cheng et al., 2013) . This requires online assessment to be redesigned for the medium of online instruction, and the primary purpose is to help students to achieve learning goals set in an online curriculum (Akimov et al., 2018; Robles & Braathen, 2002) . The design and implementation of assessment has been a major challenge for instructors of online courses (Akimov & Malin, 2020; Benson, 2003; Gaytan & McEwen, 2007) , and the Covid-19 pandemic has magnified this issue. Since the pandemic broke out in spring 2020, face-to-face teaching has been suspended in more than 144 countries, affecting about 1.2 billion students around the world (UNESCO, 2020) . Most of those students rely on online courses to continue with learning, and many teachers have to cope with the abrupt shift from offline to online teaching (MacIntyre et al., 2020; Moser et al., 2020) . In this paper, we report on the findings of a multiple-case study that investigated how Chinese EFL school teachers implemented online assessment during the Covid-19 crisis. In the field of teaching English as a second/foreign language (ESL/EFL), online instruction has become popular long before the pandemic, and there is an increasing demand for enhancing the quality of online assessment. Research on online assessment in ESL/EFL courses mainly focus on methods of assessment and students' reactions (e.g. Ebadi & Rahimi, 2019) , and little research has examined the nature of assessment from the perspective of curriculum design. In our study, by analysing the challenges that EFL teachers are faced with during the pandemic, we aim to identify the inherent difficulties in their online assessment literacy, particularly regarding ways in which they integrate online assessment into online curriculum. Generally speaking, the major purpose of assessment, in both online and offline learning environments, is to collect information about student learning progress and to enhance teaching and learning (Speck, 2002; Stiggins & Chappuis, 2005) . Based on existing literature, online assessment practice can be conceptualised as having three major components: objects, methods, and utilisation. In this section, we define the three components based on a brief review of literature. Objects of online assessment refer to the domains of learning that is being assessed in an online assessment event, summative or formative. According to existing research, online assessment usually focuses on three domains of learning: cognitive, behavioural, and affective (Wei et al., 2021) . The cognitive domain of learning involves three aspects: the amount of knowledge learned and the level of knowledge comprehension (Benson, 2003; Challis, 2005) ; the application of higher-order thinking skills used for processing taught knowledge, as well as social learning skills such as collaborative learning (Benson, 2003; Cheng et al., 2013) ; the level of cognitive competence of a certain kind (Benson, 2003) . The behavioural domain of learning refers to visible learning behaviour demonstrated in specific learning events and throughout the learning process. For instance, in MOOCs, learning behaviour includes the frequency of watching video lectures, the number of posts in online discussion forums, and the number of assignments completed and submitted (Wei et al., 2021) . Assessment of learning behaviour can reveal students' level of engagement in learning, their strategies for coping with particular tasks, and their behavioural patterns that may explain certain learning outcomes (de Lima & Zorrilla, 2017; Lee, 2019) . The affective domain refers to students' perceptions and attitudes related to learning, including course satisfaction, perceptions of learning experiences, and perceptions of how certain learning experiences affect their confidence, motivation, self-efficacy, sense of progress, and so on (Wei et al., 2021) . Most empirical research on online assessment focuses on assessing the cognitive domain of learning, discussing how certain assessment methods can accurately reflect and effectively enhance cognitive learning outcomes (e.g. Chang et al., 2011 Chang et al., , 2014 Yu, 2014; Ng 2020) . Whilst the cognitive domain of learning is found to correlate with learning in behavioural and affective domains (e.g. Jung & Lee, 2018) , it is not known how the three domains of learning are dealt with in frontline teachers' online assessment practice. Research on online assessment is mostly concerned with the procedure and effects of particular assessment methods. Methods of online assessment involve three elements: evidence selected to reflect learning in certain domains, tools for collecting evidence, and rubrics designed for assessments. All kinds of assessment replies upon certain observable evidence as the basis for instructors to make inference about students' learning (Oosterhof et al., 2008) . Most online assessment uses students' written performance as evidence, as demonstrated through online exams, in virtual discussion boards, in portfolios, and by written essays (e.g. Cheng et al., 2013; Gikandi et al., 2011; Robles & Braathen, 2002; Rovai, 2000) . Some online assessment also examines students' oral performance, through oral examinations and video presentations (e.g. Akimov & Malin, 2020; Fluck, 2019) . A range of tools are used for collecting and displaying evidence for online assessment, including web-based testing software, internet-based audio and video platforms, electronic portfolio software, and other digital tools like Google docs and other word-processing software (e.g. Ebadi & Rahimi, 2019; Fluck, 2019; Hricko & Howell, 2006; Moffitt et al., 2020) . Based on evidence collected, instructors also need to follow certain rubrics to conduct assessment. Researchers are highly aware that clear rubrics are essential for assessment activities (Gaytan & WcEwen, 2007; Wei et al., 2021) , and some studies have specified generic criteria for assessing certain types of student performance (e.g. Burch, 1999; Chang et al., 2011) . Little research has investigated the extent to which assessment methods and rubrics are intentionally designed in relation to specific learning goals in frontline teachers' online assessment practice. Utilisation of online assessment refers to how assessment results are used to facilitate students' learning, including how feedback is provided and how teaching plans are adjusted to respond to students' learning issues. Feedback is widely acknowledged as a core component of online formative assessment and an indispensable means of uniting assessment, teaching, and learning (Gaytan & McEwen, 2007; Gikandi et al., 2011) . Insufficient research, however, is found to investigate how online assessment results are used by frontline teachers, particularly the extent to which they are used as an integrated part of an online curriculum. Overall, researchers are well aware that the alignment between assessment, teaching, and learning is crucial if an online course is to effectively enhance students' learning (Gikandi et al., 2011) , but few studies have investigated frontline teachers' assessment practice in online settings regarding such alignment. In this paper, we report on a study that investigated into how EFL school teachers in China conducted online assessment during the Covid-19 pandemic. By analysing the objects and methods of their online assessment, as well as ways in which they made use of assessment results in their teaching, we hope to discover the extent to which their online assessment was integrated with learning goals. Our research question is: How did EFL school teachers implement online assessment during the Covid-19 pandemic? To address the research question, we adopted a case study design (Yin, 2009) to track the changes of participant teachers' online assessment practices in relation to the development of online curriculum during the Covid-19 pandemic. Specifically, participants' assessment practices were examined and understood as changing social practice situated in the individual online curriculum that each of them designed and implemented. Therefore, the dataset should first of all encompass data which depicted participants' employment of assessment techniques and implementation of assessment-orientation activities (Xu, 2017) . Data that revealed the context of assessment practices and that pertained to the dynamic relationship between assessment and curriculum development should also be collected and analysed. Normal teaching in the spring semester, which began in late February 2020, was disrupted by the outbreak of the Covid-19 pandemic. The epidemic initially broke out in late January in China, and by the middle of February, i.e. towards the end of winter vacation. Teaching was all moved online for secondary and elementary schools until mid-May to early June, depending on public health situations in different regions. Online teaching thus lasted for at least 10 weeks, from late February to early May. Within the duration of online teaching, we invited seven teachers from seven regions of China to participate in the current study to which they gave their formal consent as research participants. None of them was our acquaintances, but as we had common friends and colleagues, they were happy to participate and willing to share their teaching experience. All participants were senior high school EFL teachers with 15-20 years of teaching experience (Table 1 ). All names are pseudonyms. Data were collected via individual interview, teachers' journals, instructional designs for online teaching, online classroom observation, and other protocols. We conducted interviews through online conferencing with each of the participants in the first week of March, April, May, and June, which totalled up to 28 interviews ranging from 31 min to 1 h 15 min. Although all the interviews focused on teachers' conceptions and practices of online teaching, the questions that we asked each participant varied as we tried to contextualise them, e.g. referring to situations mentioned in a previous interview. Some examples of interview questions are shown below: What did you attach the greatest importance to when you were making online classroom assessment? What assessment methods did you use that were the same or similar to those you had used before the pandemic? What methods were new then? What difficulty, if any, did you experience as you conducted online classroom assessment? We also encouraged teachers to share their teaching journals, instructional designs, teaching manuals, students' assignments, and teachers' course plans related to online teaching during the pandemic. When teachers began using online conferencing services in March to deliver lessons and to interact with their students, we observed some of their online classes as well. Table 2 below shows a summary of the collected data. Data were collected in Chinese and then translated into English when presented in this paper. Interview data were transcribed before being included in the dataset to be further analysed. We (the two authors) first analysed the data individually and then compared and discussed our analyses to reach consensus in data interpretation. In our first round of data analysis, we identified events and statements that were relevant to teachers' conceptions and/or practices of classroom assessment in online teaching. Then in the second round of analysis, we coded the selected episodes and excerpts according to their specific themes or contents. For instance, teachers' conceptions of assessment were coded as ''a regular check of students' learning'', ''activity-based testing'', and so on; their assessment practices were reflected in such codes as ''implementing quizzes on language points'' and ''organising online group work''. The third round of analysis focused on sorting out stages of online teaching that were characterised by different kinds of assessment practices. This was followed by the last round of analysis which aimed to conceptualise the complex dynamics of assessment in relation to the development of online curriculum. Finally, we came up with a set of themes that corresponded with the three important aspects of classroom assessment: domain or object of assessment, methods or tools of assessment, and utilisation of assessment results. Interview data were primarily analysed by the second author, the tentative results of which were checked by the first author and further discussed between us for inter-coder agreement. As to data obtained from other sources, we browsed them after analysis of the interview data, and used them as a supplement to triangulate with findings from interview data analysis, or to raise issues that needed reconsideration when discrepancies emerged between these data and interview data. This section delineates the changes in teachers' assessment practices when their teaching was forced to move online during the pandemic. Specifically, this section first shows, as a background, how teachers responded to the needs of online teaching in the very initial stage, and then presents data that reveals the object or domain of teachers' online assessment, methods of such assessment, and how assessment results were used in different stages of online teaching. When teachers were asked to prepare for teaching from home for at least half a semester, they were most concerned about how lessons should be delivered online. All teacher participants were faced with the challenge of lesson delivery as they had had little prior experience of teaching online. They were not sure how to manage classroom teaching or curriculum design in an online context. The ''real teaching'' they conceived was the design and implementation of an online curriculum. With regard to Teachers were creative in their efforts to lecture on language points in online teaching: they enhanced the quality of their lesson videos by multiple recordings, and they provided students with more resources to increase their language knowledge. However, the delivery of teaching content, language knowledge in particular, seemed to be the sole focus of their online curriculum design and implementation. For instance, in Yu's ''Course Plan during Covid-19'', which was submitted to the school for review and approval, she did not write anything about how to assess students' learning progress. In February and March, we collected 17 instructional designs, and none of them explicitly incorporated classroom assessment activities. Although teachers did not seem to explicitly incorporate assessment in the curriculum, they did conduct assessment, without much awareness though, as they tried to ensure that their students were investing required efforts in and after online class. Specifically, they attempted to detect behaviours that could be taken as evidence for students' engagement in learning, as Yu said: Today [in class] I suddenly felt the urge to check if all of my students were listening to me attentively. I made up a few questions that required but such short responses as ''yes'' or ''no'' and randomly asked a few of them to answer. I was astonished that none of the first three students I asked gave me any response. They did not even turn on their microphone, which meant that they were not with me in class … I will ''raid'' the online classroom more often in such a way, and see if they are still there. I have to ascertain they are online. (Yu, journal on March 19) Another participant also mentioned such attempts to assess learners' behaviours: I have difficulty in making sure that the students do their homework … I ask them each to take a photo of their written assignment and post it in a Wechat group … I don't really check them [written assignments] for the quality, I check the number of posts. The first thing I need to know is that they do finish the work. (Ling, interview in March) These extracts show that in the early stage of online teaching, teachers tried to collect evidence of students' learning behaviours both in and after online class. Normalising Assessment: ''When Class is Regularly Online, Assessment Should be as Normal as Regular'' Whilst teachers were engaged with making teaching contents available online as well as detecting certain behaviours as evidence of their students' engagement in learning, they gradually realised that they were poorly informed of students' learning progress. They began to perceive this problem towards the end of March when online teaching had been on for 5-6 weeks: Today it occurred to me that we haven't had any test so far … Normally, we would have a test every two or three weeks to see who was improving and who was falling behind. As can be seen, teachers realised the problem of lack of assessment in their online teaching from a variety of sources, such as teachers' own reflective comparison between now and past in terms of conventional assessment practice and parents' request for feedback. Obviously, Examining Changing Assessment Practices in Online Teaching students' progress was what the teachers believed that they ''should have known''. However, as the curriculum was hastily put online on a massive scale in China's mainland in spring, assessment seemed to have been excluded from key considerations for a well-structured curriculum. As they became aware of the problem, teachers began to design and incorporate real assessment in the online curriculum. I've started to hold a short quiz towards the end of each online class … I use a few PPT slides to present 8-10 fill-in-the-blank questions, each testing on a new vocabulary item taught in class. The students write their answers on a piece of paper, take a photo of it, and send it to me via Wechat … I can then see how well they have gained the language knowledge. (Fu, interview in April) I've asked students to record videos of oral presentations … I write a few questions based on the reading text. The students answer them without referring to the text … I check their video recordings the same way I checked their presentations in a normal class. I record an audio message telling each of them where they've got something wrong and the like … I believe when class is regularly online, assessment should be as normal as regular. (Yu, interview in April) It is obvious that teachers, in this stage of online teaching, have already begun thinking about normalising classroom assessment again as a regular part of teaching. In the previous stage when teaching was moved online in a hectic manner, assessment on students' learning was both technically hindered and pedagogically neglected. Fu made this point even clearer in the June interview as she looked back: What we can't easily implement is more easily noticed … When the whole stuff just began, we couldn't meet the students to lecture to them. Then we went all out to lecture online … Then we noticed another thing [i.e., assessment] that we can't easily implement. (Fu, interview in June) In summary, having survived a transitional period, teachers began conducting classroom assessment by means of classroom quiz, oral presentation, etc., as they had been so accustomed to, on their students' language knowledge and skills, which was closely related to the teaching objectives of a particular lesson. Overdoing Assessment: ''We Need to Conduct Assessment More Comprehensively'' From the fifth or sixth week during the spring semester, teachers began to incorporate various kinds of classroom assessment in their online teaching. For instance, in Zhou's online class on April 20, we observed that she integrated pre-recorded videos with online interaction. In each phase of the lesson, she first played a video which presented and elaborated language knowledge and then interacted with her students using an online conferencing service, asking students questions and guiding students to talk to each other. At the end of each phase, she conducted a quiz of multiple-choice questions or blank-filling that tested on related language points. She asked all students to do the quiz, but only asked some of them to present their answers, based on which she gave her feedback. This shows that in this stage, teachers have begun conducting assessment via a greater variety of means. Some teachers like Ling also incorporated a great number of activities of self-assessment. She described how she encouraged students to make self-assessment in her teaching journal: A self-assessment task will be given to students at the end of each class, including self-ratings and reflec-tions… Those who complete it on time will obtain 10 extra points. Those whose reflections are selected as good examples to be posted will obtain another 10 extra points. (Ling, journal on May 15) Teachers also encouraged or asked students to engage in peer-assessment. For example, Yu asked students to work in groups of three or four (via Wechat groups, a social media app function) to evaluate each other's written assignments and provide corrective feedback (classroom observation, May 8). As can be seen, the teachers conducted a huge amount of classroom assessment when they had realised its importance to the curriculum. Consequently, they seemed to begin considering connecting their daily assessment practices to evaluating the effectiveness of the curriculum: The nature of online teaching has changed into a matter of check, check, check … I'm checking home assignments and tests every day … We need to conduct assessment more comprehensively, not more repetitively. (Xin, interview in May) I believe that students have spent too much time doing tests. The school asks us teachers to include too many tests. Many of them are a waste of students' time … We should be more concerned about students' learning achievements. The many tests we administer aren't of sufficient help to inform us of what the students have become of, or how they are developing themselves. (Jun, interview in May) As can be seen, the teachers seemed to have ambivalent feelings towards assessment. On one hand, they regained the initiative to conduct classroom assessment in a ''normal'' way and, gradually, with a good variety of techniques; on the other, they began to notice the limitation of confining assessment to the teaching of individual lessons as well as that of ''repetitive'' assessment. They became increasingly and intuitively aware of the importance of conducting assessment in a more ''comprehensive'' way, i.e. conducting assessment to track and understand students' development at the curricular level. The findings reveal that, whilst the teacher participants increased their awareness of the importance of online assessment, online assessment was not organised systematically to facilitate teaching and learning. In other words, assessment was separated from the online curricula. In this section, we analyse three major issues underlying this problem. The teacher participants did not consciously conceptualise assessment as a means for enabling students to achieve certain learning goals. Initially, the teachers only focused on methods of lesson delivery without thinking of how to conduct assessment. It was only during the fifth and sixth weeks that they became aware of their lack of knowledge about students' learning progress. This indicates that they did not regard assessment as an integrated part of curriculum. For if they had, they would not have ''forgotten'' about assessment, as assessment was the only way for them to understand students' level and learning needs, and such understanding should determine how teaching should be designed to bridge the gap between students' current level and desired learning goals (Cheng et al., 2004) . The teacher participants did not seem to have a clear conception of what objects to focus on in their online assessment. For instance, checking students' completion of assignments stood out as a major form of online assessment, which was essentially an assessment of students' learning behaviour. Whilst such assessment can reflect students' engagement in learning (Wei et al., 2021) , it cannot be used to identify students' cognitive learning progress. During the later stages of online teaching, it was only when teachers became overwhelmed by the large amount of work required for checking assignments that they started to reflect upon what should determine the type and amount of assessment. Underlying that question was the root questions of what domains of learning should be assessed and for what purposes. Without a clear answer to those questions, it would be very difficult for teachers to conceive effective assessment methods. The teacher participants did not plan for assessment in order to provide systematic support for students' learning. During the whole semester, the teachers shifted from assessing learning behaviour, to measuring learning achievements and then, to contemplating the nature of assessment related to learning. Usually, from the perspective of curriculum planning, various forms of assessment should be involved, such as diagnostic, summative, and formative assessment, with each serving for a distinct purpose (Benson, 2003; Hargreaves, 2007; Oosterhof et al., 2008) . Whilst the teacher participants did implement various forms of assessment, they did not seem to know when to conduct what forms of assessment and for what purposes. In other words, online assessment was not systematically planned. Overall, while previous research shows that a vast majority of frontline teachers lack assessment literacy in offline teaching environment (e.g. Gu, 2014; Lam, 2015; Xu & Brown, 2016) , our study shows that teachers also lack online assessment literacy, i.e. their transition from neglecting assessment to overdoing assessment during the Covid-19 pandemic demonstrated that they did not have sufficient understanding of principles of effective online assessment. Whilst it can be partly explained by the abrupt shift to online teaching for which many teachers were not fully prepared, it is also likely to result from teachers' insufficient understanding of the nature of learning, of assessment, and of the intrinsic link between the two, in an integrated online curriculum. It was the lack of such understanding that made it difficult for teachers to conceptualise and resolve assessment issues that they intuitively noticed in their online teaching. This study investigated Chinese EFL school teachers' online assessment practices in relation to the development of online curriculum during the Covid-19 pandemic, and the findings reveal that online assessment was not systematically incorporated into online courses as an inherent part of curriculum. Analysis shows that the challenges that teachers faced with are most likely to result from their lack of understanding of the nature of online assessment as an intrinsic component of curriculum and a major means of enhancing students' learning. Pedagogically, this conclusion implies that teachers need to better understand how assessment, teaching, and learning should be integrated in an online curriculum. It also implies that more research is needed for understanding what teachers think and do in their online assessment practice. Meanwhile, it has been recognised that assessment practice is situated in particular social and institutional contexts (Scarino, 2013) . Our study has limitations in this regard. Methodologically, our conclusion may have restricted generalizability, as our data were collected only from selected urban areas in China. Future research can conduct similar investigations in less well-off areas by taking into consideration the geographical and economic disparity across China. Besides, it has been recognised that EFL classes in secondary schools in China are mostly oriented towards high-stakes tests, and curriculum only plays a marginal role in teaching and assessment practice (Qi, 2007) . It is beyond the scope of this paper to analyse the complications involved and their implications for online teaching, and we suggest that future research needs to analyse such social and cultural factors in order to understand issues in teachers' online assessment practice and to identify possibilities for resolving them. When old becomes new: A case study of oral examination as an online assessment tool. Assessment and Evaluation in Higher Education Determinants of student success in finance courses Assessing participant learning in online environments. New Directions for Adult and Continuing Education Inside the portfolio experience: The student's perspective. English Education Committing to quality learning through adaptive online assessment. Assessment and Evaluation in Higher Education Reliability and validity of Web-based portfolio peer assessment: A case study for a senior high school's students taking computer course Is reflection performance correlated to the learning effect in a Web-based portfolio assessment environment for middle school students? The Asia-Pacific Education Researcher ESL/EFL instructors' classroom assessment practices: Purposes, methods, and procedures Reconsidering assessment in online/hybrid courses: Knowing versus learning Online learning: Concepts, strategies and application Social networks and the building of learning communities: An experimental study of a social MOOC Mediating EFL learners' academic writing skills in online dynamic assessment using Google Docs An international review of eExam technologies and impact Effective online instructional and assessment strategies Online formative assessment in higher education: A review of the literature The unbearable lightness of the curriculum: What drives the assessment practices of a teacher of English as a foreign language in a Chinese secondary school? Assessment in Education: Principles The validity of collaborative assessment for learning Online assessment and measurement: Foundations and challenges Learning engagement and persistence in massive open online courses (MOOCs). Computers and Education Language assessment training in Hong Kong: Implications for language assessment literacy Using self-organizing map and clustering to investigate problem-solving patterns in the massive open online course: An exploratory study Language teachers' coping strategies during the Covid-19 conversion to online teaching: Correlations with stress, wellbeing and negative emotions Accessibility and emotionality of online assessment feedback: Using emoticons to enhance student perceptions of marker competence and warmth Remote teaching during Covid-19: Implications from anational survey of language educators Active learning in undergraduate mathematics tutorials via cooperative problembased learning and peer assessment with interactive online whiteboards. The Asia-Pacific Education Researcher Assessing learners online Is testing an efficient agent for pedagogical change? Examining the intended washback of the writing task in a highstakes English test in China Online assessment techniques Online and traditional assessments: What is the difference? Language assessment literacy as self-awareness: Understanding the role of interpretation in assessment and in teacher learning Learning-teaching-assessment paradigms and the online classroom Using student-involved classroom assessment to close achievement gaps. Theory into Practice Education: From disruption to recovery Assessment of cognitive, behavioral, and affective learning outcomes in massive open online courses: A systematic literature review Exploring novice EFL teachers' classroom assessment literacy development: A three-year longitudinal study. The Asia-Pacific Education Researcher Teacher assessment literacy in practice: A reconceptualization. Teaching and Teacher Education Case study research: Design and methods Are there any joint effects of online student question generation and cooperative learning? The Asia-Pacific Education Researcher Acknowledgments We sincerely thank the three anonymous reviewers for their constructive feedback and suggestions during the review process.