key: cord-0048433-xs0g5npq authors: Wyse, Adam E.; Stickney, Eric M.; Butz, David; Beckler, Amanda; Close, Catherine N. title: The Potential Impact of COVID‐19 on Student Learning and How Schools Can Respond date: 2020-07-23 journal: nan DOI: 10.1111/emip.12357 sha: e8d61215b67b3ac872442cd8412e9ba5a56fae2e doc_id: 48433 cord_uid: xs0g5npq There is no denying the impact that the coronravirus disease (COVID‐19) outbreak has had on many aspects of our lives. This article looks at the potential impact of COVID‐19 on student learning as schools abruptly morphed into virtual learning environments using data from several instructional, practice, and assessment solutions offered by Renaissance. First, three hypothetical learning scenarios are considered using normative data from Star assessments to explore the potential impact on reading and math test performace. Next, data on Focus Skills are used to highlight which grades may have missed the most foundational math and reading content if instruction was stopped or reduced. Last, data from two of Renaissance's practice tools are used to evaluate whether students were practicing key skills following school closures. The article concludes that academic decline will likely occur but may be tempered by the increased use of practice tools; effects may look different for math and reading; and may impact grades and schools differently. As such, schools may need to leverage decision‐making frameworks, such as the Multi‐tiered Systems of Support/Response‐to‐Intervention (MTSS/RTI) framework, more than ever to identify needs and target instruction where it matters most when school begins in fall 2020. The pandemic brought on by the novel coronavirus disease has had far reaching impacts throughout the world. In the United States, COVID-19 has resulted in the loss of close family members and friends, changes in the way that hospitals, businesses, churches, universities, and schools operate, pay cuts and job loss, and a host of other changes. The measurement and assessment industries are not immune to the impacts of COVID-19. The closure of schools, universities, and testing centers has meant the cancellation of K-12 summative state tests (Gewertz, 2020) , the inability for some candidates to qualify or sit for credentialing exams (Lambert, 2020) , changing of assessment formats (Richards, West, & Altavena, 2020) , suspension of college entrance exams, and universities deciding to remove college entrance exams from admissions requirements (Falk, 2020) . Not all changes are bad, however, as COVID-19 has sparked exploration of new approaches to assessment (Richards et al., 2020) and greater availability of some educational technology products as some companies have made their tools freely available (Schaffhauser, 2020) . In the context of K-12 education, teachers, parents, and students faced new challenges as schools were closed and education was transitioned to a home environment from March 2020 through the end of the school year. This transition brought many changes as teachers grappled with how to continue to instruct and communicate with students, provide students with activities to help them practice what they were learning, and in some cases still try to assess what students may have learned at home. Teachers' approaches to instruction, practice, and assessment were varied as was the level of participation in the instruction, practice, and assessment by different students (Hamilton et al., 2020) . Many teachers decided to greatly reduce the amount of instruction, practice, and assessment given to their students and the amount of time that students spent learning seems to be much less than it would have been if students had been physically present in school buildings (Hamilton et al., 2020) . In extreme cases, there were students that did not access any materials provided by their teachers (Goldstein, Popescu, & Hannah-Jones, 2020) . In addition, it is widely recognized that parental assistance with and support of student learning has also likely varied depending on the age of the student as well as other social, economic, and family-related factors. An obvious question given the wide sweeping changes that COVID-19 brought to education is what potential impact it may have had on student learning. The purpose of this article is to talk about the potential impacts of COVID-19 from the perspective of one widely used educational technology provider, Renaissance. Renaissance has an integrated system of instruction, practice, and assessment solutions, which are currently used by approximately 16.7 million students in approximately 45,000 schools worldwide. To frame the discussion, we focus on the content areas of math and reading, and we discuss the potential impacts of COVID-19 in terms of assessment, instruction, and practice using data from several resources offered by Renaissance. The article concludes with additional discussion and implications of the results presented in the main text of the article. With the closure of schools in the United States, many districts and schools seemingly stopped testing altogether, while others greatly reduced their use of assessments (Strauss, 2020) . This was probably due in part to many K-12 summative assessments being canceled, and the fact many other assessments were not designed to be administered at home. In addition, some assessment providers may not have the infrastructure to offer their assessments remotely to students at home, while others may have been concerned about test security and cheating. Renaissance offers the ability for students to take its Star computerized adaptive tests (CATs) at home. Traditionally, most Star assessments have been administered online in traditional school settings, but some assessments have been taken in other settings, including at home. The Star assessments are a suite of K-12 math, reading, and early literacy CATs that are offered in English and Spanish. The tests are linked to grade level content standards in every U.S. state, the District of Columbia, and the United Kingdom, and are used to monitor student progress and growth. They also can be used to figure out placement into learning progressions and several of Renaissance's practice programs. The tests can also be used to predict end-of-year performance on summative tests. While usage of Star Reading and Star Math assessments did not stop when the COVID-19 pandemic struck, utilization did decrease dramatically in the spring 2020 testing period. Preliminary investigations of Star Reading and Star Math assessments suggested that approximately 5% of the students that had completed assessments during the winter 2019-2020 testing period completed an assessment during the spring 2020 testing period. Taking a closer look at the students who completed the Star Reading and Math assessments, it became clear that these students were not necessarily representative of the larger population of students that typically take the Star assessments and that some of the tests scores were different than expected with students in early grades showing somewhat higher test scores than they had in prior school years; possibly from students receiving parental assistance when completing assessments. Given these oddities, we decided to turn our attention to a simpler and more idealized method to quantify the potential impact of COVID-19 using data from Star. Our method involves using the norms provided in the Star Reading and Star Math assessments and proposing several hypothetical student learning scenarios. The Star assessments provide nationally representative norms of students and schools in the United States with the most recent set of norms being introduced in 2017 (Renaissance Learning, 2019a Learning, , 2019b . Each test has separate norms for each grade placement; the grade placements roughly correspond to the month that a student tests. Star Math covers grades 1-12 and Star Reading covers grades K-12. The norms are often used to define normative benchmarks to classify students into different performance categories in a response-tointervention (RTI) framework. The most commonly used normative benchmarks are set at the 10th percentile, the 25th percentile, and the 40th percentile, and are used to classify students into the performance categories of urgent intervention, intervention, on watch, and at/above benchmark (Renaissance Learning, 2019a, 2019b). The three hypothetical scenarios we consider are if a student stopped learning when COVID-19 started, if a student advanced a month in their learning after COVID-19, and if a student declined a month in their learning after COVID-19. We defined stopped learning as typical learning at the end of March, advanced a month as typical learning at the end of April, and declined a month as typical learning at the end of February. These scenarios represent several hypothetical points in which one may expect students to be in their learning during the school year given that it is expected that student learning was much less than it would be in a typical school year. To quantify the potential impact of COVID-19, one can look up the scaled scores associated with the 10th, 25th, and 40th percentile ranks in each scenario and figure out what percentile rank would be associated with those scaled scores at the end of the year. The differences approximate how many students one would expect to observe at each benchmark compared to the baseline values of 10%, 25%, and 40% at the end of the year. Figures 1 and 2 show line graphs of the three scenarios for Grades 1, 3, 5, 7, 9, and 11 for Star Reading and Star Math, respectively. The black lines show the benchmark between urgent intervention and intervention (i.e., the 10th percentile), the gray lines show the benchmark between intervention and on watch (i.e., the 25th percentile), and the light gray lines show the benchmark between on watch and at/above benchmark (i.e., the 40th percentile). One can see, as expected, that all three scenarios resulted in a greater percentage of students below the benchmarks than the typical end-of-year values with the effects being greatest if students declined a month in learning. The graphs also show that students in the lowest grades would appear to be impacted to a greater degree as the lines were further from the baseline values in these cases. The effects showing greater growth in lower grades is consistent with the notion of scale deceleration often observed on K-12 tests (see Dadey & Briggs, 2012) . The observed effects also appear to be slightly more pronounced for Star Math than for Star Reading. An important implication if these results were to hold when schools reconvene in the fall is that there may be a need to think about providing more resources at early grades to help students catch up in their learning or to prioritize what skills are taught in these grades. In the context of using assessments in a Multi-tiered Systems of Support/Response-to-Intervention (MTSS/RTI) framework, schools may need to be prepared for more students to be identified as needing urgent intervention or intervention than in the past with larger impacts in lower grades. Given that many students are likely to be behind when schools reconvene in the fall, education leaders at the state and local levels will likely have to make key instructional decisions regarding what material to cover in the 2020-2021 school year. Teaching all the skills and standards that the students missed from the spring of 2020 may not be logistically feasible, in addition to having to cover the full year of standards for 2020-2021. And more importantly, it may not be necessary. Fogarty, Kerns, and Pete (2017) argue that all skills are not created equal and that in some cases educators struggle to take this information into account when planning instruction. One strategy developed by some states and assessment vendors to help educators in this regard is to identify what are known as power skills, focus skills, or core concepts, which represent the key foundational building blocks that a student needs to know (Ainsworth, 2013; Reeves, 2002) . Over the past decade, Renaissance has developed learning progressions unique to each U.S. state and the National Curriculum of England. The progressions were initially developed using content expert judgment and then refined using quantitative skill difficulty information collected from Star CATs. Note. For math, Algebra 1, Geometry, and Algebra 2 standards have been tagged to grades 9, 10, and 11, respectively, although students may encounter those subjects in different grades. Grade 12 math Focus Skills are not identified in the table due to the fact that many states do not establish requirements for all 4 years of high school. Within each progression, a set of Focus Skills have been identified consistent with guidelines outlined by Kirkup, Jones, Everett, Stacey, and Pope (2014) , meeting three criteria: (1) are essential to the progression, involving concepts learners must master in order to move to the next step; (2) support the development of other skills, serving as prerequisites for skills to come; and (3) reflect the emphasis of the curriculum. In short, Focus Skills are the key building blocks of understanding. Information on the Focus Skills by state and subject and can be found at: www.renaissance.com/focus-skills. Table 1 summarizes the average number of Focus Skills and Focus Skills as a percent of all skills by subject and grade, across all 50 states and the District of Columbia. One can see that for reading the pressure will be greatest in grades K-2, where critical Focus Skills comprise half or more of the skills to be taught and there are a greater number of Focus Skills. Critical skills in math are more evenly distributed by grade, though there is also greater variability across states as to when math skills are emphasized. For example, Grade 9/Algebra 1 represents the greatest relative concentration of Focus Skills across all grades in Texas and Virginia, whereas in many other states, including California, Illinois, and Washington, the greatest relative concentration of Focus Skills are apparent in grades 1, 2, 4, 5, and 7. Interpreting Table 1 in the context of COVID-19 suggests that in terms of instruction the impact may differ for reading compared to math. For reading, one would probably anticipate, consistent with what was observed for the assessment results, that the earliest grades may be impacted to a greater degree and that more skills may have been missed in the lowest grades. More intensive instruction and catch up may be needed in early grades for reading. For math, the impacts appear to be more spread out and differential across states, so instructional focus may need to change depending on state with the important caveat that growth in student math ability, as noted in the assessment section, often tends to be higher for early grades. A key component of any curriculum, and particularly important during school closings, is tools to facilitate student practice of key skills. Renaissance offers several practice tools, including Freckle Math ® and myON ® Reader. Freckle Math provides students with online practice covering K-9 standards that is tailored to each student's skill level. Educators can assign particular types of math practice to students and monitor student progress through reports that capture current practice, student performance in each standard, and progress toward goals. MyON ® Reader provides students with a digital library consisting of unlimited access to thousands of digital books. In myON Reader, students can select titles from recommended reading lists that are personalized based on their interests, age, and reading level or browse the extensive digital library. Educators using myON Reader can track digital reading progress through reports that provide insight into reading activity and student reading habits. One can use high-level data from Freckle Math and myON Reader to evaluate whether learning may have stopped under COVID-19. Data from Freckle Math and myON Reader reveal sharp increases in signups and usage when compared to statistics from earlier in 2020 prior to many school closures. Usage of myON Reader, which was offered free of charge for a limited time, increased by over 51% compared to the early part of 2020 prior to the school closures. A large portion of myON Reader's increase was due to expanded usage among students who had been using myON Reader prior to school closures. Signups for Freckle Math, which has a limited free version available for parents and teachers, were more than five times higher following school closures compared to earlier in the year. This dramatic increase in Freckle Math users was accompanied by an increase in the number of days students were using Freckle Math each week. These data are consistent with findings reported for other educational technology tools (Molnar, 2020) and indicate that school closures may not have led to a pause for many students in practicing key skills. The purpose of this article was to investigate the potential impact of COVID-19 using data from several products offered by Renaissance. Results suggested that assessment declined dramatically during COVID-19 and that there was the potential for achievement gaps to exist under several idealized scenarios compared to typical end-of-year performance. These achievement gaps would appear to be greatest for early grades where growth in math and reading ability tends to be highest. Instructional data on Focus Skills suggested that if learning was stopped or reduced that impacts in reading would appear to be greatest in early grades, while impacts in math appeared to be more evenly distributed across grades and may differ across states. Data from math and reading practice products suggested increased utilization during COVID-19, and that many students were still practicing some key skills even if assessment and instruction may have been reduced or stopped. Taken together, these results suggest that one may expect lower performance when students start school in 2020, there may be more students potentially in need of support and intervention, and that schools may need to leverage decision-making frameworks such as MTSS/RTI more than ever to identify needs and target instruction where it matters most. In addition, it seems likely that impacts may differ across subjects, grade levels, and states. Impacts may also differ for different students within and across schools. It is important to point out that our study used data from one educational technology provider. While it is possible that results may be differ for other providers, it seems likely that many of our findings would generalize given that findings of increased student growth in early grades are common in K-12 tests, our instructional data was from all U.S. states and the District of Columbia, and the increased utilization of Renaissance's practice programs was consistent with data reported by other providers. It is also important to point out that there are many ways our analyses could be expanded. Future analyses could look at impacts for different demographic groups and schools, perform more detailed analyses of student assessment and practice data, and consider other factors that may impact student learning, such as summer learning loss. Targeted surveys of teachers, principals, and administrators to understand how they approached COVID-19 in spring 2020 and how they plan to approach instruction, practice, and assessment in 2020-2021 would also be valuable. Clearly, the impacts of COVID-19 may not be fully documented for years to come, and in many cases all one can do is take a best guess. Finally, it is important to mention that school for 2020-2021 will probably look much different than it has in the past. Some schools may decide to move back to face-to-face instruction, other schools may opt to use remote learning, and still other schools may use hybrid approaches that are a combination of remote learning and face-to-face instruction either by choice or out of necessity if there is another wave of COVID-19 outbreak. Future research should also consider how these different school configurations may impact student learning. Prioritizing the common core: Identifying specific standards to emphasize the most A meta-analysis of growth trends from vertically scaled assessments The decline of standardized tests: The pandemic may accelerate the demise of these measures of student achievement Unlocking student talent: The new science of developing expertise As classes go online, many students just aren't showing up See which states have cancelled spring tests because of coronavirus COVID-19 and the state of K-12 Schools: Results and technical documentation from the spring 2020 American educator panels COVID-19 surveys California takes steps to remove hurdles faced by prospective teachers whose training is stymied by coronavirus Developing national curriculum-based learning progressions: Mathematics Usage surges for 18 Ed-Tech products during COVID-19 quarantine Making standards work Renaissance Learning (2019a) Star Math technical manual Star Reading technical manual Amid coronavirus, AP exams went online and had tech problems. College Board says it's investigating. USA Today Updated: Free resources for schools during COVID-19 outbreak It looks like the beginning of the end of America's obsession with student standardized tests The authors would like to thank Katie McClarty, Gene Kerns, Julianne Robar, and Jess Youngblood for their work to build, analyze, and communicate the value of learning progressions and the role Focus Skills can play in fall 2020 and beyond. Any opinions, findings, conclusions, or recommendations expressed in this manuscript are those of the authors and are not necessarily the official position of Renaissance.