key: cord-0048446-exda82fl authors: Camara, Wayne title: Never Let a Crisis Go to Waste: Large‐Scale Assessment and the Response to COVID‐19 date: 2020-07-23 journal: nan DOI: 10.1111/emip.12358 sha: 1d1ae7930c69f897d0a80f5e073c3e7a0eafd3ee doc_id: 48446 cord_uid: exda82fl In early spring 2020 the vast majority of US colleges and schools closed for the year due to COVID‐19 with no clear direction on when or how these institutions will reopen for in‐person instruction. School closures and the associated health concerns haulted large scale admissions testing and required alternative models such as remote proctoring at home, additional flexibility in test sites and administrative conditions, and additional testing dates for the fall. It is clear that access to admissions testing has been greatly reduced despite these efforts resulting in extended deadlines in graduate admissions and the wide‐scale adoption of test‐optional polcies in undergraduate admissions. This paper traces the efforts undertaken by admissions testing programs to adapt to COVID‐19 and the measurement issues which emerge from these efforts. The original quote used in the title comes from Rahm Emanuel, former advisor to President Obama. The phrase was used in response to the 2008 financial crisis and meant that a serious crisis affords opportunities to do things that would be opposed during normal times. There are many examples of this sentiment when examining how national testing organizations have responded to the pandemic and the extended closure of schools and colleges across the United States this spring. This article describes how standardized national admissions testing programs have attempted to pivot and respond to the pandemic, how universities and colleges have reacted to the changes implemented by assessment providers and how the circumstances and events might influence the admissions testing in the future. The difficulty in tackling these issues is that testing organizations are still exploring and evaluating various options and events around the pandemic are in flux, such as whether a second wave will emerge, which schools and colleges will open for in-person instruction this fall, attitudes of parents and students about returning to schools for instruction or testing and the outlook/timeline for a vaccine. By now, educators, students and the general public recognize the extent that COVID-19 has disrupted schools, learning and assessments in spring of 2020. School closures due to coronavirus affected at least 124,000 of the estimated 132,000 U.S. public and private schools and over 56 million students. Forty-eight states, four U.S. territories, the District of Columbia, and the Department of Defense Education Activity (DoDEA) have ordered or recommended school building closures through the end of the 2020 academic year, affecting Wayne Camara, waynecamara@gmail.com approximately 50.8 million public school students (Education Week, May 15, 2020) . School closures started in mid-March and generally extended through the end of this academic year in all but nine states where district or regional determination was permitted. The loss of instructional time may have ranged from 6 weeks to over 11 weeks. U.S. colleges and universities followed similar patterns of shutting down in-person instruction from March to the end of the academic year. At the time of writing this piece there is no consensus on what number of schools and colleges will return to full-day, in-person instruction this fall. It appears that in-person instruction may return to normal in relatively few areas and locations by early fall, while many others will propose different variations of instruction that will range from online-only to in-person learning with shorter or staggered days which will reduce instructional time. A number of districts have already stated they will not be open on the first day of school. California State Universities and California Community Colleges have similarly announced they will continue online instruction through the fall semester and a number of other colleges have made similar announcements. It appears that the decision for schools and colleges to open for in-person instruction with or without major modifications (e.g., schedule, number of students, enrollment size), or to continue with online instruction, will be made at the regional level or by individual institutions in the case of higher education. This situation poses difficulties for national testing programs which rely on schools and colleges which serve as test sites, as well as general communication with students about testing. The following sections examine the effects of COVID-19 and potential solutions proposed to mitigate these effects for both national assessment programs administered in high schools and admissions tests for graduate and professional programs. Certainly the current pandemic will have a long-lasting impact on our behaviors in many ways which are yet undefined and it is particularly difficult to predict how it might impact admissions and admissions testing. But, there is no doubt that COVID-19 will have such effects. First, the most immediate impact may be on undergraduate admissions processes. While 12-17% of all 4-year colleges or universities neither recommend or require admissions tests, which represents about 1% of all freshmen in the United States, 1 over 150 colleges have adopted a test optional policy for 2021 or beyond given COVID-19. The GRE (GRE) General Test, TOEFL iBT Test, Graduate Management Admissions Test (GMAT), and Law School Admissions Test (LSAT) each suspended in-person testing due to the coronavirus and adopted remote proctoring solutions. Each organization described slightly different versions of remote proctoring to allow students to complete their test at home during the pandemic. This article begins with a basic description of some features of remote proctoring and the requirements common across at-home testing for these high-stakes tests. Remote proctoring or online proctoring, refers to the use of automated processes and/or human proctors to monitor delivery of testing with a webcam and microphone via the Internet which was first introduced in large-scale assessment by Kryterion in 2008 (Foster & Layman, 2013) . Assessments are digitally delivered with software that allows test takers to complete an assessment anywhere there is a web-enabled device and a typical Internet connection. Although remote proctoring can rely solely on live human proctors or AI alone, most highstakes testing programs will use both AI and human proctors. Live human proctors may be assigned on a 1 proctor to 1 test taker ratio, but are more commonly used to monitor multiple simultaneous test takers (1:2, 1:4, 1:8) with 1:16 generally the largest number of test takers recommended. In addition, monitoring software typically relies on artificial intelligence (AI) to track test takers through a camera and microphone in order to increase the reliability, credibility, and authenticity of testing. AI monitoring of test takers may identify questionable movement, sounds, interruptions, and the presence of additional individuals in a testing location. AI may be used to flag such issues for human review during the session or to identify risks that can be used to review video following testing if live-human proctoring is not employed. Remote proctoring has increased in popularity across online degree programs offered in higher education, other distance learning opportunities in education and the workplace, and with some certification tests. Some of the most popular vendors offering remote proctoring to high-stakes tests include ProctorU, Proctorio, PSI Bridge, Examity, Proctortrack, Pearson Vue, and Prometric ProProctor (Chin, 2020) . Test sessions are also typically recorded (audio and video). Human proctoring can be either conducted live (during the test session), which allows a proctor to terminate testing if irregularities are discovered, or after testing and prior to release of scores. The human review of a tape may be conducted on each testing session, used on a fixed or variable schedule (e.g., every 5th test administration) to deter cheating, or used only when AI has identified one or more risks when human proctoring is not conducted or used with many test takers. High-stakes testing programs are advised to utilize multiple and sophisticated methods to monitor testing and deter cheating, but each additional procedure comes with additional costs and requires additional capabilities that may limit the number of test takers participating in a session at one time. Some testing programs may establish a specific time and date for online test administration which would be similar to national in-person testing, allowing them to use the same form for all test takers. The downside to such an event-based session is that there may simply be insufficient numbers of human proctors to cover the event. For example, the ACT and SAT may expect 300,000 test takers on a national Saturday administration and testing all those students on line with remote proctoring would require over 18,000 trained proctors using a 1:16 ration-a requirement that no vendor may have available today. In addition, the bandwidth required to administer a test to that number of test takers may be an additional challenge. Finally, candidate authentication and registration, requiring a photo ID, and some 1-on-1 questioning may be required, along with a system test of your computer. Some testing programs will utilize some human check in tasks while others may employ an automated system. This can be a relatively "high touch" process from the vendors. Cell phones may be required during check in but are prohibited once testing begins. An alternative model is to stagger testing across several days and times, but then a single form is insufficient to provide the type of deterrence and protections associated with most high-stakes testing. Fully adaptive testing programs with several item banks may be able to employ this model of remote proctoring, but linear assessments will require more forms to circulate over days and times. Item and form inventory become an important requirement. The potential loss of intellectual property (e.g., test items) possesses a significant risk to online-remote proctoring despite the procedures incorporated into test protocols. Some assessment programs may restrict testing in specific nations such as China and Iran because of these concerns. Research on remote proctoring has been described as lacking in volume and rigor, as well as relevance, for high-stakes testing, in a report issued by the Institute for Credentialing Excellence (Plaus, Boren, Brazell, Wickett, & Weber, 2015) . Weiner and Hurtz (2017) noted that most research has focused on test security, usability, and score equivalence and have produced inconsistent findings across fields (e.g., employment, education, and certification). Many of the studies focused on kiosk-based remote proctoring models, popularized for employment screening, with less research on at-home applications. The GRE, TOEFL, GMAT, and LSAT at-home testing solutions prohibit tablets and mobile devices, and restrict administration to desktops and laptops. Generally, Windows or Mac computers are permitted, but there are additional requirements for Macs which may limit their use. 2 ETS, Graduate Management Admissions Council (GMAC) and Law School Admissions Council (LSAC) use live-remote proctors which requires a built in camera or webcam able to move and show the proctor a 360°view of the room including the computer surface before testing begins. A microphone and speaker that are not be part of a headset are required. A wired connection is recommended and most allow a Wi-Fi connection, but some programs explicitly prohibit tethering to a mobile hotspot. Several programs provide additional recommendations/requirements on testing environments, clothing and nontest materials, primarily for reasons of security, such as: • Placing the computer or keyboard on a desk or flat surface clear of all items not approved for testing • Sitting on a standard chair (not a bed, couch, overstuffed chair) • Prohibiting anyone beyond the test taker to be in the room • Prohibiting food and drink during testing • Avoiding wearing of jewelry, tie clips, headbands, etc. • Prohibiting taking notes on regular paper; allowing white board or transparent page protectors (test taker will be required to erase all notes in view of the proctor at the end of testing) The GMAC also adopted a similar human remoteproctoring solution using Pearson OnVue proctoring system. The GMAT Online exam is similar to the test center version, but omits the Analytical Writing Assessment Section, with the quantitative, verbal and integrated reasoning sections remaining unchanged (same number of questions and testing time) and use of the same scoring algorithm and score scale. Section scores, total scores, and reliability are comparable. GMAT Online is offered 24 hours a day, 7 days a week during the initial testing window (April 20 to July 17, 2020) and future testing windows may be offered if needed (GMAC, 2020) . No repeat testing is currently allowed. In announcing the establishment of a live-human remote proctoring for at-home testing on the GRE and TOEFL IBT, ETS noted that the testing experience would be identical to center-based testing including the format and timing of each test, the ability to preview and skip questions, review and change answers on the GRE and the ability to view reading and listening scores at the end of the TOEFL iBT test (ETS, 2020a). ETS further stated that "test takers can expect the same valid and reliable tests that are administered in test centers from the comfort of home." However, if differences in platforms, on-screen tools, permissible screen-sizes, server response times, or other issues that could impact performance exist between the two administrative models score comparability must be established (Camara & Harris, 2020) . ETS utilizes ProctorU to monitor testing. Similar to GMAT, tests are offered 24 hours a day but restricted to 4 days per week during extended testing windows that 2 For example, Mac OS X 10 or higher. currently run through September 20 and testing is offered globally with some restrictions (ETS, 2020b) LSAC has offered at-home testing with remote proctoring for the LSAT which is labeled LSAT-Flex, and uses a shorter version of the LSAT. Three 35-minute sections are administered (compared to four 35-minute sections for the traditional test). In addition, the LSAT Writing task is administered separately. In 2019, the LSAT transitioned to exclusive digital administration utilizing Microsoft Surface Go tablets (Margolis, 2018) with nine national administrations annually. Tablets are stored centrally and distributed to universitybased test centers prior to the administration. Since LSAT-Flex requires desktop and laptop computers, differences in performance due to device have been examined and appear to be minimal. Scores from the LSAT-Flex will be reported on the same 120-180 score range, and will be annotated. 3 LSAT-Flex was first administered to nearly 10,000 applicants across different testing times between May 18 and 22. Additional administrations of LSAC-Flex are scheduled for June 14-15 and July 12-13 with possible additional administrations under consideration. LSAC is also using ProctorU as their remote proctoring vendor and has similar hardware requirements as other testing programs, but notes that they will try to assist test takers who may not have the necessary equipment (e.g., webcams) with loaner devices and solutions. While requiring a quiet room with no additional individuals present, they will review situations if this is not available if contacted in advance (LSAC, 2020). These programs now offer most major accommodations for students with disabilities including 50% and 100% extended time, extended breaks, screen magnification, and selectable colors with their online tests. Screen reader and refreshable braille compatibilities are offered by ETS (2020b). Additional modifications may be available such as a Braille test form or paper test form where remote proctoring is used to monitor the testing experience but delivery and responding are not performed digitally. The Medical College Admissions Test (MCAT) chose a different route since it is normally requires over 6 hours of testing. MCAT is still only available at Pearson VUE testing centers but the examination has been shortened and three additional dates have been added. All operational sections are still administered but the pre-test section has been temporarily removed. Test takers will receive the same section and total scores (AAMC, 2020, June 8). The ACT, SAT, and Advanced Placement assessments are primarily delivered as paper-and-pencil (PNP) tests in high schools. The ACT and SAT, undergraduate admissions tests, are administered to an estimated 3.6 million students annually (Camara, Liu, & Mattern, under review) , and an estimated 2.9 million students expected to complete over 5 million AP exams in 2020 (College Board, 2019; Wanneh, 2020) . The AP program consists of 38 subjects and exams are typically administered in paper-and-pencil format in schools or larger testing centers on a single test date during a 2-week window in May, with late testing (make-up dates) scheduled the following weak. Exams are 3 hours in length and generally divided evenly between selected-response items and openended student produced response tasks (or short questions). COVID-19 was particularly disruptive for testing in the Advanced Placement program because testing occurs on a single day in the year and over 2.8 million students completed over 5 million examinations last year (College Board, 2019). Developing a new administrative model, in less than 90 days, that could scale up for nearly 3 million students to test and maintain some semblance of test security appeared to be a herculean task and the College Board arrived at the following solution for 2020 4 : • Traditional in-person, school-based AP exams will not be administered. • Instead each AP exam will be shorted from a 3-hour session with time generally split between selected response and free response items to a 45-minute web-based exam composed of one or two free response tasks. • Exams are open book. • Each AP exam will be offered on a single-day and singletime globally, with a single make-up session. • Students can complete the exam with computer, tablet, smartphone, or paper/pencil. • Students can submit their work by attaching a text file, copying and pasting their response into the exam application, or attaching photos of handwritten responses. 5 • Remote proctoring was not feasible and other deterrents to cheating included use of plagiarism detection software, teacher review of student responses, threatening to cancel scores and notify colleges if cheating is detected. • Test preparation materials were provided which included a list of content covered and omitted from each exam, study sheets, and a guided demo (College Board, 2020; Ingersoll, 2020) . The solution was greeted with both controversy and some technical difficulties during implementation. Many students were unable to submit their exams as instructed at the end of testing. One report claimed this affected nearly 20% of test takers in the first 3 days of testing, and the College Board asserted it was less than 1% of the 2.9 million test takers. Other students faced difficulties finishing the exams or logging into the platform. High school students sued the College Board claiming its AP administration for 2020 discriminated against students without sufficient resources, those in remote locations, and the disabled, according to a proposed classaction complaint filed in Los Angeles Federal Court on May 19 (Pettersson & Lorin, 2020; Wanneh, 2020) . The complaint alleged that many of the problems arising with at-home exams were known in advance and not addressed by the testing organization. The complaint also stated that "it is unrealistic to think that all students have quiet, private spaces at home in which to test. Lower-income students are much more likely to face cramped housing, siblings and parents sharing the same workspace, internet connectivity problems, noisy environments, and less comfortable testing spaces" (J.P. on behalf of her minor son, R.P. et al. vs. Educational Testing Services et al., No. 2:2020 -cv-04502 C.D. Cal. 2020 . The AP administrative model for 2020 was likely the only feasible alternative that could be developed, tested and implemented within a 90-day period given its administrative requirements (each exam administered on a single day) and large numbers of test takers. It appears that many colleges and universities understood the limitations associated with AP scores in 2020 and were sympathetic to the effort given that about 60% indicated they would confer credit as usual (Ingersoll, 2020) . The AP model can best be described as an online, timed, unproctored administration with abbreviated forms that are likely unequated and noncomparable to previous longer forms. Clearly, administering the 3-hour forms prepared for the regular May administration during an at-home testing event must have been considered too demanding and the decision was made to substitute a greatly modified form of the exams-one that was represented about 1 4 of the typical test length and time, and relied solely on free response items. Specifically, it is likely the 2020 AP exams have a number of deficiencies such as: • Lower reliability (basing a score on one to two items, even though they are longer items) • Inequivalent scores to previous AP exams; absence of information on score linking, the differences in exams (reduced content coverage, change in item format, changes in section and item order, etc.), timing, and administrative modes indicate that scores should not be treated as equivalent across years • AP grades set with a separate standard setting only for the 2020 forms • Potential differences in the population of test takers, especially if particular groups of students are unlikely to test because of difference levels of access • Unknown comparability issues that emerge with a speeded test administered across different devices and modes (2020) • Greatly increased opportunity for students to cheat (e.g., no way to verify the identity of test takers or if other parties are assisting the test taker) • Technical limitations and difficulties reported above Clearly the 2020 AP exams may be the least standardized tests administered among standardized tests. Numerous risks emerge in terms of the integrity, accuracy, reliability, validity, and fairness of scores from the 2020 AP administration and likely mean score trends and comparability cannot be maintained. These are serious limitations for any high-stakes testing program that render scores of questionable value. However, given the circumstances and timing there was likely no alternative model that could have been implemented to scale that would have minimized these risks. The only viable alternative would have been to cancel all AP examinations for the year which would carry an enormous penalty for millions of students who worked hard all year in these courses. Despite real challenges to how to interpret AP scores for 2020, it is certain that the vast majority of students who completed these examinations did so with integrity, honestly, and absent assistance. The effect of COVID-19 has been to force every national and state testing program to grapple with options which are far from perfect. Often the choice is whether to offer testing under significantly different, and often less rigorous conditions, or to forego testing for a year. Psychometricians can articulate the risks and potential consequences of these alternatives, but the decision generally rests squarely in social policy and is made by the institutions which use and require tests. The majority of students complete admissions tests at schoolbased test centers on one of seven to eight Saturdays designated as national testing dates by either testing organization. A second administrative model applies to 11th graders in about half of the states which contract for delivery of the ACT or SAT during a school-day administration. A small proportion of admissions tests are delivered in school-based testing on computers. School closures resulted in the cancellation of the May and June administrations of the SAT in 2020, as well as the April administration of the ACT. The class of 2021 will be most heavily impacted as juniors typically first take an admissions test in their spring semester with many retesting in fall of their senior year. In addition, most of the state contracts for junior testing appear to have been disrupted for 2020. The pandemic gained more public prominence immediately prior to the March administration of the SAT. The College Board did not cancel testing as many test centers were closed and many students were unable to take the exam (Jaschik, 2020) . The ACT was administered to about 55,000 students across a smaller number of test centers that opened in June and is planning to deliver the test to available test centers in July, also with greatly reduced volumes expected. Approximately 300,000 test takers normally register to take the ACT and SAT on each of these national testing dates (March, April, May, and June) and approximately one million juniors who had planned to first take a national admissions test this spring may have been displaced from testing with the school closures. What options or alternatives exist for testing students to account for student demand this fall The first option is to increase capacity for scheduled national test dates for the remainder of year which could entail increasing testing seats in available centers, as well as opening new test centers (additional high schools) in areas with highest demand. Table 1 shows the scheduled national testing dates for the ACT and SAT in 2020-2021. Clearly there are many unknowns which make it impossible to project when school-based testing can return to normal practices and what it may look like until it does. For example, as of June it is difficult to answer most of the following questions: • What schools will be open, able and willing to administer tests on Saturdays in August, September, October, November, etc.? • What social distancing guidelines will be required, how will they differ by state or school, and will there be available capability to implement these practices? • Will students feel safe to come to a school-based test center to complete the test and will educators feel safe enough to proctor the test and manage on-site logistics? • What is the impact of social distancing on available seats? • Do schools have capacity and a willingness to dramatically increase space available for testing? • Can sufficient numbers of test site supervisors and proctors be recruited to support demand and additional dates? • Will student demand for testing be reduced because of health fears, changes in college-going plans or the increase in colleges waiving testing for admission in 2021? Guidelines for centralized test administration required to maintain safety and protect the health of test takers and staff impose significant operational and financial burdens on testing organizations and school-based test centers by reducing both seating capacity and the number of schools that can implement the required protocols. For example, a requirement of 6-feet of distancing between students could reduce seat capacity by 65-75% and three to four times as many rooms and proctors would be needed to manage a similar number of test takers. Partitions and other equipment may also be required to separate test takers. As previously noted, most Saturday national test administrations involve hundreds of test centers and over 300,000 students. It is unlikely that all test centers can implement all safety and health guidelines by the fall. The effect may differ by region and state as different requirements and protocols are used by states and local school districts. The College Board reported that while only 25% of seats for its August 29 national test where filled on June 1, several states had already filled 60-75% of seats (Binkley, June 2, 2020). A second way to increase access to tests is to add in-person testing dates. ACT added three fall testing dates and the College Board added a September 26 administration to relieve some of the pent up demand. New administration dates require an additional test form(s) and significant logistical planning and support from high schools to administer the test. A third option is to deliver the test digitally for students at home. Both ACT and College Board indicated they were developing plans to provide a digital version of the tests, likely with vendors who provide remote proctoring for at-home testing, if the pandemic continued to make it difficult to open enough test centers in the fall. However, more recently, the College Board stated that the technology challenges of developing an online test that all students could take led it to drop its plans. "Taking it would require three hours of uninterrupted, video-quality internet for each student, which can't be guaranteed for all according to the College Board" (Hartocollis, 2020) . The announcement also followed technical problems experienced while trying to deliver the shortened online Advanced Placement exams. ACT affirmed its continued planning to deliver the test at home in late fall if needed (Binkley, June 2, 2020). At-home testing for high school students presents unique challenges not necessarily present when testing adults. ACT conducted a survey of college-bound high school students who registered to take the April or June 2020 ACT and asked about their experiences during the coronavirus outbreak. A total of 13,000 students completed the survey which was administered between March 26 and April 1 (Moore & Croft, 2020) . Almost all students had access to at least one type of technological device and at least an "OK" Internet connection at home. However, the type of device and quality of that Internet connection varied, most students (81%) having access to a laptop, desktop, or Chromebook at home that would comply with requirements for remote proctoring applications from many vendors and only small differences were found between subgroups of students, but differences in connectivity surfaced. Recent estimates report that between 9 and 12 million students do not have Internet access at home. And while some Internet providers are offering to provide free access to students during the pandemic, gaining such access can be a challenge. Approximately 80% of students had access to the Internet that was separate from their cell phone. However, the quality of Internet connection at home appeared to be of greater concern with only 35% of students describing their home connection as great, with 52% saying it was OK and 14% indicating their Internet connection was unpredictable or terrible. Figure 1 illustrates that first-generation college students, students from rural communities, and Hispanic and African American students were more likely than their counterparts to report that their Internet connection was unpredictable or terrible (Moore & Croft, 2020) . Even though a single ACT or SAT test has not been administered at-home or even scheduled for at-home administration, there has been plenty of criticism of the concept. The National Association of College Admissions Counseling (NACAC, 2020) released a statement sent to ACT and the College Board which read in part "public health measures have forced the cancellation of test administrations, disad-vantaging students wishing to test this semester; and we understand that ACT and College Board must plan ahead in the event schools aren't able to open in the fall. But we're also mindful that practitioners-both counselors and admission officers-have raised legitimate questions about the effects of these adaptive testing measures, particularly on those who are already at risk of dropping out of the college pipeline-our low-income, first-generation students." They listed a number of concerns with at-home administrations and remote proctoring, including: (a) the digital divide that may adversely impact low-income students, (b) the ability to provide appropriate accommodations for students with disabilities, (c) potential limitations in providing the same solution to test takers outside of the United States, and (d) limited information on the reliability, validity, and integrity of scores. Some argue that that there has been little confidence in test security before at-home testing was discussed and several colleges noted that at-home testing had not yet "been validated" and others express concerns with student privacy in their homes when cameras and microphones are allowed to scan the room and record a student testing session (Wertheimer, 2020) . Of course, the severity of these same concerns is heightened in testing 15-to 17-year-olds at home who are applying to college as compared to 22-to 30-year-olds applying to graduate school or law programs. Experience in living with the pandemic will influence assessment practices and innovation, just as such experience will change the way schools/colleges operate, and instruction is delivered, as well as virtually all areas within education. Just as we had no playbook in hand on how to adjust testing for a pandemic, we also have no technical manual documenting lessons learned from this experience-at least, not yet. There are trends, behaviors, and solutions that have been observed that can inform predictions of ways assessments may change in the future, based on what we have experienced in the past several months. First, at the undergraduate level there may be less demand for admissions tests in the near future, due to COVID-19 and the political environment. More than half of the 3,330 fouryear colleges and universities in the country will not require applicants to submit test scores for the 2021-2022 admissions cycle, including about 85 of the top 100 liberal arts colleges, as ranked by U.S. News. An informal count conducted by ACT in early June found approximately 28% institutions have gone test optional on a permanent basis since VOCID-19, with about 60% implementing test optional policies for a limited period of time (Fall 2021 or Fall 2021 and Fall 2022 . Several institutions are implementing a 3-year pilot of test optional policies which allows them to evaluate the performance if students admitted under test optional policies before additional action is taken. The primary reason cited has been concern for students who may not have access to the ACT and SAT (e.g., fewer dates, less flexibility in when and where to test), as well as concerns about student safety and a desire to reduce student anxiety about testing while they are dealing with other effects of COVID-19 on their learning and home situations. Most of these colleges continue to welcome test scores from students who have tested, but they are not requiring them during this period. Nearly two dozen colleges have announced they will be test blind, meaning they will not consider test scores from applicants if submitted (Hoover, 2020; O'Shaughnessy, 2020) . COVID-19 appears to be the driving reason for such sweeping policy changes in undergraduate admissions, but the recent debate about diversity and access at the University of California, and other events in the news that highlight the long-standing disadvantages experienced by racial minorities likely had some influence in the decisions made by some institutions. As social distancing, delayed openings in schools, and continued spread of the virus extend into the fall it is likely many more institutions will adopt a test optional policy at least for fall 2021 and perhaps fall 2022. Even if admissions tests are offered this fall, access is likely to be uneven across all regions (whether due to lack of seats or fewer test centers), potentially restricting access and causing additional anxiety for students. Both testing organizations have emphasized their understanding of these factors and the need to prioritize health and safety concerns for test takers and testing staff in the schools. It does appear that student demand for testing has not diminished, as evidenced by record volumes from students attempting to register for testing or rescheduling their test dates. However, the number of schoolbased sites able and willing to administer the tests has been a fraction of the normal availability and led to frustration for many students and parents (O'Shaughnessy, 2020) . Research on the validity of admissions tests clearly indicates that they add incremental validity to other admissions measures and that decisions absent test scores will have less precision and accuracy (Camara et al., under review; UC Standardized Testing Task Force, 2020 ). Yet the loss of some precision in prediction may be a trade-off some colleges are willing to accept if they find they can implement the admissions process with efficiency and relatively positive outcomes in terms of the quality of admitted students and yield of accepted students who matriculate. Of course, data which can evaluate the outcomes from such an admissions policy for fall of 2021 are likely to not be available till fall of 2022, 6 which is why many institutions are proposing a 2-year or 3-year suspension of test requirements. While I find it highly unlikely that virtually most colleges will permanently abolish their admissions testing requirements, changes to testing policies at the undergraduate level are highly likely. Such policy changes could result in a dramatic increase in test optional policies across public systems, use of cut scores for high school grades which can reduce the work required for comprehensive review of applications at public institutions, and an increase in test-flexibility policies where institutions accept other measures (state tests, AP) in addition to ACT/SAT. A second trend that may have been anticipated is the widespread use of remote proctoring to provide access and flexibility to test takers. The expectation that testing companies can provide remote proctoring to all students in their homes with the same quality, fidelity, and conditions seems improbable, making at-home testing for high school students (ACT, SAT, AP) improbable. The requirement of a video quality testing environment, 3-4 hours of uninterrupted testing in the home and concerns about privacy and cheating appear to cloud the future of remote proctoring in this segment. Tech-nology and administrative requirements could exacerbate inequalities across groups of students. However, remote proctoring as an accommodation, as a supplemental method for delivery of assessments between test dates, or for students needing special testing, in remote areas, and in response to emergencies, may provide limited or temporary solutions, especially if it can be conducted outside of the home (e.g., local school. test center, library, and community center). In early April the author conducted interviews with a number of higher education leaders to ascertain their attitudes toward test scores produced from at-home, proctored admissions tests and the major takeaways were that any score was considered helpful, but because of the different administrative conditions, most respondents felt such scores had to be identified or "flagged" to allow institutions to differentiate them from scores resulting from standard administrations. The greatest concerns voiced by higher education leaders concerned equity and what could be done for students who do not have the required technology or quiet room for testing at home. The most likely route for remote proctoring with undergraduate admissions tests may be in the institutionally based programs. Many colleges administer the ACT or SAT as part of a program housed at their institution when students enroll without a score. The institutional administration of these tests is not considered an "official test score" and cannot be used for admissions or sent to other institutions. Yet, the test score are used for placement decisions, program qualification or institutional research purposes. These students could just as easily be offered the test online, at home, or in another setting prior to arriving on campus and since scores are unofficial it is conceivable that human proctoring may be optional. Remote proctoring for graduate or professional admissions testing may be more feasible as a permanent solution or innovation, because we can more easily expect college students or college graduates to have access to the technological and testing (e.g., quiet room) requirements and as legal adults the concerns about privacy may be minimized. Unlike access to undergraduate degrees, entrance into a law school, medical school, or selective graduate program is not considered a right for all, but may be comparable to certification and employment situations where remote proctoring has thrived. The success of Duolingo which provides a language proficiency score, video interview and writing sample using an online remote proctoring system that is accepted by many institutions is important to consider (2020). This model appears efficient and cheap and has gained acceptance with higher education. It may also be acceptable for graduate and professional assessment delivery in the future. One of the most desirable features of the temporary online solutions provided by ETS, GMAC, and LSAC is the ability to access testing more frequently (days and times) and at a home or office. In the next year or two there will be a wealth of data that can help us understand how scores produced from online testing compare to those from more traditional methods and whether there are risks to test security, reliability, score equivalence, prediction and subgroup differences. Assessment professionals should also determine if the populations of test takers differs by approach, what technology or security risks emerge, and the reaction by test takers and institutions receiving scores. In-person testing may also change as a result of the coronavirus. First, much of the guidance issued by the Center for Disease Control (CDC) for schools and workplaces to reopen are applicable to in-person testing and may be viewed xxxx 2020 as necessary or prudent for years to come. Ensuring testing surfaces, digital devices, and the general environment (rest rooms, door handles) are sanitized and there is adequate separation between test takers may be perceived as best practice to mitigate safety and health concerns long after this specific pandemic is mitigated. The overall design of testing rooms may need to be reconsidered and greater flexibility in testing (more days and times) may both be necessary and highly desirable. The seven or eight national test dates for the ACT/SAT and the single test date for AP exams where hundreds of students may be assembled for testing in a single school are easily disrupted (e.g., weather related cancelations, security threats that require a different form, scheduling changes) and are primarily driven by the desire to reduce the number of test forms/items and offer convenience to school proctors. This model needs to be reconsidered at a time when tests can easily be delivered digitally and offer greater flexibility to students and testing sites. COVID-19 may be the stimulus that moves most or all schools to providing a digital device to each student and if that occurs the remaining excuses and preferences for paper-and-pencil testing for these tests will disappear. In fact, COVID-19 may be the tipping point for transitioning all high school assessments to some form of digital delivery. In conclusion, COVID-19 has disrupted the existing standardized testing paradigm and forced each program to innovate and pilot new types of delivery, expanded form development and use, and created a need to rethink established testing procedures and practices. The industry has adapted to the demands quickly and implemented most changes with overall success. The criticism that standardized testing is unable or unwilling to change quickly should be partly debunked by how the industry adapted to changing circumstances surrounding the pandemic. Changes are often slow and painful in large-scale testing not because the industry isn't agile, but because customers (colleges, universities, college counselors, and other test users) may not see change in assessment as desirable. The current pandemic has required adaptation and innovations, most notably in where, when and how assessments are delivered. Some tests have been shortened, deleting a section or optional essay, most programs have introduced greater flexibility and more options for testing, and remote proctoring options have been offered for many programs. These changes may be viewed positively by test takers and consumers, but we should expect skepticism. The answer will come after COVID-19 is mitigated and we take a long hard look at how we responded and the impact to students, institutions and learning. SAT drops plans for home exam amid internet access concerns Impact of technology, digital devices, and test timing on score comparability Assessment for admission, placement, and outcomes in higher education Exam anxiety: How remote test-proctoring is creeping students out. The Verge Annual AP program participation 1956-2019 ETS introduces at-home solution for TOEFL iBT and GRE General Test amid coronavirus pandemic TOEFL at home: Frequently asked questions GMAT Online exam: An online GMAT TO support business school goals College Board scraps plans for SAT at home Testing shakeup: Caltech won't consider ACT/SAT scores for next 2 years. The Chronicle of Higher Education College Board announces 2020 APs will be 45 minutes long, taken at home, online. Compass Education Group Testing turmoil. Inside Higher Education Educational Testing Services Coronavirus and the MCAT exam: Update LSAC Announces Technology Collaboration with Microsoft NACAC Statement on standardized testing during the COVID-19 emergency. National Association of Collegiate Admissions Counselors College Advanced Placement tests called unfair to tech-poor students. Bloomberg News Remote proctoring test delivery: A report on options and considerations Getting into elite colleges may have just gotten easier Report of the UC Academic Council Standardized Testing Task Force Technical Glitches and Lawsuit Dampen experiment with remote AP tests A comparative study of online remote proctored versus onsite proctored high-stakes exams Giving remote SAT and ACT exams is a bad idea Thanks to Shane King and Terry Ackerman for consultation and review in preparing this article.