key: cord-0989052-cxb406bd
authors: Engelhardt, Bryan; Johnson, Marianne; Meder, Martin E.
title: Learning in the time of Covid-19: Some preliminary findings
date: 2021-06-30
journal: International Review of Economics Education
DOI: 10.1016/j.iree.2021.100215
sha: d2e06863b3cec91567d8f773d4f67353b1ffa28f
doc_id: 989052
cord_uid: cxb406bd

In response to the Covid-19 pandemic, universities closed to face-to-face learning, shifting entirely to online instruction midway through the spring 2020 semester. In this paper, we compare student performance in the Covid-19 affected semester to that of the previous three unaffected semesters. We consider both student grades and student performance on standardized post-tests in introductory macroeconomics, microeconomics, and statistics courses. We conclude that there were no significant differences in performance across the semesters. Despite concerns that low-income, first-generation, and minority students could suffer disproportionately, we identify no measurable effect for these population subgroups. Women are found to overperform in the Covid-affected semester when compared to previous terms. Women at our institution could be expected to earn 0.15 of a grade-point less in introductory economics courses than otherwise comparable men in pre-Covid semesters. In the Covid-affected semester, this difference disappeared, with women earning higher grades, on average, than men.

As SARS-CoV-2, the novel coronavirus, first swept through communities in March 2020, universities responded by closing campuses to face-to-face learning and moving courses online. 1 By March 26, more than a thousand colleges and universities in the United States had closed their doors and commenced online instruction, a move that affected at least 14 million students (Hess, 2020) . Considering recent advancements in online learning technology and the mechanisms by which the virus spreads, fully online learning presented an attractive alternative. Decades of investment in learning technologies have vastly expanded the quality and capabilities of online learning management systems. 2 Yet, despite a myriad of technologies to facilitate online instruction, many instructors reported finding the transition disruptive and dissatisfying (Thompson, 2020) . Universities lacked contingency plans appropriate to a pandemic of this magnitude; decisions changed daily, confounding instructors and students (Knight, 2020; Lemoine and Richardson, 2020) . A Tyton survey of more than 4000 faculty following the spring 2020 semester found that many struggled to keep students engaged (60 %); additional concerns included administering exams (29 %), transitioning instructional content (35 %), and adjusting habits and practices to online teaching (38 %). Faculty complaints echoed much of what was already found in the literaturenotably that teaching online was an inferior experience when compared to face-to-face instruction (Alpert et al., 2016; Arbaugh et al., 2009; Bettinger et al., 2017) .

Educational experts, however, were quick to caution against comparing courses moved online in an emergency to those offered online with forethought and planning (Lederman, 2020) . Bessette et al. (2020) point out that the shift to remote learning relied heavily on the simplest and most accessible technologies. A survey by Cengage found that 97 percent of institutions relied at least partly on faculty with no prior online teaching experience. Nearly 56 percent of faculty surveyed reported using entirely new teaching methods with little or no preparation. As a result, half of faculty surveyed reduced the amount of work required of students and one-third lowered expectations for the quality of student work. 3 Universities, seeking to ease instructional burdens, suspended online assessments, student evaluations of teaching, and participation in national assessment and certification exams. Such decisions, however, deprived policy and education analysts with important data necessary to evaluate university responses to the Covid-19 pandemic (García-Peñalvo et al., 2021) and undermined evidence-based practices (Lemoine and Richardson, 2020) .

In this paper, we investigate the impact of the mid-semester disruption in the modality of teaching, or the move from entirely inperson to entirely virtual instruction. By focusing on student learning outcomes in the first semester to be disrupted, our study offers real time insights into how pandemic contingency plans impacted students, or more broadly, how students were affected by a major mid-semester change in course delivery. Our analysis considers two different angles. First, we compare student-learning outcomes in the disrupted semester to those of the previous three semesters. We consider two different measures of student performance: course grade and students' scores on a standardized, course-specific, multiple-choice post-test which our department unanimously agreed to administer despite a university policy allowing for the suspension of such assessments. 4 Second, we examine whether the shutdown had a heterogeneous impact on different subpopulations of students. 5 We find no evidence that the first wave of partial-semester campus closures in the spring of 2020 negatively affected student learning outcomes. Despite concerns that low-income, first-generation, and minority students would suffer disproportionately (Adedoyin and Soykan, 2020) , we identify no measurable effect for these population subgroups. One intriguing outcome we observe is that women overperformed in the Covid-affected semester when compared to previous terms. Generally, women at our institution could be expected to earn 0.15 of a grade-point less in introductory economics courses than otherwise comparable men. In the Covid-affected semester, this difference disappeared, with women earning higher grades, on average, than men.

Mid-semester disruptions can happen for a variety of reasons at the instructor level, and in extremes situations, disruptions can happen campus-wide. Understanding the impact of such shutdowns on student learning is therefore important for university planning, particularly when developing quality assurances for online teaching or policies for assessing student learning in emergency situations. These efforts can help to clarify the tradeoffs associated with choices about continued online learning, the re-opening of face-to-face learning, and possible short or long-term re-closures.

Over the past thirty years, the National Survey of Teaching Methods in Undergraduate Economics Courses has consistently documented economists' preferences for in-person classes and "chalk and talk" lecturing (Watts and Becker, 2008; Watts and Schaur, 2011; Allgood et al., 2015) . 6 A survey of recent attendees of the Conference on Teaching and Research in Economics Education (CTREE) found that 83 percent of introductory economics instructors taught in person, 14 percent taught hybrid or blended classes, and only three percent regularly taught online (Walstad and Miller, 2016) . Early studies of online learning documented that students in such courses tended to perform worse than students in live classroom settings (Brown and Liedholm, 2002; Coates et al., 2004; Pyne, 2007) . More recently, economists have attempted to pinpoint the reasons why this may be. One well-understood problem is that while computers provide new ways to deliver academic content, they also provide new opportunities for distraction (Aguilar-Roca et al., 2012; Patterson and Patterson, 2017; Sana et al., 2013) . Bettinger et al. (2016) suggest that the absence of personal interactions between instructors and students reduces engagement, thereby affecting learning. Lack of engagement may also lead students to inappropriately discount the costs of cheating, skipping, and failing to turn in assignments (Lederman, 2020) . Another possibility is that the steeper requirements for self-direction and self-motivation in online classes proves too challenging for some students, particularly those in their first year or those with weaker academic backgrounds (Figlio et al., 2010; Alpert et al., 2016) .

Some studies have attributed inferior online outcomes to selection bias amongst students, e.g., compared to students who choose face-to-face classes, students who chose to take online courses may simply be prone to poorer outcomes (Anstine and Skidmore, 2005; Brown and Liedholm, 2002; Xu and Jaggers, 2013) . In an attempt to disentangle latent student characteristics that may influence both choices about online course enrollment and course performance, Alpert et al. (2016) conducted a randomized control trial to assess learning outcomes in face-to-face, blended, and online courses. They found that students in the fully online section scored five to ten percentage points lower on a common final exam, despite random student assignment across sections, concluding the delivery method was to blame. Bowen et al. (2014) and Joyce et al. (2014) examined the effect of in-class time on academic performance by randomly assigning students between traditional and hybrid courses. Both studies found that students in the traditional courses exhibited modest gains over students in the hybrid format.

Beyond course delivery, several additional factors have been identified as contributing to student success in introductory economics. These include academic aptitude, mathematics preparedness, and previous exposure to economics (Anderson et al., 1994; Ballard and Johnson, 2005) . Since the 1970s, studies have also routinely reported on the importance of gender in predicting student success in economics. This research suggests women in introductory economics courses see a grade penalty that is both steeper and more pervasive than in other STEM fields (Bolch and Fels, 1974; Johnson et al., 2014; Siegfried, 1979) . Haley et al. (2007) reported the intriguing finding that in introductory business statistics courses taught by economics faculty, women outperformed men by roughly three percentage points, suggesting the penalty is specific to economics courses.

The reasons given for the gender gap in economics have been various, ranging from the influence of socio-cultural factors to fieldspecific idiosyncrasies such as the lack of female role models, inflexible teaching styles, and an emphasis on topics such as profitmaximization. Anderson et al. (1994) found men earned higher grades in a traditional introductory economics course despite controlling for instructor gender, student mathematics background, and student familiarity with economics. Ballard and Johnson (2005) confirmed this result, concluding that men scored about 1.8 percentage points higher than otherwise equivalent women in introductory microeconomics. A decade later, Bosshardt and Chiang (2016) found that women performed about a third of a standard-deviation worse than otherwise equivalent male students in a general principles of economics course. There is some evidence, however, that course delivery modalities might have various effects on students by gender. Brown and Leidholm (2002) found that women in a live classroom setting scored nearly six percentage points worse; however, for online students, the penalty observed by women was reduced to two percent. Gratton-Lavoie and Stanley (2009) confirm the premium enjoyed by male students in a traditional classroom setting but find it disappears by the end of the semester when students are fully online. They conclude that "the distance-learning environment [is] especially favorable to female students" (p. 12).

It would be unfair to treat the Covid-19 shutdown as experiment in the efficacy of online learning. For example, Adedoyin and Soykan (2020) emphasize the notable pedagogical and preparation differences between traditional online learning and emergency remote teaching, as well as differences in student expectations. University closure to face-to-face learning and the corresponding shift to online instruction constituted a radical experiment. Faculty and students had little time for preparation or training. Thus, while recognizing that Covid-19 induced online instruction should not be considered reflective of online courses generally, we argue that this research provides important clues for researchers seeking to understand the impact of dramatic changes in modality, whether due to Covid-19 closures or other reasons.

To consider how student learning was impacted by the change in course modality due campus closures related to the pandemic, we examine student course grades and performance on standardized pre-and post-tests in semester-long, introductory-level microeconomics, macroeconomics, and business statistics courses at a regional state university. For nearly a decade, our college has required students in introductory business and economics courses to complete a pre-test (first two weeks of semester) and post-test (final week of semester). While each course has a unique test, all have a similar format, being composed of 20 multiple-choice questions. Tests are taken through the university's course management system; a central administrator manages the delivery and scoring of the tests, rather than individual faculty. The tests were written in-house and were approved at the department level; neither the questions nor the format has changed in several years. 7 The standardized content and delivery of these tests provides an unusual opportunity to be able to assess the impact of the coronavirus closure on student learning, especially as many other such exams were significantly altered or cancelled. 8 Pre-and post-test questions are provided in the appendix. Data for this study comprise two categories: (i) student background information and course information as well as final grades provided by the university's Office for Institutional Research, and (ii) assessment data from departmental pre-and post-tests administered through a learning management system (Canvas).

In Table1, we begin by comparing the students affected by the Covid-19 closure to students who completed the same courses in the previous three semesters. 9 Students who withdrew or received an incomplete are not included in the analysis. We note that two incompletes occurred during the Covid-affected semester; none were recorded in the previous three semesters. Twelve withdrawals occurred during the Covid-affected semester; sixteen withdrawals occurred in the previous three semesters.

During the unaffected semesters, 2,261 total students were enrolled across 46 sections of introductory microeconomics, macroeconomics, and statistics. In the Covid-19 affected spring semester, 616 students were enrolled across 18 sections. 10 Grades awarded during the Covid-affected semester were statistically significantly higher across all three courses; overall, students averaged a 2.77 instead of the 2.52 of previous semesters (Table 2 ). Comparing only spring semesters, yields the same result, as students in the Covid-affected spring semester averaged 0.17 of a grade-point higher than students in the previous spring semester. However, essentially the same proportion of students failed in the Covid-affected semester as in previous semesters.

An alternative approach to measuring the impact of the shutdown on student learning is to consider student performance on departmental pre-and post-tests. Comparing the affected semester to the previous three semesters, we find no evidence that students performed more poorly in the semester with the shutdown. In previous semesters, students averaged 60.4 % correct; in the Covid- Though one might be surprised that more students did not withdraw from the spring semester, we attribute this to an institutional policy that allowed students to be able to withdraw after completing the course and being assigned a final grade. Since the risk of failure was substantially lowered, and tuition refunds were not an option, most students chose to persevere. 10 The triad of introductory microeconomics, macroeconomics, and statistics is required for entrance into our business college; students must earn a minimum grade of 2.0 in each course and a 2.5 GPA overall for their pre-business classes. Thus, many students appear in our data set more than once. Of the 2,877 observations (students who enrolled and received a grade), we record 558 students who appear in the data once, 488 students who appear twice, 292 students who appear three times, 99 students who appear four times, 16 students who appear five times, and two students who appear six times. In all regressions, we weight by the number of student occurrences and cluster by student. Overall, we lose a total of 30 observations -28 withdrawals and 2 incompletes.

affected semester, students averaged 61.6 % correct. The mean difference of scores between the post-test and the pre-test across all three course was also essentially unchanged. Mean post-test scores and mean grades may not reflect important changes in the underlying distributions. In Fig. 1 , we compare the distribution of post-test scores and course grades across the semesters using a kernel density plot. Fig. 1 suggests perhaps slightly fewer observations in the left tail in the Covid-affected semester grade distributions, suggesting that the distribution of grades in the Covid-affected semester has shifted right. While we speculated that this could be due to more students withdrawing and/or more students failing to complete the post-tests in the spring semester, the evidence does not bear this out. Students withdrew at roughly equal rates each semester; additionally, approximately the same percentage of students failed to complete the post-tests in each semester. A Kolmogorov -Smirnov equality-of-distributions test of course grades confirms that the distributions are indeed unequal (p < 0.01). The same test examining the difference between post-and pre-test scores gives no evidence that the distribution shifted. 

Our university closed to face-to-face learning on March 13, at the end of the sixth week of a fourteen-week semester. To examine whether the closure had a heterogeneous impact on various subgroups of students, we consider such student characteristics as gender, race, age, university class-status, credits completed, first-generation college student, cumulative GPA, and transfer status, in addition to whether a student was Pell-Grant eligible. A summary of the student data is available in Table 2 ; data reflect our university's position as a medium-sized, public university in the upper-Midwest. Roughly one-third of students in our sample are women, typical of prebusiness and business majors on our campus. Slightly less than 20 percent of students are non-white; this includes students who identify as African-American, Asian, Hispanic, and/or Native American. About 40 percent of students are first-generation college students; about 20 percent are Pell-Grant eligible, e.g., they come from circumstances with a high degree of unmet financial needs. About a quarter of our students transferred from another institution, most commonly from a two-year school. Introductory economics and statistics courses serve primarily first-year and sophomore students. There are no statistically significant differences in the characteristics of students between the two spring semesters as demonstrated in Table 2 , with the exception of course grades as previously discussed.

Regarding the composition of faculty in our study, twelve different instructors taught at least one section of introductory microeconomics, macroeconomics, or statistics between Fall 2018 and Fall 2019. Eleven instructors taught at least one section of these courses in the Covid-affected spring semester. In response to the campus closure to in-person learning, faculty did not change how the pre-or post-tests entered students' grades, nor did they substantially alter the format of exams or quizzes. Teaching proceeded with a mix of synchronous and asynchronous instruction delivered through Canvas, which allowed for both live-streamed lecturing and lecture capture. Faculty characteristics are summarized in Table 3 .

We measure the effect of the Covid-19 shutdown through the estimation of Eq. 1, a variation of the standard regression specification used in the economics of education literature (e.g., Anderson et al., 1994; Ballard and Johnson, 2005; Bosshardt and Chiang, 2016) . Eq. 1 posits that student performance is a function of their academic background, demographic characteristics, and the course in which the student was enrolled.

Eq. (1) is estimated using weighted least squares with clustered standard errors by student and weighted by the number of times an individual student appeared in the sample. The dependent variable (Y) is a measure of academic performance conditional on the student (i), the semester (t), and the course (k). We explore three different measures of student performance. The first is an individual's overall grade earned in a course. The second is a student's raw post-test score. The third is a measure that takes the difference between a student's post-test score and pre-test score. The key explanatory variable, c t , is equal to 1 if the student was enrolled in the semester with the Covid-19 closure, and zero otherwise. Student characteristics and demographic data are represented by the vector X i,t . In creating the specification, we sought to include variables typically used in the literature and known to be associated with student performance. These include gender, race, university class status, cumulative GPA, and the course of enrollment. We also include whether the student is a first-generation college student (i.e., neither parent has a four-year college degree), whether the student transferred to our institution, and their eligibility for Pell-Grants. 11 The regressions with all the data or estimates of grades include course related information, Z i,k,t . In the cases of estimating grades as the dependent variable, we include the post-test scores as they are Note: Standard deviations are in parentheses. The proportions are the student weighted average of each professor type. There were 11 professors in the entire sample.

often used to determine an individual's grade. We also include indicator variables for each course, macroeconomics, microeconomics, or statistics, to control for differences in course material and diagnostic tests as well as controls for instructor age and gender to control for changes in faculty composition across semesters. 12 Finally, we include a dummy variable for students who did not take the pre-test or post-test. Unlike many studies of online learning, we do not have to address the issue of student course selection; the fall-semester offered only face-to-face courses. All spring courses were initially scheduled as face-to-face, and all courses moved online on the same date.

We use a weighted least squares approach to estimate three versions of Eq. 1 that consider the determinants of student performance as measured by course grade, post-test score, and the difference between the pre-and post-test. Because some students appear in our sample multiple times, having taken several of the courses under consideration, we cluster standard errors by student and weight by Note: Standard errors in parentheses and clustered by student. Estimates are de-rived from Weighted Least Squares where observations in the regression are weighted by the number of times a student is in the sample. The comparison category for class status is sophomore, the comparison course is Microeconomics. * Indicates statistical significance at the 10% level. ** Indicates statistical significance at the 5% level. *** Indicates statistical significance at the 1% level.

12 Due to limited variation in changes in teaching modality at the instructor level, those variables have been omitted from use in the parsimonious model. When included, the estimated effects are not statistically significant, with the exception of a weakly significant positive effect of virtual teaching experience on student grade. These results are available from the authors upon request.

the number of appearances. 13 The results are reported in Table 4 . We remind readers that grade is measured on a typical four-point GPA scale; post-tests and pre-tests are measured as the proportion of questions scored correct, or in [0,1].

Our results comport well with the economic education literature generally. We confirm the result of multiple studies on the importance of cumulative GPA as a control for academic aptitude and effort (Anderson et al., 1994; Anstine and Skidmore, 2005; Ballard and Johnson, 2005; Gratton-Lavoie and Stanley, 2009 ). Cumulative GPA is positively and statistically significantly associated with all measures of student performance. There were no statistically significant differences by race, Pell-Grant eligibility, or first-generation college student. There is modest evidence that seniors may do more poorly on the post-test and difference between the post-test and difference in post and pre-test scores. Transfer students earned about 0.147 of a grade-point higher and scored 1.6 percent better on the post-test than non-transfer students; these differences were statistically significant at the 5% level. As expected, students who failed to take the post-test saw much lower measures of learning outcomes across the board as they didn't get the extra credit/missed an assignment in relation to the grade metric while they received a zero on the post-test. Of the instructor covariates, we find some sensitivity by student outcome to instructor gender and age. Students of female professors can expect to earn 0.098 of a grade-point lower and score about 1.6 percent less on the post-test; there is no impact for the difference of pre-and post-test scores. Younger instructors are associated with notably lower grades but higher post-test scores.

Women receive about 0.104 of a grade-point lower than otherwise comparable men (standard error = 0.038). Additionally, we observe that women score 1.3 percent lower on the post-test (standard error = 0.006). If we examine this outcome by course, we see that women received lower grades in microeconomics and macroeconomics, but not in statistics. 14 Results are reported by course in the first three tables of the appendix. Other factors held constant, women see 0.235 of a grade-point lower in Macroeconomics and 0.162 of a grade-point lower in Microeconomics (standard errors of 0.050 and 0.059, respectively). The pervasiveness of this finding in the economics literature has long been a source of dismay for economists (Block and Fels, 1974; Emerson et al., 2018; Siegfried, 1979) . We find it particularly concerning as performance in introductory courses has been shown to affect the likelihood of majoring in economics (Calkins and Welki, 2006; Rask and Tiefenthaler, 2008) . What we do not find is any evidence that the Covid-affected semester impacted measures of student learning outcomes in aggregate (Table 4 ). To examine this result more carefully, we consider courses individually and then consider whether this affect is heterogeneous by student characteristics.

In considering whether the Covid-19 shutdown affected different students differently, we interact the Covid-19 indicator, c t , with the other independent variables used in Eq. 1. Specifically, we estimate the following, where all terms are equivalent to their specification in Eq. 1, and the interaction coefficients are represented by superscript "c", or Y i,k,t = β + δ c t + γ X i,t + ρ k Z i,k,t + c t (γ c X i,t + ρ k c Z i,k,t ) + ε i,k,t .

(2)

Eq.

(2) is estimated using weighted least squares with clustered standard errors by student and weighted by the number of times an individual student appeared in the sample. The results for the entire sample, by Grade, Post-Test, and Difference (Post-Test -Pre-Test), are provided in Table 5 .

The baseline results are relatively unchanged from Table 4 . Students still see gains across the board from holding a higher cumulative GPA: a one-point increase in cumulative GPA is associated with an increase of 0.858 in course grade in the non-Covid-affected semesters. Similarly, a one-point increase in cumulative GPA is associated with 6 percentage points more correct on the post-test and a gain from pre-to post-test of 2.6 percentage points. We also observe similar statistically significant results relative to gender (student and professor), whether a student transferred as well as factors with large standard errors relative to their estimated association. What is new, and the focus of the second model, is the effect of the Covid-related shutdown on students with these characteristics. For example, in the Covid-affected semester, the GPA bonus is reduced by 0.226 of a grade-point; in other words, a one-point increase in cumulative GPA is associated with only a 0.632 increase in course grade. However, we did not see this on post-test scores or difference between the pre-and post-test scores.

Despite what we might expect from previous studies of online learning, we did not find that either first-year students or firstgeneration students performed worse in the Covid-19 affected semester across the board, with the only notable finding that firstyear students scored about 4.2 percentage points less on the post-test that semester (standard error = 0.014). Pell-grant eligible students were also seemingly unaffected, as were nonwhite students.

Of particular interest, however, is that the usual grade penalty observed for women was eliminated in the Covid-affected semester. This result comports well with what Gratton-Lavoie and Stanley (2009) found in their studythat men outperform women in 13 As noted previously, many students appear in our sample multiple times because they took multiple courses. Since we are interested in determinants of student success across courses and across semesters, we retain these multiple observations as they represent unique course enrollments. However, we weight the observations by the inverse of the number of times and individual is in the regression and cluster the errors by student to avoid problems of interdependence of errors. We have chosen not to employ a fixed-effects strategy to correct for the repeated observation of the some of the students in our sample because the strategy causes us to lose the ability to investigate many of the variables of interest (e.g., gender, race). For completeness, however, we did estimate the same regressions in Table 4 with student fixed effects and found no qualitative difference in the results. 14 In statistics, women earn essentially equivalent grade. They also perform indistinguishably from men on the post-test and when examining the difference between pre-and post-test scores. traditional classroom settings. However, while men evidenced an early grade advantage in online courses, the advantage disappeared by the end of the semester. We find that in the pre-Covid semesters, women could expect to earn a 0.173 of a grade-point less (standard error = 0.044), on average, across all courses. However, in the Covid-affected semester, women would actually expect to earn 0.052 of a grade-point higher (the combination of the estimated coefficient for female and for the female-Covid interaction). The change in course delivery modality seems to have had a positive impact on female students. If we examine the outcome by course, as reported in the latter three tables in the appendix, we see that the grade penalties observed for women in introductory microeconomics and macroeconomics disappeared. In macroeconomics, the penalty became statistically insignificantly different from zero. In microeconomics, women generally received 0.209 of a grade-point less than otherwise equivalent male students before Covid. However, in the Covid-affected semester, women gained 0.09 of a grade-point. 15 We see no effect in statistics courses, where women regularly earn comparable grades to men.

The closure of university campuses to face-to-face learning in response to the dangers posed by the novel coronavirus was radical and unprecedented. Faculty, students, and administrators all formed anecdotal impressions of the effectiveness of the shift to online learning; in many cases, however, institutions lack hard data by which to evaluate outcomes as university contingency plans suspended standardized testing and assessment In this paper, we report on the impact of the Covid-19 shutdown on various student-learning outcomes in introductory microeconomics, macroeconomics, and statistics courses in at a regional state university. Across a sample of more than two thousand students, we find little evidence that the first shutdown in Spring 2020 significantly affected studentlearning outcomes, measured by course grade or a fixed diagnostic test provided to all students in the course. Students demonstrated only small losses on standardized post-tests; despite this, in many cases, students earned higher grades in the Covid-19 affected semester. One important concern is that the Covid-19 shutdown may have had an unequal impact on different population subgroups. We find no evidence that first-generation college students, transfer students, non-white students, or Pell-Grant eligible students were negatively affected.

While this is largely a story of 'no harm,' one particularly interesting outcome we document is the leveling of the playing field for women students. In the semesters prior to the pandemic, women at our institution regularly earned lower grades in introductory economics compared to male students, despite comparable academic backgrounds and experience. In the Covid-affected semester, this penalty disappeared. Previous studies have identified similar outcomes for women in online economics courses (Brown and Leidholm, 2002; Gratton-Lavoie and Stanley, 2009 ). Some suggest the explanation lies in the fact that online courses reward effort at a higher rate and that women tend to exert more effort in their course work (Hadsell, 2002) . Others suggest the anonymity of online learning reduces instructors' implicit bias (Carlana, 2019) . While we cannot infer causality in this case, the finding does suggest new pathways for considering how to address the gender imbalance in economics.

Three important caveats to this discussion should be noted. First, our study only considered students in introductory-level courses at an aggregate level. Although a more nuanced, head-to-head comparison of student performance by topic and by before-and-after the shut-down would be interesting, we are unable to delve this deeply due to the subtleties of the professorial changes, the varying rates at which instructors cover material, the 'elective' topics instructors chose to cover, and potential issues of student recall. Second, we did not attempt to examine the sophistication of student knowledge in economics or statistics, but rather general, subject-knowledge competency that can be measured predominantly through a multiple-choice test. Third, we are only looking at outcomes in economics at a regional state university in the United States in the first semester affected by the Covid pandemic. That said, we do not find overwhelming evidence that institutions should fear delayed re-opening or re-closure on the grounds that online instruction is associated with significantly poorer quality educational outcomes. 

a You are planning to study eight hours this week for your economics final and are considering studying a ninth hour. You should: b compare the benefits of one more hour of study with the cost of one less hour of sleep compare the the benefits of one more hour of study with the cost of one less hour of studying calculus c compare the benefits of one more hour of study with the cost of one less hour of work at your part time job d make your decision based on the next best alternative use of your time compared to the benefit of one more hour of study 2. Increase in total output realized when individuals specialize in particular tasks and trade are known as: a the gains from trade b the profits obtained from sales of a good or service c the multiplier effect d an excess supply 3. Suppose there are a series of large forest fires, which affect the lumber industry while, at the same time, consumers prefer less wooden furniture. The wooden furniture market would experience: a An increase in price and an increase in quantity. b An increase in quantity and an indeterminate change in price. c A decrease in quantity and an indeterminate change in price. d An increase in price and an indeterminate change in quantity.

a its production possibility frontier shifts outward b its production possibility frontier shifts inward c it necessarily produces at an inefficient point inside its production possibility frontier d it has moved to a more consumer-oriented position on its production possibility frontier 5. GDP is the total dollar value of all a intermediate goods and service produced in the economy in a given period b final goods and services produced by U.S. citizens c final goods and services produced in the economy in a given period d government production in a given period 6. If the aggregate price level increased by 15 % and your nominal wage rose from $10 to $11 then your real wage a increased b decreased c remained constant d cannot be determined without more information 7. Those who are interested in assessing the relative standard of living of different countries over a given time period are most likely to look at: a Percentage change in GDP. b Population. c GDP. d Per capita GDP.

a An increase in aggregate demand. b A decrease in aggregate demand. c An increase in short-run aggregate supply. d An increase in long-run aggregate supply.

9. To be counted as unemployed by the Bureau of Labor Statistics, one a must be actively looking for a job and working less than 2 h per week b must be out of work and be actively looking for a job c must be out of work but not necessarily actively looking for a job d be actively looking for a job and have at least a high-school diploma or its equivalent 10. Worsened consumer confidence would:

a Shift the aggregate demand curve leftward. b Shift the aggregate demand curve rightward. c Move the economy down along the aggregate demand curve. d Move the economy up along the aggregate demand curve. 11. A recession can be defined as a when aggregate output rises relative to its long-term trend. b when economic output is less than its potential c when aggregate prices are falling d when unemployment is below the natural rate of unemployment 12. In which of the following cases would the aggregate supply curve shift to the right? a American consumers suddenly acquire a greater taste for Japanese products, ceteris paribus. b The stock markets totters on the brink of panic as people attempt to sell a record -breaking volume of securities for whatever they will bring. c The value of the dollar plummeted on international currency markets, causing foreigners to buy more American goods. d Productivity in many US industries increases because of technological advances.

13.An increase in the money supply that leads to a decrease in interest rates is likely to result in a a decrease in consumption spending. b a decrease in investment spending. c an increase in investment spending. d an increase in government spending.

14. Because of diminishing returns, an economy can continue to increase real GDP per hour worked only if a there continues to be decreases in capital per hour worked b there is technological change c there are decreases in human capital d the per-worker production function shifts downward 15. The key factors influencing productivity in the long-run are everything EXCEPT a the enforcement of property rights b fixed prices and interest rates c political stability d increases in technology and human capital per worker 16. An automatic stabilizer that works when the economy contracts is: a a rise in tax receipts. b a fall in government purchases. c a discretionary decrease in government purchases. d a rise in government transfers as more people receive unemployment insurance benefits 17. If government spending increases and taxes decrease, then: a the government's implicit liabilities will increase. b the government will expect a fall in economic output. c the public debt will increase d the public debt will decrease 18. The primary tools that the Federal Reserve System can use to increase the money supply and stimulate the economy are: 19. If real interest rates rise while nominal interest rates fall, then expected inflation has: a risen b fallen c stayed the same d decreased and then offset by an equal increase 20. If the exchange rate changes from 100 yen per dollar to 120 yen per dollar, a the dollar has depreciated. b the dollar has appreciated. c the yen has appreciated. d none of the above. Which Area represents producer surplus when the price is P2? a. BCG b. ACH c. ABGD d. AHGB 4. If UWO students suffer a drop in income, what will happen to the market for UWO t-shirts (assuming they're a normal good)?

a The price will rise, the quantity will fall b The price will rise, the quantity will rise c The price will fall, the quantity will rise d The price will fall, the quantity will fall 5. If drought conditions greatly reduced the world's cotton crop, what will happen to the market for UWO t-shirts?

a The price will rise, the quantity will fall b The price will rise, the quantity will rise c The price will fall, the quantity will rise d The price will fall, the quantity will rise 6. If demand for potatoes is inelastic, a 4% increase in the price of potatoes will:

a decrease the quantity of potatoes demanded by more than 4% b decrease the quantity of potatoes demanded by less than 4% c increase the quantity of potatoes demanded by less than 4% d increase the quantity of potatoes demanded by more than 4% 7. When output is 100 units, the firm's total fixed cost is $500. What will this firm's total fixed cost be if output doubles to 200 units? a. $1000 b. $500 c. $250 d. Can't tell from the information provided 8. Suppose a 4% increase in income in the economy leads to an 8% increase in soft drink consumption. Then the income elasticity of demand is: a negative, so soft drinks are an inferior good b negative, so soft drinks are a normal good c positive, so soft drinks are a normal good d positive, so soft drinks are an inferior good 9. m the market equilibrium price for taxi rides is $2 per mile. Then a price celling of $1 per mile would result in: a a surplus of tax service per year b a decrease in the availability of taxis c a decrease in the number of people hailing taxis d no change in taxi service per year 10. Students at a Midwestern university pay $40 a year for parking permits, but many complain they can't find an available parking space. This suggests that: a parking permits are under priced b student incomes are too low c parking permits are overpriced d the university should make parking free 11. The supply of walnuts is very price inelastic, but the demand for walnuts is quite price elastic. If the government imposes a tax on walnut sales, then:

a walnut producers will have to absorb all the tax. b walnut consumers will end up paying the tax, even though the government collects it from the walnut producers. c walnut producers will pass almost all of the tax onto walnut consumers. d walnut producers will only pass a small fraction of the tax onto walnut consumers 12. A firm in a competitive market has the following cost structure If the market price is $16, this firm will:

a produce 4 units of output in the short run and exit in the long run. b produce 5 units of output in the short run and exit in the long run. c produce 5 units of output in the short run and face competition from new market entrants in the long run. d shut down in the short run and exit in the long run.

13. What do wages paid to factory workers, interest paid on a bank loan, forgone interest, and the purchase of component parts have in common?

a None are either implicit or explicit costs. b All are opportunity costs. c All are implicit costs. d All are explicit costs.

14. Game theory:

a is the analysis of how people (or firms) behave in strategic situations b is best suited for analyzing purely competitive markets c reveals that mergers between rival firms are self-defeating d reveals that price-fixing among firms reduces profits 15. Refer to the figure below.

Regarding a firm's costs, this figure suggests that: a when marginal cost is zero, total cost is at a minimum. b when marginal cost lies above average cost, average cost is rising. c when marginal cost lies below average cost, average cost is rising. d when total cost is at a maximum, so is marginal cost and average cost.

16. The competitive market price has recently dropped below your average total cost, but above average variable cost. You should: a continue to produce in the short run, since your earning an economic profit. b continue to produce in the short run, since you're in the breakeven range. c continue to produce in the short run, even though you're losing money. d shut down in the short run, to avoid losing money.

a exit or entry drives firms' economic profits to zero. b firms are producing at their maximum capacities. c quantity supplied consistently exceeds quantity demanded. d the market price more than covers average total cost 18. Economist say monopoly is inefficient because: a monopolists try to price gouge consumers. b monopolists produce and sell less output than a competitive market would. c monopolies have only one firm. d monopolists frequently engage in price discrimination.

19. Use the figure below.

To maximize profits, this firm should charge at: a. at $A. b. at $B. c. at $C. d. at $0. 20. Which of the following would not be considered price discrimination? a A car dealer selling the same type of car to different people at different prices. b Airline fares are cheaper if you reserve several weeks in advance. c The price of lettuce is 59 cents a head or two for a dollar. d The price of a brand-name prescription drug is higher than the price of a generic brand.

1. In a questionnaire, students are asked to mark whether they live on campus or off campus. This is an example of a: a quantitative variable. b qualitative variable. c none of these alternatives is correct. d qualitative or quantitative variable, depending on how the respondents answered the question.

2. If the variance of a data set is correctly computed with the formula using n-1 in the denominator, which of the following is true?

a The data set could be either a sample or a population. b The data set is a sample. c The data set is from a census. d The data set is a population. 5. As the sample size increases, the:

• standard error decreases.

• standard deviation of the population decreases.

• standard error of the mean increases.

• population mean increases.

6. x is a normally distributed random variable with a mean of 22 and a standard deviation of 5. The probability that x is less than 9.7 is: a 0.0069 b 0.4931 c 0.000 d 0.9931 7. A simple random sample of 64 observations was taken from a large population. The sample mean and standard deviation were determined to be 320 and 120 respectively. The standard error of the sample mean is: a. 5 b. 1.875 c. 40 d. 15 8. The t distribution is applicable when:

• the variance of the population is known.

• the sample standard deviation is used to estimate the population standard deviation.

• the standard deviation of the population is known.

• the population has a mean of less than 30.

9. In hypothesis testing if the null hypothesis is rejected,

• the data must have been accumulated incorrectly.

• the alternative hypothesis is likely to be true.

• the sample size has been too small.

• no conclusions can be drawn from the test.

10. When the p-value is used for hypothesis testing, the null hypothesis is rejected if:

11. The school's newspaper reported that the proportion of students majoring in business is more than 30 %. You plan on taking a sample to test this claim. The correct set of hypotheses is: 12. Referring to the table below, which of the following statements is true?

The following Microsoft Excel table is obtained when "Score received on an exam (measured in percentage points)" (Y) is regressed on "percentage attendance" (X) for 22 students in an Economics and Business Statistics course. a If attendance increases by 1%, the estimated score received will increase by 0.341 percentage points. b If attendance increases by 1%, the estimated score received will increase by 39.39 percentage points. c If the score received increases by 39.39 % the estimated attendance will go up 1%. d If attendance increases by 0.341 %, the estimated score received will increase by 1 percentage point.

The table above, the proportion of the variation in "Score received on the exam" that can be explained by the variation in attendance is given by: a Standard error. b Intercept. c The P-value d R square

Using the Excel output reported above, if we were to test to see whether "Attendance" is statistically significantly associated with "Score received on the exam," we would conclude that we should: a Accept the null hypothesis and conclude that Attendance IS statistically significantly associated with "Score received on the exam." b Reject the null hypothesis and conclude that Attendance IS statistically significantly associated with "Score received on the exam." c Reject the null hypothesis and conclude that Attendance is NOT statistically significantly associated with "Score received on the exam." d Fail to reject the null hypothesis and conclude that we cannot say that Attendance is statistically significantly associated with "Score received on the exam."

15. The average gasoline price of one of the major oil companies in Europe has been $1.25 per liter. Recently, the company has undertaken several efficiency measures in order to reduce prices. A random sample liter. Furthermore, assume that the standard deviation of the population is $0.14.

Using the information in the paragraph above, the value of the test statistic for this hypothesis test is:

16. Based on the test statistic computed in the previous question (and using the information in the paragraph above), we would:

a Fail to reject the null hypothesis and conclude prices are higher. b Reject the null hypothesis and conclude that prices are lower. c Fail to reject the null hypothesis and conclude that prices are lower.

d Accept the null hypothesis and conclude that we cannot say whether prices are lower.

17. When outliers are present in continuous data, the (fill in the blank) is typically the best choice to locate the center of data. • Accept that the null hypothesis is true.

• Reject the null hypothesis.

• Fail to reject the null hypothesis.

• Accept that the null hypothesis is false.

See Tables A1-A6 . Note: Standard errors in parentheses and clustered by student. Estimates are de-rived from Weighted Least Squares where observations in the regression are weighted by the number of times a student is in the sample. The comparison category for class status is sophomore. * p < .1, **p < .05, ***p < .01. Note: Standard errors in parentheses and clustered by student. Estimates are derived from Weighted. Least Squares where observations in the regression are weighted by the number of times a student is in the sample. The comparison category for class status is sophomore. * p < .1. ** p < .05. *** p < .01. Note: Standard errors in parentheses and clustered by student. Estimates are derived from Weighted. Least Squares where observations in the regression are weighted by the number of times a student is in the sample. The comparison category for class status is sophomore. * p < .1. ** p < .05. *** p < .01. Note: Standard errors in parentheses and clustered by student. Estimates are derived from Weighted. Least Squares where observations in the regression are weighted by the number of times a student is in the sample. The comparison category for class status is sophomore. * p < .1. ** p < .05. *** p < .01.

Covid-19 pandemic and online learning: the challenges and opportunities

The impact of laptop-free zones on student performance and attitudes in large lectures

Research on teaching economics to undergraduates

A randomized assessment of online learning

The determinants of success in university introductory economics courses

A small sample study of traditional and online courses with sample selection adjustment

Research in online and blending learning in the business disciplines: key findings and future directions

Gender, expectations, and grades in microeconomics

5 myths about remote teaching in the covid-19 crisis

Connections matter: how interactive peers affect students in online college courses

Virtual classrooms: how online college courses affect student success

Economics, sex, and gender

A note on sex and economic education

Lecture Capture Learning: Do Students Perform Better Compared to Face-to-Face Classes?

Interactive leanring online at public universities: evidence from a six-campus randomized trial

Can Web Courses Replace the Classroom in Principles of Microconomics?

Factors that influence the choice of major: why some students never considered economics

Implicit stereotypes: evidence from teachers' gender Bias

No significant distance' between face-to-Face and online instruction: evidence from principles of economics

The gender gap in economics degrees: an investigation of the role model and quantitative requirements hypotheses

Is It Live or Is It the Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning

Recommendations for mandatory online assessment in Higher education during the Covid-19 crisis context

Teaching and learning principles of microeconomics online: an empirical assessment

Not for want of trying: effort and success of women in principles of microeconomics

Student and professor gender effects in introductory business statistics

How Coronavirus Dramatically Changed College for Over 14 Million Students

A meta-analysis of the gender gap in performance in collegiate economics courses

Does Classroom Time Matter? A Randomized Field Experiment of Hybrid and Traditional Lecture Formats in Economics

Pandemic communication: a new challenge for higher education

The Best Way to Stop Cheating in Online Courses? 'Teach Better

Planning for higher education institutions: chaos and the Covid-19 pandemic

Computers and productivity: evidence from laptop use in the college classroom

Does the choice of introductory microeconomics textbook matter?

The role of grade sensitivity in explaining the gender imbalance in undergraduate economics

Laptop multitasking hinders classroom learning for both users and nearby peers

Male-female differences in economic education: a survey

Advice for Faculty Members in a Turbulent Time

What's in a grade? Grading policies and practices in principles of economics

A little more than chalk and talk: results from a third national survey of teaching methods in undergraduate economics courses

Teaching and assessment methods in undergraduate economics: a fourth national quinquennial survey

The impact of online learning on students' course outcomes: evidence from a larger community and technical college system