key: cord-0065354-8dxz1oa2
authors: Chisadza, Carolyn; Clance, Matthew; Mthembu, Thulani; Nicholls, Nicky; Yitbarek, Eleni
title: Online and face‐to‐face learning: Evidence from students’ performance during the Covid‐19 pandemic
date: 2021-03-31
journal: Afr Dev Rev
DOI: 10.1111/1467-8268.12520
sha: dcacd9855942618c76eba9cdc38c82e6ad864b2d
doc_id: 65354
cord_uid: 8dxz1oa2

This study investigates the factors that predict students' performance after transitioning from face‐to‐face to online learning as a result of the Covid‐19 pandemic. It uses students' responses from survey questions and the difference in the average assessment grades between pre‐lockdown and post‐lockdown at a South African university. We find that students' performance was positively associated with good wifi access, relative to using mobile internet data. We also observe lower academic performance for students who found transitioning to online difficult and who expressed a preference for self‐study (i.e. reading through class slides and notes) over assisted study (i.e. joining live lectures or watching recorded lectures). The findings suggest that improving digital infrastructure and reducing the cost of internet access may be necessary for mitigating the impact of the Covid‐19 pandemic on education outcomes.

ways in which inequality is perpetuated in South Africa, and Africa as a whole, is through access to education (Anyanwu, 2016; Coetzee, 2014; Tchamyou et al., 2019) ; therefore, understanding the obstacles that students face in transitioning to online learning can be helpful in ensuring more equal access to education.

Using students' responses from survey questions and the difference in the average grades between pre-lockdown and post-lockdown, our findings indicate that students' performance in the online setting was positively associated with better internet access. Accessing assisted study material, such as narrated slides or recordings of the online lectures, also helped students. We also find lower academic performance for students who reported finding transitioning to online difficult and for those who expressed a preference for self-study (i.e. reading through class slides and notes) over assisted study (i.e. joining live lectures or watching recorded lectures). The average grades between pre-lockdown and post-lockdown were about two points and three points lower for those who reported transitioning to online teaching difficult and for those who indicated a preference for self-study, respectively. The findings suggest that improving the quality of internet infrastructure and providing assisted learning can be beneficial in reducing the adverse effects of the Covid-19 pandemic on learning outcomes.

Our study contributes to the literature by examining the changes in the online (post-lockdown) performance of students and their F2F (pre-lockdown) performance. This approach differs from previous studies that, in most cases, use between-subject designs where one group of students following online learning is compared to a different group of students attending F2F lectures (Almatra et al., 2015; Brown & Liedholm, 2002) . This approach has a limitation in that that there may be unobserved characteristics unique to students choosing online learning that differ from those choosing F2F lectures. Our approach avoids this issue because we use a within-subject design: we compare the performance of the same students who followed F2F learning Before lockdown and moved to online learning during lockdown due to the Covid-19 pandemic. Moreover, the study contributes to the limited literature that compares F2F and online learning in developing countries.

Several studies that have also compared the effectiveness of online learning and F2F classes encounter methodological weaknesses, such as small samples, not controlling for demographic characteristics, and substantial differences in course materials and assessments between online and F2F contexts. To address these shortcomings, our study is based on a relatively large sample of students and includes demographic characteristics such as age, gender and perceived family income classification. The lecturer and course materials also remained similar in the online and F2F contexts. A significant proportion of our students indicated that they never had online learning experience before. Less than 20% of the students in the sample had previous experience with online learning. This highlights the fact that online education is still relatively new to most students in our sample.

Given the global experience of the fourth industrial revolution (4IR), 2 with rapidly accelerating technological progress, South Africa needs to be prepared for the possibility of online learning becoming the new norm in the education system. To this end, policymakers may consider engaging with various organizations (schools, universities, colleges, private sector, and research facilities) To adopt interventions that may facilitate the transition to online learning, while at the same time ensuring fair access to education for all students across different income levels. 3

Online learning is a form of distance education which mainly involves internet-based education where courses are offered synchronously (i.e. live sessions online) and/or asynchronously (i.e. students access course materials online in their own time, which is associated with the more traditional distance education). On the other hand, traditional F2F learning is real time or synchronous learning. In a physical classroom, instructors engage with the students in real time, while in the online format instructors can offer real time lectures through learning management systems (e.g. Blackboard Collaborate), or record the lectures for the students to watch later. Purely online courses are offered entirely over the internet, while blended learning combines traditional F2F classes with learning over the internet, and learning supported by other technologies (Nguyen, 2015) . 2 The 4IR is currently characterized by increased use of new technologies, such as advanced wireless technologies, artificial intelligence, cloud computing, robotics, among others. This era has also facilitated the use of different online learning platforms (https://www.brookings.edu/research/the-fourth-industrialrevolution-and-digitization-will-transform-africa-into-a-global-powerhouse/). 3 Note that we control for income, but it is plausible to assume other unobservable factors such as parental preference and parenting style might also affect access to the internet of students.

Moreover, designing online courses requires several considerations. For example, the quality of the learning environment, the ease of using the learning platform, the learning outcomes to be achieved, instructor support to assist and motivate students to engage with the course material, peer interaction, class participation, type of assessments (Paechter & Maier, 2010) , not to mention training of the instructor in adopting and introducing new teaching methods online (Lundberg et al., 2008) . In online learning, instructors are more facilitators of learning. On the other hand, traditional F2F classes are structured in such a way that the instructor delivers knowledge, is better able to gauge understanding and interest of students, can engage in class activities, and can provide immediate feedback on clarifying questions during the class. Additionally, the designing of traditional F2F courses can be less time consuming for instructors compared to online courses (Navarro, 2000) .

Online learning is also particularly suited for nontraditional students who require flexibility due to work or family commitments that are not usually associated with the undergraduate student population (Arias et al., 2018) . Initially the nontraditional student belonged to the older adult age group, but with blended learning becoming more commonplace in high schools, colleges and universities, online learning has begun to traverse a wider range of age groups. However, traditional F2F classes are still more beneficial for learners that are not so self-sufficient and lack discipline in working through the class material in the required time frame (Arias et al., 2018) .

For the purpose of this literature review, both pure online and blended learning are considered to be online learning because much of the evidence in the literature compares these two types against the traditional F2F learning. The debate in the literature surrounding online learning versus F2F teaching continues to be a contentious one. A review of the literature reveals mixed findings when comparing the efficacy of online learning on student performance in relation to the traditional F2F medium of instruction (Lundberg et al., 2008; Nguyen, 2015) . A number of studies conducted Before the 2000s find what is known today in the empirical literature as the "No Significant Difference" phenomenon (Russell & International Distance Education Certificate Center (IDECC), 1999) . The seminal work from Russell and IDECC (1999) involved over 350 comparative studies on online/distance learning versus F2F learning, dating back to 1928. The author finds no significant difference overall between online and traditional F2F classroom education outcomes. Subsequent studies that followed find similar "no significant difference" outcomes (Arbaugh, 2000; Fallah & Ubell, 2000; Freeman & Capper, 1999; Johnson et al., 2000; Neuhauser, 2002) . While Bernard et al. (2004) also find that overall there is no significant difference in achievement between online education and F2F education, the study does find significant heterogeneity in student performance for different activities. The findings show that students in F2F classes outperform the students participating in synchronous online classes (i.e. classes that require online students to participate in live sessions at specific times). However, asynchronous online classes (i.e. students access class materials at their own time online) outperform F2F classes.

More recent studies find significant results for online learning outcomes in relation to F2F outcomes. On the one hand, Shachar and Yoram (2003) and Shachar and Neumann (2010) conduct a meta-analysis of studies from 1990 to 2009 and find that in 70% of the cases, students taking courses by online education outperformed students in traditionally instructed courses (i.e. F2F lectures). In addition, Navarro and Shoemaker (2000) observe that learning outcomes for online learners are as effective as or better than outcomes for F2F learners, regardless of background characteristics. In a study on computer science students, Dutton et al. (2002) find online students perform significantly better compared to the students who take the same course on campus. A meta-analysis conducted by the US Department of Education finds that students who took all or part of their course online performed better, on average, than those taking the same course through traditional F2F instructions. The report also finds that the effect sizes are larger for studies in which the online learning was collaborative or instructor-driven than in those studies where online learners worked independently (Means et al., 2010) .

On the other hand, evidence by Brown and Liedholm (2002) based on test scores from macroeconomics students in the United States suggest that F2F students tend to outperform online students. These findings are supported by Coates et al. (2004) who base their study on macroeconomics students in the United States, and Xu and Jaggars (2014) who find negative effects for online students using a data set of about 500,000 courses taken by over 40,000 students in Washington. Furthermore, Almatra et al. (2015) compare overall course grades between online and F2F students for a Telecommunications course and find that F2F students significantly outperform online learning students. In an experimental study where students are randomly assigned to attend live lectures versus watching the same lectures online, Figlio et al. (2013) observe some evidence that the traditional format has a positive effect compared to online format. Interestingly, Callister and Love (2016) specifically compare the learning outcomes of online versus F2F skillsbased courses and find that F2F learners earned better outcomes than online learners even when using the same technology. This study highlights that some of the inconsistencies that we find in the results comparing online to F2F learning might be influenced by the nature of the course: theory-based courses might be less impacted by in-person interaction than skills-based courses.

The fact that the reviewed studies on the effects of F2F versus online learning on student performance have been mainly focused in developed countries indicates the dearth of similar studies being conducted in developing countries. This gap in the literature may also highlight a salient point: online learning is still relatively underexplored in developing countries. The lockdown in South Africa therefore provides us with an opportunity to contribute to the existing literature from a developing country context.

South Africa went into national lockdown in March 2020 due to the Covid-19 pandemic. Like most universities in the country, the first semester for undergraduate courses at the University of Pretoria had already been running since the start of the academic year in February. Before the pandemic, a number of F2F lectures and assessments had already been conducted in most courses. The nationwide lockdown forced the university, which was mainly in-person teaching, to move to full online learning for the remainder of the semester. This forced shift from F2F teaching to online learning allows us to investigate the changes in students' performance.

Before lockdown, classes were conducted on campus. During lockdown, these live classes were moved to an online platform, Blackboard Collaborate, which could be accessed by all registered students on the university intranet ("ClickUP"). However, these live online lectures involve substantial internet data costs for students. To ensure access to course content for those students who were unable to attend the live online lectures due to poor internet connections or internet data costs, several options for accessing course content were made available. These options included prerecorded narrated slides (which required less usage of internet data), recordings of the live online lectures, PowerPoint slides with explanatory notes and standard PDF lecture slides.

At the same time, the university managed to procure and loan out laptops to a number of disadvantaged students, and negotiated with major mobile internet data providers in the country for students to have free access to study material through the university's "connect" website (also referred to as the zero-rated website). However, this free access excluded some video content and live online lectures (see Table 1 ). The university also provided between 10 and 20 gigabytes of mobile internet data per month, depending on the network provider, sent to students' mobile phones to assist with internet data costs.

High data costs continue to be a contentious issue in Africa where average incomes are low. Gilbert (2019) reports that South Africa ranked 16th of the 45 countries researched in terms of the most expensive internet data in Africa, at US$6.81 per gigabyte, in comparison to other Southern African countries such as Mozambique (US$1.97), Zambia (US $2.70), and Lesotho (US$4.09). Internet data prices have also been called into question in South Africa after the Competition Commission published a report from its Data Services Market Inquiry calling the country's internet data pricing "excessive" (Gilbert, 2019) .

We use a sample of 395 s-year students taking a macroeconomics module in the Economics department to compare the effects of F2F and online learning on students' performance using a range of assessments. The module was an introduction to the application of theoretical economic concepts. The content was both theory-based (developing economic growth models using concepts and equations) and skill-based (application involving the collection of data from online data sources and analyzing the data using statistical software). Both individual and group assignments formed part of the assessments. Before the end of the semester, during lockdown in June 2020, we asked the students to complete a survey with questions related to the transition from F2F to online learning and the difficulties that they may have faced. For example, we asked the students: (i) how easy or difficult they found the transition from F2F to online lectures; (ii) what internet options were available to them and which they used the most to access the online prescribed work; (iii) what format of content they accessed and which they preferred the most (i.e. self-study material in the form of PDF and PowerPoint slides with notes vs. assisted study with narrated slides and lecture recordings); (iv) what difficulties they faced accessing the live online lectures, to name a few. Figure 1 summarizes the key survey questions that we asked the students regarding their transition from F2F to online learning.

Before the lockdown, the students had already attended several F2F classes and completed three assessments. We are therefore able to create a dependent variable that is comprised of the average grades of three assignments taken before lockdown and the average grades of three assignments taken after the start of the lockdown for each student. Specifically, we use the difference between the post-and pre-lockdown average grades as the dependent variable. However, the number of student observations dropped to 275 due to some students missing one or more of the assessments. The lecturer, content and format of the assessments remain similar across the module. We estimate the following equation using ordinary least squares (OLS) with robust standard errors: 

where Y i is the student's performance measured by the difference between the post and pre-lockdown average grades. B represents the vector of determinants that measure the difficulty faced by students to transition from F2F to online learning. This vector includes access to the internet, study material preferred, quality of the online live lecture sessions and pre-lockdown class attendance. X is the vector of student demographic controls such as race, gender and an indicator if the student's perceived family income is below average. The ε i is unobserved student characteristics. Table 2 gives an overview of the sample of students. We find that among the black students, a higher proportion of students reported finding the transition to online learning more difficult. On the other hand, more white students reported finding the transition moderately easy, as did the other races. According to Coetzee (2014) , the quality of schools can vary significantly between higher income and lower-income areas, with black South Africans far more likely to live in lower-income areas with lower quality schools than white South Africans. As such, these differences in quality of education from secondary schooling can persist at tertiary level. Furthermore, persistent income inequality between races in South Africa likely means that many poorer black students might not be able to afford wifi F I G U R E 1 Summary of survey data CHISADZA ET AL.

| S119

connections or large internet data bundles which can make the transition difficult for black students compared to their white counterparts. A higher proportion of students reported that wifi access made the transition to online learning moderately easy. However, relatively more students reported that mobile internet data and accessing the zero-rated website made the transition difficult. Surprisingly, not many students made use of the zero-rated website which was freely available. Figure 2 shows that students who reported difficulty transitioning to online learning did not perform as well in online learning versus F2F when compared to those that found it less difficult to transition.

In Figure 3 , the kernel density shows that students who had access to wifi performed better than those who used mobile internet data or the zero-rated data. F I G U R E 2 Transition from F2F to online learning. Notes: This graph shows the students' responses to the question "How easy did you find the transition from face-to-face lectures to online lectures?" in relation to the outcome variable for performance S120 |

The regression results are reported in Table 3 . We find that the change in students' performance from F2F to online is negatively associated with the difficulty they faced in transitioning from F2F to online learning. According to student survey responses, factors contributing to difficulty in transitioning included poor internet access, high internet data costs and lack of equipment such as laptops or tablets to access the study materials on the university website. Students who had access to wifi (i.e. fixed wireless broadband, Asymmetric Digital Subscriber Line (ADSL) or optic fiber) performed significantly better, with on average 4.5 points higher grade, in relation to students that had to use mobile internet data (i.e. personal mobile internet data, wifi at home using mobile internet data, or hotspot using mobile internet data) or the zero-rated website to access the study materials. The insignificant results for the zero-rated website are surprising given that the website was freely available and did not incur any internet data costs. However, most students in this sample complained that the internet connection on the zero-rated website was slow, especially in uploading assignments. They also complained about being disconnected when they were in the middle of an assessment. This may have discouraged some students from making use of the zero-rated website. Students who expressed a preference for self-study approaches (i.e. reading PDF slides or PowerPoint slides with explanatory notes) did not perform as well, on average, as students who preferred assisted study (i.e. listening to recorded narrated slides or lecture recordings). This result is in line with Means et al. (2010) , where student performance was better for online learning that was collaborative or instructor-driven than in cases where online learners worked independently. Interestingly, we also observe that the performance of students who often attended in-person classes before the lockdown decreased. Perhaps these students found the F2F lectures particularly helpful in mastering the course material. From the survey responses, we find that a significant proportion of the students (about 70%) preferred F2F to online lectures. This preference for F2F lectures may also be linked to the factors contributing to the difficulty some students faced in transitioning to online learning.

We find that the performance of low-income students decreased post-lockdown, which highlights another potential challenge to transitioning to online learning. The picture and sound quality of the live online lectures also contributed to lower performance. Although this result is not statistically significant, it is worth noting as the implications are linked to the quality of infrastructure currently available for students to access online learning. We find no significant effects of race on changes in students' performance, though males appeared to struggle more with the shift to online teaching than females.

For the robustness check in Table 4 , we consider the average grades of the three assignments taken after the start of the lockdown as a dependent variable (i.e. the post-lockdown average grades for each student). We then include the pre-lockdown average grades as an explanatory variable. The findings and overall conclusions in Table 4 are consistent with the previous results.

As a further robustness check in Table 5 , we create a panel for each student across the six assignment grades so we can control for individual heterogeneity. We create a post-lockdown binary variable that takes the value of 1 for the lockdown period and 0 otherwise. We interact the post-lockdown dummy variable with a measure for transition difficulty and internet access. The internet access variable is an indicator variable for mobile internet data, wifi, or zerorated access to class materials. The variable wifi is a binary variable taking the value of 1 if the student has access to wifi and 0 otherwise. The zero-rated variable is a binary variable taking the value of 1 if the student used the university's free portal access and 0 otherwise. We also include assignment and student fixed effects. The results in Table 5 remain consistent with our previous findings that students who had wifi access performed significantly better than their peers.

F I G U R E 3 Access to online learning. Notes: This graph shows the students' responses to the question "What do you currently use the most to access most of your prescribed work?" in relation to the outcome variable for performance The Covid-19 pandemic left many education institutions with no option but to transition to online learning. The University of Pretoria was no exception. We examine the effect of transitioning to online learning on the academic performance of second-year economic students. We use assessment results from F2F lectures before lockdown, and online lectures post lockdown for the same group of students, together with responses from survey questions. We find that the main contributor to lower academic performance in the online setting was poor internet access, which made transitioning to online learning more difficult. In addition, opting to self-study (read notes instead of joining online classes and/or watching recordings) did not help the students in their performance.

The implications of the results highlight the need for improved quality of internet infrastructure with affordable internet data pricing. Despite the university's best efforts not to leave any student behind with the zero-rated website and free monthly internet data, the inequality dynamics in the country are such that invariably some students were T A B L E 3 Results: Predictors for student performance using the difference on average assessment grades between pre-and postlockdown negatively affected by this transition, not because the student was struggling academically, but because of inaccessibility of internet (wifi). While the zero-rated website is a good collaborative initiative between universities and network providers, the infrastructure is not sufficient to accommodate mass students accessing it simultaneously. This study's findings may highlight some shortcomings in the academic sector that need to be addressed by both the public and private sectors. There is potential for an increase in the digital divide gap resulting from the inequitable distribution of digital infrastructure. This may lead to reinforcement of current inequalities in accessing higher education in the long term. To prepare the country for online learning, some considerations might need to be made to make internet data tariffs more affordable and internet accessible to all. We hope that this study's findings will provide a platform (or will at least start the conversation for taking remedial action) for policy engagements in this regard. | S123

We are aware of some limitations presented by our study. The sample we have at hand makes it difficult to extrapolate our findings to either all students at the University of Pretoria or other higher education students in South Africa. Despite this limitation, our findings highlight the negative effect of the digital divide on students' educational outcomes in the country. The transition to online learning and the high internet data costs in South Africa can also have adverse learning outcomes for low-income students. With higher education institutions, such as the University of Pretoria, integrating online teaching to overcome the effect of the Covid-19 pandemic, access to stable internet is vital for students' academic success.

It is also important to note that the data we have at hand does not allow us to isolate wifi's causal effect on students' performance post-lockdown due to two main reasons. First, wifi access is not randomly assigned; for instance, there is a high chance that students with better-off family backgrounds might have better access to wifi and other supplementary infrastructure than their poor counterparts. Second, due to the university's data access policy and consent, we could not merge the data at hand with the student's previous year's performance. Therefore, future research might involve examining the importance of these elements to document the causal impact of access to wifi on students' educational outcomes in the country. Notes: Coefficients reported. Robust standard errors in parentheses. The dependent variable is the assessment grades for each student on each assignment. The number of observations include the pre-post number of assessments multiplied by the number of students. * * * p < .01. * * p < .05. * p < .10. S124 | CHISADZA ET AL.

An empirical study of face-to-face and distance learning sections of a core telecommunication course

Characteristics and macroeconomic determinants of youth employment in Africa

Accounting for gender equality in secondary school enrolment in Africa: Accounting for gender equality in secondary school enrolment

Virtual classroom versus physical classroom: An exploratory study of class discussion patterns and student learning in an asynchronous internet-based MBA course

On-line vs. face-to-face: A comparison of student outcomes with random assignment. e-Journal of Business Education and Scholarship of Teaching

How does distance education compare with classroom instruction? A meta-analysis of the empirical literature

Can web courses replace the classroom in principles of microeconomics?

A comparison of learning outcomes in skills-based courses: Online versus face-to-face formats

No significant distance" between face-to-face and online instruction: Evidence from principles of economics

School quality and the performance of disadvantaged learners in South Africa

How do online students differ from lecture students

Does educational investment enhance capacity development for Nigerian youths? An autoregressive distributed lag approach

Blind scores in a graduate test. Conventional compared with web-based outcomes

Is it live or is it internet? Experimental estimates of the effects of online instruction on student learning

Exploiting the web for education: An anonymous asynchronous role simulation

The most expensive data prices in Africa

Determinants of HIV knowledge and condom use among women in Madagascar: An analysis using matched household and community data

Education and economic development in Africa

Comparative analysis of learner satisfaction and learning outcomes in online and face-to-face learning environments

Do online students perform better than face-to-face students? Reflections and a short review of some empirical findings

Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies

Economics in the cyber-classroom

Performance and perceptions of distance learners in cyberspace

Learning style and effectiveness of online and face-to-face instruction

The effectiveness of online learning: Beyond no significant difference and future horizons

Online or face-to-face? Students' experiences and preferences in e-learning

The no significant difference phenomenon: A comparative research annotated bibliography on technology for distance education: As reported in 355 research reports, summaries and papers

Twenty years of research on the academic performance differences between traditional and distance learning: Summative meta-analysis and trend examination

Differences between traditional and distance education academic performances: A meta-analytic approach

The role of ICT in modulating the effect of education and lifelong learning on income inequality and economic growth in Africa

Performance gaps between online and face-to-face courses: Differences across types of students and academic subject areas

The authors acknowledge the helpful comments received from the editor, the anonymous reviewers, and Elizabeth Asiedu.

http://orcid.org/0000-0002-9340-210X