key: cord-0136975-krh24rzy authors: Haldolaarachchige, Neel title: Student performance analysis of virtual introductory calculus-based physics class date: 2022-01-05 journal: nan DOI: nan sha: aeb9a01f12668121d12a939daf763391107298b4 doc_id: 136975 cord_uid: krh24rzy Student performance of virtual introductory physics class (calculus-based mechanics) is analyzed. A fully web-enhanced class was done synchronously. The analysis is done in two categories, averaging all mid-exams (or chapter exams) and cumulative final exams. It shows that students perform well with shorter timed mid-semester assessments (or chapter exams) than the longer timed assessments. Results were then compared with two survey questions to investigate the effect of the background knowledge of physics and calculus on their performance. It shows two interesting results, a) students with basic conceptual background knowledge of physics perform well, b) calculus knowledge does not correlate well with student performance. These findings can be used to rethink and redesign the assessments for calculus-based introductory physics classes. Also, the findings can be used to develop a supplemental support program to enhance student performance. Generally, introductory level calculus-based physics classes are not offered fully online at the university level. However, due to the COVID19 effect, it was forced all the classes move to online. There are various types of online teaching formats now available, such as online synchronous (class meets virtually on scheduled date/time), online asynchronous (there are no virtual meetings), hybrid (part of the class is virtual and the other part is inperson). [1, 2, 3] All of the above-mentioned modalities follow a structured schedule for the semester. On the other hand, there are fully online classes which have a flexible schedule that follows students' phase of learning. Virtual (or online) synchronous classes, got strong attention in recent times mainly because such modality can simulate the real in-person learning experience. In synchronous classes students and the teacher meet virtually via video conference technology. Therefore, student-teacher interaction is real-time. [4, 5, 6, 7] Introductory calculus-based physics at the university level is one of the most challenging classes for most freshman/sophomore students. [8] These classes are required to advance into most science and engineering majors. And also these physics classes consist of very important fundamental background knowledge about the development of scientific models to understand the physical processes of nature. Also, introductory-level physics classes are extremely important to build the analytical skills of future science/engineering students. Therefore, the design and delivery of virtual physics classes must be done carefully. The standard of such teaching modality must be at a very high level to match the students learning outcomes of the course. This can be achieved by using a learning management system (LMS), which consists of functionalities to engage students in various learning activities, and also students can track their progress in the course. However, the concern of the virtual teaching modalities is on the assessments. [9, 10] There are various functionalities (lockdown browser, plagiarism checker, online proctoring software's, etc.) built into learning management systems (LMS) to support assessments and minimize (or stop) possible academic integrity issues. [11, 12, 13, 14, 15, 16] The problem is non of those can be used with calculusbased physics assessments because students must solve very detailed problems that include drawing diagrams, building mathematical models, and finding algebraic solutions. Therefore, it is extremely important that assessments (quizzes and exams) be proctored in real-time (or live). Another important question is the correlation of student's background knowledge of basic physics concepts and calculus with the performance in the assessments of calculus-based physics classes. [17, 18] This question becomes one of the heated discussion topics when setting the prerequisites for the introductory calculus-based physics courses at the university level. Usually, at community colleges, only pre-calculus is a prerequisite and calculus-I is a co-requisite when registering for calculus-based physics I (calculus-based mechanics). There are no prerequisites of physics at the high school level or conceptual physics at the university level and it only recommends students to complete a conceptual physics class only if one does not have a high school physics background. Most students register for calculus-based physics-I (calculus-based Me-chanics) without a proper understating of the level of the class. Therefore, it is important to investigate the correlation of student's background knowledge of conceptual physics and calculus to the performance of calculus-based Mechanics (Physics I). This study was based on the introductory calculus-based physics-I (mechanics) at the largest community college (Bergen community college) in the state of New Jersey. All the students enrolled were engineering majors. In general, calculus-based physics is considered the most challenging class for almost every engineering major student at Bergen community college. Historically, this class shows a very high dropout rate and failure rate. Here we present the details of the design and delivery of virtual calculusbased physics-I (Mechanics) and the analysis of student performance. The details of the comparison of two types of assessments are presented. Also, we investigated the correlation of class performance and the student's background knowledge of conceptual physics and calculus. In-person and virtual classes are fully web-enhanced with a learning management system (LMS). [19] There are various LMS available and we have used the most widely used two LMS (Moodle and Blackboard). [20, 21, 22] All class activities were fully done via LMS. All classwork was submitted into LMS and was graded electronically via LMS. The electronic pen was used to comment on submissions that simulate the real in-person paper submission and corrections. The details of the class design and delivery are published elsewhere and details can be found in the following references [23, 24] . There are few widely used video conference technologies available such as Zoom, Webex, Microsoft Teams, Google Meeting. [25] We used the Webex platform to deliver the virtual synchronous classes. [26] Most importantly class assessments (quizzes and exams) were proctored live via Webex. [27, 28, 29, 30] Students were asked to align the webcam to show the working area. The teacher can see student behavior during the quizzes/exams. Students were asked to keep the microphone on during the quizzes/exams to make sure that they do not talk to someone else with a second electronic item. All the quizzes and exams were closed book and only equation sheet was allowed during the quizzes/exam. All three semesters (FA19, FA20, SP21) classes were done with the same teaching methods that include short concept, theory/principle lecture followed by the detailed problem-solving. Simulations were used to visualize almost every concept/theory/principle and also simulations were used with the discussion of in-class questions. Students were trained to use simulations very in detail during the laboratory experiments of the class. New series of lab manuals were developed for simulation-based laboratory experiments and detailed step-by-step lesson video for each simulation-based experiment were provided to students. [31, 32, 33] During the fall 2020 and spring 2021 semesters students learning was supported with supplemental instructions that are similar to recitation sessions. Two sessions (each one hour) per week were done and the sessions were focused on the most challenging problems per each chapter. [34, 35] Microsoft Excel (MS Excel) software package was used for data analysis and all the tables and graphs were done with Excel. A survey at the beginning of the semester was used to collect student responses to understand their background knowledge of conceptual physics and calculus knowledge. The survey was given to students within the first week of the semester via LMS (Moodle or Blackboard) and responses were collated anonymously. All the students completed the survey so the response rate is 100%. Then survey results were investigated to find a correlation of student's background knowledge with class performance. The investigation of student performance is done by using the results of the virtual-synchronous calculus-based introductory physics (level I, mechanics) class from two consecutive semesters (fall 2020 and spring 2021, both during COVID19). Then the results are compared to the same class done in-person (traditional) during fall 2019 (before COVID19). Analysis of assessments are done in two aspects to understand the student performance, 1) percent of students below and above 70% of the mid-semester assessments and 2) percent of students below and above 70% of cumulative final exam. Mid-semester assessments were done with a similar set of questions in all three semesters. However, the method of mid-semester assessments was changed in spring 2021. Mid-semester assessments were longer (each 2.45hrs) timed, closed-book exams and there were three mid exams (one per every 3-4 chapters) during fall 2019 (in-person) and fall 2020 (virtual-synchronous). During spring 21 shorter (each 30 min) timed, closed-book exam was done per each chapter (total of 13 chapter exams). Also, in the SP21 semester optional practice questions for each chapter exam were given via Moodle. More than 90% of students were completed weekly practice questions. The final exam format (3.00hrs, longer timed and closebook) was the same in all three semesters. New exam questions (for both mid-exams and final exams) were written from scratch for all three semesters but the format, concepts, theory/principles were kept the same. Each exam question was searched via Google (before each assessment) to make sure that the exact question and solution does not available on the internet. Students submitted their written solutions to LMS and the submitted solutions were carefully analyzed. There were no academic integrity issues detected. All the mid-exams (or chapter exams) and final exams were proctored live via video conference. Use of a new set of questions and live proctoring are extremely important on virtual classes to minimize (or stop) any possible academic integrity issues. First, each student's mid exams (chapter exams) total and final exam total points were normalized relative to passing mark of 70%, This makes it easier to find the total number of students who got above and below the passing mark (70%) because the normalized value is positive for students with above the passing mark (70%) and it is negative for students with below passing mark (70%). Then, the percentage of students who got above and below the passing mark (70%) were calculated. This study focuses on a few different research questions as given below and the research questions are addressed in the following section, • Does class modality (virtual or in-person) affect student performance? To investigate this comparison of student performance is done between in-person traditional and virtual-synchronous. • Does the type of assessment affect student performance? To investigate this, two types of mid-semester assessments were used (FA2019 and FA2020 -3 mid exams, each 2.45 hrs) and SP2021 -chapter exams, each 30 min). • Does student's background knowledge of conceptual physics affect student performance? • Does student's math level (completed calculus-I or not) affect student performance? 3.1. Analysis of student performance in mid-semester assessments and final exam One of the most interesting and important questions is that the effect of class modality (in-person and virtualsynchronous) on student learning. We investigated this with student performance in three semesters by comparing the class averages of mid-semester assessments and the final exams between in-person vs virtual-synchronous class modalities. Figure 1 (a) shows analysis of class average percent difference (C.A.P.D) of mid exams (dark gray color) and the final exam (light gray color) between semesters. It can be observed that C.A.P.D of mid exams and final exams between FA19 (in-person) and FA20 (virtual) is negligible. On the other hand, C.A.P.D is very high between SP21 (virtual) vs FA20 (virtual) and SP21 (virtual) vs FA19 (inperson). C.A.P.D is significantly high for mid-exams and considerable for final exams. This shows that something very specific happens during SP21 and it does not correlate with the type of the learning (virtual or in-person). Because the C.A.P.D of final exams is the same between virtual two semesters (FA20 and SP21) and virtual-in-person two semesters (SP21 and FA19). Analysis of student performance in three semesters (FA19-in-person, FA20, and SP21 -virtual synchronous) of mid-exams shows in figure 1b. Percent of students with above 70% of mid exams (the bright blue color) is about the same in FA20 (virtual) and FA19 (in-person). This confirms that the class type (virtual or in-person) does not affect student learning. However, student's percent above 70% in mid-exams is significantly higher in SP21 compared with that of FA20 and FA19. This indicates that something very specific did happen in the SP21 semester. The increase percent above the passing percentage of students in SP21 correlates with the format of mid-exams during spring 2021. In which shorter timed closed-book midexams (one per each chapter) were done. Both the other semesters (fall 2019 and fall 2020) mid-exams were longer timed closed book (one per each 3-4 chapters). Therefore, it is clear that the shorter mid exams are more effective to student performance. On the other hand, the final exam analysis of three semesters (FA19-in-person, FA20, and SP21 -virtual synchronous) shows in the figure 1(c) . It can be observed that the percent of students above 70% is very much similar in all three semesters. This a very good indication that the class modality (in-person or virtual synchronous) has no effect on student learning. This further confirms that the students do not perform well in longer timed closedbook exams (final exams are 3.0 hrs long and cumulative). When comparing the percent of the student with above 70% of mid-exams and final exams in the SP21 semester, significant difference can be observed. The higher percent of the student with above 70% in mid-exams during the SP21 semester confirms that students can perform well in shorter timed closed-book exams. Figure 2(a and b) shows the end of the semester analysis of percentage of students (P.S.) who got below 70% (including grades of W -withdrawal, F -less than 60% and D between 60% and 70%). Figure 2 (a) shows total percentage of students with below 70%. SP21 (virtualsynchronous) semester shows better student performance compared to the previous two semesters (FA19 -in-person and FA20 -virtual synchronous). The percent of students difference (P.S.D) with lower than 70% between two semesters can be seen in figure 2 (b). It shows that the failure percentage dropped by 20% in SP21 when compared to the previous two semesters (FA19 and FA20). There is no difference between FA19 (in-person) and FA20 (virtualsynchronous). It is clear that the student's performance enhances significantly in the SP21 semester. This shows that something specific affects student performance in the SP21 semester. As confirmed with the analysis in figure 1 and the reason for better student performance in SP21 is the type of mid-semester assessments. Figure 2 (c and d ) shows the final grade analysis for all three semesters (FA19 in dark gray color, FA20 in light gray color and SP21 in orange color). There is no difference in final class grade distribution between FA19 (in-person) and FA20 (virtual) semesters. Both graphs show that the percent of students below 70% (final grades of W, F, D) is very similar to FA19 (in-person, pre-COVID) and FA20 (virtual-synchronous, during COVID). Also, it shows in FA19 and FA20 semesters it is not possible to get the bell curve (see figure 2d ) for the distribution of final grades. this is due to a large percentage (about 40%) of students below 70%. However, grade distribution in SP21 (virtualsynchronous) semester shows nice bell curve in figure 2(d ) . Student percent below 70% (final grades of W, F, D) in SP21 drop by about 20% compared to FA20 and FA19. This is a significant improvement in student performance. The analysis of mid-semester assessments and final exams in figure 1 and final class grades in figure 2 suggests that something very specific happens in the SP21 semester. Therefore, it is very important to discuss the reasons for this development. Classes were done with the same teaching style in all three semesters. Resources used for teaching were also the same in three semesters. Final exam assessments (3.0 hrs, proctored, closed-book) were the same in three semesters. Similar student performance in final exams (see figure 1c) in three semesters confirmed that there were no issues of academic integrity in the exams in virtual classes during FA20 and SP21. The exams were live proctored via video conference technology and this confirms that the proctoring method is effective in virtual classes. Now, it is clear that the student performance only can be impacted due to one parameter which changes during the SP21 semester. That is the mid-semester assessments. Usually, in calculus-based physics classes, there are three mid-exams (each exam 2.45 hrs, closedbook, proctored, and covers 3-4 chapters contents). That type of mid-exam was used during FA19 (in-person), and FA20 (virtual-synchronous). However, the type of midsemester assessments was changed to chapter exams during SP21 (virtual-synchronous) which were shorter-timed (each 30min), closed-book, proctored and one short quiz per chapter. And at the end of the semester two lowest chapter quiz grades were dropped. The effect of continuous assessments through the semester can be observed in figure 1(b) . The continuous mid-semester assessments affect most of the students to perform well at the end of the semester which can be observed in figure 2 . Also, figure 2(d ) shows the bell curve nature of final grade distribution in SP21 semester. Therefore, the enhancement of student performance correlates well with continuous assessment in the SP21 semester. On the other hand, When compared to all three semesters it can be confirmed that class modality (in-person or virtual) does not impact on student performance. Another important and interesting aspect of this study is the effect of student background knowledge of conceptual physics and calculus on the performance in calculusbased physics at the university level. Figure 3(a) shows analysis of student performance in mid-exams passing percentage of students (blue color) with respect to their background knowledge of conceptual physics (green color) and calculus-I level math (gray color). This shows that student percentage with conceptual physics background knowledge from high school (students who did physics at high school) correlate well with the passing percentage of students than that of the percent of students with calculus-I level math knowledge (who started the class after completing calculus-I). Students with conceptual physics background in SP21 (see figure 3a) are higher than the students with calculus-I level math in both FA20 and SP21 semester. This is because calculus-I is only a co-requisite for calculusbased physics-I. Therefore, most students simultaneously take both calculus-based physics I and calculus-I. It also shows that the student percent with above 70% in SP21 is higher than that of FA20 and student percent with concept physics background in SP21. The reason for this as discussed earlier is due to the effect of better student performance in shorter timed mid-semester assessments in SP21 compared with longer-timed mid-semester assessments in FA20. Figure 3 : (Color online) (a) Student percent with above 70% in mid exams (blue color) compared to student background knowledge (at the beginning of the semester) of conceptual physics (green color) and calculus-I (gray color). (b) Student percent with above 70% in final exam (orange color) compared to student background knowledge (at the beginning of the semester) of conceptual physics (green color) and calculus-I (gray color). It is evident that students who started with the background of conceptual physics perform well in mid-exams than of the students who started the class with background of calculus-I level math. Figure 3 shows that students passing percentage in mid-exams is higher than that of the final exam in the SP21 semester. It is because the mid-exam assessment method was based on the shorter chapter exams in SP21. When observing SP21 students passing percentage in mid exams it can be seen that about 8% of students did class materials without previous conceptual physics background and this is an even higher percentage compared with student percentage did well in the mid exams without calculus-I (about 25%). Percent of students with a passing grades of the final exam in FA20 (see figure 3b) is higher than the student percent with concept physics and calculus-I. However, figure 3(b) shows that student percent with a passing grades on the final exam is very similar to the percent of students with conceptual physics backgrounds in the SP21 semester. Also, it shows that the percent of students with calculus-I is much lower than the passing percentage of the students in the final exam in the SP21 semester. Therefore, analysis in figure 3 (b) confirms that student performance in the final exam correlates well with the conceptual physics background than calculus-I. Analysis of student performance in mid-semester assessments and the final exams was done and compared in three semesters. The results show that online live proctoring of assessments with video conference technology works well when considering academic integrity. The analysis also shows that students can perform well in shorter-timed assessments than of longer-timed assessments. The effect of shorter-timed assessments confirms with further analysis of final class grades. Student's background knowledge of conceptual physics and calculus-I into class performance was done. It shows that the students who started the class with a conceptual physics background did well in class assessments than the students with the calculus-I background. The results of this study suggest that the student performance depends on the type of assessments of calculusbased physics class. It further suggests that the prerequisites of the calculus-based physics-I class must be reevaluated. Also, the results can be used to develop effective learning support mechanisms. NH acknowledges the Cerullo Learning Assistance Center (CLAC) that provided learning supports (supplemental instruction -SI and tutoring) to students. Supplemental instruction (SI) was financially supported by the STEMatics grant from Department of Education. NH acknowledges the academic freedom and support provided to investigate new teaching methods by the dean of STEM division Dr. Emily Vandalovsky (and former dean -Dr. P.J. Ricato) and department chair of Physical science Dr. Ara Kahyaoglu (and former chair -Dr. Lynda Box) at Bergen community college. Education and Information Technologies Mike Watts and Pascal Sylvain Nadal Khalid Berrada, Radical Solutions and eLearning 30th International Conference on Information Technology Interfaces 37th Annual Frontiers In Education Conference -Global Engineering: Knowledge Without Borders, Opportunities Without Passports Learning management system -Moodle Learning management system -Blackboard The Cambodian 1st International Conference on Mentoring Educators (CICME 2020 This research was done according to ethical principles outlined by Institutional Review Board (IRB) at Bergen Community College. That is in accordance with the ethical policies out lined by Institute of Physics (IOP). This article does not contain any personal data, and it does not identify any individual (students and/or teachers). All students and teachers have given explicit consent to participate in a research project.