7-IJTLHE 3386 International Journal of Teaching and Learning in Higher Education 2018, Volume 30, Number 2, 262-276 http://www.isetl.org/ijtlhe/ ISSN 1812-9129 The Impact of Open Educational Resources on Various Student Success Metrics Nicholas B. Colvard University of Georgia C. Edward Watson Association of American Colleges and Universities Hyojin Park Ewha Womans University There are multiple indicators which suggest that completion, quality, and affordability are the three greatest challenges for higher education today in terms of students, student learning, and student success. Many colleges, universities, and state systems are seeking to adopt a portfolio of solutions that address these challenges. This article reports the results of a large-scale study (21,822 students) regarding the impact of course-level faculty adoption of Open Educational Resources (OER). Results indicate that OER adoption does much more than simply save students money and address student debt concerns. OER improve end-of-course grades and decrease DFW (D, F, and Withdrawal letter grades) rates for all students. They also improve course grades at greater rates and decrease DFW rates at greater rates for Pell recipient students, part-time students, and populations historically underserved by higher education. OER address affordability, completion, attainment gap concerns, and learning. These findings contribute to a broadening perception of the value of OERs and their relevance to the great challenges facing higher education today. The Impact of Open Educational Resources on Student Success Metrics The Association of American Colleges and Universities (AAC&U) performed a member survey of its 1,400-member institutions in 2017 to better understand the challenges facing colleges and universities today (AAC&U, 2018). In regard to students, student learning, and student success, among the greatest challenges were issues surrounding retention and completion, the quality and assessment of student learning, and the affordability of higher education. As you survey the higher education landscape and consider state and national initiatives with the widest presence, it comes as little surprise that these challenges are being voiced. As an example, with 39 states currently in their alliance, Complete College America exists to “significantly increase the number of students successfully completing college and achieving degrees… and close attainment gaps for traditionally underrepresented populations” (Complete College America, 2018). Their recommendations for higher education focus predominantly on how to keep students in college and accelerate their time to a degree. Furthermore, a key component of the larger completion agenda involves attainment gaps (AAC&U, 2015; Perna & Finney, 2014; Tinto, 2012). The attainment gap refers to the rates at which different ethnicities earn college degrees. The U.S. Census Bureau tracks educational attainment, and in 2016, they reported that 37.3% of White Americans over the age of 24 had received a bachelor’s or higher degree. For African Americans in 2016 the attainment rate was 21.8%, and for Hispanic Americans the rate was 15.4% (National Center for Education Statistics, 2016). AAC&U encourages the use of equity-minded practices to enable higher education to better address attainment gaps. Among the recommendations they promote is encouragement for institutions to disaggregate their student data to better understand disparities in student learning outcomes and degree attainment by considering socioeconomic status, as well as race and ethnicity (AAC&U, 2015; Gavin, Bolton, Fine, & Morse, 2018). In truth, the attainment gap has long been recognized, but as demographics continue to shift in the United States, it is becoming a national imperative that higher education better serve all populations. While strategic attention is being placed on issues of retention, completion, and attainment, it is also argued that “the quality shortfall is just as urgent as the attainment shortfall” (AAC&U, 2010, p. 1), and there are a number of initiatives and organizations nationally that are designed to address quality. The Professional and Organizational Development (POD) Network in Higher Education exists to promote quality through improved teaching and learning practice and is the central professional association for those engaged in faculty development. Quality is central to the work of AAC&U’s LEAP Initiative, which promotes excellence in learning through faculty development, general education reform, high impact educational practices, and authentic assessment (Finley & McNair, 2013; Kuh & O’Donnell, 2013). In truth, most institutions are at work today developing a portfolio of solutions that address issues of quality, retention, completion, and attainment. Tuition, Textbooks, and Student Debt Although completion and quality are central to higher education’s work, the dominant public concern for most outside of higher education is cost (Humphries, 2012). Since the mid-1980’s, the cost of a post-secondary degree in the United States has been rapidly increasing (Kuh, Kinzie, Buckley, Bridges, & Hayek, 2006) due to increased tuition and associated miscellaneous costs, such as textbooks Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 263 (Paulson & St. John, 2002; Senack & Donoghue, 2016). Indeed, increases in tuition have been a direct response to the shift of cost away from the public in the form of taxes to students and/or their families (Humphries, 2012). Years of cuts in state funding for public colleges and universities have driven up tuition and harmed students’ educational experiences by forcing faculty reductions, fewer course offerings, and campus closings. These choices have made college less affordable and less accessible for students who need degrees to succeed in today’s economy (Mitchell, Leachman, & Masterson, 2016). Although tuition has been the largest contributor in the equation of student debt, textbooks and ancillary materials are a key variable as well, especially since many students find it challenging to budget for the cost of books because they typically don’t learn about the true scope of those expenses until the beginning of a semester. Depending on the specific course or discipline, the associated traditional commercial textbook can cost students several hundred dollars each semester (Fischer, Hilton, Robinson, & Wiley, 2015; Hilton, 2016). While the increasing costs of attending college affect all students, low-income individuals and their families face greater difficulties than other socioeconomic groups in paying rising tuition and textbook fees (Kuh et al., 2006). This can directly affect their decision regarding where to apply and ultimately decide to attend college. Students with unmet financial need are more likely to delay their college enrollment or may not even attend college (Paulsen & St. John, 2002; Provasnik & Planty, 2008). This, of course, can have a cascading impact on future career decisions and employment opportunities (St. John, Paulson, & Carter, 2005). For individuals who do enroll in higher education institutions, some will make the financial decision to take courses without purchasing the textbook (Watson, Domizi, & Clouser, 2017), presumably negatively affecting their understanding of the course material, their subsequent performance (i.e., grade) in the class, and potentially their persistence in the discipline (Buczynski, 2007; Fischer et al., 2015). Open Educational Resources In an effort to curb the inflating cost of a postsecondary education and reduce student debt, there has been a growing movement in higher education regarding the authoring, adoption, and use of Open Educational Resources (OER) in course settings. OER are broadly defined as “the open provision of educational resources, enabled by information and communication technologies, for consultation, use, and adaptation by a community of users for non- commercial purposes” (UNESCO, 2002, p. 24). Within the higher education context, OER typically encompass free, online learning content, software tools, and accumulated digital curricula that are not restricted by copyright license and available to retain, reuse, revise, remix, and redistribute (Hilton, Fischer, Wiley, & Williams, 2016). Within the context of this study, OER refer to free, open textbooks, which replaced previously adopted expensive, traditional, commercial textbooks. The narrative traditionally supporting the adoption and implementation of OER textbooks has focused on cost savings by making high-quality educational resources freely available to the students. It is well documented in the literature that high- quality OER can lead to significant financial benefits for students and/or institutions, as well as reduce the potential of financial debt (Bliss, Robinson, Hilton, & Wiley, 2013; de los Arcos, Farrow, Perryman, Pitt, & Weller, 2014; Farrow et al., 2015; Fischer et al., 2015; Hilton, Gaudet, Clark, Robinson, & Wiley, 2013; Watson, Domizi, & Clouser, 2017). In empirical studies by Bliss, Robinson, Hilton, and Wiley (2013) and Hilton, Robinson, Wiley, and Ackerman (2014), college teachers and students reported significant cost savings on textbooks due to the implementation of OER in classes. Furthermore, several studies have shown evidence that the affordability of OER can effectively support at-risk learners in their efforts to finish their studies (de los Arcos et al., 2014; Farrow et al., 2015; Winitzky-Stephens & Pickavance, 2017). Additionally, previous studies have found that a majority of faculty and students perceive OER to be equal to, or better than, commercial textbooks in terms of quality (Allen & Seaman, 2014; Bliss et al., 2013; Watson, Domizi, & Clouser, 2017). Many students preferred using OER instead of traditional textbooks (Feldstein et al., 2012; Petrides, Jimes, & Hedgspeth, 2012), citing the benefits of cost, access, and attributes of online textbooks (Bliss et al., 2013; Watson, Domizi, & Clouser, 2017). When evaluating faculty perception, a majority of the faculty rated OER equal or superior to traditional resources in terms of current content (91.2%), ease of use (88.1%), efficacy (84.6%), trusted quality (73.6%), and cost (97.9%) (Allen & Seaman, 2014). While studies focusing on cost savings and student and faculty perceptions have dominated the OER research landscape, there has been less research that has looked at the impact OER have on student learning. Several studies have shown that implementations of OER may result in similar or improved academic performance in addition to saving students’ money (Bowen, Chingos, Lack, & Nygren, 2014; Feldstein et al., 2012; Hilton & Laman, 2012; Lovett, Meyer, & Thille, 2008; Pawlyshyn, Braddlee, Casper, & Miller, 2013). It was found that students enrolled in courses that have implemented OER as the textbook perform just as well, if not better, in comparison to students enrolled in courses that use traditional commercial textbooks (Hilton, 2016; Hilton et al., 2016). Faculty also described OER as having prepared Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 264 students at the same level of rigor, and in some cases more so, as traditional textbooks (Bliss, Hilton, Wiley, & Thanos, 2013; Bliss et al., 2013). Further, some studies suggest that OER may indirectly improve student performance through increased satisfaction, engagement, and interest in the subjects (de los Arcos et al., 2014; Farrow et al., 2015; Pitt, 2015). In regard to measures of student performance (i.e., final grades), several studies suggest that courses that have implemented OER result in higher student grades (Feldstein et al., 2012), higher pass rates (Fischer et al., 2015; Pawlyshyn et al., 2013), or lower failing and withdrawal rates (Feldstein et al., 2012) than courses that do not use OER materials. However, other studies do not find any significant difference in grades between OER adoption and traditional textbook use (Croteau, 2017; Feldstein et al., 2012; Lovett, Meyer, & Thille, 2008). Of the studies that have evaluated student performance in OER vs. non-OER courses, we have not found any that examine differences between full- and part-time student performance, although research has shown that part-time students are less likely than full- time to graduate (Shapiro & Bray, 2011). Further, we are not aware of any research that has evaluated student performance with regard to student financial need or disaggregated student data to better understand the impact OER might be having on various student sub- populations, especially those that might be at the greatest risk of leaving college. In truth, one would not necessarily anticipate that OER would positively impact the performance of a student who would have otherwise been able to purchase a traditional commercial textbook; however, one would imagine that a free textbook would indeed help those students who might choose to forgo a textbook in a course due to the cost. Purpose and Research Questions The purpose of this research, then, was to better understand how courses employing OER impact student success metrics and student academic achievement by disaggregating student performance based upon federally determined financial need (Federal Pell Grant status), ethnicity, and registration status (part-time vs. full-time). We predicted that students from low socioeconomic backgrounds that require substantial financial assistance to attend college would exceedingly benefit from courses that have adopted a free textbook when compared to previous semesters when traditional, commercial textbooks were used (for the purposes of this paper are referenced as “non- OER” courses). Additionally, we predicted that all students perform better in courses that have adopted OER— regardless of socioeconomic or demographic background— as all students will indeed possess the materials needed to succeed in the course. In order to address these research predictions, we sought to answer the following questions: 1) What is the impact of OER textbooks on student academic performance, quantified by evaluating final grades and DFW (D, F, and withdrawal letter grades) rates? 2) Does the use of OER textbooks affect students from a low socioeconomic background (quantified by Federal Pell Grant eligibility status) disproportionately compared to students who do not qualify for Federal Pell Grant status? 3) Does student performance increase significantly for those from underserved populations when a free OER textbook is used instead of a traditional textbook? Ultimately, we sought to determine if OER might address all three of the great challenges facing higher education today. Method Context of Study The Center for Teaching and Learning (CTL) at the University of Georgia (UGA) began encouraging faculty to adopt OER in the summer of 2013. Like many institutions pursuing OER, the goal was to decrease the cost of higher education and student debt by helping faculty find and adopt free, high quality, online textbooks. With limited resources, the CTL developed a model that they anticipated would maximize cost savings for students while also minimizing the scope of work for the Center. They chose to pursue faculty who taught large enrollment courses and who were also currently using an expensive textbook or textbook/technology package. In this way, it was theorized that significant savings would be had by students with only a relatively small number of faculty adoptions of free textbooks. As a result of this course profile, the majority of the courses transitioned were large enrollment general education courses at the 1000-level. By the end of the Fall 2017 semester, it is estimated that 35,985 students had been enrolled in a UGA course that had switched from an expensive textbook to a free textbook. It is further estimated that these students had collectively saved $3,266,930 as a result of this adoption (Watson & Colvard, 2018). While several different OER textbooks were used in this initiative, the majority were created by OpenStax, a nonprofit OER textbook publisher based at Rice University that is largely funded through philanthropic foundations, including the Bill & Melinda Gates Foundation, the William and Flora Hewlett Foundation, and several others (OpenStax, 2018a). The OpenStax publication process mirrors processes implemented by the “big five” textbook publishers: faculty author and Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 265 Figure 1 Timeline of the eight courses and adoption of OER. The black cells represent when the instructor did not teach that respective course for the given semester. The white cells represent when the instructor taught the respective course but used a traditional, commercial textbook. The gray cells represent when the instructor taught the respective course and used an OER for the course text. Table 1. Count of Student Grade-Level for All Students Enrolled in non-OER and OER Courses. The Grade-Level: Other accounts for Transient, Graduate, and Unclassified students. Grade-Level Non-OER OER Freshmen 4328 3689 Sophomore 5001 3782 Junior 1560 1735 Senior 768 908 Other 24 27 Total 11681 10141 peer review of these textbooks. OpenStax’s textbooks are 100% free and openly licensed (OpenStax, 2018b). The open license enables faculty to make changes to the textbooks if they so choose. As a result of OpenStax’s publication approach and their OER’s editable attributes, the CTL chose to focus the majority of their OER adoption efforts on titles provided by OpenStax. Courses This study evaluated historical student academic performance data (i.e., final grades) for eight different undergraduate courses at the University of Georgia (UGA) from Fall 2010 – Fall 2016. These courses were selected because they adopted OpenStax OER textbooks in place of traditional commercial textbooks. The eight courses in question span a range of disciplines, including science and social science courses: • American History since 1865 • American History to 1865 • Anatomy and Physiology II • Basic Concepts in Biology • Elementary Psychology • Introduction to Sociology • Organismal Biology • Principles of Biology All of these are large introductory courses within their respective departments. Some of the courses are designed for majors, whereas most are designed to satisfy UGA’s general education requirements. While UGA launched its OER initiative in Fall 2013, the semester of adoption of the OER differed across these eight courses, but all courses used OpenStax OER textbooks. The courses under consideration used OER textbooks between two and seven semesters (see Discipline Course Fall 2010 Spring 2011 Fall 2011 Spring 2012 Fall 2012 Spring 2013 Fall 2013 Spring 2014 Fall 2014 Spring 2015 Fall 2015 Spring 2016 Fall 2016 Biology Basic Concepts in Biology Organismal Biology Principles of Biology Anatomy and Physiology II History American History to 1865 American History since 1865 Psychology Elementary Psychology Sociology Introduction to Sociology Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 266 Figure 1 for course by course specifics). Additionally, only sections of courses taught by the same instructor were considered. This was done to control for instructor bias in the analysis of pre- and post-OER adoption. For example, we did not consider sections of Principles of Biology taught by anyone other than the instructors who eventually adopted OER for their courses. Participants The timeframe selected for this study provided two large groups of students bridging multiple disciplines, as well as provided two groups of students of similar size. Specifically, there were 11,681 students in the group who were in courses using traditional commercial textbooks, and there were 10,141 students in the group who were in courses using free, OER textbooks. The grade-level breakdown of students enrolled in the non-OER courses and OER courses is listed in Table 1, with a majority of the students enrolled in the eight courses of interest for this study being largely comprised of lower level classmen (Freshmen and Sophomores, n=9,329 students for non- OER courses and n=7,471 students for OER courses) compared to the number of upper level classmen (Juniors and Seniors, n=2,328 students for non-OER courses and n=2,643 students for OER courses). In total, there were 21,822 students in this study. Of those, 5,427 (24.9%) were Federal Pell Grant recipient students. Our study’s Pell eligibility percentage closely approximates UGA’s overall Pell eligibility percentage of 23.8%. In Fall 2016 UGA had a total undergraduate enrollment of 27,951 students with a sex distribution of 43.7% male and 56.3% female students. In this study, 35% of the students were male while 65% were female. For the purposes of this study, Pell eligibility served as a proxy for student socioeconomic status, and therefore, by evaluating student performance within the context of Pell eligibility, it allowed us to make an inference on how OER affected the grades of students from lower socioeconomic backgrounds. In Fall 2016, the ethnic origin characteristics of UGA undergraduate students consisted of 4,835 non- White students (17.30%; not accounting for the Asian student population = 3,226, 11.54%) and 19,672 White students (70.38%). The ethnic origin characteristics of the students enrolled in courses under consideration for this study were 4,078 non-White students (18.69%; not accounting for the Asian student population = 2,549, 11.68%) and 14,938 White students (68.45%). Therefore, the breakdown of student ethnic origin in this study is representative of the student demographics of the university. All student ethnicity data were self-reported, so students that were classified as “Not Reported” were removed from the analysis (n= 257 students, 1.18%). Additionally, the aggregation of “non-White” student ethnicities did not account for Asian students who are outperforming White students in terms of degree attainment (National Center for Education Statistics, 2016). Our non-White category represents ethnicities that have been historically underserved by higher education and are attaining college degrees at significantly lower rates than White students and Asian students. The non-White category is comprised of American Indian or Alaskan Native, Black or African American, Hawaiian or Other Pacific Islander, Hispanic or Latino, and “Two or More Races” students. Additionally, the registration status of undergraduate students enrolled at UGA in Fall 2016 was 26,328 (94.19%) full-time students and 1,623 (5.81%) part-time students. There were 19,419 (88.99%) full-time students and 2,403 (11.01%) part- time students enrolled in the courses of interest. However, when evaluating the registration status respective for the OER courses (between Fall 2013 and Fall 2016), the number of full-time students (9,649; 95.15%) and part-time students (492; 4.85%) more closely follows the breakdown in student registration status for the university in Fall 2016. Data and Sources Examination of student academic performance consisted of a multi-level approach. First, we evaluated academic performance of all students enrolled in select courses pre- and post-OER adoption. We then disaggregated the data to evaluate differences in academic performance for Federal Pell Grant recipient students and for non-Pell grant recipients. Finally, we again disaggregated based on student demographic data—student ethnic origin (White and non-White) and registration status (full-time and part-time)—and again compared academic performance pre- and post-OER adoption. Our data set consisted of all letter grades (+/-) and aggregated DFW grades, and all were de-identified to ensure student anonymity. All letter grades were converted to numerical representations (i.e., A = 4, A- = 3.7, B+ = 3.3, and so on) for statistical analyses. For all three sets of comparisons, we evaluated grade distribution, average course grade, and percent DFW grades for these respective student populations. At UGA the Office of Institutional Research (OIR) possesses student course grade information and most student demographic information; however, the Office of Student Financial Aid (OSFA) is the institutional steward of Federal Pell Grant status. OSFA, working within strict and emerging federal guidelines, required that each course grade grouping contain at least 20 students within each category. This requirement was designed to protect student identities and thus required that we collapse the D, Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 267 Table 2. Percent Student Grade Distribution Data for All Students Enrolled in non-OER and OER Courses. Grade Non OER OER A 17.96 23.46 A- 11.33 19.06 B+ 12.99 14.13 B 22.10 17.02 B- 9.25 7.94 C+ 6.75 3.90 C 7.75 5.55 C- 1.01 0.74 DFW 10.87 8.19 Table 3. Percent Student Grade Distribution Based on Pell Eligibility in non-OER and OER Courses. Non-Pell Recipients Pell Recipients Grade Non-OER OER Non-OER OER A 19.48 24.90 13.48 18.97 A- 11.72 19.83 10.17 16.66 B+ 13.70 13.90 10.88 14.84 B 22.49 16.46 20.95 18.77 B- 8.92 7.54 10.20 9.16 C+ 6.30 3.87 8.11 4.01 C 6.88 5.20 10.30 6.65 C- 0.89 0.72 1.35 0.81 DFW 9.62 7.57 14.56 10.13 Figure 2 Average grade (Final grade) of students enrolled in courses pre-OER adoption (Non-OER) and post-OER adoption (OER). This analysis compared students that were not recipients of the Federal Pell Grant (Non-Pell) and students that did receive the Federal Pell Grant (Pell). The numbers over each bar represent the total number of students in that respective classification. 2.2 2.4 2.6 2.8 3.0 3.2 3.4 Non-Pell Pell Av er ag e G ra de Non-OER OER (8721) (7674) (2960) (2467) Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 268 Figure 3 Percent of DFW students comparing Non-Pell and Pell recipients in course pre-OER adoption (Non-OER) and post- OER adoption (OER). F, and W (Withdrawal) letter grades into a single DFW grade category. This collapsed category is also a metric of interest at UGA and many other institutions interested in DFW rates. Further, all “other” final grade classifications (e.g., Medical Leave, Military Leave, etc.) were deleted prior to analysis as such reasons for course withdrawal would not be related to course performance, financial need, or OER adoption. To analyze the data for all students and groups involved in the study, two sample t-tests were used to compare non-OER to OER courses. To compare student financial aid status (Pell and non-Pell recipients), ethnic origin characteristics (White and non-White students), and registration status (full- time and part-time) with regard to enrollment in non- OER and OER courses, we used two-way ANOVAs with grade as the dependent variable and OER status and student demographic information as fixed factors. All analyses were completed using IBM SPSS Statistics for Macintosh, Version 22.0. This study received IRB approval from the University of Georgia Human Subject Division in the Office of Research. All data received from OIR and OSFA were de-identified in order to maintain student privacy and anonymity. In compliance with the IRB approval, all data were stored, analyzed, and interpreted on one computer device. Results All Students We first compared academic performance of all students categorized into two groups – non-OER courses and OER courses – without stratification based upon financial need or student demographics, and there was a statistically significant improvement in final course grades for students in the OER courses (M = 3.048, SE = 0.011) compared to non-OER courses (M = 2.806, SE = 0.011) (t(21,820) = -15.95, p < .001). Table 2 displays the grade distributions for both groups of students, showing there was a decrease in the percent of DFW through B grades and an increase in the percent of B+ through A grades in courses using OER. For A grades, there was a 5.50% increase after OER adoption, a 7.73% increase for A- grades, and an 1.14% increase for B+ grades. Importantly, the presence of OERs decreased the DFW rate by 2.68% for all students enrolled in the respective courses. Federal Pell Grant Recipient Students Analysis of student performance for Federal Pell Grant recipients maintained the same trend as described for all students, with a statistically significant difference when comparing student Pell eligibility status (F(1,21818) = 0 2 4 6 8 10 12 14 16 Non-Pell Pell % o f S tu de nt s Non-OER OER Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 269 173.54, p < .001), OER use (F(1,21818) = 232.161, p < .001) and Pell eligibility ́OER use, F(1,21818) = 9.348, p = .002). This study found there was a notable increase in B+ through A grades and a decrease in B through DFW grades. For non-Pell recipients, after OER adoption there was a 5.42% increase for A grades, a 8.11% increase for A- grades, and a 0.20% increase for B+ grades. For Pell recipients, after OER adoption we observed a 5.49% increase for A grades, a 6.49% increase for A- grades, and a 3.96% increase for B+ grades (see Table 3). For non-OER courses, the final average course grade was 2.878 ± 0.012 (±SE) for non-Pell recipients and 2.594 ± 0.022 for Pell recipients; for OER courses, the final average course grade was 3.091 ± 0.012 for non-Pell recipients and 2.914 ± 0.023 for Pell recipients (Figure 2). This resulted in a 6.90% increase in non-Pell recipients’ end-of- course grade and a 10.98% increase for Pell recipients end-of-course grade with the adoption of OER into the courses. In this analysis, OER adoption resulted in a 2.05% reduction in DFW grades for non-Pell recipients and a 4.43% decline in DFW grades for Pell recipients (Figure 3) Student Ethnic Origin When evaluating White and non-White students’ academic performance, there was a statistically significant difference in student ethnic origin (F(1,19012) = 195.56, p < .001), OER use (F(1,19012) = 306.98, p < .001), and student ethnic origin ́OER use (F(1,19012) = 10.374, p = .001). There were statistically significant differences in grade distribution for White and non-White students’ academic performance; however, both groups’ academic performance increased in the OER courses. Additionally, non-White students had a greater increase in B through A grades relative to the grade distribution of White students (Table 4). When comparing average course grades for these two demographic groupings, the results demonstrated a narrowing in the gap in academic performance between these student groups following the adoption of OER (Figure 4). In non-OER courses, White students (n = 8152) had an average course grade of 2.925 ± 0.012 compared to 2.525 ± 0.027 for non-White students (n = 2029). Once OER were adopted for these courses, the average course grade increased for both groups, specifically to 3.132 ± 0.013 for White students (n = 6,786), and to 2.857 ± 0.025 for non- White students (n = 2,049) (Figure 4). This resulted in a 7.09% increase in average grade for White students and a 13.13% increase for non-White students. Additionally, there was a large decline in DFW grades once OER were adopted in these courses. For White students, DFW grades accounted for 8.70% of the final grades before OER adoption, and that percentage dropped to 7.19% after OER adoption. For non-White students, we observed that DFW final grades accounted for 15.28% when traditional Figure 4 Average grade (Final grade) of students enrolled in courses pre-OER adoption (Non-OER) and post-OER adoption (OER). This analysis compared self-identified White students and Non-White students – aggregating all other self-identified ethnicities, excluding Asian. The numbers over each bar represent the total number of students in that respective classification. 2.0 2.2 2.4 2.6 2.8 3.0 3.2 3.4 White Non-White Av er ag e G ra de Non-OER OER (8152) (2029) (6786) (2049) Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 270 Table 4. Percent Student Grade Distribution Based on Ethnicity in Non-OER and OER Courses. White Students Non-White Students Grade Non-OER OER Non-OER OER A 20.22 26.27 11.83 15.96 A- 12.51 19.95 8.33 17.23 B+ 13.85 14.65 10.45 13.91 B 22.42 16.05 22.08 19.52 B- 8.91 7.54 10.40 8.44 C+ 5.96 3.24 9.27 5.47 C 6.59 4.48 10.89 8.10 C- 0.85 0.62 1.48 1.22 DFW 8.70 7.19 15.28 10.15 Figure 5 Percent of DFW students for non-OER and OER based courses for White and Non-White students. Students classified as “Asian” were removed from the analyses. textbooks were used, and we noted a disproportionally greater decline in DFW grades to 10.15% with the adoption of OER (a decline of 5.13%) (Figure 5). Student Registration Status Finally, we evaluated the impact of OER when considering student registration status by comparing full-time and part-time students. When evaluating grade distribution data for full-time and part-time students before and after OER adoption, there were two striking results that emerged. First, the shift to higher- level grades, while present for both groups of students, was more pronounced for part-time students than full time students after OER were implemented. Second, DFW grades dropped significantly more for part-time students than full-time students with OER (Table 5). We found a significant difference in student registration status (F(1,21818) = 141.90, p < .001), OER use (F(1,21818) = 968.41, p < .001), and student registration status ´ OER use (F(1,21818) = 59.68, p < .001) for both full-time and part-time students. For both groups, OER adoption helped to raise average course grades (full-time: M = 3.080, SE = 0.011; part- time: M = 2.420, SE = 0.067) compared to course grades prior to OER adoption (M = 2.986, SE = 0.010; 0 2 4 6 8 10 12 14 16 18 20 White Non-White % o f S tu de nt s Non-OER OER Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 271 Table 5 Percent Student Grade Distribution Based on Registration Status in Non-OER and OER Courses. Full-Time Students Part-Time Students Grade Non-OER OER Non-OER OER A 20.25 23.70 6.28 18.70 A- 12.67 19.47 4.45 10.98 B+ 14.05 14.41 7.54 8.74 B 22.85 17.15 18.26 14.43 B- 9.11 7.80 9.94 10.57 C+ 6.32 3.87 9.00 4.67 C 7.48 5.49 9.11 6.71 C- 0.99 0.73 1.10 1.02 DFW 6.28 7.38 34.33 24.19 Figure 6 Average grade (Final grade) of students enrolled in courses pre-OER adoption (Non-OER) and post-OER adoption (OER). This analysis compared students enrolled in the university at least 12 credit hours per semester (Full-time) to those students enrolled in at least 6, but no more than 12 credit hours per semester (Part-time). The numbers over each bar represent the total number of students in that respective classification. part-time: M = 1.889, SE = 0.033). OER helped to narrow the gap in performance by increasing average course grades by 3.18% for full-time students and by 28.13% for part-time students (Figure 6). When evaluating the impact OER had on DFW rates, we observed a slight increase from 6.28% to 7.38% in DFW grades for full-time students, though for part-time students OER adoption resulted in a decrease in DFW grades from 34.28% to 24.19%, which was a 10.14% decline (Figure 7). Closer analysis of these data showed the trend in DFW grades increasing for full-time students in OER courses, and this was attributed to more reported Withdrawal grades (from 173 to 405 students) and fewer D and F grades (299 and 142 to 171 and 136 students, respectively), when compared to full-time students enrolled in non-OER courses. However, we did not evaluate why students withdrew from a course. Discussion While the financial benefits of OER are well- documented (Dimeo, 2017; Lieberman, 2018; Watson & Colvard, 2018), this study sought to determine if OER 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Full-time Part-time Av er ag e G ra de Non-OER OER (9770) (9649) (1911) (492) Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 272 adoption (in our case, free OpenStax textbooks) by faculty in course settings has additional benefits beyond saving students money. Statistically significant and important additional benefits were discerned. Without disaggregating the data, it was first found that students tend to perform better in course settings when OER textbooks were used in place of expensive, commercial textbooks. DFW rates also decreased. Following recommendations from AAC&U (AAC&U, 2015; Gavin, Bolton, Fine, & Morse, 2018), we obtained demographic information which allowed us to disaggregate our data by Pell eligibility status, ethnicity, and registration status. This enabled us to look more deeply into the data to better understand course performance outcomes for subpopulations of interest. While end of course grades increased for all groups considered, DFW rates decreased dramatically for student populations we hypothesized would benefit the most from free textbooks (e.g., Pell eligible students, underserved populations, and part-time students). When considering Federal Pell eligibility, we observed an increase in A through B+ letter grades and a decrease in B through DFW grades when evaluating courses that have implemented OER at the University of Georgia. A significant decrease in DFW rates for Pell- eligible students was found (a 4.43% change) when OER were adopted as the textbook for the class. These results reveal a measurable decrease in the number of students failing or withdrawing from a course when OER are adopted, and that decrease in the number of failing or withdrawal grades is more significant for students from low socioeconomic backgrounds (see Figure 3). This research also evaluated student demographic metrics – ethnic origin and registration status – which helped to provide a more nuanced understanding of student academic performance with regard to OER adoption. This research revealed significant differences in academic performance (average final grade) for both White and non-White students enrolled in OER courses compared to previous semesters when OER were not yet adopted. The finding that students’ final grades improved in courses that adopted OER is encouraging, but the magnitude in which non-White students’ grades improved is very compelling. Additionally, the benefits of OER are significant for part-time students. This study found a 53.12% increase in average course grade and a 29.54% decrease in DFW rates for students who were not enrolled full- time at UGA. These findings uniquely highlight the impact openly accessible content has on this non- traditional student population. Part-time students are an often overlooked population in higher education, and 71% are on their own financially (Bombardieri, 2017). It is not surprising that those enrolled part-time in college benefitted from free textbooks. Figure 7 Percent of DFW students comparing Full-time and Part-time students in courses pre-OER adoption (Non-OER) and post-OER adoption (OER). 0 5 10 15 20 25 30 35 40 Full-time Part-time % o f S tu de nt s Non-OER OER Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 273 As noted earlier, students at UGA have collectively saved approximately $3,266,930 since the launch of the initiative in 2013. The cost of higher education and the associated debt have a well-documented connection to drop-out rates (Goldrick-Rab, 2016); however, there is more to the OER story than simply reducing debt. Given the findings of this large-scale study, we believe the conversation regarding OER should change significantly. While compelling, the argument for OER as primarily a cost saving measure is incomplete and minimizes the value of OER. This study suggests that OER speaks to all three of the great challenges facing higher education today: affordability, retention and completion, and quality of student learning. Although drop-out rates were not examined as part of this study, it is logical to deduce that reducing the number of students who fail would have a positive impact on retention. As noted above, OER were found to significantly decrease DFW rates across a range of demographics. They also have a more pronounced impact on grades for those who start further behind, are in financial need, and/or are among populations that have been historically underserved by postsecondary education. OER speaks to the aforementioned attainment gap as well. Still further, there is an expectation that grades are an indicator of student achievement within course settings, and by simply ensuring that all students, regardless of need or background, have access to course materials on the first day of class, the quality and extent of learning appear to be improved. Study Limitations It should be noted that there are limitations and assumptions made for this study. The analysis provided within this article only considers students at a single, large, doctoral-granting research university. This should be taken under consideration as readers evaluate the generalizability of these findings. Some of the course transitions to OER textbooks represented in this study included assistance from UGA’s CTL, and it is probable that the adoption of the OER-based textbook served as a catalyst to further the instructors’ engagement with their own teaching. Additionally, this study only evaluated end of course grades, though there are a number of course assessments that went into generating the final grades for these respective classes. The degree to which OER influenced individual assignment or assessment grades was not explored by this study and could not be determined based on the nature of the data set. Finally, this study evaluated large, introductory courses spanning a range of disciplines; therefore, upper class (juniors and seniors) students were a small percentage of the population under consideration. Conclusion This research suggests OER is an equity strategy for higher education: providing all students with access to course materials on the first day of class serves to level the academic playing field in course settings. While additional disaggregated research is needed in a variety of postsecondary contexts such as community college, HBCU, and other higher education settings to increase the generalizability of this notion, this study provides an empirical foundation on which to begin to change the advocacy narrative supporting OER. A new opportunity appears to be present for institutions in higher education to consider how to leverage OER to address completion, quality, and affordability challenges, especially those institutions that have higher percentages of Pell eligible, underserved, and/or part-time students than the institution presented in this study. References AAC&U. (2018). 2018-22 strategic plan: Educating for democracy. Washington, DC: Author. Retrieved from https://www.aacu.org/sites/default/files/files/ab out/AACU_StrategicPlan_2018-22.pdf AAC&U. (2015). Step up and lead for equity: What higher education can do to reverse our deepening divides. Washington, DC: Author. AAC&U. (2010). The quality imperative: Match ambitious goals for college attainment with an ambition vision for learning. Washington, DC: Author. Allen, I. E., & Seaman, J. (2014). Opening the curriculum: Open educational resources in U.S. higher education, 2014. Retrieved from http://www.onlinelearningsurvey.com/reports/open ingthecurriculum2014.pdf Bliss, T. J., Hilton III, J., Wiley, D., & Thanos, K. (2013). The cost and quality of open textbooks: Perceptions of community college faculty and students. First Monday, 18(1). Retrieved from http://scholarsarchive.byu.edu/facpub/1378/ Bliss, T. J., Robinson, J. T., Hilton III, J., & Wiley, D. A. (2013). An OER COUP: College teacher and student perceptions of open educational resources. Journal of Interactive Media in Education, 25. Retrieved from http://doi.org/10.5334/2013-04 Bombardieri, M. (2017). Hidden in plain sight: Understanding part-time college students in America. Washington, DC: Center for American Progress. Retrieved from https://www.americanprogress.org/issues/education -postsecondary/reports/2017/09/06/438341/hidden- plain-sight/ Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 274 Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. I. (2014). Interactive learning online at public universities: Evidence from a six-campus randomized trial. Journal of Policy Analysis and Management, 33(1), 94-111. Buczynski, J. A. (2007). Faculty begin to replace textbooks with “freely” accessible online resources. Internet Reference Services Quarterly, 11(4), 169–179. Complete College America. (2018). About. Retrieved from https://completecollege.org/about/ Croteau, E. (2017). Measures of student success with textbook transformations: The Affordable Learning Georgia Initiative. Open Praxis, 9(1), 93-108. de los Arcos, B., Farrow, R., Perryman, L.-A., Pitt, R., & Weller, M. (2014). OER evidence report 2013- 2014. OER Research Hub. Retrieved from http://oerresearchhub.files.wordpress.com/2014/11/ oerrh-evidence-report-2014.pdf Dimeo, J. (2017). Saving students money. Inside Higher Ed. Retrieved from https://www.insidehighered.com/digital- learning/article/2017/06/28/report-saving- students-money-oer Farrow, R., Pitt, R., de los Arcos, B., Perryman, L. A., Weller, M., & McAndrew, P. (2015). Impact of OER use on teaching and learning: Data from OER Research Hub (2013-2014). British Journal of Educational Technology, 46(5), 972–976. Feldstein, A., Martin, M., Hudson, A., Warren, K., Hilton III, J., & Wiley, D. (2012). Open textbook and increased student access and outcomes. European Journal of Open, Distance, and E- Learning, 15(2). Retrieved from http://www.eurodl.org/?p=archives&year=2012&h alfyear=2&article&article=533 Finley, A., & McNair, T. (2013). Assessing underserved students’ engagement in high impact practices. Washington, DC: Association of American Colleges and Universities. Fischer, L., Hilton III, J., Robinson, T. J., & Wiley, D. A. (2015). A multi-institutional study of the impact of open textbook adoption on the learning outcomes of post-secondary students. Journal of Computing in Higher Education, 27(3), 159–172. Gavin, M., Bolton, K., Fine, R., & Morse, A. (2018). Building and sustaining an equity-minded campus. In A vision for equity (pp. 1-7). Washington, DC: AAC&U. Goldrick-Rab, S. (2016). Paying the price: College costs, financial aid, and the betrayal of the American Dream. Chicago: University of Chicago Press. Hilton III, J. L. (2016). Open educational resources and college textbook choices: A review of research on efficacy and perceptions. Educational Technology Research and Development, 64(4), 573-590. Hilton III, J. L., Fischer, L., Wiley, D., & Williams, L. (2016). Maintaining momentum toward graduation: OER and the course throughput rate. International Review of Research in Open and Distributed Learning, 17(6), 18–27. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/ 2686/3967 Hilton III, J. L., Gaudet, D., Clark, P., Robinson, J., & Wiley, D. (2013). The adoption of open educational resources by one community college math department. International Review of Research in Open and Distributed Learning, 14(4). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/152 3/2652 Hilton III, J., & Laman, C. (2012). One college’s use of an open psychology textbook. Open Learning: The Journal of Open, Distance and e-Learning, 27(3), 265-272. Hilton III, J. L., Robinson, T. J., Wiley, D., & Ackerman, J. D. (2014). Cost-savings achieved in two semesters through the adoption of open educational resources. International Review of Research in Open and Distance Learning, 15(2), 67–84. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1 700/2833Humphrey, D. (2012). What’s wrong with the completion agenda: And what we can do about it. Liberal Education, 98(1), 8-17. Kuh, G. D., Kinzie, J., Buckley, J. A., Bridges, B. K., & Hayek, J. C. (2006). What matters to student success: A review of the literature. San Francisco: Jossey-Bass. Kuh, G. D., & O’Donnell, K.. (2013). Ensuring quality and taking high-impact practices to scale. Washington, DC: Association of American Colleges and Universities. Lieberman, M. (2018). Textbook alternatives take hold at community colleges. Inside Higher Ed. Retrieved from https://www.insidehighered.com/digital- learning/article/2018/05/09/two-year-institutions- report-significant-savings-oer-initiatives Lovett, M., Meyer, O., & Thille, C. (2008). The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education, 14, 1–16. doi:10.5334/2008-14 Mitchell, M., Leachman, M., & Masterson, K. (2016). Funding down, tuition up: State cuts to higher education threaten quality and affordability at public colleges. Washington, DC: Center on Budget and Policy Priorities. National Center for Education Statistics. (2016). Digest of education statistics: 2016. Washington, DC: U.S. Department of Education. Retrieved from https://nces.ed.gov/programs/digest/d16/ OpenStax. (2018a). About us. Retrieved from https://openstax.org/about Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 275 OpenStax. (2018b). Peer-reviewed. Openly licensed. 100% free. Retrieved from https://openstax.org/subjects Paulsen, M. B., & St. John, E. P. (2002). Social class and college costs: Examining the financial nexus between college choice and persistence. Journal of Higher Education, 73(2), 189-236. Pawlyshyn, N., Braddlee, D., Casper, L., & Miller, H. (2013). Adopting OER: A case study of cross- institutional collaboration and innovation. EDUCAUSE Review. Retrieved from http://er.educause.edu/articles/2013/11/adopting- oer-a-case-study-of-crossinstitutional- collaboration-and-innovation Perna, L. W., & Finney, J. E. (2014). The attainment agenda: State policy leadership in higher education. Baltimore, MD: Johns Hopkins University Press. Petrides, L. A., Jimes, C., & Hedgspeth, C. (2012). Knowledge sharing and collaboration as indicators of learning in OER communities. In A. Okada, T. Connolly, & P. J. Scott (Eds.), Collaborative Learning 2.0: Open Educational Resources (pp. 38–50). New York: IGI Global. Pitt, R. (2015). Mainstreaming open textbooks: Educator perspectives on the impact of OpenStax college open textbooks. International Review of Research in Open and Distributed Learning, 16(4). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/238 1/3497 Provasnik, S., & Planty, M. (2008). Community colleges: Special supplement to the condition of education 2008. Statistical analysis report. Washington, DC: National Center for Education Statistics, US Department of Education. Retrieved from https://nces.ed.gov/pubs2008/2008033.pdf Shapiro, J., & Bray, C. (2011). Improving retention and enrollment forecasting in part-time programs. Continuing Higher Education Review, 75, 121-129. St. John, E. P., Paulsen, M. B., & Carter, D. F. (2005). Diversity, college costs, and postsecondary opportunity: An examination of the financial nexus between college choice and persistence for African Americans and Whites. Journal of Higher Education, 76(5), 545-569. Tinto, V. (2012). Completing college: Rethinking institutional action. Chicago, IL: University of Chicago Press. UNESCO. (2002). Forum on the impact of open courseware for higher education in developing countries: Final report. Retrieved from http://unesdoc.unesco.org/images/0012/001285/12 8515e.pdf Watson, C. E., & Colvard, N. B. (2018). Open educational resources (OERs): Another high impact practice? Paper presented at the annual meeting of the Association of American Colleges and Universities, Washington, DC. Watson, C. E., Domizi, D., & Clouser, S. (2017). Student and faculty perceptions of OpenStax in high enrollment courses. International Review of Research in Open and Distance Learning, 18(5). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/246 2/4299 Winitzky-Stephens, J. R., & Pickavance, J. (2017). Open educational resources and student course outcomes: A multilevel analysis. The International Review of Research in Open and Distributed Learning, 18(4). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/3118 ____________________________ NICHOLAS B. COLVARD, PhD, is a Lecturer and Academic Coach in the Division of Academic Enhancement (DAE) at the University of Georgia (UGA). He teaches Developing Literacies in the STEM Fields in addition to other university-level courses which support students as they transition into higher education and sustain their progress through the University’s unique academic environment. He is a 2017-2018 William and Flora Hewlett Foundation Open Educational Resource Research Fellow and is helping to champion OER adoption and implementation across UGA. Prior to his teaching role in DAE, he served as a Research Associate in the UGA Center for Teaching and Learning where he examined the impact and effect of OER on student success at UGA. He also served as the Instructional Laboratory Manager for the Odum School of Ecology at UGA. C. EDWARD WATSON, PhD, is the Associate Vice President for Quality, Advocacy, and LEAP Initiatives for the Association of American Colleges and Universities (AAC&U) where he leads the association’s national and state-level advocacy and policy efforts to advance quality in undergraduate student learning. Prior to joining AAC&U in 2017, Watson was the Director of the Center for Teaching and Learning at the University of Georgia (UGA). In that role, he designed, launched, and led UGA’s OER initiative; wrote and received multiple grants supporting UGA’s OER development, adoption, and advocacy efforts; led multiple OER research projects; served as the Affordable Learning Georgia Campus Champion for UGA; and is now a national advocate for OER as an equity strategy. He is also the co- author of the Care Framework (http://careframework.org), an OER stewardship model published in 2018. He continues to serve as a Fellow in the Institute of Higher Education at UGA. HYOJIN PARK, PhD, is a graduate of the Learning, Design, and Technology Program in the College of Education at the University of Georgia. Her dissertation is entitled Enhancing Veterinary Students’ Clinical Decision- making Skills by Promoting Revision of their Decision- making Process in Case-based Learning. Following graduation, she served as a post-doctoral fellow in UGA’s Colvard, Watson, and Park Open Educational Resources, Learning, and Retention 276 Center for Teaching and Learning where she worked on several research projects associated with Open Educational Resources. Park is currently a lecturer at Ewha Womans University in Seoul, South Korea. Acknowledgements The authors would like to thank the University System of Georgia’s Affordable Learning Georgia Initiative and the Bill & Melinda Gates Foundation for their support of OER projects and this research at the University of Georgia.