Volume 84 Issue 2 Spring 2010 The Journal of Dental Hygiene 75 An Analysis of Student Performance Benchmarks in Dental Hygiene via Distance Education Jodi L. Olmsted, RDH, BS, MS, EdS, PhD Abstract Purpose: Three graduate programs, 35 undergraduate programs and 12 dental hygiene degree completion programs in the United States use varying forms of Distance Learning (DL). Relying heavily on DL leaves an unanswered question: Is learner performance on standard benchmark as- sessments impacted when using technology as a delivery system? A 10 year, longitudinal examination looked for student performance differences in a Distance Education (DE) dental hygiene program. The purpose of this research was to determine if there was a difference in performance between learners taught in a traditional classroom as compared to their counterparts taking classes through an alternative delivery system. Methods: A longitudinal, ex post facto design was used. Two hundred and sixty–six subject records were examined. Seventy–seven individuals (29%) were lost through attrition over 10 years. One hundred and eighty–nine re- cords were used as the study sample, 117 individuals were located face– to–face and 72 were at a distance. Independent variables included time and location, while the dependent variables included course grades, grade point average (GPA) and the National Board of Dental Hygiene Examination (NB- DHE). Three research questions were asked: Were there statistically signifi- cant differences in learner performance on the National Board of Dental Hy- giene Examination (NBDHE)? Were there statistically significant differences in learner performance when considering GPAs? Did statistically significant differences in performance exist relating to individual course grades? T– tests were used for data analysis in answering the research questions. Results: From a cumulative perspective, no statistically significant differ- ences were apparent for the NBDHE and GPAs or for individual courses. Conclusions: Interactive Television (ITV), the synchronous DL system ex- amined, was considered effective for delivering education to learners if simi- lar performance outcomes were the evaluation criteria. Key Words: Distance Education, Distance Learning, Outcomes, Benchmarks, Dental Hygiene, Assessment, Student Performance, Allied Health This study supports the NDHRA priority area, Professional Education and Development: Validate and test measures that evaluate student critical thinking and decision making skills. ResearchResearch Introduction Educational opportunities con- tinue to grow exponentially in colleges and universities across the United States because of the implementation of technologi- cally based delivery systems.1 Distance learning (DL) or dis- tance education (DE), defined as the learner and facilitator separated by physical space, is facilitated by rapid advances in today’s technology.2 Technologi- cal changes, including computer use and fiber–optic cabling, syn- chronous interactive television systems (ITV) and other hybrid configurations, allow learning in other than the traditional face–to– face classrooms.3 In 2002, 1.6 million students took courses on–line across the United States. By 2003, the num- ber of online course enrollees climbed to 1.97 million, succeed- ed by 2.33 million in 2004.4,5 De- livery of online courses requires a technological platform and com- puter.6 Continued growth and ex- pansion of online coursework is now an integral element of main- stream higher education – 63% of schools offering undergraduate face–to–face programs also offer undergraduate programs online.7 Three graduate programs, 35 un- dergraduate and 12 dental hygiene degree completion programs in the United States now use vary- ing forms of DL.8,9 Even though a significant body of literature ex- ists documenting participant sat- isfaction using alternative mediums for delivering dental hygiene cours- es, 10–12 only one published quantita- tive research study existed verifying actual performance while receiving dental and dental hygiene education using various alternative delivery systems.13 Quantitative analyses of performance benchmarks from a programmatic perspective, such as grade point average (GPA), course 76 The Journal of Dental Hygiene Volume 84 Issue 2 Spring 2010 Review of the Literature The ADA CODA Standard 1 requires that “benchmarks or mea- sures of learning are used to demon- strate effectiveness”14 and are docu- mented in the areas of planning and assessment. Addressing the quality of educational programs is the in- tent of this Standard. Educational programs have a responsibility to assure the ongoing cycle of edu- cational assessment, planning and implementation. Evaluation assures learners receive current, relevant and cost efficient educational pro- gramming. Programs are account- able to stakeholders in assuring educational programs meets the ADA CODA Quality Standards. A key consideration of the Standards is broad–based, systematic, contin- uous data collection and analysis, which is designed to evaluate and promote achievement of program goals while maximizing academic success of enrolled students. The CODA Standard allows programs to identify their own goals, assur- ing competence within the disci- pline. Ongoing program assessment Methodology This research was designed to address the question: To what extent was learner performance on stan- dard benchmark assessments im- pacted when using technology as a delivery system? The evaluation of NBDHE scores, course grades and grades and scores on national ex- aminations provide research evi- dence of individual’s educational experiences in alternative delivery programs. The purpose of this study was determining whether face–to–face learners performed differently on established benchmark assessments than their classmates located at a distance. The defined benchmarks include course grades, GPAs and the National Board of Dental Hy- giene Examination (NBDHE). A longitudinal examination of bench- mark scores during a 10 year period looked for statistically significant differences in performance. Evalua- tion of entire educational programs, rather than single courses, addressed issues concerning data reliability, validity and the American Dental Association Commission of Dental Accreditation (ADA CODA) Stan- dard 1, relating to Institutional Ef- fectiveness. uses the NBDHE, GPAs and course grades to provide direct benchmark measures that are indicators of In- stitutional Effectiveness. Indirect measures, including retention, attri- tion, program transfer and employer satisfaction data also identify and document areas for continuing pro- cess and program improvements. Educational technology contin- ues transforming dental hygiene ed- ucation at a rapid pace. A review of the current dental hygiene literature indicates various program delivery models are being used by educa- tional institutions and program ad- ministrators to implement alterna- tive delivery methods for teaching and learning, as well as ongoing assessment.15–17 As institutions con- sider expanding learning using DL as a delivery modality, theoretical constructs need to be considered, such as those offered by Gussy et al18 and Magnussen,19 or theorized by this researcher. Additionally, as institutions consider advancing the profession at large by expand- ing educational opportunities for degree completion as suggested by Monson and Engeswick,20 it is cru- cial the concrete, statistical mea- sures of predictive performance be examined.21–22 A focus on statistical performance measures for authenti- cation instead of learner satisfaction data can provide a solid, evidence– based groundwork for continued use of DL for dental hygiene education. As dental hygienists seek to pursue degree completion and additional certification, it is important that both institutional and programmatic decisions are made based on solid predictors of academic performance over time rather than only student satisfaction surveys or outlier data from 1 or 2 courses.10–12 GPAs may ascertain program ef- fectiveness in the delivery of dental hygiene education. Learners were self–selected for statistical analysis based on location. The first group consisted of face–to–face (host site) learners while the second group consisted of learners located at a distance from the instructor via ITV. The study design evaluated if face– to–face and distance participants had statistically significant differences in performance.22 Independent vari- ables included location and time. Location determined group assign- ment while time included examin- ing graduate populations from 1997 through 2006. Dependent variables included course GPAs, cumulative GPAs and NBDHE scores. Research Questions Three questions were asked re- garding documentation of perfor- mance between face–to–face and distance college learners: Were there significant statistical 1. differences in performance on the NBDHE? Were there significant statistical 2. differences in GPAs? Were there significant statistical 3. differences in individual course grades? Population This study’s research population began with 266 students from the graduating classes of 1997 through 2006. One hundred and eighty–nine students graduated from the pro- gram in 10 years. There were 117 (44%) face–to–face instructor–lead learners, while 72 (27%) used the ITV system. All data files for the learners who graduated (n=189) were used for the study. Over the 10 year period, 77 stu- dents (29%) did not complete all ed- ucational coursework and graduate. Learners were required to complete exit interviews with a neutral party upon deciding to leave the program and institution. The data collected from individuals who did not com- plete the program (n=77) was used as part of the program effectiveness Volume 84 Issue 2 Spring 2010 The Journal of Dental Hygiene 77 Results Each of the study’s research questions are addressed based upon the statistical analyses while look- ing at each individual supposition. Discussion highlights findings from a program perspective, examining program effectiveness over time that may be of concern to allied health administrators and educators. Research Question 1 The first research question asked: Did significant statistical differenc- es in performance on the NBDHE exist between face–to–face and distance college learners? Cumula- tive data demonstrated no statisti- cally significant difference existed between host and distance learners’ performance over a 10 year period (Table I. Research Question 2 The second research question asked: Did significant statistical differences in GPAs exist between face–to–face versus DE college– level learners? No statistically sig- nificant differences existed between host and distance learners’ perfor- mance related to cumulative GPAs for the 10 cohorts (Table II). Research Question 3 The third research question asked: Did significant statistical dif- ferences in individual course grades Year Location N Mean df t–value 2–Tailed Prob. 1997–2006 Host 114 89.19 183 –.109 .914 Distance 71 89.79 *p < .05 Table I: Ten year Cumulative Analysis of National Board Dental Hygiene Examination Scores by Host and Distance Learners Year Location N Mean df t–value 2–Tailed Prob. 1997–2006 Host 117 3.37 186 –1.079 .282 Distance 71 3.43 *p < .05 Table II: Ten year Analysis of Cumulative GPAs by Host and Distance Learners process. Thirty–six persons (13.5%) not completing were located face– to–face and 41 persons (15.5%) re- ceived didactic education through ITV. Academic performance, while one reason for attrition, was not con- sidered a major factor. Attrition was equally distributed between both groups and was attributed to several categories beyond academic perfor- mance, including personal, health, ethical conduct considerations, mil- itary commitments, the profession not being “right” for the individual and death of 1 participant. It should also be noted the majority of attri- tion occurred during the first year of the program. The state where the research data was gathered has a statewide curriculum in place for dental hygiene education. This per- mits students to change academic location, if space is available, with- in the state. As individual personal situations change and people move, they can still complete educational programs elsewhere. While consid- ering the data, it should be noted a substantial number of individuals (n=14) transferred to different pro- grams within the state, taking ad- vantage of the statewide curriculum if personal situations warranted its necessity. Ten persons withdrew, citing they did not like instructor. This information is also important from a programmatic perspective as we consider CODA Standard 1. Seventy–five percent of participants completed the program and gradu- ated. Colleges and universities have the latitude under the auspices of CODA Standard 1 to determine ac- ceptable levels of attrition. While 29% of individuals not completing the program over a 10 year period might seem high, when considering Institutional Effectiveness, the pro- gram used the data to make continual revisions to the admissions process, faculty teaching assignments and tenure decisions. Positive impacts on increasing the state’s health care provider numbers and cost/benefit to the state’s economy were also factors documented to meet CODA Standard 1. Upon closer review, the data demonstrates, despite this attri- tion, the program was effective in providing and graduating dental hy- gienists using DL as an educational delivery mechanism. Data Analysis For each research question, inde- pendent group t–tests determined if any statistically significant differ- ences existed. Data was analyzed year–by–year, course by course and through cumulative comparisons. Furthermore, a t–test was applied to the aggregated group’s data. Only a portion of the data findings are re- ported here. Data analysis was con- ducted for documenting program effectiveness of DL. The results of this study, in documenting learner performance for an entire program, addresses one of 2 “gaps” noted in the scientific literature by Phipps and Merisotis23 – research learner outcomes for entire academic pro- grams and not just for individual courses. Phipps and Merisotis also recommended proposing and using a conceptual framework for consid- eration and potential testing for fur- ther DL research, which is posited separately by this author. Statistical Significance Statistical level of significance (p–value) was set at p<0.05. This value level is a routine alpha–level for probability testing of null hy- potheses. 78 The Journal of Dental Hygiene Volume 84 Issue 2 Spring 2010 exist between face–to–face and dis- tance college learners? Data was analyzed answering this question in 3 different ways: cumulatively, year by year and course by course, to identify any existing statistically significant trends. Only the results of the cumulative analysis are ad- dressed here. Comparing cumula- tive learner performance for the core dental hygiene courses revealed no statistically significant differences (Table III). It was determined no statistically significant differences existed in learner performance for the program’s entire didactic aca- demic curriculum over a 10 year period. It is important to note that year by year and course by course, statistically significant results were identified. The results were evaluated from a programmatic perspective while examining trends that might be of concern when considering CODA Standard 1. The results of the study, reflecting a 10 year period of com- parative data, identified no statisti- cally significant performance dif- ferences between face–to–face and distance learners on the various benchmark measures evaluated. Analysis of NBDHE scores, GPAs and cumulative core course grades were used to determine if CODA Standard 1 was met. Pro- gram completion rates, graduate success on the analyzed bench- marks, program improvement and change based on assessment data, plans, timelines and programs ef- fectiveness in meeting the stated missions, goals and strategic plans are all used as evidence document- ing Institutional Effectiveness. The data analyzed here as a component of effectiveness assessment sug- gests DE was as effective as tradi- tional methods for delivering edu- cational programming. Question 1 As a trend over time, no major differences were observed in per- formance on the NBDHE between host and distance learning cohorts. The research presented in Table I Oral Anatomy (508–101) Course Averages by Host and Distance Learners Student Year Location n Mean df t–value 2–Tailed Prob. 1997–2006 Host 107 2.96 134 1.223 .223 1997–2006 Distance 69 2.84 Dental Hygiene Theory I (508–113) Course Averages by Host and Distance Learners Student Year Location n Mean df t–value 2–Tailed Prob. 1997–2006 Host 114 3.73 184 1.445 .150 1997–2006 Distance 72 3.64 Nutrition (508–114) Course Averages by Host and Distance Learners Student Year Location n Mean df t–value 2–Tailed Prob. 1997–2006 Host 107 3.36 177 .001 .999 1997–2006 Distance 72 3.36 Periodontology (508–115) Course Averages by Host and Distance Learners Student Year Location n Mean df t–value 2–Tailed Prob. 1997–2006 Host 111 3.05 179 .853 .395 1997–2006 Distance 70 2.97 Oral Pathology (508–122) Course Averages by Host and Distance Learners Student Year Location n Mean df t–value 2–Tailed Prob. 1997–2006 Host 109 3.41 129 1.19 .233 1997–2006 Distance 72 3.30 Dental Pharmacology (508–123) Course Averages by Host and Distance Learners Student Year Location n Mean df t–value 2–Tailed Prob. 1997–2006 Host 116 3.35 185 .322 .748 1997–2006 Distance 71 3.32 Dental Hygiene Theory II (508–124) Course Averages by Host and Distance Learners Student Year Location n Mean df t–value 2–Tailed Prob. 1997–2006 Host 117 3.65 187 1.199 .232 1997–2006 Distance 72 3.57 Community Dental Health (508–131) Course Averages by Host and Distance Learners Student Year Location n Mean df t–value 2–Tailed Prob. 1997–2006 Host 116 3.70 186 –1.09 .913 1997–2006 Distance 72 3.71 *p < .05 Table III: Ten year Analysis of Core Dental Hygiene Courses provides exploratory, longitudinal DL data for this national benchmark, providing documentation of learner success as an example of evidence meeting CODA Standard 1. Question 2 Based on the GPA evidence pre- sented in Table II, the data suggests DE can be effective for dental hy- giene education. This data may also Volume 84 Issue 2 Spring 2010 The Journal of Dental Hygiene 79 Conclusion Allied health and dental hygiene programs should continue offering education using DL as an alterna- tive delivery mechanism. This re- search identified learners at various locations who performed equally well on standard benchmark as- sessments documenting program effectiveness. DL was considered an effective medium for delivering educational programming, and the use of DL should be continued. Ad- ditionally, DL could be used for de- livering not only other allied health programs, but could also deliver other educational programs in the Acknowledgement The author thanks Mary Jo Ti- etge and Amelia Ann Fox, techni- cal writer, and the administration, faculty and staff of the Wisconsin Technical College System dental hygiene program, which allowed this study to be conducted. Discussion The statistical analyses of per- formance outcomes suggest CODA Standard 1 was met as evidenced by the 10 year cumulative NBDHE and GPA data, and the cumulative course by course data. The results docu- mented learner performance for an entire academic program rather than only analyzing individual cours- es, addressing one of the research gaps noted by Phipps and Meriso- tis.23 The data also helps answer the question: Is it prudent for colleges’ and universities’ dental and dental hygiene programs to continue ex- pansion of the use of DE as a means of course and program delivery? Based on these preliminary research findings, the answer appears to be yes. It is important to note the study design limits findings to dentistry and might not be applicable to other provide preliminary evidence for this national performance bench- mark for DL programs. Further- more, from a program perspective, this data also documents learner success as another example of evi- dence meeting CODA Standard 1. Even though some learners might not prefer DL for educational deliv- ery, their individual performance, as evidenced by GPAs, was not im- pacted by its use. Question 3 Cumulative analysis of core di- dactic course grades also provides a pattern of evidence documenting meeting the intent of Standard 1. An examination of the data for signifi- cant trends indicates these courses’ cumulative averages have remained consistent and stable over time. The 8 core dental hygiene courses did not show statistically signifi- cant differences in performance between face–to–face and distance students. This study documents one programs’ successful delivery of a dental hygiene educational program while using DL. educational programming. Another question posed is: Can a perspective body of knowledge be generated regarding learner perfor- mance on given standard outcome measures? Performance measured by GPAs and course grades as na- tional benchmarks provide gener- alized, external validity to other programs because this study data lays the foundation for DL research related to academic programs. The findings of this research also pro- vide insights into the use of DL as a viable delivery mechanism for education. In addition, it provides a foundational basis for benchmark comparison for future DL research for programs considering using this method for program delivery. If similar performance outcomes are the decision–making factor, the study findings suggest the use of DE is a viable mechanism for educa- tional delivery. Finally, this research provides an analysis of performance over time, rather than 1 or 2 years, for an entire educational program using DL technologies. This data might assist administrators at vari- ous colleges or universities in de- cision–making processes regarding the implementation of DL program- ming for general education, allied health or, specifically, dental and dental hygiene programs. same manner. A path analysis should be under- taken as we consider inferring data back to larger populations for both GPAs and course grades. It must be stated that DL is not for every- one. Further research may include analyzing this national benchmark data and determining what factors promote student success in dental hygiene DL programs. Once research is conducted identifying persons for whom the DL option is not a preferred de- livery mechanism, strong recom- mendations could be made to that individual or individuals to pro- vide guidance into taking program courses with face–to–face cohorts for improving performance results, retention, learner satisfaction, per- severance and, ultimately, program completion and graduation. There is a need to replicate this study using data from asynchronous DL programs, which have replaced the synchronous model studied in the current paper. . Jodi L. Olmsted is an Assistant Professor at the University of Wis- consin Stevens Point in the College of Professional Studies, Health Sci- ences Department and was with the first DH program that began offer- ing its entire didactic curriculum to students via distance education. Jodi was the 2008 recipient of the ADHA Irene Newman Professional Development Award. She is the owner, president and distance edu- cation consultant for Kaleidoscope Consulting, LLC. 80 The Journal of Dental Hygiene Volume 84 Issue 2 Spring 2010 Wilson M. Distance Degrees. Oakland (OR): Um-1. pqua Educational Resource Alliance; 2001. Wahlstrom C, Williams BK, Shea P. The successful 2. distance learning student. Belmont (CA): Scratch- gravel Press; 2003. O’Lawrence H. The influences of distance learning 3. on adult learners. Techniques. 2006;81:47–49. Allen IE, Seaman J. Seizing the opportunity: The 4. quality and extent of online education in the United States, 2002 and 2003. Wellsley (MA): Sloan Con- sortium; 2003. Allen IE, Seaman J. Entering the mainstream: The 5. quality and extent of online education in the United States, 2003 and 2004. Wellsley (MA): Sloan Con- sortium; 2004. Rossman MH, Rossman ME. Facilitating distance 6. education. San Fransisco (CA): Jossey–Bass; 1995. Allen IE, Seaman J. Growing by degrees: Online ed-7. ucation in the United States, 2005. Wellsley (MA): Sloan Consortium; 2005. Dental hygiene programs. American Dental Asso-8. ciation [Internet]. 2004. Available from: http:www. ada.org/prof/ed/programs/dahlt/alliedus.asp. Dental hygiene education programs offered by state. 9. American Dental Hygienists’ Association [Internet]. 2008. Available from: http://www.adha.org. Moore W. An assessment of online learning in a 10. dental hygiene baccalaureate degree completion program. J Dent Hyg. 2007;81:84–84. Mitchell TV, Gadbury–Amyot CC, Bray KK, Sim-11. mer–Beck M. Advanced degree seeking students’ satisfaction with online courses at UMKC––an early investigation. J Dent Hyg. 2007;81(3):62–62. Grimes EB. Student perceptions of an online dental 12. terminology course. J Dent Educ. 2001;66(1):100– 107. Olmsted JL. Longitudinal analysis of student perfor-13. mance in a dental hygiene distance education pro- gram. J Dent Educ. 2002;66(9):1012–1020. American Dental Association, Commission on Den-14. tal Accreditation. Accreditation standards for dental hygiene education programs. Chicago (IL): Ameri- can Dental Association; 2005. Gadbury–Amyot CC, Fried JL, Syme, SE. Technol-15. ogy in teaching and online (distance) learning: Two model programs. Access. 2007;21:10–17. Fried JL. Innovations in Education and Technology. 16. J Dent Hyg. 2007;81(1):1–9. Bray KS, Gadbury–Amyot CC, Mitchell TV. Pro-17. viding advanced degrees in dental hygiene via com- puter mediated distance learning: A model program. J Contemp Dent Pract. 2006;7(5):96–107. Gussy MG, Knevel RJ, Sigurdson V, Karlberg G. 18. Theoretical and practical considerations for the development of online international collaborative learning for dental hygiene students. Int J Dent Hyg. 2006;4(3):154–159. Magnussen L. Applying the principles of significant 19. learning in the e–learning environment. J Nurs Educ. 2006;47(2):82–86. Monson AL, Engeswick LM. ADHA’s focus on ad-20. vancing the profession: Minnesota’s dental hygiene educators’ response. J Dent Hyg. 2007;81(2):1–12. Alzahrani MJ, Thomson EM, Bauman DB. Pre-21. dictors of Student Success in an entry–level bac- calaureate dental hygiene program. J Dent Hyg. 2007;81(2):1–13. Bernard RM, et al. How does distance education 22. compare with classroom instruction? A meta– analysis of the empirical literature. Rev Educ Res. 2004;74(3):379–434. Phipps R, Merisotis J. What’s the difference? A re-23. view of contemporary research on distance learning in higher education. Washington, DC. National Edu- cation Association; 1999. References http://www.ingentaconnect.com/content/external-references?article=1043-254x(2007)81:3L.62[aid=9141097] http://www.ingentaconnect.com/content/external-references?article=0022-0337(2002)66:9L.1012[aid=7935139] http://www.adha.org http://www.ada.org/prof/ed/programs/dahlt/alliedus.asp http://www.ada.org/prof/ed/programs/dahlt/alliedus.asp