ajpe90 1..6 INSTRUCTIONAL DESIGN AND ASSESSMENT Using Performance-based Assessments to Evaluate Parity Between a Campus and Distance Education Pathway Thomas L. Lenz, PharmD, MA, Michael S. Monaghan, PharmD, Amy F. Wilson, PharmD, Jennifer A. Tilleman, PharmD, Rhonda M. Jones, PharmD, and Mary M. Hayes Creighton University Medical Center Submitted September 8, 2005; accepted January 12, 2006; published August 15, 2006. Objectives. To compare the performance of campus-based students with that of distance students during the first 2 years of a doctor of pharmacy program to evaluate parity between the pathways. Methods. Twelve cases were created for each year of the program along with performance criteria. The cases were converted into computer-based simulations for programmatic assessment at the end of the 2002-2003 and 2003-2004 school years. All first-professional year (P1) and second-professional year (P2) students participated in the assessments. Overall class means were calculated and used to compare student performances between campus and distance education pathways. Results. Overall scores for the 2003 P1 class were 56.4% for the campus-based students and 62.4% for the distance students, (p 5 0.002); overall scores for the 2003 P2 class were 48.8% and 55.5%, re- spectively (p , 0.0001). The 2004 overall scores for P1 campus and distance students were 59.0% and 65.7%, respectively, (p 5 0.001); and for 2004 P2 scores the results were51.8% and 56.5%, respec- tively (p 5 0.049). Conclusions. Students receiving their pharmacy education via a distance pathway scored higher on performance-based assessments compared with students receiving their pharmacy education via the traditional campus-based pathway. This indicates that distance students are receiving at least an equivalent curricular experience in the P1 and P2 years compared to that received by campus-based students. Keywords: performance-based assessment, distance education, abilities-based curriculum INTRODUCTION The aging population and corresponding increase in prescriptions has created a current and future demand for pharmacists.1 This workforce demand has in turn produced an unprecedented period of growth in phar- macy education. Universities are opening new schools of pharmacy, while existing schools and colleges are de- veloping innovative ways to increase their class sizes. They are graduating more practitioners by developing satellite campuses, and employing distance education technology. 2 Our school addressed this shortage of pharmacists through the development of a distance-based doctor of pharmacy degree pathway in which all didactic courses, with the exception of 2 laboratory-based courses and the clinical rotations, are delivered via the Internet. A more complete description of this pathway and its development is available in a previous publication. 3 This increase in enrollment was not without chal- lenges such as ensuring that we effectively increase class size without sacrificing quality. The courses are identical in content to the campus-based pathway, and when pos- sible, are delivered at the same time via distance technol- ogy. But not all courses are taught simultaneously or by the same instructors. Some of the distance courses are taught out of sequence and by different instructors. Also, differences in examination administration exist with the differences in pathways. For example, the distance-based pathway may use QuestionMark to administer examina- tions electronically, while in the campus-based course, a paper-based format is used. Therefore, we required a controlled testing condition to account for potential course-to-course variations in examinations. The purpose of this project was to provide a standardized testing environment to compare the performance of campus- based students to that of distance-based students for the Corresponding Author: Tom Lenz, PharmD. Address: Assistant Professor of Pharmacy Practice, School of Pharmacy and Health Professions, Creighton University Medical Center, 2500 California Plaza, Omaha, NE 68178. Tel: 402-280-3144. Fax: 402-280-3320. Email: tlenz@creighton.edu. American Journal of Pharmaceutical Education 2006; 70 (4) Article 90. 1 first 2 years of the program to evaluate parity between the pathways. METHODS The development of test methods for this project has been described in detail in a previous publication.4 Our program adopted an abilities-based curriculum and estab- lished 12 terminal educational ability-based outcomes. These ability-based outcomes became the basis for com- paring the 2 curricular pathways. The first step was case development based on these ability-based outcomes. Twelve faculty teams were formed: 1 team per ability- based outcome. When developing the faculty teams, care was taken to select faculty memberswho taughtcourses in which the outcomes were listed in the course syllabus as a primary focus. Faculty teams took the terminal educa- tional ability-based outcomes and defined expected out- comes for student performance by each year in the curriculum. In other words, the terminal ability-based outcome (Appendix 1) was defined by producing student performance expectations or tasks for each year in the curriculum. Twenty-four different cases (12 for P1 and 12 for P2) were authored for the computer-based clinical simulations; each computer-based clinical simulation as- sessment that students completed consisted of 12 case stations. Each station was composed of a case, directions to the student, case performance criteria, and any refer- ences deemed necessary to complete the problem or clin- ical task(s) required in the case. The faculty members were instructed to use their pro- fessional/clinical experience to describe a problem that focused on the ability-based outcome level in question. This real-life problem then constituted the case scenario. For each case brief descriptions of the context of the prob- lem were provided to ensure congruence between stu- dents’ perception and actual case construct. These directions allowed the students to quickly identify the tasks required to complete the case. An example of stu- dent directions for a P1-level case is provided in Appen- dix 2. Case writers defined the case scenario, directions to the student, performance criteria used for evaluating stu- dent performance, and identified any references neces- sary for solving the case. The performance criteria were in the form of rating scales or checklists, which are proven means of evaluating clinical skills in health professional education. 5,6 These checklists consisted of performance indicators (ie, items) or observable behaviors that could be scored by a faculty grading panel. Validations of case content and checklists were based on a previously described method.7 Briefly, the case con- tent,ratingscale,anddirectionswerereviewedbyafaculty validation panel. The panel was composed of 5 pharmacy faculty members from different pharmacy schools: 2 rep- resentatives from the East Coast (Pittsburgh, Shenan- doah), 1 from the Midwest (Drake), 1 from the West Coast (UCSF), and 1 from the South (Houston). Two of the faculty members were employed in private pharmacy schools (Shenandoah and Drake) while the remainder came from state schools. The purpose of the validation was twofold. First, to ensure that each case appropriately tested the specific outcome in question (ie, construct) and that the case was true to the context in which it was set (ie, problem or clinical condition). Second, the panel addressed the generalizability of the cases to their institu- tion and region of the country. Comments from the panel were used to produce the final cases. The cases were then developed into online computer simulations and pilot tested at the pharmacy school by P4 clerkship students. Twenty minutes were allotted for each case. Case refer- ences and props (eg, prescription bottles) were case- dependent and could not be moved; thus, students physi- cally rotated from one computer to another until all cases were completed. The students provided feedback and the assessment procedures were amended based on their com- ments. The revised assessments were then administered to both campus and distance students. Wired classrooms were utilized for access to computers at each case. All student responses were scored by a 2-member faculty grading panel according to the performance crite- ria indicated in the checklists. Each item on a rating scale was scored dichotomously (ie, ‘‘yes, the student com- pleted/preformed the item successfully’’ or ‘‘no, the stu- dent did not’’). The number of items varied by case, and each case was treated as an independent unit (question) of the entire performance-based assessment since each case assessed a different outcome. The 2 scores generated by the grading panel were averaged and this mean repre- sented the students’ score. Overall class means were calculated and used to com- pare student performances for both campus and distance pathways using a Mann-Whitney U test. This project was reviewed by the Institutional Review Board for Research and deemed to be exempt. RESULTS All P1 and P2 students in the program completed the assessments. Data were similar with no statistical differ- ence found between the groups in entering grade point averages (Table 1). Overall scores for the 2003 academic year are listed in Table 2. The mean score for P1 campus students’ was 56.4% and for P1 distance students, 62.4%, (p 5 0.002). The mean score for P2 campus students was 48.8% and for P2 distance students, 55.5% (p , 0.0001). Table 3 lists the overall scores for the 2003-2004 American Journal of Pharmaceutical Education 2006; 70 (4) Article 90. 2 academic year. Scores for P1 campus students compared to those of distance students were 59.0% vs. 65.7%, re- spectively (p 5 0.001). Scores for P2 campus students compared to those of distance students were 51.8% vs. 56.5%, respectively (p 5 .049). Course-grade compari- sons between the 2 pathways for P1 and P2 are provided in Tables 4 and 5, respectively. DISCUSSION Studentsreceivingtheirpharmacyeducationviaadis- tance-based pathway scored higher on performance- based assessments compared with students receiving their pharmacy education via the traditional campus-based pathway. This indicates that distance students are receiv- ing at least an equivalent curricular experience in the P1 and P2 years compared to that received by campus-based students. Severalreasonsexistthatjustifytheneedforthisstudy. An examination of the course grade comparisons between the 2 pathways is provided in Tables 4 and 5 and is difficult to interpret. Knowing that there is some variation in course sequence, course instructor, and minor differences in test- ing conditions, some controlled means of assessing parity between the pathways is necessary. Also, since our pro- gram has an abilities based curriculum, and not all tests utilized are performance-based, it is programmatically es- sential to identify how well students are achieving or per- forming the program’s ability-based outcomes. The use of performance assessments to compare stu- dent outcomes between the 2 pathways was important. The Accreditation Council for Pharmacy Education (ACPE) and the American Association of Colleges of Pharmacy (AACP) encourage the adoption of an abilities-based curriculum. 8,9 Our school has adopted an abilities-based curriculum, and therefore when evaluating such a program it would be appropriate to use perfor- mance assessments as they require students to demon- strate or produce evidence of learning.4 Our data indicate that learning occurred; that students were able to demonstrate competencyof theabilities. This is unique in that one may assume student experiences were different based on the pathway. That is, learning was clearly documented even though the learning expe- riences may have differed based on the pathway. Even though the distance students scored signifi- cantly higher than the campus students in our study, nei- ther of the groups performed well on the assessments. One possible explanation for the low scores may be that our students arenotprimarilyassessedviaperformance-based methods during the first 2 years of the program. This may account for the discrepancy between expectations (ability-basedoutcomesareperformance-based)andscores. The data from this analysis only included students who were in the first- and second-professional years of the program. To get a clearer and more overall picture of students’ abilities, we must also look at the assessment data from the third- and fourth-professional years. The P3 and P4 assessments were conducted using standardized participants that included the Objective Structured Clin- ical Examination (OSCE) approach. 4 These data are cur- rently being analyzed. In addition, looking at individual course scores, overall GPA, and NAPLEX results may give additional information to assess parity between the 2 pathways. These analyses are underway. Once all of our performance assessment data are an- alyzed, it will be used in aggregate to determine the ef- fectiveness of the curriculum’s content in terms of preparing students to meet the program’s educational ability-based outcomes. 4 The combination of student per- formance data regarding specific programmatic outcomes may be used to monitor overall effectiveness, demon- strate improvement, and provide accountability of student achievement to all constituents.4,10 The data can then be related back to the instructional methods and used for Table 1. Demographics of Pharmacy Students Enrolled in Distance and Campus Pathways Variable P1 P2 Distance Campus Distance Campus Enrolled, No. 60 111 54 102 Mean GPA 3.31 3.36 3.18 3.21 Sex Male 19 (32%) 36 (32%) 22 (41%) 39 (38%) Female 41 (68%) 75 (68%) 32 (59%) 63 (62%) Previous degree 31 (52%) 36 (32%) 34 (63%) 40 (39%) Mean age, y 31 25 34 26 P1 5 first-year students; P2 5 second-year students; GPA 5 grade point average Table 2. Comparison Results for of 2002-2003 P1 and P2 Campus and Distance Students Campus P1 Distance P1 Campus P2 Distance P2 Number of students 96 52 96 45 Maximum number of points possible 54 54 57 57 Mean score (SD) 30.4 (7.3) 33.68 (5.7) 27.8 (6.3) 31.61 (3.85) Mean score as a percent 56.4 62.4 48.8 55.5 American Journal of Pharmaceutical Education 2006; 70 (4) Article 90. 3 program planning, such as the incorporation of increased active-learning strategies in the classroom.4 Further, the information in this study may be useful to other schools of pharmacy in making decisions regarding adopting a dis- tance-based pharmacy pathway or to model the perfor- mance-based assessment approach we have taken with their own abilities-based curriculum. Limitations Because the distance students take their didactic courses online, the distance students may be more adept at online assessments than the campus students. This may confound our results. Also, because different instructors were used in a few of the courses, our results may reflect differences in pedagogical effectiveness versus absolute student learning. Finally, because this is the first large performance-based assessment of hundreds of students, no previous data exist with which to compare. Therefore, it is difficult to interpret the low scores. To determine whether other students would perform similarly or better, more research is needed. CONCLUSIONS The performance-based assessment data indicate that our distance pathway provides a similar learning experi- ence in the P1 and P2 years to that provided by the campus pathway based on student performance. This study pro- vides data to help support the evaluation of parity between the 2 pathways to date based on student performance. Table 3. Comparison of Results for 2003-2004 P1 and P2 Campus and Distance Students Campus P1 Distance P1 Campus P2 Distance P2 Number of students 99 38 87 52 Maximum number of points possible 54 54 57 57 Mean score (SD) 31.97 (4.94) 35.48 (5.24) 29.56 (6.98) 32.23 (5.13) Mean score as a percent 59.0 65.7 51.8 56.5 Table 4. Comparison of Grades Between P1 Students Enrolled in Campus and Distance Pathways Class Campus Mean (n)* Distance Mean (n)* P Anatomy 3.81 (94) 3.83 (52) 0.259 Biochemistry 3.02 (106) 2.78 (55) 0.216 Pharmacy Calculations 3.58 (111) 3.46 (61) 0.236 Physical Pharmacy 3.79 (110) 3.73 (59) 0.899 Health Care Systems 3.80 (112) 3.76 (64) 0.691 Communication Skills 3.37 (110) 3.79 (55) 0.000y Introduction to Drug Information & Pharmaceutical Care 3.73 (110) 3.50 (60) 0.000y Physiology 2.97 (84) 2.91 (55) 0.914 Introduction to Disease 3.86 (109) 3.74 (59) 0.055y Dosage Forms and Drug Delivery Systems 3.42 (109) 3.42 (61) 0.553 Biostatistics & Research Design 3.30 (99) 3.24 (48) 0.980 Pharmacy Practice Management 3.80 (109) 3.82 (61) 0.470 Early Pharmaceutical Care Experience I 3.95 (109) 4.00 (55) 0.056 *Mean based on the following scale A 5 4.0, B1 5 3.5, B 5 3.0, C1 5 2.5, C 5 2.0, D 5 1, F 5 0 yp , .05; significance determined using a Mann-Whitney U analysis Table 5. P2 Student Grade Comparison Between the Campus and Distance Pathways Class Campus Mean (n)* Distance Mean (n)* P Microbiology 3.37 (111) 3.11(62) 0.021y Pharmacology I 3.01 (110) 3.05 (58) 0.980 Parenteral Drug Products 2.99 (107) 3.27 (55) 0.038y Chemical Basis of Drug Action I 2.56 (113) 1.73 (59) 0.000y Pharmacology II 2.93 (105) 2.71 (58) 0.039y Basic Pharmacokinetics 3.99 (105) 3.99 (55) 0.660 Chemical Basis of Drug Action II 3.00 (107) 2.77 (57) 0.095 Nonprescription Therapeutics 3.80 (106) 3.64 (59) 0.003y Patient Assessment 3.38 (109) 3.76 (55) 0.000y Early Pharmaceutical Care Experience II 3.98 (105) 3.98 (58) 0.561 *Mean based on the following scale A 5 4.0, B1 5 3.5, B 5 3.0, C1 5 2.5, C 5 2.0, D 5 1, F 5 0 yp , .05; significance determined using a Mann-Whitney U analysis American Journal of Pharmaceutical Education 2006; 70 (4) Article 90. 4 ACKNOWLEDGMENT This project is sponsored in part by the Fund for the Improvement of Postsecondary Education (FIPSE), US Department of Education and by the Creighton University 2004 Summer Assessment Grant, Office for Excellence in Teaching, Learning, and Assessment, Creighton Univer- sity, Omaha, Nebraska. REFERENCES 1. Pedersen CA, Doucette WR, Gaither CA, Mott DA, Schommer JC. Final Report of the National Pharmacist Workforce Survey: 2000. Available at: http://www.aacp.org/site/ page.asp?TRACKID5&VID51&CID5360&DID53490. Accessed September 7, 2005. 2. Davison M, Medina MS. Addressing industry needs through innovative technologies. Am J Pharm Educ. 2003;67(3): Article 100:3-4. 3. Malone PM, Glynn GE, Stohs SJ. The development and structure of a web-based entry-level Doctor of Pharmacy pathway at Creighton University medical center. Am J Pharm Educ. 2004;68:Article 46. 4. Monaghan MS, Jones RM. Faculty Case Writing Teams. Designing an assessment for an abilities-based curriculum. Am J Pharm Educ. 2005;69: Article 19:118-25. 5. Stillman PL, Gillers MA. Clinical performance evaluation in medicine and law. In: Berk RA, ed. Performance Assessment: Methods and Applications. Baltimore MD: Johns Hopkins University Press; 393-445. 6. Andrew BJ. The use of behavioral checklists to assess physical examination skills. J Med Educ. 1977;52:589-91. 7. Monaghan MS, Vanderbush RE, Gardner SF, Schneider EF, Grady AR, McKay AB. Standardized patients: an ability-based outcomes assessment for the evaluation of clinical skills in traditional and nontraditional education. Am J Pharm Educ. 1997;61:337-44. 8. American Council on Pharmaceutical Education. Accreditation Standards on Guidelines for Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree Adopted June 14, 1997. Available at: www.acpe-accredit.org/frameset_Pubs.htm. Accessed August 19, 2005. 9. Center for the Advancement of Pharmaceutical Education Advisory Panel on Educational Outcomes. Alexandria, Va: American Association of Colleges of Pharmacy; 1998. 10. Arrasmith DG, Kushman JW, Olson N. Using performance information for improvement. In: Blum RE, Arter JA, eds. A Handbook for Student Performance Assessment in an Era of Restructuring. Alexandria, Va: Association for Supervision and Curriculum Development; 1996:VII-1. Appendix 1. Ability-based educational outcomes for graduates. Pharmaceutical Care Abilities 1. Patient Assessment - The student shall contribute to the database of information about the patient by: a) performing a medication history, review of systems, and physical assessment; b) recommending and interpreting laboratory tests; c) assessing medical, psychosocial, behavioral, and economic status; and d) identifying potential drug-related problems. 2. Pharmaceutical Care Plan Development - The student shall develop pharmaceutical care plansa that maximize the patients’ response to drug therapy and prevent or resolve a drug-related problem(s) in order to ensure positive out- come(s). The student shall appropriately document the implementation of and outcomes related to the care plan. The pharmaceutical care plan shall include medical devices, as needed, and educational information (e.g., nutrition, lifestyle, etc.) intended to promote general health and prevent or minimize disease progression. 3. Drug Therapy Evaluation - The student shall assess and monitor the patient’s drug therapy, including a consideration of the chemical, pharmaceutical, pharmacokinetic, and pharmacological characteristics of the administered medications. 4. Pharmacotherapy Decision-Making - The student shall make pharmacotherapy decisions and support those decisions based on knowledge of biomedical, pharmaceutical, administrative, and clinical sciences.b The student shall recommend patient use of prescription and nonprescription medications, as well as nondrug therapy. 5. Medication Preparation, Distribution, and Administration –The student shall compound and/or dispense drug products consistent with patient needs and in harmony with the law. The student shall demonstrate the ability to accurately interpret the prescription, select the appropriate dosage form, route and method of administration, and appropriately package and label the product. The student shall demonstrate the ability to administer medications, when appropriate. 6. Systems Management - The student shall use and evaluate acquisition, inventory control and distribution systems, while documenting and maintaining quality. The student shall plan, organize, direct and control pharmaceutical care systems and human, material, and financial resources, utilizing management theories and information technology. General Education Abilities 7. Communication Skills - The student shall read, write, speak, listen and use multimedia to communicate effectively. The student shall counsel and educate patients, as well as communicate with other healthcare professionals. 8. Critical Thinking - The student shall acquire, comprehend, apply, analyze, synthesize, and evaluate information. The student shall integrate these abilities to identify, resolve, and prevent problems and make appropriate decisions. The student shall understand the research process. 9. Professional Ethics and Responsibility - The student shall represent the profession in an ethical manner. The student shall identify, analyze, and resolve ethical problems. American Journal of Pharmaceutical Education 2006; 70 (4) Article 90. 5 10. Social Interaction, Citizenship, Leadership, Professionalism - The student shall demonstrate appropriate interpersonal behaviors. The student shall provide service to the profession, as well as the community. The student shall be proactive in collaborating with other health care professionals. 11. Life-long Learning - The student shall continuously strive to expand his or her knowledge to maintain professional competence. 12. Information Management – The student shall apply technology to pharmacy practice and science. The student shall demonstrate the ability to interpret and evaluate data for the purpose of assessing the suitability, accuracy, and reliability of information from reference sources. aThe pharmaceutical care plan shall include subjective and objective patient information, an assessment of that information, and a plan to resolve and monitor any drug-related problems that were identified. bPharmacotherapy decisions determine what, why, when, where, and how drug therapy is provided. The making of pharmacotherapy decisions is the foremost expressions of the professional knowledge, responsibility and authority of pharmacists. Appendix 2. Directions to the Student Directions You are a pharmacist in the clinic pharmacy serving a medical building containing physicians from many different practices. Dr. Baker just prescribed Avandamet for a 34-Year-old woman just diagnosed with type 2 diabetes today. Please review the prescription and clarify any problems using the resources available. If you find any problems, please document them on the form provided. The following is a list of tertiary sources; please check which would be pertinent to determine if problems exist in this case (Check all that apply) s Briggs Drugs and Pregnancy s Facts and Comparisons s Micromedex s Pharmacotherapy (old edition) s American Hospital Formulary Service Drug Information Using only Micromedex determine if problems exist with the following prescription. American Journal of Pharmaceutical Education 2006; 70 (4) Article 90. 6