Untitled-5 Computer-Assisted Instruction 19 Computer-Assisted Instruction: Is It an Option for Bibliographic Instruction in Large Undergraduate Survey Classes? Joan Kaplowitz and Janice Contini Developing effective library instruction for large undergraduate survey courses can be a difficult task. In 1993, librarians at UCLA’s Louise M. Darling Biomedical Library decided to develop a computer-assisted li­ brary instruction program for the roughly 800 to 900 undergraduate biol­ ogy students per year who are enrolled in the department’s basic, re­ quired-for-the-major introductory course. A formal summative evalua­ tion of the effectiveness of this CAI, as compared to the lecture method, also was developed. The evaluation utilized both a quantitative pre- and posttest design using objective questions and a qualitative follow-up survey featuring open-ended questions. eveloping effective library in­ struction programs that effi­ ciently deal with large under­ graduate courses is often extremely difficult. In 1993, librarians in the Reference Division of UCLA’s Louise M. Darling Biomedical Library decided to develop a computer-assisted instruc­ tion (CAI) program to replace the library skills instruction sessions previously of­ fered to the three hundred biology under­ graduate students per quarter. This course, Biology 5L (Organismic and En­ vironmental Biology Laboratory), is a re­ quired preparation course for the biology major. This instruction, which took place during the first week of each quarter, con­ sisted of a fifty-minute lecture/tour fol­ lowing a two-hour field trip in which undergraduate students were taught measuring and surveying techniques used by biologists. Many of these stu­ dents are unfamiliar with the organiza­ tion of the biology literature, yet the em­ phasis of their assignments in this course is on locating current journal literature. Because little formal evaluation of CAI effectiveness as a mode for library instruc­ tion appears in the literature, the biomedi­ cal library reference librarians decided to develop a formal, summative evaluation that would compare the effectiveness of the CAI process to the then in-place lec­ ture-type presentation. Traditional approaches such as lectur­ ing to the whole class in a one-shot ses­ sion or scheduling presentations for the typically smaller discussion or laboratory Joan Kaplowitz is a Reference Librarian in the Louise M. Darling Biomedical Library at UCLA; e-mail: jkaplowi@library.ucla.edu. Janice Contini is Head of Reference at the Louise M. Darling Biomedical Li­ brary at UCLA; e-mail: jcontini@library.ucla.edu. 19 mailto:jcontini@library.ucla.edu mailto:jkaplowi@library.ucla.edu 20 College & Research Libraries January 1998 sections have both advantages and draw­ backs. The large lecture hall approach ensures that everyone is exposed to the same material at the same time. It requires minimal personnel (usually only one li­ brarian) and often is done in the student’s own classroom, thereby minimizing the facility demands made on the library. However, large group presentations can seem impersonal and out of context. Moreover, it is extremely difficult to in­ volve the students actively in the learn­ ing process. Although small-group pre­ sentations can be more interactive and personal, they are more labor-intensive for librarians, can be difficult to sched­ ule, and can place enormous demands on both space and equipment. In addition, as the number of people scheduled to make the presentation increases, the con­ sistency of the presentation can decrease. The authors estimate that it took roughly five hundred hours of librarian time to design the program and an additional five hundred hours of programmer time to implement the design. CAI has many characteristics that make it an appealing way to teach basic library skills to large numbers of students. It is interactive, self-paced, and self-di­ rected, and gives students immediate feedback on their progress. Moreover, it presents exactly the same content to all students, thereby eliminating the un­ avoidable variations that can result when six or more instructors are involved in the lectures. Finally, CAI offers an alternative learning approach that may be appealing to students who have experienced only the lecture method. However, development of a CAI pro­ gram is not a trivial task. The time and expertise needed to design and develop CAI are crucial considerations. The au­ thors estimate that it took roughly five hundred hours of librarian time to design the program and an additional five hun­ dred hours of programmer time to imple­ ment the design. An examination of the applications of CAI to library instruction revealed that most CAI addressed the notion of teaching some specific skill, most often use of an OPAC. The intent of the biomedical library’s CAI project was to address larger, more global instruction issues such as the nature of scientific com­ munication and methods of information access in the life sciences. Online searches of the PsycINFO®, ERIC, and Library Literature databases revealed that very little has been done in the area of objectively evaluating CAI in general. Even less has been done in evalu­ ating CAI applications to library instruc­ tion. Most evaluations are either forma­ tive and used as a feedback mechanism in the development process itself, or summative and deal primarily with the affective outcome of having students use CAI. Summative evaluations generally focus on whether the students liked the process, not whether they learned the material presented in this format. This probably is due to the fact that it is very difficult to assess the cognitive effective­ ness of any library instruction. Although objective tests are easy to administer and score, the questions used to assess learn­ ing are out of context and do not neces­ sarily test how well a student may func­ tion in a real-life situation. More qualita­ tive measures, such as reviewing stu­ dents’ papers or projects, certainly assess a more real-life situation but are difficult to design and quantify. (For further back­ ground on this topic, please see the anno­ tated, selected bibliography on the bio­ medical library’s Web page: www.library. ucla.edu/libraries/biomed/adm/per­ sons/caibib.htm). The librarians involved felt strongly that the two modes of presentation had to be evaluated and their effectiveness compared before any recommendations could be made regarding elimination of the lecture mode in favor of CAI. The ex­ www.library Computer-Assisted Instruction 21 pectation was that CAI would be as good as or better than the lecture method in providing this type of instruction. The li­ brarians also expected that the students would find the CAI experience more en­ joyable and less stressful than the lecture method because CAI is self-paced and interactive, and makes extensive use of graphics. Conceptual Framework for Instruction In formulating the outlines for the CAI, the design team reexamined the concep­ tual framework that had been in use for the lecture. The standard lecture that had been in place for many years used a tool- based framework. After an initial over­ view segment that discussed scientific communication, types of material, and the elements of a citation, each major life sci­ ence index was examined one at a time. In discussing how to organize the CAI, the team began to question the tool-based framework. Ultimately, it was decided that the CAI would be more useful as a concept-based design, so the team devel­ oped a decision tree approach organized around the types of questions the stu­ dents usually have. The CAI would illus­ trate how to use library materials based on what the individual student’s infor­ mation need was at the time. This seemed a more practical and real-life type of situ­ ation. In addition, the supporting material that appears in the students’ lab manual was reorganized along the same lines. The team then was faced with an ethical di­ lemma. Now that a new conceptual framework for both the CAI and the lab manual had been determined, what should be done about the lecture? For re­ search purposes, the lecture should re­ main untouched so that the new tech­ nique could really be compared to the old technique. However, that would mean using what now was felt to be a less-than­ effective conceptual framework with half the students. Ethically, the team felt that the best possible instruction should be de­ livered to all students. Therefore, the lec­ ture material was reorganized to match the conceptual framework being used in the CAI and the lab manual. Thus, all three modes of instruction (CAI, lecture, lab manual) used the decision tree model. However, this decision, which undoubt­ edly was beneficial to the students, ended up interfering with the ability to really evaluate the relative effectiveness of the different modes of instruction. Methodology The Biology 5L evaluation featured a pre- and posttest design directed toward ex­ amining both the affective and the cogni­ tive outcomes of the two forms of instruction. (Samples of these tests may be obtained by e-mailing the Louise M. Darling Biomedical Library Reference Di­ vision at biomed-ref@library.ucla.edu.) A follow-up study then was used to gather additional qualitative, affective data from students to supplement the information obtained through the quantitative pre- and posttest design. Pre- and posttests were based on ma­ terial previously designed by Joan Kaplowitz1, 2 and were developed to evaluate the educational outcomes of both presentation modes (lecture versus CAI). The variables measured included usage factors, affective attributes, and cognitive skills. The evaluation also attempted to examine questions regarding learning preferences as they relate to presentation technique. In addition, the researchers hired a consultant from the UCLA Statis­ tical Consulting Center to assist in the fi­ nal design of these instruments. The study was undertaken in the 1994– 95 academic year. The pretest was admin­ istered during the first lab section meet­ ing of the quarter. Library instruction us­ ing one of the two modes followed the pretest administration. A hands-on home­ work assignment that utilized the mate­ rial being taught was required of all stu­ dents. The posttest was administered dur­ mailto:biomed-ref@library.ucla.edu 22 College & Research Libraries January 1998 ing a lecture period following completion of the instruction and this assignment. The CAI package was evaluated for one full year to account for possible sea­ sonal variations. The teaching assistants involved in the class were randomly as­ signed either the CAI or the lecture as the The CAI package was evaluated for one full year to account for possible seasonal variations. mode of instruction to be used with their students. As a result, roughly half the stu­ dents used the CAI and half attended the lecture. Scores on the pre- and posttests were compared for each group of stu­ dents (i.e., those receiving the lecture pre­ sentation and those receiving the CAI presentation). The difference between the pre- and posttests for each group was analyzed to determine the effectiveness of each presentation mode. The scores for the two groups were compared to see if any difference in effectiveness could be detected. Additional data regarding the stu­ dents’ feelings about using the CAI were gathered the following year via a more open-ended, qualitative-type follow-up survey. This survey also asked questions about the written instruction that appears in the students’ biology lab manual. Li­ brarians wanted to find out if the students were making use of this written material, and if so, what comments they might have about it. This survey was adminis­ tered in the individual lab sections about halfway through the spring quarter of 1996. Results Although roughly eight hundred stu­ dents were enrolled in Biology 5L during the 1994–95 academic year, the study in­ cluded only data from those students who completed both a pre- and posttest. The net result was data from a total of 423 stu­ dents. Two hundred of these students at­ tended the lecture and the remaining 223 used the CAI program. The average num­ ber of correct responses on the pretests was 13.02 (68.9%) for the lecture group and 13.33 (69.71%) for the CAI group. The average number of correct responses on the posttests was 15.30 (79.85%) for the lecture group and 15.12 (79.58%) for the CAI group. Statistical analysis of these responses using t-tests failed to reveal any significant differences between the groups. Students’ posttest responses seemed equivalent regardless of whether they had used the CAI or had attended the librarian’s lecture. On the basis of these data, a decision was made to elimi­ nate the lecture method and use CAI as the mode of presentation for all students in the future. Although the survey did not reveal any clear difference between the two modes of instruction, CAI was still deemed a better choice for the library because it was less labor-intensive. In addition, informal discussions with the faculty, teaching as­ sistants, and students seemed to indicate that students liked using CAI better than just listening to a lecture. Moreover, be­ cause the conceptual framework of the lecture had been changed to match that of the CAI and the lab manual, an addi­ tional artifact may have been introduced inadvertently. In effect, the new CAI was not being compared to the original in­ struction. Because the lecture had been changed, the study ended up comparing two new forms of instruction rather than comparing a new form to an old one. Although the decision had been made to move to CAI as the preferred mode of instruction, the authors still felt that some additional data were needed. The objec­ tive pre- and posttest analysis indicated that CAI was indeed a reasonable option for delivering instruction to this group of students. Now it seemed appropriate to examine why the students felt this mode of instruction worked so well. What ex­ actly did they like or dislike about using a computer to learn the material? How Computer-Assisted Instruction 23 did the CAI and the revised pages in the lab manual work together to provide in­ struction? Did students prefer this mode of instruction over the standard (though revised) lecture method? Was anything needed to improve the CAI or the lab manual? The follow-up survey, done in the spring of 1996, addressed some of these concerns. Because CAI effectiveness was no longer the primary issue, the authors felt that a more qualitative survey featur­ ing open-ended questions would more likely elicit the types of responses they were seeking. The results from this sur­ vey are summarized in table 1. Surveys were distributed to all 175 students en­ rolled in the course, and 139 surveys were returned for a response rate of 79.43 per­ cent. The breakdown of students by class was: 8.6 percent freshman, 23.1 percent sophomores, 38.1 percent juniors, 15.8 percent seniors, and 4.3 percent other (pri­ marily university extension students). In general, the survey responses were favorable and reinforced the librarians’ informal impressions of how the students were interacting with the program. Most students finished the program within the allotted time frame of forty-five to sixty minutes and felt that it contained exactly the right amount of material for them to complete their assignments. Questions three through seven were designed as a Likert scale, with 1 repre­ senting “strongly disagree” and 5, “strongly agree.” The intent of this sec­ tion was to assess the students’ overall impression of CAI. Most students liked the program, thought it was clear and well organized, and felt it was an easy way to learn the material. When asked whether they would prefer a lecture or CAI as a mode of instruction, responses broke down roughly into thirds with 37 percent preferring CAI, 32 percent pre­ ferring the lecture method, and the rest undecided. Roughly a third of the stu­ dents also indicated they felt they had a better understanding of the material fol­ lowing use of the CAI program, com­ pared to 23 percent who felt they did not have a better understanding and 30 per­ cent who were undecided. The questions in the next section were open-ended and aimed at eliciting de­ scriptive comments from the students about CAI. The fact that it was fun, easy to use, and graphical won high marks from many of the students. The students also liked the organization of the mate­ rial and the interactive nature of the pro­ gram. The fact that they could work at their own pace also was a plus. The students’ biggest criticism of the program was that it was much too long and needed more realistic examples. Some students did not like the organiza­ tion of the material, and felt that it was boring and tedious and did not accurately represent what they were going to find in The students’ biggest criticism of the program was that it was much too long and needed more realistic examples. the library. Suggestions for change all cen­ tered on the issues of length, repetitiveness, and a desire for more real-life simulations. Despite these criticisms, however, almost half the students indicated they would recommend use of this program to oth­ ers. The remainder of the survey concen­ trated on the library instruction pages in the lab manual. Of the 139 survey respon­ dents, eighty-three (59.7%) said they had used the manual. Almost 90 percent of these students indicated that the material in the manual was helpful. They liked the way the material was organized and the fact that they could refer back to pages as needed, especially when working on their assignments in the library itself. Many students commented that although they liked CAI as a means of original learn­ ing, they also liked having something portable, such as the lab manual, to refer 24 College & Research Libraries January 1998 TABLE 1 Follow-up Survey Results: Summary Statistics Question % Response Question % Response 1. How long did it take you to complete the library's Computer-Assisted Instruction pro- gram? 0-15 minutes 8.6 16-30 minutes 32.4 31-60 minutes 46.8 More than 60 minutes 2.9 2. The Computer-Assisted Instruction pro- gram contained: More information than I 33.1 needed to complete my assignments/papers Less information than I 10.8 needed to complete my assignments/papers Just the right amount of 51.8 information to complete my assignments/papers None of the information I 0.7 needed to complete my assignments/papers Not applicable 3.4 3. The Computer-Assisted Instruction pro- gram was clear and well organized. Strongly agree 12.2 Agree 38.8 Undecided 36.7 Disagree 10.1 Strongly disagree 2.2 4. The directions for using the program were clear and easy to follow. Strongly agree 20.1 Agree 41.0 Undecided 30.0 Disagree 7.5 Strongly disagree 0.0 5. I would rather listen to a lecture than use a computer program to learn new material. Strongly agree 12.9 Agree 19.4 Undecided 30.2 Disagree 15.8 Strongly disagree 21.6 6. After completing the Computer-Assisted Instruction program, I had a better understand- ing of how to use the various indexes to search for journal articles on my subject. Strongly agree 9.4 Agree 25.9 Undecided 41.7 Disagree 17.3 Strongly disagree 5.8 7. The Computer-Assisted Instruction pro- gram was a difficult way to learn how to use indexes and other library material. Strongly agree 2.8 Agree 12.2 Undecided 30.2 Disagree 38.8 Strongly disagree 15.8 8. Did you feel that the Computer-Assisted Instruction program helped you complete your assignments in Biology 5L? Please describe why or why not. Yes 64.0 No 35.0 9. What did you like best about the Com- puter-Assisted Instruction program? Easy to use/fun 17.9 Graphics 16.5 Organization 9.4 Interactive nature 9.4 Work at own pace 9.4 Examples/overview 5.8 Link to manual 1.2 Nothing 5.7 Not applicable 24.5 10. What, if anything, did you dislike about the Computer-Assisted Instruction program? Too long 21.6 Unclear/disorganized 5.7 Boring/tedious 5.0 Not enough detail 5.0 Didn't teach/too simple 5.0 Individual misc. comments 10.0 Not applicable 21.6 Computer-Assisted Instruction 25 TABLE 1 cont. Follow-up Survey Results: Summary Statistics Question % Response Question % Response 11. Would you recommend this Computer­ Assisted Instruction program to other students who need to learn how to find journal articles on topics in the life sciences? Why or why not? Yes 49.6 No 28.1 Not applicable 22.3 12. If you could change the Computer­As­ sisted Instruction program, what changes would you make? Shorten it 14.4 Add examples/make it more 10.1 realistic Improve organization 7.2 Get rid of it 5.0 Less repetitious 2.1 Make it faster 2.1 Make it less childish 1.4 Individual misc. comments 6.3 Not applicable 49.6 13. Did you make use of the Libraryf Re­ sourcesfforfthefLifefSciences section of your Biology 5L laboratory manual to help you complete your class assignment? Yes 57.5 No n .7 Not applicable 16.5 14. Did you find the Library Resources for the Life Sciences pages in the lab manual help­ ful when doing your 5L assignments? Please describe why or why not. Yes 89.2 Somewhat 2.7 No 2.7 Not applicable 6.0 to when needed. There were very few negative com­ ments or suggestions for change regard­ ing the lab manual section. When asked for a final summation, more than 50 per­ cent of the students indicated they would recommend keeping both the CAI and the manual section because they were a good 15. What did you like best about the Library ResourcesfforfthefLifefSciences pages in the lab manual? Clear organization 44.5 Portablelcould refer to it 12.0 Easy to use 10.8 Examples used 6.0 Short 2.4 Informativelhelpful 2.4 Not applicable 20.5 16. What, if anything, did you dislike about the Library Resources for the Life Sciences pages in the lab manual? Various individual answers 10.0 Not applicable 90.0 17. What would you change about the Library ResourcesfforfthefLifefSciences pages in the lab manual? Various individual answers 10.0 Not applicable 90.0 18. What are your overall comments about the Computer-Assisted Instruction program, the LibraryfResourcesfforfthefLifefSciences pages in the lab manual, or both? Both usefullkeep both 54.7 Keep manual only 31.3 Still needed personal 9.3 instruction from librarian Individual misc. comments 4.8 Note: The responses to questions 14-18 weretabulated on the 83 out of the total 139 whoanswered yes to question 13. combination for learning the material. Discussion The responses to the follow-up survey went a long way toward supporting the librarians’ feelings that the CAI in com­ bination with the lab manual was an ef­ fective and user-friendly way of offering 26 College & Research Libraries January 1998 instruction. Students appear to be func­ tioning quite independently when they come to the library and usually need only be pointed in the right direction to locate the indexes in question, or be referred back to the correct pages in the lab manual to refresh their memory about what they learned using CAI. Many of the comments made on the survey reinforced the authors’ hypothesis about why there appeared to be so little difference between the CAI and the lec­ ture method in the original study. Al­ though many students love to use com- By combining the CAI with written backup, it is possible to offer variety and reach most of the students. puters, many still are intimidated by them. Further, many students still prefer the hu­ man touch offered by a lecture or a one- on-one session with a reference librarian. Still others prefer to read the material and refer to it as needed. If the authors had in fact been successful in randomly assign­ ing students to the CAI and lecture groups, these differences also would be equally distributed. Both groups would have a cer­ tain number of students who love comput­ ers and an equal number who hate them. Differences between these groups would then disappear because the responses of those who love computers and responded favorably to CAI would be negated by those who hate computers and were un­ able to learn well using that mode of in­ struction. The reverse of this situation would appear in the lecture groups with “good” lecture learners’ responses being negated by those who do not learn by lis­ tening to a lecture. The authors were especially delighted to read the comments on use of the lab manual in conjunction with the CAI—and even in lieu of it. Offering instruction in many different modes is really the ideal and addresses the differing learning styles issue. However, practically speak­ ing, most libraries cannot offer more than one, or at best two, different modes of instruction for the same topic. By com­ bining the CAI with written backup, it is possible to offer variety and reach most of the students. Those who do not like to use computers can rely more heavily on the lab manual and, of course, can still ask questions at the reference desk. Those who learn well from computers do not need to use the lab manual, although it is there for future reference if needed. The CAI program also is available in the li­ brary itself should students want to re­ fresh their memories electronically. Conclusion The students in this study definitely viewed CAI as a very viable option for bibliographic instruction. However, from the library’s perspective, this mode of in­ struction clearly is not a cost-effective approach. So what does the future look like? Although librarians in the biomedi­ cal library have been quite happy with the move from lecture to electronic delivery of instruction, they have found the CAI pro­ gram to be very expensive and labor-in­ tensive to develop. They now have used the program for more than two years and the time has come to think about updating and revising it. To do so, however, will re­ quire hiring a programmer and spending many hours working on the project. Cur­ rently, it is unclear whether that endeavor would be worthwhile. At the same time, the library is explor­ ing new possibilities. As always, technol­ ogy continues to march on and now there exists a very viable alternative to CAI— the World Wide Web. Because the concep­ tual framework that was developed for the CAI and the lab manual seems to be working well, the hours that went into designing the intellectual content of the CAI were well spent. The aspects of CAI (graphical interface, move at own pace, interactive) are all things that now can be done fairly easily on a Web page. Computer-Assisted Instruction 27 Use of the Web instead of CAI also would address some of the students’ com­ plaints, especially those concerning length and repetitiveness. The CAI pro­ gram was structured in a very linear man­ ner. Students began at the beginning and had to move through the program, step­ by-step, to the end. They were forced to go over material they already might have been familiar with, and could not skip around as needed. Forcing the students through the program was a conscious design decision. It was felt that all stu­ dents should be taught how to search by scientific name as well as by general topic, regardless of their immediate information need. In other words, the program was designed to follow the “just-in-case” phi­ losophy rather than simply teach students what they currently needed to know. The fact that students would not have easy access to CAI at a later date contributed to this decision. However, using this de­ sign philosophy lengthened the program, and in the students’ eyes, made it tedious and repetitive. Using the Web as the mode of deliv­ ery certainly would address these prob­ lems. Due to the hyperlink characteristic of the Web, students could interact with the material in any order they liked and could concentrate on just the material they felt they needed to review. Those who wished to look at all parts of the in­ struction could do so, and those who felt they needed only certain aspects would be free to pick and choose. Furthermore, mounting this material on a Web page would increase the stu­ dents’ ability to access it. Right now, they can only use CAI in their own biology lab and in the library. Putting the information in Web format means they could use it any­ where they had Web access. Students could use the program in any computer lab on campus and even from their own homes. The authors definitely learned a lot about electronic delivery of instruction by devel­ oping the CAI program and look forward to applying this knowledge to the next phase of this instruction. Notes 1. Joan Kaplowitz, “A Pre-and Post-test Evaluation of the UCLA English 3 Library Instruc­ tion Program at UCLA,” Research Strategies 4 (winter 1986): 11–17. 2. Thomas K. Fry, “The English 3 Library Instruction Program at UCLA: A Follow-up Study,” Research Strategies 6 (summer 1988): 101–8.