College and Research Libraries LASLO A. NAGY AND MARTHA LOU THOMAS An Evaluation of the Teaching Effectiveness of Two Library Instructional Videotapes Two library instructional videotapes, developed by the University of Cali- f?rnia, Irvine , General Library, were evaluated for their teaching effec- tweness. An experimental design was utilized. Results indicated that after viewing the two videotapes, students in the experimental group performed significantly better on a posttest of library skills than students in the control group. The implication of these findings is discussed with special reference to the future use of "mechanical" media in library instruction. INTRODUCTION As the financial constraints upon universi- ties and university libraries tighten, attempts to increase the efficient use of cur- rently available library resources have been initiated. Consequently, a variety of library utilization classes have emerged over the past decade to promote effective use of librar- ies. 1 - 4 Although some library instruction programs have been evaluated, s-H evaluation is still relatively uncommon. 9, 10 Especially infrequent are evaluations using pre- and posttest experimental designs. 11 This study reports such an evaluation of a library in- struction program at the University of Cali- fornia, Irvine, General Library. The pro- gram consisted of two videotapes explaining library research procedures for successful computer data base searching and research- paper writing. A brief description of the two videotapes is provided in a subsequent sec- · Laslo A. Nagy is a graduate student and teaching assistant in the program in social ecolo- gy, University of California , Irvine. Martha Lou Thomas is a librarian in the General Library, University of California , Irvine. The authors wish to thank the students and the librarians who par- ticipated in the study . The valuable advice and. encouragement of Ross Conner was greatly appreciated. 26 I tion. The tapes are relatively important be- cause, on their own , they form a mini-li- brary-instruction program that can be ad- ministered at a comparatively low cost. Both videotapes were written and produced by one of the authors (Thomas) with funding from the UC Irvine Committee for Instruc- tional Development and UC Irvine Library Administration in response to the growing need for bibliographic instruction at a time when library staff and financial resources were becoming increasingly more limited. Formulated with the idea of broad audience appeal and the possibility of utilization in a variety of instructional settings, the videotapes are used by the UC Irvine Li- brary during student orientation, public re- lations functions, and in the instruction of bibliostrategy classes. In r esponse to appeals in the literature, 12- 15 it was decided to evaluate the teaching effectiveness of the two videotapes by using an experimen tal design (with an experimental and a control group) and statistical tests for the analysis of results. METHODS Description of Videotapes The first videotape, titl ed Searching Computer Data Bases to Meet Your In- formation Needs centers around a mime and his search for information in the library . The esse ntially humorous character of the mime is counterbalanced by an invisible narrator who helps the mime in his search. The narrator explains the importance of appropriate subject headings and the use of the multiple search term strategy, which is the simultaneous use of several appropriate keywords during a literature search. The narrator also explains the use of citations . and abstracts. The information presented is utilized by the mime in his preparation of a computer data base search. The tape is thir- teen minutes long, with the narrator and mime interacting to produce a humorous yet potentially enlightening atmosphere. The second videotape, The Romance of Writing a Research Paper, focuses on two young lovers who need to write research papers. Once again the humorous element is offset by the narrator , who explains approaches to research-paper writing, use of library catalog cards, citations, search terms , and the multiple search term strategy. During the production's eighteen minutes, the couple makes use of the pre- sented information in attempts to write competent research papers . Description of Questionnaire Two twenty-question multiple-choice questionnaires were formulat e d with the aim of assessing students' library utilization skills, with special emphasis on their knowl- edge of library card catalog use , citations and their meaning, and the multiple search term strategy. These three components of library utilization skills were also the ones most emphasized in the content of the videotapes . From approximately eighty possible questions, forty were randomly selected and divided between the two ques- tionnaires. The order of the questions with- in each questionnaire was then randomly determined. In this way, two different questionnaires of potentially equal difficulty were produced. (Copies of these question- naires are available from the authors .) A randomly selected group of librarians from the UC Irvine Library was then asked to rate the questionnaires as a test of library skills on a scale from one to seven, with one being very unsatisfactory and seven Instructional Videotapes I 27 being very satisfactory. The librarians were also asked to complete the questionnaires and to indicate the answers they regarded as suitable. The librarians ' answers were used to determine the "correct" responses to the questions asked. These correct re- sponses were then used in the scoring of students' answers to the questionnaires. Description of Subjects and Testing Procedure Students in an undergraduate class (with a majority of third- and fourth-year stu- dents) at the University of California, Irvine, were selected for the study and were randomly assigned to the experimental or the control group. The first question- naire, referred to as the pretest, was com- pleted in class at the same time by both ex- perimental and control groups. Attached to this questionnaire were questions asking students their status at the university and the frequency of their library borrowings. Students in the experimental group were then e ncouraged to view the two videotapes at predetermined early evening hours dur- ing three days immediatel y following the pretest. Unlike the experimental group, stu- dents in the control group were not exposed to th e videotapes. The week following the pretest, both groups were once again tested for their library utilization skills, this time by the second questionnaire. This question- naire , referred to as the posttest , was potentially similar in difficulty to the first questionnaire . After the completion of the posttest, the control group was encouraged to view the videotapes. This particular pro- cedure (indicated in figure 1) was followed because it would have been unethical to withhold the videotapes, which most likely had some educational benefit, from the stu- dents in the control group. Students were asked to write their names . on the questionnaires, and only those stu- dents who completed both pretest and post- test and who were (1) initially assigned to the experimental group and actually observed the videotapes and (2) initially assigned to the control group and did not observe the videotapes were included in the analysis of the results . The experimental group was composed of twenty-four sub- jects, whereas the control group contained 28 I College & Research Libraries • January 1981 randomly assigned into Experimental group 01 X 02 Subjects Control group 01 = pretest (i.e., questionnaire 1) X = two videotapes 02 = posttest (i.e., questionnaire 2) Fig. 1 A Summary Diagram of The Experimental Design Utilized in the Study twenty-six. The main statistical analysis was a t-test which compared the performance of the experimental and the control group on the pretest and then on the posttest. 16 FINDINGS Of the eleven randomly selected librar- ians, six completed the questionnaires. Their average evaluation (on a scale from one to seven) was five, indicating that the questionnaires were generally a satisfactory operationalization of library skills. The average scores of the experimental and the control group on the pretest and the posttest are indicated in table 1. Stu- dents in both groups answered, on the aver- age, twelve to thirteen questions correctly out of a possible twenty. The difference of 0.81 units between the two means was not statistically significant (at p < 0.05). This in- dicated that statistically the experimental and control groups performed equally well on the pretest. However on the posttest, students in the experimental group answered, on the aver- age, fifteen questions correctly, whereas those in the control group still only answered, on the average, twelve to thir- teen questions correctly. The observed dif- ference of 2. 08 units between the two means was statistically significant (at p< 0.05). An analysis of results of individual im- provement in the experimental group indi- cated that students in their third, or junior, year improved their score considerably more than students in their fourth, or senior, year . As indicated in table 2, third- year students showed a statistically signif- icant improvement of 3.43 units, whereas fourth-year students showed no significant change in their score. TABLE 1 AVERAGE SCORES OF THE EXPERIMENTAL AND CONTROL GROUPS ON THE PRETEST AND THE POSTTEST (MAXIMUM POSSIBLE SCORE = 20) STANDARD DEVIATIONS ARE IN PARENTHESES Experimental group Control group Pretest Mean (s. d. ) 13.00 (2.1) 12.19 (2.7) Posttest Mean (s .d .) 15.04 (3. 0)* 12.96 (2.9) *Difference between e xperimental and control groups at posttest is significant at p less than 0.05. TABLE 2 AVERAGE SCORES OF THIRD- AND FOURTH-YEAR STUDENTS IN THE EXPERIMENTAL GROUP ON THE PRETEST AND THE POSTTEST (MAXIMUM POSSIBLE SCORE = 20) STANDARD DEVIATIONS ARE IN PARENTHESES Third-year students Fourth-year students Pr~test Mean (s. d. ) 12. 7l (1.9) 13.38 (2.6) *Difference between pretest and posttest scores is significant at p less than 0.001. Posttest Mean (s.d.) 16.14 (2.1)* 12.50 (3.0) Numbe r of Subjects 24 26 Number of Subjects 14 8 Attached to the pretest were questions requesting the student's status at the uni- versity and his or her frequency of library borrowings. Although it was hoped that this ·information would correlate with their score on .the pretest, no relationship was found between either of the above two parameters (i.e ., student status and frequency of library borrowings) and students' score on the pre- test . DISCUSSION An attempt was made in this study to evaluate, using an experimental design and statistics , two library instructional video- tapes produced at the University of Cali- fornia , Irvine. The results of the experiment appear to indicate that viewing the two videotapes significantly improved students' knowledge of library utilization skills. Although these results were not unex- pected, the actual magnitude of the im- provement was. It would seem that a short, thirty-one-minute presentation was enough to produce , on the :1verage, a sixteen per- cent increase in the library utilization skills of university students. If one assumes that students' library utilization skills are accum- ulated throughout many years of high school and university work, the possibility of a 16 percent increase produced by a single half- hour presentation was unexpected. Also un- expected was the observation that although both third- and fourth-year students are ap- proximately equal in terms . of library utiliza- tion skills (table 2) , third-year students were far more likely to benefit from the presenta- tion than fourth-year students. This finding . Instructional Videotapes I 29 is somewhat difficult to explain, although it may be possible that fourth-year students, close to graduation, are less motivated or interested in learning new library research skills than students who still have several quarters of university study to complete. Although the comparative effectiveness of mechanical (i.e., slide shows) over more tra- ditional (i.e., library tours) methods of li- brary instruction has been questioned, it appears that the videotapes performed rel- atively wellY One possible reason for this is the potentially greater attractiveness of the TV-videotape medium over other mechanical media such as slide/tape shows. Most mem- bers of our society are accustomed to view- ing TV screens and probably find any mate- rial presented through this medium intrinsi- cally attractive. Libraries in the future will have to look increasingly toward the more mechanical forms of library instruction, and as indicated by some studies, these forms of instruction can be just as efficient as traditional methods, 18 ' 19 and in some cases more so. 20 These mechanical media can be high- quality, relatively attractive, educationally valuable, yet economically inexpensive methods of providing library instructional services. The systematic evaluation of these instructional services will become in- creasingly more important as the financial constraints upon universities and university libraries tighten . Only through systematic evaluation of library .services will it be possible to increase the efficiency of cur- rently available library resources. REFERENCE S 1. Barbara H . Phipps , "Library Instruction for the Undergraduate," College & Research Li- braries 29:411-23 (Sept. I968). 2. Elisabeth B. Davis and others , "A Two-Phase Model of Library Instruction ," Bulletin of the Medical Library Association 65:40-45 (Jan . I977). 3. Allan J . Dyson , " Organizing Undergraduate Library Instruction-The English and Amer- ican Experience ," journal of Academic Li- brarianship 1:9-I3 (March I975) . 4. Marvin E. Wiggins , "The Development of Library Use Instructional Programs ," College & Research Libraries 33:473-79 (Nov. 1972). 5. Ellen H. Keever and James C. Raymond , "Integrated Library Instruction on the Uni- versity Campus: Experiment at the Universi- ty of Alabama," journal of Academic Librar- ianship 2 :185--87 .. 6 . Linda L. Phillips and E. Ann Raup , "Com- paring Methods for Teaching Use of Periodi- cal Indexes ," journal of Academic Librarian- ship 4:420-23 (Jan. 1979). 7. Shelley Phipps and Ruth Dickstein, "The Li- brary Skills Program at the University of Ari- zona: Testing , Evaluation and Critique , " Journal of Academic Librarianship 5 :205-14 (Sept . 1979). 8. Mitsuko Williams and Elisabeth B. Davis , " Evaluation of PLATO Library Instruction 30 I College & Research Libraries • ] anuary 1981 Lessons," Journal of Academic Librarianship 5:14-19 (March 1979). 9. Thomas Kirk , " Bibliographic Instruction-A Review of Research, " in Evaluating Library Use Instruction , Papers Presented at the Uni- versity of Denver Conference on the Evalua- tion of Library Instruction , December 1~14 1973 (Ann Arbor, Mich.: Pierian Pr. , 1975). 10. Peter Heron , " Library Lectures and Their Evaluation: A Survey," Journal of Academic Librarianship 1:14-18 Guly 1975) . 11. Stuart Glogoff, " Using Statistical Tests to Evaluate Library Instruction Sessions ," Jour- nal of Academic Librarianship 4:438-42 Gan. 1979) . 12. Heron , " Library Lectures and Their Evalua- tion ." 13. Glogoff, " Using Statistical Tests. " 14. Kenneth H . Cook and Constance M. Greco, " The Ugly Duckling Acknowledged : Ex- perimental Design for Decision-Making, Part I," Journal of Academic Librarianship 3 :~ 28 (March 1977). 15. Kenneth H. Cook and Constance M. Greco, " The Ugly Duckling Acknowledged: Ex- perimental Design for Decision-Making, Part II ," Journal of Academic Librarianship 3:85- 89 (May 1977). 16. W. J. Dixon and M. B. Brown , BMDP-77 Biomedical Computer Programs P-Series (Ber- keley, Calif.: Univ . of California Pr. , 1977). 17. Julia F. Baldwin and Robert S. Rudolph , " Comparative Effectiveness of a Slideffape Show and a Library Tour," College & Re- search Libraries 40:31-35 Qan. 1979). 18. Diane D . Hardison and David M. Moore , " Library Skills via Videocassette : Back to the Drawing Board," International Journal of In- structional Media 6, no . l:l01-5 (1978-79). 19 . Thomas Kirk , " A Comparison of Two Methods of Library Instruction for Students in Introductory Biology," College & Research Libraries 32:465-74 (Nov. 1971). 20. Harriett Genung, "Can Machines Teach the Use of the Library?" College & Research Li- braries 28:25-30 Gan. 1967).