College and Research Libraries Research Notes End-User Training in the Use of a Small Swedish Database Nancy Fjallbrant, Elisabeth Kihlen, and Margareta Malmgren INTRODUCTION In recent years there has been a consid- erable increase in the availability of computer-stored information both in bib- liograghic .a~d nonbi~liogr.aphic data- bases. Thts mcrease m onlme informa- tion has resulted in a need to teach people how to use the available facilities. As Lan- caster pointed out in 1973, a critical factor affecting the success or failure of an on-line information retrieval system is the ef- fectiveness of the procedures employed to teach people how to use the facilities. 3 Several groups are concerned with on- line orientation, training, and education: (1) database producers; (2) systems opera- tors; (3) institutions responsible for termi- nal operation for example libraries or in- formation centres within academic institutions; (4) library schools; (5) inter- mediaries; (6) end-users. This paper will describe how a group of end-users (engineering undergraduates) were taught to carry out interactive online searching on a small, specialized Scandi- navian.database. Their search techniques and attitude to online searching will be ex- amined and evaluated. THE BYGGDOK DATABASE The BYGGDOK database has been de- veloped and built up by the Swedish Na- tional Building Documentation Centre. 4 BYGGDOK is a bibliographic database containing some 25,000 references (with an annual growth rate of 5,000). The data- base covers the highly specialized field of building and construction with references dealing with the following subjects: archi- tecture, building materials, building plan- ning, civil engineering, constructim.:t tech- nology, environmental technology, ~stallation technology, and urban plan- nmg. Input to the database is mainly Scandi- navian material, in the form of journal arti- cles, reports, product information stan- dards, etc. The abstracts are written in the language of the country of origin (mainly Swedish, Danish, or Norwegian). Each reference is described with Swedish key- words. The database can, however, be used by persons with no knowledge of Swedish, by means of a reduced UDC (universal decimal classification) code sys- tem developed by the Swedish National Building Documentation Centre. BYGGDOK is available to users throughout Scandinavia via SCANNET, the Nordic data network. The database is widely used by practising engineers and architects. The concentration on Scandi- ~ancy Fjiillbrant is deputy librarian; Elisabeth Kihlen is library assistant; and Margareta Malmgren is research asszstant, all of Chalmers University of Technology, Gothenburg, Sweden. · 161 162 College & Research Libraries navian material is regarded as being very practical, due to the differences in build- ing standards between Scandinavian and other countries. LIBRARY USER EDUCATION PROGRAMME AT CHALMERS UNIVERSITY Chalmers University Library devel- oped, from 1974 to 1980, an extensive user education programme5 that can be divided into the following stages: 1. Orientation for new users. 2. A fourteen-hour introduction to in- formation retrieval for third- and fourth- year engineering undergraduates. 3. Advanced courses in information re- trieval (including computerized methods) for doctoral students-a seventy-hour course. 4. Courses for practising engineers, lab- oratory assistants, etc. In 1980, Chalmers Library received a grant from the Office of the Chancellor of Swedish Universities to investigate the possibility of teaching end-users how to carry out direct online searches on BYGGDOK. More than 300 undergradu- ates from the schools of civil and mechani- cal engineering have so far taken part in practical online training. The BYGGDOK online training programme will be de- scribed in the following sections. GOALS AND OBJECTIVES FOR THE BYGGDOK ONLINE INSTRUCTION The main goal for the undergraduate online programme was to enable the stu- dent to carry out online information searches within a subject field on the BYGGDOK database as, and when, re- quired in connection with information needs. Specific objectives were described as follows-after completing the course the end-user should: 1. Be aware of different methods for in- formation retrieval: manual, batch pro- cessing, SDI, and interactive online. 2. Be aware of different kinds of infor- mation searches: current awareness, ret- rospective, factual. 3. Be able to carry out logging-on proce- dures for a network database. March 1983 4. Be able to use a teletype or VDU (vis- ual display unit) terminal. 5. Be able to use the basic commands and features of the BYGGDOK retrieval system. 6. Be familiar with a typical unit record. 7. Be able to express a search topic in suitable terms and parameters. 8. Be able to develop a search strategy in terms of Boolean logic. 9. Be able to evaluate the results ob- tained. TEACHING METHODS CHOSEN AND ORGANIZATION OF COURSE There is a wide variety of methods and media available for online teaching. A se- lection of methods was used-lectures, small group seminars and demonstra- tions, audiovisual demonstrations, printed manuals and checklists, and direct "hands-on" training. The introductory course lecture was used to present a frame of reference, with emphasis on the pat- terns of communication and the relation- ship between computerized information retrieval and manual methods. Different types of information searches were pre- sented. The main part of the course was based on small group methods- laboratory sessions with demonstrations and practical searches on a topic of the stu- dent's own choice. The first laboratory session (five hours) was mainly devoted to manual informa- tion searching, using a variety of tools. The last hour was used for an orientation on computerized information retrieval us- ing audiovisual, multimedia MEDIA- TRON programmes. The MEDIATRON teaching aid was developed at the Central Information Service of the University of London by Vickery and Pratt. 6 The ME- DIATRON is a modified stereotape re- corder that is designed to carry out simul- taneous recordings of audio commen- taries, trigger pulses for photographic slides, and digital signals from computer- ized information retrieval systems. A mul- timedia online-orientation programme was produced using synchronized slides, audio commentaries, and recorded digital signals (examples of a search). This online orientation was displayed via slide projec- tor, loudspeaker, VDU, and monitors. The BYGGDOK database was described and demonstrated. Students were then given a short manual that introduced com- puterized information retrieval, described the BYGGDOK database, logging-on techniques, the search command lan- guage, and search techniques. They were told that at the next session, in a week's time, they would carry out their own hands-on search. The second laboratory session started with a short review, including a MEDIA- TRON demonstration programme of BYGGDOK. Students then prepared their searches, planning strategies with the aid of search-term frequency lists. They usu- ally worked in groups of two or three per- sons. The group then carried out their search on a teletype terminal, with a BYGGDOK checklist of commands imme- diately adjacent to the terminal. On completion of the course the stu- dents handed in a list of references rele- vant to their search topic, found by means of both manual and online searching. They also presented copies of their online search in which the references obtained were assessed for relevance and novelty. EVALUATION Evaluation of the online learning was carried out in a number of ways: 1. Students were asked to complete a questionnaire about: • overall gain from the course • the teaching material used • the practical terminal sessions • future use of the BYGGDOK database . and of online searching • the teaching methods used 2. Direct observation of how the end- users managed to log on and off and carry out a search. 3. Comparative search analysis with re- gard to: • the process (search effort) • the product (in terms of recall and preci- sion) A number of end-user searches were re- peated by a trained BYGGDOK interme- diary. Searchers' efforts were compared with respect to: • Numbers of commands used • Different types of commands used • Numbers of search descriptors Research Notes 163 • Sets viewed (any set which was re- quested for display or printing) • Search modification • Errors made • Connect time The choice of these parameters was based on measures for discrimination among users with different levels and ex- perience of online searching described by Fenichel in 1979.7 The searches were also compared with respect to recall and preci- sion. QUESTIONNAIRE RESPONSES Students were asked to state their opin- ions as to "overall gain" from the course, on the new teaching material produced, and were asked to evaluate the practical terminal sessions (see tables 1-3). In addition, students were asked to state their attitudes to this kind of training in computerized information retrieval (see table 4). For comparison, a group which had merely received online orientation is included. Students were asked if they thought they would use the BYGGDOK database at some future time. Fifty-five percent thought they would use the database, 3 percent thought it too expensive, and 42 percent were not sure. Eighty-four per- cent stated that, if they had access to a ter- minal, they would like to carry out their own online searching in the future, whereas 17 percent were not sure about this. Eighty percent of the students said that they preferred computerized infor- mation retrieval to manual methods. Rea- sons given were speed, convenience, and flexibility. A number of students pointed out that it was possible to combine more search terms in computerized searching TABLE 1 "OVERALL GAIN" FROM COURSES IN INFORMATION RETRIEVAL Online Manual Course Course 1980 1975 " Overall Gain " % % Very good 17 15 Good 67 58 Satisfactory 16 24 Little 0 3 Nothing 0 · 0 (N = 70) (N = 220) 164 College & Research Libraries March 1983 TABLE2 OPINIONS ABOUT TEACHING MATERIAL AND MEDIA BYGGDOK MEDIATRON Manual Demonstration Checklist Very good 14 9 19 Good 73 47 61 Satisfactory 12 36 20 Hardly satisfactory 0 8 0 Unsatisfactory 0 0 0 TABLE3 PRACTICAL TERMINAL SESSIONS Very good Good Satisfactory Attitudes to 23% 70% 6% Very easy Easy 'About right' Difficulty 23% 63% 14% TABLE 4 DO YOU THINK THIS IS A GOOD WAY FOR TEACHING ABOUT ONLINE INFORMATION RETRIEVAL? Very good way Good way No opinion Not particularly good Not a good way than by use of the manual tools. One par- ticularly interesting fact that emerged from the evaluation was that students were themselves evaluating the different approaches. This can be illustrated by the following comments: "Both manu~l and computerized · searching are useful. Online techniques are convenient and quick: manual search- ing allows a rapid glance through subject headings in an index. You need to com- bine both methods.'' "You can't say that you prefer manual or computer searching. They have differ- ent advantages and disadvantages. The computer's strength is in its speed, and the fact that the references are printed out. In manual searching it is easy to limit a search by suitable choice of search term." ''The database contained only a limited amount of information-we found more foreign language references by manual methods." ''Both methods are necessary.'' "I found mor~ references through man- ual searching.lt is important to know how to perform manual searches, as many li- 1980 (Includes Practical Searching) 61% 38% 1% 0% 0% 1978 Orientation Only 10% 66% 9% 7% 0% braries don't yet have terminals for com- puter searching. Computer methods were interesting and I enjoyed the searching.'' Observations All the students (some 300) who have so far taken part in online training sessions, have managed to carry out the various logging-on procedures and terminal searching. In the first sessions the main difficulty was finding the On/Off switch on the terminals. (This was easily remedied by writing On and Off marks in large red letters on the appropriate parts of the ma- chine!) Most of the students worked in small groups, and this appeared more sat- isfactory than individual terminal work. In cases of difficulty the group could often work out a joint solution, whereas indi- vidual searchers tended to ask the assis- tant in charge of the session for advice. There was no doubt that the students enjoyed the terminal sessions. Often it was difficult to stop students searching! This sometimes led to grumbles amongst those waiting. We increased the number of terminals from two to four in order to limit waiting time. A considerable number of students have come back to the library and asked to carry out an online search in connection with some other project. Comparative Search Analysis Eleven end-user searches were com- pared with similar searches carried out by a trained BYGGDOK intermediary. The latter not only had extensive experience in searching, but also in input and construc- tion of BYGGDOK. She was conversant with the use of search techniques based on the special reduced UDC code developed at the Swedish Institute of Building Docu- mentation for this purpose. It should be noted that the students had not been taught the use of the classification code for their online searching. At the same time, the control searches were carried out in the absence of an end-user, so that the in- teractive relevance judgement of the user was missing. The intermediary said that, as a non-engineer, she experienced some difficulty in relevance judgements. The student group was highly motivated, as they were searching for information needed for their research projects. With respect to external conditions, variables such as type of terminal, search tools, database, and search command language were controlled and identical. System re- sponse varied to a small extent from search to search. Comparisons of search efforts between end-users and intermedi- ary are shown in table 5. Research Notes 165 From this table it can be seen that end- users expended greater efforts on search- ing than did the intermediaries. The latter carried out short, economical searches. The end-users showed considerable de- termination and flexibility in their at- tempts to obtain relevant references from the BYGGDOK database, with repeated restarts and the use of a variety of descrip- tors. Their inexperience can be seen in the errors (four per search) and in the domi- nance of use of non-truncated descriptors with respect to truncated terms. The expe- rienced intermediary made greater use of truncated descriptors. Connect time was greater for end-users than for the experi- enced intermediary. Results of end-user and intermediary searches were compared with regard tore- call and precision. These are defined as follows: No. of relevant references retrieved Recall = ------------------- Total no. of relevant references in the database Precision = Relevant references retrieved Total no. of references retrieved The search results are seen in table 6. Recall is surprisingly high for the end- user group. This may be connected with their search motivation and aforemen- tioned determination to obtain references. Precision was considerably lower for the end-user group than for the intermediary. TABLE 5 COMPARISON OF SEARCH EFFORT -BYGGDOK SEARCHES Search Effort Variable Total no. of commands used Different types of commands Total number of descriptors searched Number of truncated descriptors searched Number of non-truncated descriptors searched Unique descriptors Sets viewed Errors/ search Connect time • Average/ search End-Users* 26 8 11 4 7 8 3 3 < TABLE6 COMPARISON OF SEARCH RESULTS Search Product Variables Recall Precision End-Users 62% 46% Intermediary* 8 5 4 1 2 4 1 0 > Intermediary 80% 76% 166 College & Research Libraries This factor is perhaps of less importance in the use of a small database than with a larger one. DISCUSSION AND CONCLUSION Most of the teaching/learning activities on computerized IR for nongraduates has been directed towards orientation and promotion-by means of demonstration searches and lectures. There is, however, evidence to show that new users can learn to perform searches after a short period of training. Engineering and science under- graduates are, today, in many cases, com- petent in the use of terminals for online computations and simulations. It seemed, therefore, reasonable to test the hypothe- sis that end-users could be trained to carry out their own interactive online searches using one system (and command lan- guage) and a limited number of databases. This paper describes how a group of seventy civil engineering undergraduates were trained to use a small Swedish database-BYGGDOK-for online search- ing. The students achieved surprisingly good recall values, in comparison with an experienced intermediary. The students expressed very positive attitudes towards online information retrieval-84 percent said that if they had access to a terminal, they would like to carry out their own on- line searching in the future. The online training appears to have increased aware- ness of methods of information retrieval- not only computerized but also manual methods. ("You can't say that you pref~r manual or computer searching. They have different advantages and disadvan- tages ... ") The positive experience from this exper- iment led to the introduction of hands-on online training for a group of 200 mechani- cal engineering undergraduates, in 1981, on MechEn-a database produced at the Royal Institute of Technology, Stockholm. Two further groups of civil engineering undergraduates, and a smaller group of postgraduates, have received training in the use of BYGGDOK. It is hoped to extend this type of training for students of architecture and engineer- ing chemistry, so that online searching be- March 1983 comes a regular part of courses in informa- tion retrieval. Search results obtained by end-users will be compared with those ob- tained by intermediaries. It is hoped that these analyses will provide useful infor- mation about possible differences in search behaviour between new users and experienced intermediaries . This type of information can then be used to modify and improve existing teaching pro- grammes. One example of this formative evaluation can be seen in the relative prev- alence of nontruncated descriptors in the searches of the civil engineering students. In future training, more emphasis will be placed on the use of truncation to simplify search techniques. It is important to study whether end- users can be taught to carry out online searching themselves, or whether all searches are best performed by an inter- mediary. It is often argued that intermedi- ary performed searches are more cost- effective than those of end-users. Cost-effectiveness is sometimes ex- pressed in terms of unit cost where: Unit cost= __ C_o_n_n_e_ct_t_irn_e_ No . of relevant references retrieved This measure of cost-effectiveness is, however, limited. In the world of the prac- tising engineer or scientist, time is pre- cious, and a truer measure of cost- effectiveness would be: Cost-effectiveness = Time used by engineer No . of relevant references In the intermediary-performed search, this increases with distance from online search centre . We conclude that it is possi- ble to train undergraduate engineering students (who will soon be practising en- gineers) to carry out searches based on one information system (and common command language) for a limited number of databases. With user-friendly inter- faces, it seems reasonable to suppose that end-users will be increasingly able to carry out their own information searches in the future . The role of the intermediary, in turn, might well develop from that of indi- vidual searcher to that of adviser and-con- Research Notes 167 sultant, with specialized knowledge for transdisciplinary or exhaustive searches. If you teach him to fish He will be fed for life If you give a man a fish -Old Chinese Proverb He will have one meal REFERENCES 1. M. E. Williams, "Data-bases: A History of Developments and Trends from 1966 through 1975," Journal of the American Society for Information Science 28:71-78 (1977). 2. A. Tomberg, ed., EUSIDIC Database Guide (1980 ed .; New York: Learned Information, 1979). 3. F. W. Lancaster and E. C. Fayen, Information Retrieval On-line (Los Angeles: Melville, 1973). 4. S. Abrahamsson, F. Backlund, and B. Eresund, "Costs of Building Up Small Databases in Highly Specialized Fields or in Restricted Language Areas'' (Paper presented at the 1980 Annual Meeting of FID, "Information forlndustry," Copenhagen, Aug. 15-17, 1980). 5. N . Fjallbrant, "The Development of a Programme of User Education at Chalmers University of Technology Library" (Ph .D. thesis, Guildford, University of Surrey, 1976) . 6. G. Pratt and A. Vickery, ''The Development of Multi-media Teaching Aids for Users of Computer- based Information Retrieval Systems," Program 11:10-15 (1977). 7. C. H. Fenichel, "Online Searching: Measures That Discriminate among Users with Different Types of Experience," Journal of the American Society for Information Science 32:23-32 (Jan. 1981). 8. F. W. Lancaster, Evaluation of On-line Searching in MEDLARS (AIM-TWX) by Biomedical Practitioners (Urbana, Ill.: Univ. of Illinois, Graduate School of Library Science, 1972). 9. W. Sewell and A. Bevan, "Nonmediated Use of MEDLINE and TOXLINE by Pathologists and Pharmacists, " Bulletin of the Medical Library Association 64:382-91 (1976) . 10. N . Fjallbrant, "European Experience in Online User Education," IATUL Proceedings 12:57-66 (1980). I plJBLI~D tJNIJP STANDING ORDER SERVICE GUIDE The United Nations, as a result of its manifold activities, publishes a great number of titles in several categories: Disarmament, Human Rights, Social Questions, International Law, Narcotic Drugs, Economics, Demography and International Statistics. To facilitate ordering and to be sure of receiving all the United Nations publications in your field of interest, use the Standing Order Service Guide, which describes in detail how the service works. Send for our catalogue, free of charge! ' UNITED NATIONS • PUBLICATIONS Room A-3315 • Palais des Nations New York, N.Y. lQOlJ ~ 1211 Geneva 10, Switzerland