norlin.p65 546 College & Research Libraries November 2000 Reference Evaluation: A Three-Step Approach—Surveys, Unobtrusive Observations, and Focus Groups Elaina Norlin Traditional reference service, where the librarian gives the patron the right or wrong answer to a question, has slowly begun to change. With the emergence of electronic resources, digitalization, Web resources, and full text, many students need more of a consultation on where to get started than on which option is correct. Librarians at the University of Arizona Libraries strive to help students gain user sufficiency and infor­ mation literacy. When on the reference desk, they tend to teach rather than do the work for the students. As the libraries are customer cen­ tered, it was decided to conduct an evaluation of its reference services. The evaluation involved using a combination of surveys, focus groups, and unobtrusive observation worksheets with a small sample of stu­ dents. The results were very meaningful. he University of Arizona is a team-based learning organiza­ tion. There are no reference, cataloging, and acquisition departments but, rather, teams such as the Social Sciences Team, the Science-Engi­ neering Team, and the Undergraduate Services Team. The Undergraduate Ser­ vices Team is in charge of coordinating and evaluating the public service desks in the main and science-engineering li­ braries. During the spring of 1998, this team decided to do an in-depth reference evaluation of the main and science refer­ ence desks, which was coordinated by the author of this article. At these reference desks are representatives from almost all the library teams, including a combina­ tion of librarians and staff members. There also are peer information counse­ lors and library school students who work on the desks at night and on week­ ends. Because they are not all librarians, these groups of people are referred to as reference service providers. Although they work during the day, evenings, and week­ ends, their hours are not consistent and there is always room to trade hours with others on a daily basis. The average ref­ erence service provider works four to six hours per week. Problem The major problem was to evaluate refer­ ence service providers who have very inconsistent schedules. Most reference service providers trade with each other pretty consistently; moreover, temporar­ ies sometimes are hired as “fill in behind” to help librarians accomplish other stra­ tegic work. Just because an individual’s name is on the schedule does not mean Elaina Norlin is an Assistant Librarian on the Undergraduate Services Team in the Main Library at the University of Arizona; e-mail: enorlin@bird.library.arizona.edu. 546 mailto:enorlin@bird.library.arizona.edu Reference Evaluation 547 he or she is going to work those hours. Another concern was that some reference service providers feel that “approachabil­ ity” is the most important quality to have on the reference desk, and others think that the ability to answer questions cor­ rectly constitutes good service. But what do the students think? If a reference ser­ vice provider does not answer a question correctly but is friendly, will the patron The team also decided to combine various qualitative methods in order to obtain both general information (surveys) and specific information (unobtrusive observation and focus groups). use the reference desk again? A third con­ cern centered on the concept of “the cor­ rect answer.” With the wave of numer­ ous electronic resources and endless choices, many students simply need ideas on how to get started rather than a yes or no answer. In this case, will the student be satisfied with reference service provid­ ers helping with information literacy skills, instead of just pointing out right or wrong answers? Solution It was going to be impossible to get indi­ vidualized reference assessments, based on the average reference desk schedule. Therefore, before beginning the process, it was explained that reference service providers would be treated as one team. This model goes along with the library’s team-based environment and allowed the Undergraduate Service Team not to overgeneralize a specific person’s perfor­ mance. Any training afterwards would be given to all reference service providers. The team also decided to combine vari­ ous qualitative methods in order to ob­ tain both general information (surveys) and specific information (unobtrusive observation and focus groups). In addi­ tion, the team wanted to have a chance to follow up on participant questions or thoughts that needed clarification. There­ fore, the reference evaluation consisted of three steps: a survey for demographic in­ formation, an unobtrusive observation to get feedback immediately after the refer­ ence transaction, and a focus group ses­ sion to obtain detailed information. Literature Review For the literature review, the Undergradu­ ate Services Team mainly wanted to learn more about unobtrusive observation studies. There is plenty of information about surveys as reference assessments, and the questionnaire in the team’s refer­ ence evaluation was used only to obtain demographic information. The Univer­ sity of Arizona also had conducted nu­ merous focus group sessions; therefore, the team already had internal resources it could tap to find out how to conduct focus groups. Nevertheless, only a few libraries have utilized unobtrusive obser­ vation studies in their reference assess­ ments. Unobtrusive observation has been used for years in the business sector, with individuals commonly known as “secret shoppers.” Some businesses actually em­ ploy secret shoppers to check out cus­ tomer relations, products, pricing, and so on.1 Most secret shoppers pose as real customers and provide a detailed report on any particular findings.2 The library’s definition for unobtrusive observation studies is that they are “a predetermined set of test questions, which are given to reference librarians and staff members without their knowledge.”3 The research­ ers popularly accredited with putting un­ obtrusive observation on the map are Pe­ ter Hernon and Charles R. McClure. They came up with the 55 percent rule on an­ swering reference questions correctly.4 Their unobtrusive model for testing refer­ ence service assumed that there is one right or wrong answer and that the librarian is responsible for providing the correct one. In their research study, participants were trained to ask the librarian specific ques­ tions of varying complexity.5 The Under­ graduate Services Team wanted to add to this groundbreaking research by somehow incorporating behavioral traits within the unobtrusive questionnaire. 548 College & Research Libraries November 2000 A few authors have written articles on some of the flaws of looking for customer satisfaction in right or wrong answers. Joan Durrance stressed that a good mea­ sure for customer satisfaction is not only if the person receives the right answer, but also “the willingness of the inquirer to return to the staff member at a later time.”6 Carolyn W. Jardine stated that patrons “will judge the service they receive not only on whether or not they get what they came in for, but also on the reference li­ brarians attitude behavior, interest and enthusiam.”7 The closest article that addressed us­ ing multiple qualitative techniques is “Using Undergraduate Marketing Stu­ dents in an Unobtrusive Observation.” In it, the librarians worked with an upper- level marketing class that participated in the unobtrusive study as a for-credit class exercise.8 The goal of the study was to evaluate the accuracy and thoroughness of the reference answer.9 After the refer­ ence encounter, students filled out the evaluation questionnaire and then met with the two librarians to discuss their experience. The focus group session lasted about an hour, and the librarians were able to expound on information they already had received from the question­ naire. The techniques of getting addi­ tional information from the participants through focus groups and developing a detailed worksheet were very beneficial to the Undergraduate Services Team’s re­ search. However, the team wanted to re­ cruit a wide range of students for the sur­ vey and did not want to be limited to a particular class. Therefore, it decided not to create questions for the students but, rather, to let the students ask questions pertaining to their particular research in­ terests. Methodology The reference evaluation took place over a period of three semesters. In the first semester, a standard survey, an unobtru­ sive worksheet, and focus group ques­ tions were created. Participants com­ pleted all three steps before they were compensated. The procedure was as fol­ lows: 1. The student came into the office, filled out the survey, picked up the unob­ trusive observation and instructions, and signed up for a focus group session. 2. The student had to have completed the unobtrusive worksheet by the time he or she attended the focus group session. 3. The student was paid $10 after com­ pleting the focus group session. The reference evaluation usually took place right around midterm and finals when students were naturally doing re­ search for papers and assignments. During the three semesters of conducting all three steps, more than a hundred students participated in the reference evaluation. Survey The survey contained ten questions and was used mainly to obtain demographic information and the students’ initial opin­ ion of the reference desk service. Other information sought included: • How often do you visit the U. of A. Libraries? • How often do you approach the desk? • Rank the following items in terms of exceptional reference service: ap­ proachability, listening, follow-up, expert knowledge of resources, and computer technology. Unobtrusive Observation The unobtrusive worksheet contained eleven questions that students had to an­ swer immediately after approaching the reference desk. Students were told not to let on that they were evaluating the ref­ erence staff and to ask questions pertain­ ing only to their research. Moreover, they were given specific instructions on the type of questions they were not allowed to ask (e.g., Where is the rest room?). The first thing all participants were required to do was observe the reference desk for two to five minutes and then write down everything they saw. The back of the sheet Reference Evaluation 549 provided room to state two things they thought reference service providers did well and two things they thought needed improvement. The unobtrusive observa­ tion worksheet questions are provided in figure 1. Focus Groups After completing the survey and the un­ obtrusive observation worksheet, three to five students at a time attended an hour- long focus group session. The focus group questions basically allowed the team to elaborate on points from the survey and unobtrusive observation. One question asked how librarians should balance tech­ nology and print resources to help stu­ dents with their research. Normally, the focus group was coordinated by a mod­ erator and included a recorder. Usually, the moderator was an Undergraduate Services Team member who had been trained on how to obtain additional information and how to ensure that everyone has a turn to talk. The recorders varied; most often, a reference service provider was involved so that he or she could get an insider look at what the students really thought about the reference desk. The recorder was respon­ sible for writing everything down and summarizing the session in a one- to two- page report. Target Groups During the three semesters of conducting all three steps, more than a hundred stu­ dents participated in the reference evalu­ ation. One important thing the team dis­ covered is that it was not necessary to interview thousands of participants in order to begin to identify areas that need improvement. Yet, to have a smaller sub­ set of participants, the team had to spend lots of time developing a target market. The goal was to ensure that the sample population roughly represented the university’s student population. Therefore, most of the time the team had to screen for freshmen, sophomores, and minorities during its recruitment efforts. Initially, ads in the library and the stu­ dent unions produced many more seniors and graduate students. Although stu­ dents at this level were more reliable in terms of completing all three steps, they had a very different outlook on what con­ stitutes good reference service than fresh­ men and sophomores did. Moreover, by taking extra steps to recruit a wide range of people (i.e., putting ads in the dorms, cultural centers, student organizations), the team was able to capture different experiences without needing a larger population. After a few times with the whole process, the team started to get people repeating the same answers, thoughts, and ways of thinking, which reaffirmed that the team was on the right track. Results Overall, the students were very pleased with the reference desk. Most of them thought it was a worthwhile and needed service to help confused and new stu­ dents get acclimated to the world of re­ search. The overall scores were very posi­ tive, and reference service providers usually advised the students correctly on how to get started with their research in­ terest. Very few students wanted the ref­ erence service provider to do the research but preferred, instead, that he or she point out which databases/print resources/ Web sites were needed to guide them in the right direction. Most of the important information the team uncovered was from the unobtrusive observation. A few of the questions had a combination of a Gantt score plus room to expand, which gave a better understanding of why the person scored higher and lower in some cases. Spring 1998 For this round, the team recruited stu­ dents by posting ads in the library. Signs were put up on bathroom walls, in the elevators, and by the e-mail computers in the main and science-engineering librar­ ies. More than seventy people initially showed interest in participating. Upper­ classmen (juniors, seniors, and graduate students) accounted for 80 percent of the 550 College & Research Libraries November 2000 FIGURE 1 Unobtrusive Observation Worksheet (Questions) 1. Observe the reference desk for approximately two to five minutes (incognito). What are the reference staff members doing? (Write down everything.) Do they look approachable? No yes 1 2 3 4 5 2. How long is the average wait for service? 3. Which person are you going to approach? Why? 4. What is your research question? (Write down the general idea.) 5. What did the person say when you initially asked the question? 6. What was the person's attitude while you explained the problem? 7. Which resources did the person recommend? 8. Did you find the resources useful? No yes 1 2 3 4 5 9. Did the sources directly answer your question? No yes 1 2 3 4 5 10. Would you approach the same person again? Why? 11. Did the person follow up to see if you needed additional help? Take this time out to list two strengths and two ideas for improvement about the reference/service Reference Evaluation 551 participants. Responses also came from some freshmen and sophomores who worked in the library but also used the reference desk frequently. Upperclassmen: • visited the reference desk more fre­ quently; • were less interested in Web search­ ing than in mastering database searching; • picked “expert knowledge of re­ sources” over approachability; • thought the reference service pro­ viders were doing a good job answering questions but needed to reach out more to incoming students (after observing the desk for five minutes); • identified roving, wearing name tags, and learning more about the new da­ tabases as the main ideas for improvement. A number of Asian and other interna­ tional students also participated in the first round. Some of these students: • did not stop at the reference desk before this evaluation because of negative recommendations from other students (focus group sessions); • looked more for approachability than expert knowledge of resources; • mentioned not feeling welcomed at the reference desk but knew service providers gave good advice/answers (fo­ cus group sessions); • identified roving, smiling, and fol­ lowing up with confused students as the main ideas for improvement. One final group of students is worth mentioning. These were freshmen and sophomores with technology-intensive class assignments who needed additional help. • Two students had been assigned to work on Web pages and asked the librar­ ian staff person for help. The only person referred to as an expert in designing Web pages was out of town. • Most of the time, the students said the Internet was their main source of in­ formation. • One person wanted to know how to incorporate Web images into PowerPoint and was told “We don’t handle those questions.” • Ninety percent of the students have not heard about the Free Workshop classes. • This group of students mentioned roving, name tags, follow up with con­ fused students, and technology training as the main ideas for improvement. Fall 1998 For this round, the team wanted to in­ volve more undergraduate students (freshmen and sophomores). In addition, more students were needed who were less familiar with the library. The team was successful in recruiting freshmen and sophomores, but these students proved less reliable than the graduate students. In addition, more African American and Latino students were recruited for this round. Most of the fliers were distributed outside the library at cultural centers and the student union. Although seventy-five students were recruited originally, only twenty-five completed the process. Nev­ ertheless, the same patterns emerged. In the focus group sessions, the student re­ sponses and perceptions of the reference desk remain surprisingly similar. Most people thought that approachability and the correct direction were equally impor­ tant or immediately agreed about going to the reference desk as the last resort. Freshman and sophomore students: • were less likely to approach the ref­ erence desk; • used the reference desk only after exhausting all other options; • were more likely to look at ap­ proachability as an initial sign to ap­ proach the desk; • would not first come to the refer­ ence desk to have Internet questions an­ swered; • believed the reference service pro­ viders were at the desk to help with print and electronic database research only; • generally were unaware (80% of these students) that the free workshop classes existed or that help was available with Web pages and multimedia; • generally identified following up with confused students, roving, name 552 College & Research Libraries November 2000 tags, and smiling more as ideas for im­ provement; • believed that the reference service providers could answer most of their re­ search questions but felt that they should be able to find answers to Internet ques­ tions on their own. Spring 1999 For the last round of the reference evalua­ tion, the team wanted to reach only those students who never visit the library or visit it only rarely. This time, no fliers were posted in the main or science-engineering library. Most of the students sought were from the campus resident halls and cul­ tural centers. Seventy-five students were recruited, and thirty-six of them partici­ pated in the different sections. By the third semester, most of the results had become repetitive and the “new-to-the-library” student experiences were similar to last semester. This group included mainly freshmen and sophomores, as well as more Native American students. (The first two semesters involved only one Native American participant.) The results are very similar to last semester ’s results. Undergraduate students: • were less likely to approach the ref­ erence desk; • used the reference desk only after exhausting all options; • were more likely to look at ap­ proachability as an initial sign to ap­ proach the reference desk; • would not think of coming to the reference desk to ask technology-related questions; • believed that the reference service providers were at the desk to help with print and electronic database research only; • generally were unaware (92% of this group) that there were free workshop classes or that help was available with Web pages and multimedia; • identified following up with con­ fused students, roving, and smiling more as ideas for improvement; • believed that the reference service providers could answer most of their re­ search questions but felt that they should be able to answer Internet questions on their own. Overall Findings from Unobtrusive Observation Following are the behaviors that received high or perfect scores from the unobtru­ sive observation. • Initially approachable: Even when reference service providers are using the computer or performing other chores, they occasionally need to notice students and smile. • Friendly/helpful/listening: This be­ havior usually determined whether the student would ever approach the desk again. • Different options: This refers to a ba­ sic understanding of how much informa­ tion to provide students. Some students said they were overwhelmed with too many options; others wished the refer­ ence service providers would have fol­ lowed up to ensure they understood the electronic databases/print resources, and so on. • Follow-up: Follow-up is especially necessary when the student is new to or confused about using the library. A sim­ ply statement such as “Please come back to the desk if you have any more ques­ tions” is enough to make students feel comfortable about approaching the desk a second time. The reference service provider who went the extra mile to follow up with a student always received higher marks overall even if his or her answer was incorrect. Overall Need for Improvement Following are efforts that would improve service: • walking around (main library, not science-engineering library); • following up with confused stu­ dents; • smiling more; • advertising technology classes; • being alert to students who might be uncomfortable about asking for help; • wearing name tags; • putting up more signage; Reference Evaluation 553 • being more consistent with refer­ rals for technology assistance. Plans for Action Following are some of the things the Un­ dergraduate Services Team either has done already to improve service or is cur­ rently working on: • The team’s graphic designer has created name tags for all reference service providers. • More visible signage has been posted in the reference areas of both the main and science-engineering libraries. • The main library instituted a “rover” slot during peak times, where reference service providers can walk around the reference room with a clip­ board to answer questions. • The team is increasing the adver­ tising about technology classes. • The team is setting up a referral list of reference service providers who have technology expertise. • A final report with results and rec­ ommendations for the reference desk has been distributed to all reference service providers. Future Plans The Undergraduate Services Team is currently working on plans for an in­ formation commons. The information commons will have more than two hun­ dred computers with multimedia capa­ bility. The team anticipates that when the information commons is completed in two years, reference desk service will change completely as more and more computer technology options are pro­ vided at the library. In addition, the team anticipates that reference service providers will have to have at least a basic knowledge about multimedia soft­ ware, in addition to electronic data­ bases. At that time, the team will con­ duct another reference evaluation to obtain feedback on how things are go­ ing. Until then, the team will continue to work on current recommendations from students and try to anticipate and exceed their expectations. Notes 1. Louise Jarvis, “Secret Shopping Service (Consulting Service)” Executive Female 19 (July– Aug. 1996): 24. 2. Ibid., 1. 3. Ibid., 2. 4. Peter Hernon and Charles R. McClure, “Unobtrusive Reference Testing: The 55 Percent Rule,” Library Journal 111 (Apr.15, 1986): 37–41. 5. Ibid., 3. 6. Joan Durrance, “Reference Success: Does the 55 Percent Rule Tell the Whole Story?” Li­ brary Journal 114 (Apr. 15, 1989): 32. 7. Carolyn W. Jardine, “Maybe the 55 Percent Rule Doesn’t Tell the Whole Story,” College & Research Libraries 56 (Nov. 1995) : 477–85. 8. Mary Tygett, V. Lonnie Lawson, and Kathleen Weesies, “Using Undergraduate Marketing Students in an Unobtrusive Reference Observation,” RQ 36 (winter 1996): 270–76. 9. Ibid., 9.