Evidence Summary
Undergraduate Students Can Provide Satisfactory Chat Reference Service
in an Academic Library
A Review of:
Keyes, K., & Dworak, E. (2017). Staffing
chat reference with undergraduate student assistants at an academic library: A
standards-based assessment. Journal of Academic Librarianship, 43(6),
469–478. https://doi.org/10.1016/j.acalib.2017.09.001
Reviewed by:
Heather MacDonald
Health and Biosciences Librarian
MacOdrum Library
Carleton University
Ottawa, Ontario, Canada
Email: heather.macdonald@carleton.ca
Received: 25 Feb. 2018 Accepted: 24 Apr. 2018
2018 MacDonald.
This is an Open Access article distributed under the terms of the Creative
Commons‐Attribution‐Noncommercial‐Share Alike License 4.0
International (http://creativecommons.org/licenses/by-nc-sa/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium,
provided the original work is properly attributed, not used for commercial
purposes, and, if transformed, the resulting work is redistributed under the
same or similar license to this one.
Abstract
Objective – To determine whether undergraduate students can
provide quality chat reference service.
Design – Content analysis of undergraduate student,
professional librarian, and paraprofessional staff responses in chat reference
transcripts.
Setting – Academic library.
Subjects – 451 chat reference transcripts.
Methods – Chat reference transcripts from May 2014–September
2016 were collected. Five categories of answerer were coded: librarian in the
reference department (LibR), librarian from another
department (LibNR), staff without a Master of Library
Science (staff), staff with a Master of Library Science (+staff), and student
employee (student). A random sample of 15% of each category of answerer was
selected for analysis. The answerer categories were collapsed to librarians,
staff, and students for the results section.
Four criteria were used to code chat reference
transcripts: difficulty of query, answerer behaviour,
problems with transcript answer, and comments from coders. Coding for
difficulty was based on the READ scale (Reference Effort Assessment Data).
Answerer behaviour was based on The RUSA Guidelines
(Reference and User Services Association). Behaviours
assessed included: clarity, courtesy, grammar, greeting, instruction, referral,
searching, sign off, sources, and whether patrons were asked if their question
was answered. All coding was done independently between the two researchers,
with very good interrater reliability. Data for variables with disagreement
were removed from the analysis. The chi-square test was used to analyze the
association between variables. Analysis also included patrons’ ratings and
comments about their chat experience. Content and tone were assessed for each
patron comment.
Main Results – Answerer behaviours
showed a significant difference between groups for 3 of the 10 behaviours assessed: courtesy (p=0.031), grammar (p=0.001),
and sources (0.041). The difference between groups for courtesy was: staff
(88%), librarians (76%), and students (73%). Grammar was correct in most
transcripts, but there was a significant difference between the answerer
groups: librarians (98%), staff (90%), and students (73%). There was a
significant difference between groups that offered sources: librarians (63.8%),
staff (62.5%), and students (43.8%).
There was no
significant difference between the answerer groups for the other seven behaviours. Overall, 31% of transcripts showed that
answerers asked if a patron’s query was answered or if they needed further
help. The analysis showed that 79% of transcripts were coded as clear or free
of jargon. Greetings were found in 65% of transcripts. Instruction was
indicated in 59% of transcripts. Referrals were offered in 27% of all
transcripts. Of the transcripts where searching was deemed necessary, 82%
showed evidence of searching. A sign off was present in 56% of all transcripts.
Transcripts with noted problems were deemed so because of lack of effort, being
incomplete or incorrect, having no reference interview, or the answerer should
have asked for help. There was no significant difference between answerer
groups with respect to problem questions.
Of the 24% of patrons
who rated their chat experience, 90% rated it as good or great, and no
significant difference was found between answerer groups. Question difficulty
was coded 50% at level 0-2 (easier), 39% at level 3 (medium difficulty), and
11% at level 4-5 (more difficult).
Conclusion – Undergraduate students are capable of
providing chat reference that is similar in quality to that of librarians and
staff. However, increased training is needed for students in the areas of
referrals, providing sources, and signing off. Students do better than
librarians and staff with greetings and are more courteous than librarians.
There is room for improvement for staff and librarians offering chat services.
Tiered chat reference service using undergraduates is a viable option.
Commentary – Bravender, Lyon and Molaro’s 2011
paper asked “Should chat reference be staffed by librarians?” They found that
non-librarian staff provided a cost effective alternative to librarians. Keyes
and Dworak take this one-step further and ask whether
undergraduate students can provide quality chat reference service. The authors
provide extensive background information on reference services in academic
libraries, tiered reference and chat service models with undergraduate
students, and assessing chat reference quality.
This commentary uses
the CAT critical appraisal tool (Perryman & Rathbun-Grubb, 2014) to guide
the appraisal. The authors clearly state their objective and explain the
impetus for this study. After their library changed chat reference platforms, a
decision was made in July 2014 to include access services staff as answerers.
In the fall of 2015, undergraduate students were added to the roster. The data
collected was from May 2014 through September 2016. It may have made more sense
to present the data starting from the date all answerer groups were involved in
the chat service, especially as this paper focused on the student group.
The results were
presented in several tables with clear explanations. For the most part the
presentation of the results was logical and easy to follow. However, there were
a few inconsistencies. The abstract stated that 451 transcripts were analyzed,
but the results talked about 454 (minus 68) transcripts. The layout of Table 6
was unclear, as the first line in the table appeared to be a header when in
fact it was actually data similar to the rest of the table. For the behavioural variable “searching,” the data in Table 6 was
not consistent with the results section. The p-value in Table 6 was 0.040, but
in the results section the p-value was 0.099. This is problematic, as the
significance threshold used was p<0.05. The numbers for the behavioural variable “sources” were different in Table 6
and the results section as well.
Although there were
some issues with tables matching the results, the methodology was solid and
well executed. This methodology can be used by libraries to analyze their own
chat reference services to identify strengths and weaknesses and improve training.
The authors’ conclusions were grounded in the evidence presented. This study
provides evidence that undergraduate students are capable of providing chat
reference services. Administrators and reference department managers may want
to consider the benefits of students working in roles more traditionally filled
by librarians. Meanwhile, reference librarians will want to consider how their
roles continue to change and what this means for the profession.
References
Perryman, C. & Rathbun-Grubb, S. (2014). The CAT: A generic critical
appraisal tool. In JotForm – Form Builder. Retrieved from http://www.jotform.us/cp1757/TheCat
Bravender, P., Lyon, C., & Molaro, A. (2011).
Should chat reference be staffed by librarians? An assessment of chat reference
at an academic library using LibStats. Internet
Reference Services Quarterly, 16(3), 111-127. https://doi.org/10.1080/10875301.2011.595255