Evidence Summary

Chat Transcript Analysis Reveals that Undergraduate Students are Open to Instruction, while Instructors and Librarians Care About Supporting Student Learning

A Review of:

Jacoby, J., Ward, D., Avery, S., & Marcyk, E. (2016). The value of chat reference services: A pilot study. portal: Libraries and the Academy, 16(1), 109-129. https://doi.org/10.1353/pla.2016.0013

Reviewed by:

Elaine Sullo

Coordinator, Information and Instructional Services

Himmelfarb Health Sciences Library

The George Washington University

Washington, District of Columbia, United States of America

Email: elainej@gwu.edu

Received: 30 Nov. 2016 Accepted: 1 Feb. 2017

2017 Sullo. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

Abstract

Objective – To investigate student, instructor, and librarian perspectives of chat reference service in the context of first-year undergraduate students conducting research for an introductory composition course.

Design – Focus groups, individual interviews, and surveys.

Setting – A large, public university in the United States of America.

Subjects – 57 library reference providers, 36 instructors of an introductory composition course, and approximately 936 undergraduate students in certain sections of the introductory composition course who were assigned a specific research project.

Methods – In spring of 2014, all participants were invited via email to respond to an anonymous chat transcript of a librarian interacting with a student working on his or her research project. Study participants could participate via a brief survey or by taking part in a focus group or individual interview. The invited instructors were asked to forward the invitation to the students in their sections, and reminder emails were sent two weeks after the initial email.

Main Results – Nine instructors, 24 students, and 25 library reference providers participated in the study, representing a response rate of 25%, 3% (estimated), and 44%, respectively. The authors conducted a qualitative analysis of key themes that were derived from both the focus groups or individual interviews and the survey questions. The themes were: students as novice researchers, question negotiation, open and closed questions, instruction, speed and convenience, customer service, and referrals. The theme of “students as novice researchers” is based on student comments related to their frustrations of being inexperienced researchers, as well as librarian comments on strategies for helping these students. Opinions regarding the traditional reference interview, including specific techniques that made the interaction successful, were categorized as “question negotiation.” The “open and closed questions” theme focused on feedback on the types of questions used by librarians in the reference interview. Several components related to chat and instruction were encompassed within the “instruction” theme, including whether those participating in the study were conscious of librarians providing instructions via chat and whether it was deemed valuable; the impact of a library instruction session in which students participated; and identification of missed teachable moments during the chat. The “speed and convenience” theme represented thoughts regarding the balance of instruction and librarian support of news skills, with the student expectation of having their question answered quickly and efficiently. The “customer service” theme focused on the service quality of the reference transaction, while the “referrals” theme encompassed thoughts related to whether students were referred to subject specialists, writing specialists, instructors, or if there was a lack of a referral altogether.

Conclusion – Based on the research results, the authors highlighted the importance of the interconnectedness of teaching that is done in the classroom, in library instruction sessions, and on the reference desk, as all three types of instruction should align. Furthermore, because students are open to instruction via the chat service when they are creating and revising their research question and delving into subject research, chat can be viewed as a key teaching and learning opportunity. Additionally, study results led the authors to conclude that chat reference services could be better marketed; some students were unaware of the extent of the chat service or that it existed at all.

Commentary

In this article, the authors address a topic that has been well researched, however they use a unique methodology in that they examine chat reference service from three perspectives – the student, the instructor, and the librarian. Furthermore, while prior studies focus on the quality of chat transactions or user satisfaction with the service, this study, in contrast, considers a specific context for the chat, which is the student and instructor perspective related to using chat to assist undergraduate students develop research strategies for a particular assignment. The authors state that the patron perspective regarding chat transcript analysis provides librarians with a unique opportunity to understand what users value and hope that this study builds upon the existing literature that uses more traditional research methodologies.

The study was evaluated using the CriSTAL Checklist for appraising a user study (CRiSTAL, n.d.). The study addresses a clearly focused issue in terms of the population studied and the qualitative outcomes: the analysis of focus groups, individual interviews, and open-ended survey questions. The survey instrument and focus group questions are listed as appendices to the article, which makes for a study that could be easily replicated, although readers do not have access to the actual chat transcripts that were analyzed. While the questions were created for the specific study population, they could be used to survey a different student and faculty group without difficulty. The authors did not mention if the survey and focus group questions were pre-tested or piloted, so the researchers did not have the opportunity to ensure that the questions made sense to participants.

The primary limitation of this study was the small sample size and low response rate; this limitation contributed to the difficulty in performing statistical analysis of the survey responses. The authors suggest that a follow-up study could be considered that would focus on only one transcript, compared to multiple transcripts in the current study. A single transcript, along with a larger sample size, would allow the researchers to conduct a comparative analysis between the study groups and better understand the themes that emerged.

Another probable limitation was the way the “librarians” group was defined. This category of participants was comprised of professional librarians, preprofessional graduate assistants, and a small number of paraprofessional staff. Professional librarians who have extensive experience may have different perspectives related to chat transcripts than those who are non-librarians, or those with little experience.

While this research focused on undergraduate students and instructors in an introductory composition course, study data can be used by both instructors and librarians in other disciplines as well. Specifically, the article provides insight into what students’ experience is like as “novice researchers.” Instructors can use the information to better understand how librarians interact with students via chat and perhaps create plans to work in tandem with librarians so the information being provided is uniform. Librarians providing chat service can examine and make changes to their service based on the themes and responses elicited in this research.

References

CRiSTAL Checklist for Appraising a User Study. (n.d.) In nettingtheevidence.pbwiki.com. Retrieved from http://nettingtheevidence.pbwiki.com/f/use.doc