What I've Been Reading
But, What Are Students REALLY Thinking When They Do Research?
Michael Fosmire
Head, Physical Sciences, Engineering, and Technology Division
Purdue University Libraries
West Lafayette, Indiana
fosmire@purdue.edu
Motivation
Have you ever wondered what your students are really thinking when they engage in research projects or otherwise solve information intensive problems? Librarians, and indeed humans in general, are very good at developing theories about how students should behave, but it is less clear whether students actually do behave in that way. How can we target our instruction to address their needs and gaps in understanding, if we don't know how they conceptualize information literacy topics? In my own practice, I have been interested in how engineering students approach design tasks and what role information plays in their solution process.
There are many ways to try and access students' cognitive understanding. Performance assessments, from multiple choice exams to search logs and research papers show how well students can recall information or carry out prescribed tasks. Surveys, interviews, and focus groups gave some information about how students recollected their actions, but I was looking for a richer source of information grounded in the activity itself. For me, think aloud verbal protocols were the key to unlocking the wealth of (often implicit) information that students possess about their research habits.
Background
Verbal protocols are the technical term for any time you are asking participants to give you a verbal response. Verbal Protocol Analysis (VPA) typically encompasses the techniques of setting up a verbal protocol situation and then extracting meaning from the data collected. Two seminal texts describing verbal protocol analysis are van Someren, van de Velde, and Sandberg (1994) and Ericsson and Simon (1993). The first is a quite readable, practical discussion of the technique aimed at the practitioner. Ericsson and Simon, on the other hand, has a more scholarly treatment, looking at the technique from a cognitive psychology perspective, including describing a cognitive model of what happens during the verbal protocol. The text answers questions about validity, common pitfalls, and ways to analyze verbal reports within a model framework.
One of the critical concepts behind the think-aloud is that it captures the participant's thoughts as they are thinking them (or as closely as possible). Simply asking students how they solve a problem, in a survey or interview, will encourage them to answer how they think they should solve the problem, or how they think they actually do, but, according to Ericsson and Simon, retrospective thinking tends to be unreliable. For example, the brain, in attempting to create order out of experience, will reconstruct activities and make them seem more coherent and purposeful than what actually happened, what van Someren calls post hoc rationalization. Conversely, "Two forms of verbal reports can claim to being the closest reflection of the cognitive processes. Foremost are concurrent verbal reports -- 'talk aloud' and 'think aloud' reports -- where the cognitive processes described as successive states of heeded information, are verbalized directly. We claim that cognitive processes are not modified by these verbal reports, and that task-directed cognitive processes determine what information is heeded and verbalized" (Ericsson and Simon 1993, p. 16). Ericsson and Simon go on to describe retrospective reports as also effective, particularly, if they are done immediately after the event, and when reviewing specific actions, for example, by co-viewing a video recording of a task. The authors acknowledge that there is some likelihood of reconstruction and other memory errors in the retrospective technique, but by grounding the participant's report in specific activities, the error is mitigated.
Summary and Consideration of the Technique
Verbal protocols appear to be very simple to set up and implement. Briefly, one sets a task for the participant to do, asking them to carry it out while they say everything that goes through their mind. The resulting activities and verbalizations are recorded and then analyzed.
There are some situations to which verbal protocols are not well suited, however. Tasks requiring a lot of translation from non-verbal information (e.g., interpreting circuit diagrams), high-speed or automatic actions (i.e., routine tasks), and tasks that contain verbalization as part of the task itself (e.g., a live reference interaction) are not well suited for verbal protocols. Basically, anything in which the process of verbalization will interrupt the cognitive process of, or actions taken by, the participant will decrease the validity of the data gathered. That said, particularly when working with experts, who use a lot of intuitive leaps, slowing them down by making them verbalize can help to unpack their decision making process can lay bare information the expert themselves weren't aware of.
Van Someren identifies several key considerations for conducting a verbal protocol: setting, instructions, warm up, prompting, recording, and transcription. All of these considerations relate to developing an environment in which participants are comfortable speaking their minds naturally without being self-conscious or attempting to interpret their thinking before verbalizing it. The setting should be comfortable and relaxed, and in my practice, I have found it effective to employ peers (typically undergraduates) to facilitate the protocol, so participants will be less inclined to try and impress an authority figure. The instructions need to be as simple and straightforward as possible. Typically, instructions might be as brief as "Perform the task and say out loud what comes to your mind," or "Perform this task in the way you are used to. It's important to say aloud everything you think or do."
Although participants usually warm up to thinking aloud pretty quickly, particularly if your participants are science and engineering majors, they may be on the introverted side. Thus, it can be helpful to engage in a quick warm up activity before the protocol begins in earnest (e.g., talking through how they make a sandwich, or how they might build a house out of playing cards), or to include some warm up action as the first part of the problem they are going to solve. This will give the participant a chance to practice verbalizing while doing something. Van Someren suggests if a participant has prolonged difficulty with the warm up activity, it might be best just to terminate the protocol, rather than trying to pull commentary out of the individual, as their solution process will be overly affected by the protocol.
When conducting the actual protocol activity, it is as simple as sitting there, saying 'please keep talking,' when the person is silent for an appreciable length of time. That's it. The idea is to let the participants do what they want in an uninterrupted manner. It's very easy to lead the participants, and suddenly you're capturing your thought process instead of theirs. I've found with librarians conducting interviews, it's very hard for us to ask a question, for example, 'how do you conduct a literature review?' without jumping in and saying something like "have you tried controlled vocabulary?" rather than letting them finish talking about their process. Similarly, if you say "did you mean this?" in response to a verbalization, it might introduce that idea into their thoughts when it otherwise would not have occurred to them. So, the bottom line is to be a good listener and a bad reference librarian! Even if you have to sit on your hands (another reason to have an undergraduate student or other, dispassionate individual conduct the protocol), make every effort not to interrupt the participants' process.
Finally, capturing the verbal protocol. Generally speaking, it is good to make at least two recordings of the protocol, in case a battery runs out or a file is corrupted or written over as it is transferred to be analyzed. The recorder should be unobtrusive, but, as almost any mobile device now comes with a video recorder, it isn't typically difficult to find equipment to use. Recording devices about the size of an iPhone (or indeed an iPhone itself) are pretty common and can easily be ignored by the participant, and flexible min-tripods can ensure that the recording is stable and focused on the participant. I tend to use an audio recording as the backup. Smart pens are pretty handy if you are conducting a protocol involving written work or sketching. The smart pen records the audio and synchronizes it with the drawing on "smart paper," which enables you to see what the participant was saying while they were making a particular sketch, for example. Usability software, such as Morae, or even Captivate in a pinch, allows one to synchronize audio with computer screen capture as well.
Once the protocol has been recorded, it typically needs to be transcribed for easier analysis (although one can code directly in in Morae, for example, tagging and commenting directly on the video stream). For this, I would suggest finding a fast typist and a foot pedal for stopping and starting the recording being transcribed. This makes the process much easier and less frustrating for all involved. As a librarian, the institution is paying you for your ability to analyze the data rather than to input the data. So, making efficient use of a scarce resource, your time and attention, means outsourcing some of the lower level tasks. Again, undergraduate students can be a good source of effort to devote to this kind of activity. Overall, I've had good success involving undergraduates in research projects from end to end (conducting the protocol through analysis, and even presentation), with the bonus that, since undergraduate research is seen as a high-impact educational practice, institutions, even small colleges, are increasingly looking for opportunities for students to get involved in these projects, even subsidizing the cost for students to participate.
Analysis
As with any qualitative research method, there needs to be a framework for analysis of the data. Sometimes, researchers will look for "emergent themes," trying not to prejudge what an event means, but rather trying to identify actions or ideas, grouping them, and then creating a label for that particular class of actions/ideas. Often, one takes a sample from the recordings, creates a classification scheme, and then uses the rest of the recordings to test the categories, adding concepts as needed. Other researchers will ground the data in a particular cognitive or process model, determining how well the data fits the model. Or, they may use a classification scheme to define "bins" for the data, or to compare different studies using the same "measuring stick."
In qualitative research, it is common to have multiple reviewers look at the data, to provide some assurance of the objectivity of the results. Reviewers may co-view a few transcripts and code them, so that they get a common understanding of what a particular code means. Then, the reviewers will code a few transcripts independently and compare results, as one way of computing an inter-rater reliability. If the reliability is too low, they will recalibrate until either the coding scheme is improved to be less ambiguous, or the reviewers reach a common understanding of what the codes mean. After coding the rest of the transcripts, reviewers may go back to some of the original transcripts and re-code them, to ensure intra-rater reliability, i.e., that the understanding of particular codes hasn't drifted in the course of the analysis.
Depending on the purpose of the verbal protocols, one may try and develop models for problem solving, to try and categorize different ways that individuals approach the problem, such as Kruger and Cross (2006), who identified problem driven or solution driven approaches to solving design problems. Or, one might develop frequency models to describe, for example, the amount of time different individuals spend on different tasks. Atman et al (2007), for example, compared information gathering habits of engineering students and professional engineers to see the differences between those groups.
Library Applications
Oh and Wildemuth (2009) provide an excellent overview of the think-aloud protocol and a review of the applications of the technique in the library literature.
In the library field, verbal protocols are most often used in usability testing (e.g., Bury and Oud 2005; Guha and Saraf 2005) and for understanding research processes (e.g., Hofer 2004). In usability testing, participants will often be interacting with a new library interface. They will be asked to complete various tasks, such as searching for a known item, finding an unknown item (e.g., subject searching), or locating a specific service on the library's web site. Think aloud protocols allow the researcher to understand what participants are thinking as they navigate the library information system, for example, what they expect to see when they click a link, or what kind of link they are looking for, which may or may not be visible on the top level of the library web site.
Pomerantz (2004), as described by Oh and Wildemuth, investigated the decision-making process of digital reference "triagers" to identify factors the influence decision making. Pomerantz was able to identify several new factors in the think-aloud sessions that a prior Delphi study wasn't able to unearth. In my own research (Moody et al. 2012), we characterized the problem solving process of technology and engineering students when solving design problems, finding that, in our small sample, technology students tended to be problem focused and engineering students solution focused. There has also been a fair amount of research from the industrial engineering research community (who are interested in process improvement) that has application to information habits of engineers, see Fosmire and Radcliffe (2014) for a summary.
Final Considerations
Think aloud protocols can be very enlightening ways to gain insight into how individuals are actually thinking about a problem or task. While interviews and surveys can also provide information, there is often a distance between what participants think they are doing, or want to believe they are doing and what actually takes place. There is nothing like getting the "raw data" of watching someone actually going through the process to really understand what is going on and realize what gaps in knowledge, skills, and abilities, or at least transfer skills to a particular problem, exist.
That said, verbal protocols have their own drawbacks. Precisely because they are undirected, if you are looking at a particular behavior, it just might not take place during a protocol. When I first devised an engineering design protocol to look at information use, it was certainly a possibility that the students wouldn't use any of the information available to them. This would be useful information, but make for a very short paper! Luckily, only one student solved their design problem entirely from first principles, i.e., not gathering any information outside of the given problem statement. Verbal protocols are also typically messy to interpret. Being fairly stream-of-consciousness, there are a lot of false-starts, incomplete thoughts, 'um's and 'ah's, that are a part of the process and reflect the fluid state of cognition happening in the moment. Thus, there might be a fair amount of content that is challenging to categorize.
Think-aloud protocols can be used effectively in combination with other techniques to get complementary information. For example, as mentioned above, viewing a recording of the protocol with the participant and debriefing immediately after they finish the task provides additional information about how they interpreted their actions, what their bigger strategies were, perhaps clarifying the less coherent verbalizations they made, or filling in the blanks of the protocol when they were being less verbal. Again, retrospective reports can be affected by post hoc rationalization, wherein the participant thinks they "knew the right answer all along," even though it was evident they were frantically trying to find a way forward in their actions. Similarly, surveys and interviews can provide complementary information to think aloud protocols, accessing the more coherent, interpretive way that participants view their action, ideas, or approaches to problem solving. Oh and Wildemuth (2009) provide several examples of studies combining think aloud protocols with other techniques.
There is a pretty low barrier to carrying out a think-aloud protocol, so if you haven't done anything like this before, you might want to try a quick and informal study. The results are often unexpected and will help you understand where your patrons are really coming from. Particularly, as I get farther away from the days when I first explored information systems, this kind of activity helps me reconnect conceptually with the novice user and keeps my reference and instruction interactions from getting too far away from what students need.
References
Atman, C.J., Adams, R.S., Cardella, M.E., Turns, J., Mosborg, S. & Saleem, J. 2007. Engineering design processes: a comparison of students and expert practitioners. Journal of Engineering Education. 96(4): 359-379.
Bury, S. & Oud, J. 2005. Usability testing of an online information literacy tutorial. Reference Services Review. 33(1): 54-65.
Ericsson, K.A. & Simon, H.A. 1993. Protocol analysis: Verbal reports as data. MIT Press: Cambridge, MA.
Fosmire, M. & Radcliffe, D.A. 2014. Integrating Information into the Engineering Design Process. Purdue University: West Lafayette, IN.
Guha, T.K. & Saraf, V. 2005. OPAC usability: assessment through verbal protocol. Electronic Library 23(4): 463-473.
Hofer, B.K. 2004. Epistemological understanding as a metacognitive process: thinking aloud during online searching. Educational Psychologist. 39(1): 43-55.
Kruger, C. & Cross, N. 2006. Solution driven versus problem driven design: strategies and outcomes. Design Studies. 27(5): 527-548.
Moody, N., Fouch, K., Kelley, T., Purzer, S. & Fosmire, M. 2012. Innovation differentiation: examining the problem-solving approaches of engineering and technologist students. Illinois/Indiana - ASEE Section Conference, Valparaiso University. [Internet]. [Cited December 3, 2014]. Available from http://ilin.asee.org/Conference2012/Papers/Fosmire.pdf
Oh, S. & Wildemuth, B.M. 2009. Think-aloud protocols. In Applications of Social Research Methods to Questions in Library and Information Science. Edited by Barbara Wildemuth. Westport, CT: Libraries Unlimited.
Pomerantz, J.J. 2004. Factors influencing digital reference triage: a think-aloud study. The Library Quarterly. 74(3): 235-264.
van Someren, B., van de Velde, W. & Sandberg, J. 1994. The think aloud method: A practical guide to modeling cognitive processes. Academic Press: London, UK. Available on Maartan Van Someren's ResearchGate page: http://www.researchgate.net/publication/215439100_The_think_aloud_method_A_practical_guide_to_modelling_cognitive_processes
This work is licensed under a Creative Commons Attribution 4.0 International License. 