199 Aha Moments and Continued Confusion: An Analysis of Threshold Concepts through Student Reflections in the ACRL Framework Nicole C. Eva, Marissa S. Rocca, and D. Bruce MacKay* With the advent of the ACRL Framework for Information Literacy for Higher Educa- tion in 2015, librarians everywhere have tried to adapt their existing information literacy sessions to incorporate the revised concepts. This article discusses how the librarian responsible for a series of four labs in a first-year course reformed the lab content around the six ACRL Frames. Student reflections from three semesters’ worth of classes were analyzed for content related to each of the six Frames, as well as for areas of enlightened understanding (evidence of crossing a threshold into higher understanding, as first outlined by Meyer and Land, 2003) and continued confusion, with applicability for all instructors trying to incorporate the Frames. Introduction When the Association of College & Research Libraries (ACRL) replaced their earlier Information Literacy Competency Standards for Higher Education with an updated Framework for Information Literacy for Higher Education in 20151, librarians everywhere began to rework their information literacy classes to include these new ideas. The Frames focus on core concepts of information literacy, which function as threshold concepts (as originally defined by Meyer and Land2) and can be troublesome for most students to cross. These include Authority Is Constructed and Con- textual; Information Creation as a Process; Information Has Value; Research as Inquiry; Scholarship as Conversation; and Searching as Strategic Exploration. These six Frames form an interconnected set of skills critical for students to adopt before becoming truly information literate. But how do we know if students “get” these concepts? By linking threshold concepts to the Framework, we can analyze the content of student reflections to see what thresholds the students have trouble crossing. This study attempts to do just that. Background The University of Lethbridge (U of L) was founded in 1967 as a primarily undergraduate in- stitution with a Liberal Education philosophy. In recent years, the principles of Liberal Educa- Nicole C. Eva is Liaison Librarian for several departments, including the School of Liberal Education at the University of Lethbridge, email: nicole.eva@uleth.ca; Marissa S. Rocca is Librarian at Grande Prairie Regional College, email: marissa.rocca@gmail.com; and D. Bruce MacKay is Professor in the School of Liberal Education at the University of Lethbridge, email: mackayb@uleth.ca. © 2021 Nicole C. Eva, Marissa S. Rocca, and D. Bruce MacKay, Attribution-NonCommercial (https://creativecommons.org/licenses/by-nc-sa/4.0/) CC BY-NC-SA. mailto:nicole.eva@uleth.ca mailto:marissa.rocca@gmail.com mailto:mackayb@uleth.ca https://creativecommons.org/licenses/by-nc-sa/4.0/ 200 College & Research Libraries March 2021 tion at the U of L have been defined by four “pillars”: breadth across disciplines; the ability to connect and integrate knowledge; critical thinking and problem-solving skills; and civic engagement. One of the foundational Liberal Education courses, LBED 1000, is a first-year undergraduate, semester-long class. It combines lectures on the theme of “knowledge” in the sciences, social science, humanities, and fine arts, together with a series of labs designed to support the development of a variety of academic success skills. For more than 15 years, these skills labs have included four sessions focused on information literacy (IL) taught by a librarian. With the advent of the new ACRL Framework for information literacy, the librarian involved in the labs decided to try to reframe these four labs around the six frames. The four information literacy labs for LBED 1000 have traditionally been allotted 8 per- cent of the final class grade and thus have included an assignment of one kind or another. Over the years, the labs have morphed somewhat in content but have always focused most on skills-building—catalogue and database searching, citation, and other bibliographic skills. When the labs were revised to focus on the Frames, this assignment was changed to a reflec- tive journal entry done by the students each week. One of the reasons for this was that the instructors of the course had observed that, despite receiving instruction in library resources and search skills, students would default to using substandard websites as sources in their final assignment. It was believed that having students journal their thoughts, associations, moments of enlightenment, and moments of frustration would allow the instructors to see what students struggled with, perhaps revealing some of the major threshold concepts that students were having trouble crossing over. Literature Review The literature linking threshold concepts to information literacy has grown in recent years; and, while not extensively researched, the linkages between liberal education and information literacy seems established. Less well-researched is the use of student reflections as an assess- ment tool for student outcomes in library labs. Meyer and Land first wrote about their theory of Threshold Concepts in 2003; since that time, countless disciplines have adopted this theory as a way to look at student struggles, including information literacy. Threshold concepts are those fundamental ideas within a discipline that are troublesome for newcomers to the field to grasp but, once understood, give learners new perspectives or levels of understanding that enable them to advance further in their learning. Grasping and incorporating such concepts into their ways of knowing is often troublesome for learners, as acquiring them often requires leaving behind what is familiar for what is new and uncomfortable. This step, or threshold, is difficult but, once crossed, is transformative. When learners have mastered such threshold concepts, they are permanently changed as older concepts are no longer sufficient in the context of their new understanding. When first introduced, the ACRL Framework explicitly referenced Meyer and Land’s work and used this idea of disciplinary doorways to enlightened understanding in creating the Frames. This lens through which to view student struggles in information literacy has been seen by many as a good way to integrate information literacy collaboratively in various disciplinary contexts;3 as such, it is a natural fit for a liberal educa- tion course that, by definition and design, focuses on multidisciplinarity. The adoption of the ACRL Framework was not without controversy, however. Rowbot- tom4 claims that there is no scientific evidence proving the credibility of threshold concepts as a theory, citing problems with a consistent definition and application. In his 2014 blog post, Aha Moments and Continued Confusion 201 Kevin Michael Klipfel5 cites Rowbottom’s critique of Threshold Concepts to question why such an unproven theory would be used as the basis of the new Framework being developed by the ACRL. Around the same time, Wilkinson6 likewise points out various flaws in the theory and application of threshold concepts. Bombaro7 went so far as to call the newly adopted Framework elitist, claiming it created divisions between theoretical/philosophical librarians and those on the ground with more practical needs, who found it difficult to understand and apply. There has also not been universal agreement on what threshold concepts in information literacy would be. Hofer, Townsend, and Brunetti8 identified slightly different thresholds than what ultimately appeared in the ACRL Framework, and Fister9 also wondered how a specific set of thresholds might be decided upon, in addition to questioning the disciplinary specificity suggested by the original definition of threshold concepts and how that might ap- ply to information literacy, which is ultimately quite cross-disciplinary. While threshold concepts are a relatively new educational concept, and linking it to information literacy even more recent, the links between liberal education and information literacy have been fairly well accepted. Other writers have considered information literacy a core feature of a liberal education. Wiebe10 indicated that, in today’s information society, both critical thinking and information literacy are more important than ever and explicitly notes the connection between information literacy, liberal education, and lifelong learning—all linked to the critical thinking skills needed to evaluate information not just for academic purposes, but for life. In the “fake news” era, information literacy is a skill as critical to today’s citizens as reading or writing. Critical thinking skills taught in a liberal education mirror the evalua- tion skills needed to be information literate and to live a successfully informed life. Even as the internet was in its infancy, Shapiro and Hughes11 noted its importance as a basic literacy and liberal art for functioning in our society: information literacy should in fact be conceived more broadly as a new liberal art [… ] as essential to the mental framework of the educated information-age citizen as the trivium of basic liberal arts (grammar, logic and rhetoric) was to the educated person in medieval society.12 The use of reflective journaling in information literacy assessments seems limited but is growing. In her presentation at LOEX 2016, Corrall13 makes a call for more information lit- eracy instructors to use reflective practices with our learners, tying this directly to threshold concepts. Reflective assessments have long been recognized by the Education community as an effective learning and assessment technique.14 McKinney and Sen15 had business students write reflections as an assessment of learning outcomes for an IL module, and Squibb and Mikkelsen16 used student reflections as one of their assessment tools in an analysis of student learning for an introductory composition course (2016). In the latter, they discovered that coding the student reflections was more useful and enlightening than their original plan of grading them according to a rubric. They did find the reflections insightful and useful for future course development. Similarly, Scott17 used a reflection assessment to analyze whether students had learned and understood the ACRL Framework. It is interesting to note that the PIs were not aware of these previous studies at the time they decided to undertake this methodology. This study brings together the various threads of threshold concepts, liberal education, and student reflection under one information literacy umbrella to tie it all together. https://rulenumberoneblog.com/author/kevinmichaelklipfel/ https://rulenumberoneblog.com/author/kevinmichaelklipfel/ https://rulenumberoneblog.com/author/kevinmichaelklipfel/ 202 College & Research Libraries March 2021 Methodology In fall 2015, the librarian teaching the newly revised LBED 1000 information literacy labs and the lead instructor of LBED 1000, Dr. Bruce MacKay, were discussing the new ACRL Framework and its links to Threshold Concepts. MacKay had written and presented on the topic of Threshold Concepts before and noted that the 2016 Threshold Concepts Conference call for presentations was out. He suggested they present on the revised lab structure, and they decided on student reflections as the assignment for the library portion of the course to see if they could find evidence of thresholds that students were having difficulty with. They received ethics approval to collect and analyze student work for spring 2016 before the library labs began that semester and presented their preliminary results in Halifax at the 2016 conference.18 They sought and received ethics approval again before the fall 2016 term began to collect student responses for fall 2016 and spring 2017. Students in each lab section during these terms were told about the reflection assignment, which asked students to write a short journal entry each week on what they learned or remained confused about during the lab. Students were encouraged to link this knowledge to other concepts or course work and note whether they had any “aha” moments during the class—basically, any thoughts that came to mind as a result of the lab. The assignment, which includes a rubric by which they were marked, is available in appendix A. The librarian instructor then left the room, and the current library co-op student would hand out the informed consent forms and as- sure students that their participation would be kept secret from the library instructor and would in no way affect their grades. The co-op student collected the signed consent forms and each week would photocopy the student reflection submissions (for which consent had been received) before the librarian graded them; at the end of the semester, the co-op student handed over the anonymized, ungraded copies after the grade appeal window had passed. While the rubric shown in appendix A was used very roughly to guide the marking process, it had no bearing on the final coding analysis done here. In total, 117 students participated during the three semesters, though not all handed in all four reflections; 398 total reflections were received and analyzed. The copies of student reflections were scanned and converted into OCR PDFs and im- ported into NVivo qualitative analysis software. The librarian involved in the project did the first round of coding, establishing both the expected nodes of the 6 ACRL frames, as well as creating new ones as they became evident. A complete list of final nodes is indicated in figure 1. She then enlisted the current library co-op student to perform a second round of analysis to validate the original findings; very few discrepancies were found, and agreement levels ranged from 92 to 100 percent. Looking at the discrepancies, it was determined that most of these were due to the second coder correctly assigning a second node to a section previously only coded to one node, and not due to outright errors or miscoding. Results Figure 1 shows the number of references coded to each node, and figure 2 shows the relative size of each coded node. Overall, among the three semesters’ worth of student comments, the largest node coded to was “understanding and learning” (UL). This node had more than double the references to the next code, which was Searching as Strategic Exploration (SSE). It is not entirely surprising that SSE is the most prominent Frame featured, as a fair amount of instruction time, including several videos and a worksheet, was spent on actual database, Aha Moments and Continued Confusion 203 catalogue, and web searching. SSE is also one of the skills that students associate with and expect to get from the library sessions. It is similarly satisfying to see that UL was so widely featured. This node was defined as “evidence of understanding (‘aha’ moments, particular insight or clarity) but also of recognition of importance of labs (thanks! everyone should take this! glad I did).” While it covers a wide territory, leading the researchers to break it down FIGURE 1 Final Nodes and Number of References FIGURE 2 Coding References Hierarchy 204 College & Research Libraries March 2021 further, it is indicative that students recognized the importance of the IL labs and provides great evidence to show to administrators to continue library involvement in LBED 1000. On the flip side, if the goal was to touch on each Frame equally, it is clear that more time could be spent on Scholarship as Conversation (SC), Information Creation as a Process (ICP), and Authority Is Created and Contextual (ACC). With reference to the latter, more re- cent iterations of the lab have morphed away from SSE to ACC, so it would be interesting to perform an analysis of how that has changed the student reflections. (Unfortunately, ethics and consent was not sought for more recent semesters but may be in the future.) Similarly, a breakdown of coding among the three semesters reveals other changes to have affected the coding by Frame (see figures 3–5): From the figures, which indicate the presence of any code in each semester, it is clear that a change in the curriculum has a direct effect on students’ takeaway. In spring 2016, FIGURE 3 Coding References by Semester Aha Moments and Continued Confusion 205 FIGURE 4 Coding References by Semester FIGURE 5 Coding References by Semester 206 College & Research Libraries March 2021 there was an entire lab dedicated to using Web of Science, which emphasized Scholarship as Conversation. Based on student feedback indicating that they had not understood the utility of the Web of Science exercise, it was dropped. The change allowed more time to be spent on the database searching worksheet, to which questions on searching Google Scholar had been added. Although SC was still mentioned as part of other labs, not having an exercise dedicated to it clearly had an effect on students’ understanding and retention of the concept. Correspondingly, SSE increased in both subsequent semesters. Breaking down more specific nodes, a stemmed word frequency query was run on the node “awareness of library resources,” with the words library, libraries removed from the word list as that was deemed to not be useful and overwhelmed the results. As evident from the word cloud below, the biggest resource that students became aware of through the IL labs was the help and research assistance available at the library (see figure 6). As our librarians will often say that, if students take away one thing from the class, they hope it’s that the students know help is available at the library, this was very pleasing to see. Furthermore, the prominence of words like research, information, resources, available, know, learning, search, and access indicates that the library is seen as a place from which to obtain information. Smaller but still present are words relating to citation, reference, and topic selection, and even words like comforting make an appearance. It seems clear that students do understand how the library might help them in their academic endeavors, at least after having completed these labs. A similar analysis was done on the Continued Confusion node, with library, libraries removed from the word list. From this, it is clear that students still feel uncertain about their FIGURE 6 Awareness of Library Resources Word Cloud Aha Moments and Continued Confusion 207 searching skills, with words such as find, search, using, databases, research, and articles promi- nent here. That students mention these words in the context of identifying places in which they are still uncertain, reveals that more time and practice could be spent on helping them acquire these search skills. The same query was run on Awareness of Past Knowledge Deficits (defined as, “boy I wish I’d known this before, I used to do x but now I know better, etc.”). They used words such as research, topic, search, information, find, and citations when speaking of these past deficits, which indicates that students realized that they hadn’t possessed the type of research and citation skills coming in to university that they now were aware of. Since knowing one has a knowledge deficit is the first step in knowing how to fix that gap, it is heartening to see the students acknowledge that they have (or had) more to learn. The same analysis done on Understanding and Learning was the least illuminating. This makes some sense, as it overwhelmingly had the most references coded to it and as such, can be expected to be more of a mixed bag. Frequent words ran the gamut from search, using, help, research, information, learning, and sources and didn’t really provide much insight into what students did learn and understand after the sessions. This, along with the coders’ knowledge of the variety of items that ended up in that particular node, led them to further subcode this particular node during the analysis phase. Discussion Because the Understanding and Learning node was so broad, with more than 400 references coded to it, it was decided that it would be beneficial to break it down to three subnodes: “Thanks for the labs,” where students expressed gratitude for having the opportunity to learn the skills presented and realized the utility of the labs for their future endeavors; “Aha mo- ments,” where it was clear that students had a breakthrough in learning directly tied to the labs; and “General learning,” where there was indication of learning but it wasn’t necessarily an “aha” moment. Breaking these apart not only helps illuminate which thresholds students crossed, but it is also useful in collecting illustrative quotes to share with administrators to ensure the longevity of the partnership between the Library and the School of Liberal Educa- tion. This resulted in a fairly even breakdown among the three subnodes, with Aha moments having the least coded references (110), General Learning with 170, and Thanks for Labs at 188. Aha Moments Looking specifically at the Aha moment category, many students had breakthroughs with the notion of choosing a research topic (broad vs. narrow, adapting as you go) as well as with research tips such as truncation and Boolean, citation chain searching, and the knowledge of the utility of the library resources themselves. These results speak mainly to Searching as Strategic Exploration, Research as Inquiry, and Scholarship as Conversation. A tree map query (run with stemmed words, and with library as a stop word) illustrates this finding quite nicely (see figure 7), and it is even clearer when limited to the 10 most frequently mentioned words in this subnode: The 10 most frequent words in Aha Moment subnode include “research” (122 mentions), “using” (92), “topic” (90), “find” (80), “help” (70), “search” (66), “sources” (64), “information” (62), “paper” (52), and “know” (46). These all indicate the presence of student learning in the most critical concepts of information literacy, in a meaningful way. 208 College & Research Libraries March 2021 Quotes from the “aha moment” subnode range from the understanding that there are real-life implications for being information literate, such as knowing the timeline on which scholarly information can be produced and becoming more aware of the value of source evaluation. Students were also amazed to find out that choosing topics and writing essays were strategic and iterative, and that citation analysis was a valid research mechanism. For a few representative quotes from this subnode, please see appendix B. Thanks for Labs Many of the comments coded to this category were quite similar, expressing gratitude to the librarians and recognizing the utility of the labs for their future academic endeavors. These comments are incredible ammunition to advocate for the continued inclusion of IL labs not only in LBED 1000, but perhaps to advocate for a wider swath of first-year classes. Students clearly see a need for these skills, even if going in they did not; as an institution, we would be remiss not to listen. A few typical quotes that illustrate this range can be found in appendix B. Awareness of Past Deficits Another illuminating category that evolved during the course of coding was the students who recognized that they had been ill-served in the past, whether wishing they’d had training in high school, received a library skills session earlier, or had a realization of what they’d been doing wrong in previous research. Many of these quotes are not only gratifying, but along with the “Thanks” quotes will prove useful in attempts to revise the curriculum to ensure students receive an IL session early in their university career; a selection can be found in appendix B. The quotes provided from particular nodes display the most interesting and salient personal anecdotes provided from the students. There were several mentions that the skills they were learning in the IL labs were transferable to other classes in all disciplines, not just Liberal Education; they included those in English, Public Health, Psychology, and Political Science. Perhaps more importantly, many realized the skills would serve them throughout their university career and beyond. While the word clouds and number of coded references provide a good overview of what students understood in the labs, highlighting the Aha moments, the moments where students realized their past limitations, and how FIGURE 7 Awareness of Library Resources Word Cloud Aha Moments and Continued Confusion 209 helpful these skills will be in the future are the true rewards of doing this analysis. They illuminate why we do this work and provide fodder for ensuring all future students have this opportunity. Continued Confusion Finally, one of the most instructive sections to review before embarking on a new set of labs is the Continued Confusion category. As found in the word frequency analysis of this node, despite most students learning the most in the area of SSE, it also continues to be one of the most confusing for them. Another way of looking at it is to pull out the 10 most common (stemmed) words in this category, which include using, search, find, articles, databases, research, information, helpful, different, and topic—all words relating to the search process. Illustrative quotes along this line include confusion around topic creation, authority, and citation; these quotes are found in appendix B. Another evident source of confusion, not always apparent to the students but apparent to the researchers, was confusion among the various search engines (library catalogue, Sum- mon, individual databases, and the open web). However, this did not occur as frequently as had been anticipated, especially once Google Scholar had been incorporated into the search worksheet (semesters 2 and 3) to try to make the differences between searches more explicit. Using student narratives to highlight their experiences in the library labs turned out to be effective not only in revealing what Frames were most successfully covered in the classes, but also to have students think a little more critically about what they were learning. The pro- cess of reflection caused some students to think more deeply about the bigger implications of research, as opposed to learning rote skills. This lends credence to the literature cited above, which shows links between liberal education and information literacy. The critical thinking so lauded by liberal education comes into play when students discover that research is about much more than pointing and clicking, or about a simple checklist for source evaluation. As such, this study has strengthened those ties. In addition, we have added to the literature on the use of student reflection in information literacy as an effective and insightful assessment tool, in addition to highlighting some thresholds that other librarians may want to pay par- ticular attention to as possible “pain points” for their own students. Embarking on this study, we wondered whether students encountered any difficulties in learning the main concepts in information literacy, as defined by the ACRL Framework. Looking at the analysis, it is clear that, while students did indeed “get” the Frames, they did so in varying levels. Part of this is due to the evolving nature of the course during the three semesters studied; it is quite clear that, the more time spent on a concept, the most students mentioned it and truly understood it. This is also evident by looking at the skills students mentioned most in learning the Frames, mostly focused around SSE and RI, which a great deal of time was spent on in the class. Perhaps more valuable, however, were the additional codes that emerged during the analysis. Students freely discussed their realization of past skills deficits, their gratitude at having learned the skills as part of the labs (even if at first they were resistant), and the wish that this type of class was available to all first-year students during their first semester. This type of data is incredibly useful when trying to map out a curriculum plan and lobby for a more inclusive model to include IL content in a mandatory first-year experience course. It shows how critical these skills are, how students lack those skills entering university, and 210 College & Research Libraries March 2021 how grateful they are for having learned them. For administrators of first-year programs, this is a message they need to hear. Limitations A limitation of sorts is that the three semesters studied had quite different numbers of en- rolled students, so an across-the-board quantifiable analysis would not be possible; similarly, the labs were not taught in the exact same way each semester, with some content morphing over time. As such, semester-to-semester analysis has limited utility. Another limitation will be that students had to opt in for the study, so not all student reflections from each semester will be included (though the majority are). An improvement for future research could be to conduct post-tests to discover student retention of the framework concepts. Conclusion Student reflections proved to be a useful assessment exercise for information literacy labs, both for student learning and for continued improvement in the evolution of the labs. It pro- vides a weekly feedback mechanism and allows the librarian-instructor to adjust if it is clear that students haven’t understood something from a prior week. Just as importantly, careful analysis of the student reflections allows the librarian to see trends across the semester and get a holistic view of what did and didn’t work, enabling an adjustment of the labs for the longer term. It is also clear that students learned more than they expected and they realized the utility of the labs, a conclusion they may not have come to had they not taken the time to reflect. Overall, this effort proved to be an extremely useful way not only to critically engage students in the learning process but to provide evidence for continuous improvement of the IL labs to assist students to grasp the major threshold concepts present in the ACRL Frame- work. Finally, it provides important evidence of the importance of information skills to a wide variety of students in a range of disciplines, particularly in their first year. This analysis provides not only a blueprint for other library instructors to engage in meaningful feedback and assessment for their own classes, but it also provides a window into the types of skills students lack, are looking for, and are grateful to receive instruction in. Acknowledgments Thank you to all of the students in the studied cohorts for participating in this project and allowing us to study and use your words. True learning is where both students and teachers gain new knowledge. Thank you also to the peer reviewers, whose comments undoubtedly improved this manuscript. Finally, thank you to Bruce MacKay for making the link to threshold concepts and encouraging us to embark on this research, and to Marissa Rocca for bringing fresh insight and careful attention to this work. Aha Moments and Continued Confusion 211 APPENDIX A. Liberal Education 1000—Spring 2016 Reflective Journal Assignment What is a reflective journal and why should you use one? A reflective journal helps you process your understanding of concepts we cover in the labs. It differs from typical class notes in which you “passively” record information given to you by an instructor; instead, it will allow you to reflect upon lessons you have learned and help you process your thoughts around it. ¾ BRIEFLY summarize the concepts discussed to show you understand them—in your OWN WORDS—but then expand your understanding by making connections with other subjects or knowledge you have. Record any thoughts, ideas, responses, reactions, questions, or confusion you may have over the subject matter. ¾ Make a strong connection between the subject matter and how it relates to your life—that is, how you have seen this work in the past, how you might use this skill in the future, how this reminds you of another concept from another class, and so on. Make clear how connections have been sparked in your mind. ¾ Reflect on what interests you about the topic, or what you would like to follow up on. ¾ Reflect on what confuses you about the topic, if anything, and what you might need further explanation of. 9 Journal entries must be typed and submitted in hard copy. 9 Late submissions will not be accepted. Entries must be submitted at the beginning of the lab every week. 9 Entries may range from one paragraph to one page—as long as you are addressing all of the points listed here, length doesn’t matter. 9 Write clearly—while marks will not be deducted for grammar or spelling, try to avoid sentence fragments, run-on sentences, abbreviations, and other things that may interfere with my understanding. 9 If diagrams or pictures help you explain an idea, concept, or feeling—feel free to use them. 9 Don’t limit yourself to the points mentioned—if inspiration strikes, write away! The point of this exercise is to have you think more deeply about the topics discussed, and for you to show me that you understand the concepts. Grading Matrix—Reflective Journal Assignment There is no “right” or “wrong” when recording your ideas. Marks will not be deducted for grammar, although it must be clearly written or I won’t be able to understand your points. The table below presents the “essentials” I will look for: Exceeds Expectations Meets the Standard Needs Improvement Weak Element Journal entry submitted on time every week Clearly written Shows understanding of concept Draws connections to other topics/real life Reflects what interests/excites you about the current topic Reflects what confuses you about the current topic (if anything) 212 College & Research Libraries March 2021 There will be four journal submissions required (one for each week of the lab). The first three will be due at the beginning of the lab the following week; the last one will be due at the beginning of your regular class on March 23. Attendance at the labs is mandatory, as it will be impossible to complete a journal entry for a lab not attended. Each journal entry will be worth 2% of your final grade for this course, for a total of 8%. Along with the weekly journal entry, you must submit the previous week’s in-class work- sheet (completed). The worksheets won’t be specifically marked, but they WILL be noted for completeness, which will figure into your 2% that week. Here are some questions that might help you reflect… remember, you do not need to answer all of them (and many may not be applicable for a given topic), but they may help to guide your thinking and spark ideas about what to write. Explore a learning experience… Awareness Evaluation Regulation Content (What) What have I learned? Do I understand what I have learned? What else do I need to learn? What can I do to gain a better understanding? Where can I find more information? Process (How) How did I learn/do it? What strategy have I used in learning this topic? How effective is this strategy? How can I make this strategy more effective? Is the way I do it the best way? Reasons (Why) Why learn it? What is learning? Why would I think so? Is this the only purpose of learning? What would be a more useful way to understand learning? How could this learning experience be interpreted differently? Think of a learning experience in relation to… Awareness Evaluation Regulation Academic Development How does this learning experience contribute to my academic development? What is/are my short- term/long-term academic goal(s)? What does this learning experience tell about my choice of academic goal and path? Am I making good progress? Am I on the right track? All things considered, is this goal a suitable goal? Are there any other options? What other paths can I take to achieve my goals? What obstacles have I encountered? What is the source of the obstacles? Am I on the right track? How can I remove those obstacles? What other paths can I take to achieve my goals? Professional Development Same as academic development, just that this time think about the learning experience in relation to your professional development instead of academic development. Personal Development What does this learning experience mean to me? How does it matter to me if I failed or succeeded? What does this learning experience tell me about my potentials, and myself as a person? What do I know about myself? How am I living the most of myself? http://www.polyu.edu.hk/learn-to-learn/es/materials/Reflective%20Learning%20Journal%20(Teacher%20Guide).doc http://www.polyu.edu.hk/learn-to-learn/es/materials/Reflective%20Learning%20Journal%20(Teacher%20Guide).doc Aha Moments and Continued Confusion 213 Sample Reflective Journal Entry (by A. Student) Last week’s lecture presented the idea that science is the most powerful form of evidence [1]. My position as a student studying both physics and law makes this an im- portant issue for me [2] and one I was thinking about while watching the “The New Inventors” television program last Tuesday [3]. The two “inventors” (an odd name consider- ing that, as Smith [2002] says, nobody thinks of things in a vacuum) were accompanied by their marketing people. The conversations were quite contrived, but also funny and enlightening. I realized that the marketing people used a certain form of evidence to persuade the viewers (us?) of the value of the inventions [4]. To them, this value was determined solely by whether something could be bought or sold—in other words, whether something was “mar- ketable.” In contrast, the inventors seemed quite shy and reluctant to use anything more than technical language, almost as if this was the only evidence required—as if no further explanation was needed. This difference forced me to reflect on the aims of this course—how communication skills are not generic, but differ according to time and place. Like in the “Research Methodology” textbook discussed in the first lecture, these communication skills are the result of a form of triangula- tion, [5] which I have made into the following diagram: 1. Description of topic en- countered in the course 2. The author’s voice is clear 3. Introduces “everyday” life experience 4. The style is relatively informal, yet still uses full sentences 5. Makes an explicit link between “everyday” life and the topic https://student.unsw.edu.au/examples-reflective-writing https://student.unsw.edu.au/examples-reflective-writing 214 College & Research Libraries March 2021 APPENDIX B. Quotes from “Aha Moment” Subnode “This discussion about the information cycle is really important for real-life application (not just for writing papers). If we keep in mind that it takes six or more months for a scholarly article about an event to be written and peer reviewed, it really makes you stop and think about the validity of sources we use to gain knowledge about day-to-day events. If something happens and the media cover it immediately, we need to remember that the whole story and investigation into it may not be available immediately and that we shouldn’t jump to conclu- sions based on one reporter’s word. From now on I’m going to give it a little more time after events before I start jumping to conclusions based on what other people have heard or said.” “It never really occurred to me that writing a paper could have such a comprehensive strategy.” “It is also useful to check how many times that particular source has been cited. I had never known that this was possible before and it really brought a new light to my research and my analyses of sources.” “I learned that modifying my topic shouldn’t be thought of as something that is to be done in addition to producing the project but as a part of producing it. If it is included in the process of writing, it changes the students’ perspective and allows for a higher-quality and more ef- fective paper.” “I had never actually considered that the initial research question is actually research in it- self. I also had no idea of the importance of selecting a good research question. The lab that we did in class was really helpful in helping me evaluate the integrity of my initial and final research question.” “Truthfully, I used to think that I was doing a great job at using the library database and I didn’t think there was much need to improve, but boy was I wrong. Although I definitely did know how to navigate my way around successfully, I was left in the dark when it comes to the little tips and tricks like how to use Boolean.” “This sounds so simple when the librarian explains it, but I never actually thought about how wording it like that would impact the search results.” “In this lecture, I think that the most important thing is we should always question and be pre- cise of our sources. Even though we find [sic] some information that with [sic] peer reviewed, and considered as an academic resource. There may still have some subjective viewpoints as well as some errors that we should be aware of.” “We compared summon to google scholar to a database search and it was eye opening to see how much more information you can get through the library! I have often viewed the library as something that is for more casual reading for people who enjoy reading to use but that is not true at all. It can make research a lot easier: when you know how to use the library, you have better access to newer reliable information.” Aha Moments and Continued Confusion 215 Quotes from “Thanks” Subnode “Without this lab I honestly don’t know how I would have figured this all out by myself.” “These labs are so helpful to first years who are sometimes too scared to ask for help.” “Overall the library labs have been extremely helpful, not just by giving us skills that are useful across multiple disciplines but by helping to reduce the overwhelming nature of the first year of university.” “Overall I did not know there was this much we could do to help find what we need. These skills will be used for a lifetime as we are using more and more the internet and technology in school and workplaces. It is a skill that will be valuable beyond school.” “Liberal education made a great decision to include this aspect of the course. These lecture present a clear cut way to do research, and the various tools the library offers.” “This library lab overall has been a very valuable lab to learn and I think requiring first-year students to attend it would do tremendous things for students. People probably won’t seek help by themselves, but if every first-year class required you to take this lab at least once, everyone would really benefit.” “I originally did not think I would enjoying [sic] this class at all, but it was better than expected. I didn’t expect to like it because I had written multiple essays last semester that required academic sources. And last semester I had no issues using the library search tool. But I did not know about the different ways you can search with the library search tool. Using the info I learned from class, I was able to find more scholarly sources for an essay I am working on this semester.” “Library lab is actually interesting! Sorry if I sound surprised… It is like a whole new world of information I never really knew about.” “Overall, these library labs have been very beneficial for helping me to learn how to access information and use the library more than I have been. I have developed numerous skills that will consistently help me in the future and that I plan to build upon during my time in university. I am very glad that the Liberal Education program provided us with these library labs, and I know I feel much more confident in many of the areas that we went over than I did before. Not to mention that I learned a lot of useful tools I had no idea existed before.” “But I wish I would’ve learned all of this sooner!” Quotes from “Awareness of Past Deficits” Subnode “This is my second year of university yet I am finding this out now. It could have helped me in my first year. I think that they should make this a universitywide situation maybe teach it in a couple of or all of the first-year classes… if one does not take liberal education they will graduate this school without knowing there is a much easier… and intelligent way to get 216 College & Research Libraries March 2021 information. I am saying this because most students like myself really did not know much about the library or went there to study will be clueless like I was.” “I honestly wish someone had taught me how to do this years ago, it would have been very helpful when researching topics in high school. Not only that but to be introduced to this ear- lier would have definitely helped prepare myself better for papers in university. The process for choosing a topic makes a lot of sense to me now that someone has laid it out.” “The number 1 topic that really got my attention was the writing process and research topic variations. Coming straight from high school has made it very difficult to adjust to the new expectations of university, but I found going through this topic in lab very useful. “ “This was the biggest tip I’ve learned in these labs, only because I didn’t realize that differ- ent formats of sources can follow different citing formats. Not pertaining to APA standards or Chicago protocols that is. Had I known this specific tip last year I wouldn’t have lost 5% grade mark on messing up a citation for a scholarly journal source I referenced in my paper.” “When I first started I was scared of the library but now I feel more confident!” “Truthfully, I used to think that I was doing a great job at using the library database and I didn’t think there was much need to improve, but boy was I wrong.” “I used to not understand what a peer-reviewed article was compared to an excerpt from a book.” “I started to recollect and remember times in my researching past, where I tried to research my topic but all I found was irrelevant information. I used to search full sentences or ques- tions into Google and expect to find appropriate and valid information.” Quotes from “Continued Confusion” “How am I able to better differentiate between sources that can be beneficial for the specific topic I am researching?” “The one thing that confused me in the lecture was truncations and wild cards. “ “I would like a follow-up on more ways to narrow the search.” “Different techniques on how to better analyze a topic in order to understand the main mes- sage in my own words and understanding.” “I would like to be able to learn how to properly identify sources that contain not only ben- eficial information for my topic that I am researching, but also being able to determine how legitimate it is.” “Importance of referencing our research as it is not always clear to me, especially when it comes time to write it.” Aha Moments and Continued Confusion 217 Notes 1. Association of College & Research Libraries, “Framework for Information Literacy for Higher Education,” available online at www.ala.org/acrl/standards/ilframework [accessed 9 December 2019]. 2. Jan H.F. Meyer and Ray Land, “Threshold Concepts and Troublesome Knowledge: Linkages to Ways of Thinking and Practising within the Disciplines,” in Improving Student Learning: Ten Years On, ed. Chris Rust (Oxford, UK: Oxford University Press, 2003), 1–16. Originally published as ETL Occasional Report 4, www.etl. tla.ed.ac.uk/docs/ETLreport4.pdf. 3. Amy R. Hofer, Silvia L. Hanick, and Lori Townsend, Transforming Information Literacy Instruction: Threshold Concepts in Theory and Practice (Santa Barbara, CA: Libraries Unlimited, 2018); Disciplinary Applications of Informa- tion Literacy, eds. Samantha Godbey, Susan B. Wainscott, and Xan Goodman (Chicago, IL: Association of College and Research Libraries, 2017); Joanna M. Burkhardt, Teaching Information Literacy Reframed: 50+ Framework-Based Exercises for Creating Information-Literate Learners (Chicago, IL: ALA Neal-Schuman, 2016); Teaching Information Literacy Threshold Concepts: Lesson Plans for Librarians, eds. Patricia Bravender, Hazel McClure, and Gayle Schaub (Chicago, IL: Association of College and Research Libraries, 2015); Colleen Burgess, “Teaching Students, Not Standards: Threshold Crossings for Students and Instructors Alike,” Partnership: The Canadian Journal of Library and Information Practice and Research 10, no. 1 (2015), https://doi.org/10.21083/partnership.v10i1.3440; Joe J. Eshle- man and Julie Obst, “Librarians and Students: Making the Connections,” in Not Just Where to Click: Teaching Students How to Think About Information, eds. Troy A. Swanson and Heather Jagman (Chicago, IL: Association of College and Research Libraries, 2015), 293–309, available online at http://scholarsarchive.jwu.edu/staff_pub/24; Lori Townsend, Korey Brunetti, and Amy R. Hofer, “Threshold Concepts and Information Literacy,” portal: Libraries and the Academy 11, no. 3 (2011): 853–69, https://doi.org/10.1353/pla.2011.0030. 4. Darrell Patrick Rowbottom, “Demystifying Threshold Concepts,” Journal of Philosophy of Education 41 (2007): 263–70, https://doi.org/10.1111/j.1467-9752.2007.00554.x. 5. Kevin Michael Klipfel, “This I Overheard: Threshold Concepts Getting Laughed Out of the Room,” Rule Number One: A Library Blog (Nov. 3, 2014), available online at https://rulenumberoneblog.com/2014/11/03/this-i- overheard-threshold-concepts-getting-laughed-out-of-the-room/ [accessed 5 June 2020]. 6. Lane Wilkinson, “The Problem with Threshold Concepts,” Sense and Reference: A Philosophical Library Blog (June 19, 2014), available online at https://senseandreference.wordpress.com/2014/06/19/the-problem-with- threshold-concepts/ [accessed 8 June 2020]. 7. Christine Bombaro, “The Framework Is Elitist,” Reference Services Review 44, no. 4 (2016): 552–63, https:// doi.org/10.1108/RSR-08-2016-0052. 8. Amy R. Hofer, Lori Townsend, and Korey Brunetti, “Troublesome Concepts and Information Literacy: Investigating Threshold Concepts for IL Instruction,” portal: Libraries and the Academy 12, no. 4 (2012): 387–405, https://doi.org/10.1353/pla.2012.0039. 9. Barbara Fister, “Crossing Thresholds and Learning in Libraries,” Library Babel Fish: Inside Higher Ed (May 22, 2014), available online at https://www.insidehighered.com/blogs/library-babel-fish/crossing-thresholds-and- learning-libraries#sthash.AsO9Tifm.dpbs [accessed 5 June 2020]. 10. Todd J. Wiebe, “The Information Literacy Imperative in Higher Education,” Liberal Education 101/102, nos. 4/1 (2016): 52–57, www.aacu.org/liberaleducation/2015-2016/fall-winter/wiebe. 11. Jeremy J. Shapiro and Shelley K. Hughes, “Information Literacy as a Liberal Art: Enlightenment Proposals for a New Curriculum,” Educom Review 31, no. 2 (1996): 31–35, available online at https://www.educause.edu/ir/ library/html/erm/31231.html. 12. Shapiro and Hughes, “Information Literacy as a Liberal Art,” para 13. 13. Sheila Corrall, Crossing the Threshold: Reflective Practice in Information Literacy Development, paper presented at LOEX Annual Conference, Pittsburgh, PA, May 5–7, 2016, http://d-scholarship.pitt.edu/28240/1/Corrall_LOEX_2016. pdf. 14. Gavin T. Brown and Lois R. Harris, “Student Self-Assessment,” in SAGE Handbook of Research on Classroom Assessment, ed. James H. McMillan (Thousand Oaks, CA: Sage Publications, 2013), available online at https:// search.credoreference.com/. 15. Pamela McKinney and Barbara A. Sen, “Reflection for Learning: Understanding the Value of Reflective Writing for Information Literacy Development,” Journal of Information Literacy 6, no. 2 (2012): 110–29, https://doi. org/10.11645/6.2.1747. 16. Sara D. Squibb and Susan Mikkelsen, “Assessing the Value of Course-Embedded Information Literacy on Student Learning and Achievement,” College & Research Libraries 77, no. 2 (2016): 14–83, https://doi.org/10.5860/ crl.77.2.164. http://www.ala.org/acrl/standards/ilframework http://www.etl.tla.ed.ac.uk/docs/ETLreport4.pdf http://www.etl.tla.ed.ac.uk/docs/ETLreport4.pdf https://doi.org/10.21083/partnership.v10i1.3440 http://scholarsarchive.jwu.edu/staff_pub/24 https://doi.org/10.1353/pla.2011.0030 https://doi.org/10.1111/j.1467-9752.2007.00554.x https://rulenumberoneblog.com/2014/11/03/this-i-overheard-threshold-concepts-getting-laughed-out-of-the-room/ https://rulenumberoneblog.com/2014/11/03/this-i-overheard-threshold-concepts-getting-laughed-out-of-the-room/ https://senseandreference.wordpress.com/2014/06/19/the-problem-with-threshold-concepts/ https://senseandreference.wordpress.com/2014/06/19/the-problem-with-threshold-concepts/ https://doi.org/10.1108/RSR-08-2016-0052 https://doi.org/10.1108/RSR-08-2016-0052 https://doi.org/10.1353/pla.2012.0039 https://www.insidehighered.com/blogs/library-babel-fish/crossing-thresholds-and-learning-libraries#sthash.AsO9Tifm.dpbs https://www.insidehighered.com/blogs/library-babel-fish/crossing-thresholds-and-learning-libraries#sthash.AsO9Tifm.dpbs http://www.aacu.org/liberaleducation/2015-2016/fall-winter/wiebe https://www.educause.edu/ir/library/html/erm/31231.html https://www.educause.edu/ir/library/html/erm/31231.html http://d-scholarship.pitt.edu/28240/1/Corrall_LOEX_2016.pdf http://d-scholarship.pitt.edu/28240/1/Corrall_LOEX_2016.pdf https://search.credoreference.com/ https://search.credoreference.com/ https://doi.org/10.11645/6.2.1747 https://doi.org/10.11645/6.2.1747 https://doi.org/10.5860/crl.77.2.164 https://doi.org/10.5860/crl.77.2.164 218 College & Research Libraries March 2021 17. Rachel E. Scott, “Part 2. If We Frame It, They Will Respond: Student Responses to the Framework for Information Literacy for Higher Education,” Reference Librarian 58, no. 1 (2017): 19–32, https://doi.org/10.1080/02 763877.2016.1196471. 18. D. Bruce MacKay and Nicole Eva, Information Literacy and Liberal Education: From Google to Scholarly Sources, presented at Sixth International Biennial Threshold Concepts Conference: Thresholds on the Edge, Halifax, NS, Canada, June 15–17, 2016. https://doi.org/10.1080/02763877.2016.1196471 https://doi.org/10.1080/02763877.2016.1196471 topofpage