Abstract

Objective - To use a multi-dimensional approach to evaluate the effectiveness of a nursing information literacy program (Pathways to Information Literacy) delivered to undergraduate nursing students. Assessment sought to track progress in both affective and cognitive spheres.

Design – This program evaluation focuses on the Pathways to Information Literacy (PIL) curriculum, which was delivered from 1988-1992. It consisted of 6 hours of librarian-delivered instruction, divided over 4 sessions. To evaluate the impact of this curriculum, the authors gathered five different data sets: informal feedback; the results of a survey measuring the affective domain of confidence; the results of a longitudinal cohort survey of graduates; and two different sets of data gathered from distinct samples but utilizing the same information literacy assessment tool. All five data sets served the greater purpose of assessing students’ mastery of information literacy.

Setting - An undergraduate Bachelor of Science in Nursing (BSN) program within a state university, the University of Northern Colorado.

Subjects - In general, the subjects were different cohorts of nursing students between 1988 and 1992. Class size hovered at just over 100. For most measures, response rate was high and the dropout rate was low. It is worth noting that one of the assessments was a longitudinal cohort survey of graduates. As the mobility of graduates often decreases the number of responses, the omission of the response rate for this measure is of concern.

Methods - The methods are one of the most striking aspects of this study. The authors employed no less than five methods of assessment:

From 1988-1992, investigators gathered informal feedback from both students and faculty members about the written assignments of the PIL program. The specific method for gathering feedback was not reported.
From 1988-1990 PIL students in their junior year took pre- and post- PIL program confidence surveys. The survey tool, which was included, contained 6 Likert-like questions which assessed their affective domain of confidence as related to their ability to perform information literacy related tasks, such as using a bibliographic index.
During the 1990/91 academic year the authors administered a general, university-wide information literacy assessment tool, which was included. It measured both self-perceived progress and objectively measured skill attainment in the cognitive domain. The survey tool lacked validation, but had been previously published in the library literature (Greer, 1991). Students from the PIL program were extracted from the data for comparison to the general student population.
During 1991/92 PIL students completed the above mentioned information literacy assessment tool as a pre-and post- test.
Both PIL (1990/91) and non-PIL (1988/89) cohorts completed a longitudinal post-graduation survey which was included. Comparison of the two cohorts aimed to examine the effect of the PIL program on subsequent scholarly professional activities.

While the methodologies were all tied to the overall purpose of program evaluation, they were not tied to specific pedagogies or content units.

Main results - Fox et al. generally utilized simple, descriptive statistical data. The data derived from the information literacy assessment tool was the exception, producing ordinal data which was analyzed using a chi-squared approach.

All outcomes supported the positive effect of the PIL program. The soft technique of gathering informal feedback from students and faculty resulted in positive feedback. Faculty reported that their students became independent in information-seeking and the quality of their papers increased. Students also reported that the assignment and instruction gave them confidence and that the written assignments were a nice break from the traditional examinations. The affective confidence survey noted a substantial improvement: pre-program only 26% reported confidence when performing information-seeking strategies compared to 76% post-instruction. When the information literacy assessment tool was administered both to 68 PIL participants and to 208 general students, the PIL students both believed themselves to be more successful and demonstrated greater knowledge. 70% of PIL students answered CD-ROM index questions correctly, compared to 49% of general students. When the same tool was given to only PIL students as a pre-and post- test, it showed statically significant increases in the use of the library and mastery of several specific search techniques: p<0.05 for 7 of 20 measures. The post-graduation survey showed that 45% of the PIL students had engaged in some scholarly activity, as compared to 10% of non-PIL graduates.

Conclusion - The authors concluded that the multidimensional assessment efforts delivered a comprehensive view of the effectiveness of the program, demonstrating student benefit in cognitive (knowledge attainment) and affective (confidence levels) domains as well as in subsequent professional behavior.

Commentary

A multidimensional evaluation of a nursing information-literacy program (MENILP) (Fox, Richter & White, 1996) contains a description of the Pathways to Information Literacy (PIL) program. This program consisted of four instruction sessions, which were integrated across two required nursing courses in the junior year of study. Though the instructional pedagogies of the sessions were not detailed, the authors did provide the goals for the instruction and mentioned several assessment methods. The goals for the instruction included:

develop an understanding of library organization and services
acquire skills in forming research questions and locating and evaluating accurate, relevant information for problem solving
apply appropriate information-seeking strategies; and
emulate the scholarly activities of professional nurses (Fox et al., 1996, p.183)

The first information literacy instruction session targeted the skills of information location and synthesis. Mastery of the course was measured by an exam and an assignment which required the students to synthesize general sources to create a topical summary of a condition and its treatment. Session two introduced more advanced and nursing-specific research techniques such as locating and using specialized handbooks and citation indexes. Librarian-led group discussion surrounding conflicting research findings was utilized as a synthesizing methodology. Session three presented the process of topic selection and research question refinement. It offered the students in-class time to work on their topics using a topic selection grid tool. Session four focused on computer search strategies, including CINAHL on CD-ROM, which was a relatively new addition to the library’s collection. Students were taught effective search strategies for CINAHL and asked to demonstrate their knowledge by submitting a print out of their strategy. A detailed description of the program’s rationale and creation can be found in an earlier work by the same authors (Fox, Richter, & White, 1989). Though well done, that project description has not had as great an impact as MENILP which focuses on program evaluation.

The program evaluation plan for the PIL program pre-dated many of the current, widely-used frameworks for program evaluation, such as the Framework for Program Evaluation in Public Health (Centers for Disease Control and Prevention, 1999). The authors did however utilize an appropriate evaluation framework, selecting Staropoli and Waltz’s model (1978) which is specific to health education programs. The model takes a familiar five question format, asking first, who will be involved in the evaluation? The faculty and the librarian. What are the purposes in conducting an evaluation? To evaluate the effectiveness of the program and measure the degree of skill acquisition. What is to be evaluated: curriculum, objectives, faculty or students? Students’ mastery of information literacy in the cognitive and affective domains. How is the evaluation to proceed? The authors summarize the past practices and detail plans for on-going evaluation “every two-three years” (Fox et al., 1996, p. 184). While this older model pre-dates the now commonly seen emphasis on evidence, the authors were ahead of their time in broadly defining their purpose as creating “[o]bjective procedures [which] should be used to facilitate the collection of dependable, unbiased data… to determine value” (Fox et al., 1996, p. 184).

MENILP significantly impacted subsequent scholarship. A three-pronged approach using the cited reference search functions of ISI Web of Knowledge, CINAHL, and Google Scholar identified 39 subsequent citations spanning 17 years. Although most of the publications were within the field of Library and Information Science, approximately 25% were from Nursing/Allied Health. Google Scholar, as expected, identified the greatest diversity of formats (books, thesis, articles and reports) as well as the most non-English language publications, six in total (Kleibel & Mayer, 2005; Meneses Placeres, 2008a, 2008b; Qun, 2011; Nodarse Rodríguez, 2005; Sundin, 2003).

Figure 1

Number of MENILP citations by year

The five-prong evaluation strategy was one of the main drivers of the article’s popularity. Each of the evaluation methods designs can be appraised in terms of evidence quality. The informal feedback, while illuminating, did not have the power to influence practice. The other four methodologies involved either survey or tests; none of which utilized validated tools. The full text of the tools was included so the reader is able to make some judgments about the content and face validity of the measures. When the information literacy assessment tool was used as a pre- and post-test, the results were determined statistically to be significant, with reported p values. When it was used to compare PIL students to the larger under-graduate population, there are concerns. The PIL students differed significantly from the general student population. No controls were put in place for the nursing students’ higher average GPAs, greater average age and their skewed gender, though the authors noted these differences. The confidence survey’s descriptive statistics were impressive but are not analyzed for statistical significance. The post-graduation survey suffers from the same limitations. It does however have the most interesting design of all the methodologies: a cohort, longitudinal survey. When considering an evidence base, this type of study design is considered strong and is especially well-suited to look at questions of etiology (MacKibbon, Wilczynski, & Eady, 2009). In this design, the PIL program was the exposure which was controlled for in two different cohorts of graduates. The PIL graduates, when compared to their non-PIL peers using simple descriptive statistics, reported more professional reading and greater participation in scholarly activity. Fox and colleagues do not discuss the internal/external validity, reliability or limitations of their methodologies at any point.

Though the individual results dim under an analysis of their strength of evidence, they do synergistically work together to form a larger preponderance of evidence. When considered as a whole, the methodologies create a body of data that delivers its own internal triangulation of results and meet the authors’ objective of a collection of data which determines value. The thoroughness and ultimate success of this work as a program evaluation goes a long way in explaining why this article has had such an impact. It was of particular relevance to the subsequent researchers who were picking up on the growing evidence-based trend that was occurring both in program evaluation and in nursing practice.

These subsequent researchers have used and continue to use this work in divergent ways. The multiple methodologies and conclusions are often considered in piecemeal fashion. This study provides support or evidence for:

the definition of information literacy (IL)
the positive effect of IL instruction in general as well as within the affective and cognitive domains
evaluation of IL instruction (including post-graduation impact)
nursing faculty/librarian collaborations
evidence-based practice

Each one of these areas will be considered separately.

Definition of Information Literacy

Because MENILP was a vanguard of information literacy scholarship within nursing, its first section was aptly titled Why Information Literacy? (Fox et al., 1996). This section defined the term and justified the need for IL, specifically within a nursing context. This provided fertile ground for subsequent authors to mine the section when defining information literacy for their readers (Cobus, 2008; Frier, 2009; Hopkins, Callister, Mandleco, Lassetter, & Astill, 2011; Jacobs, Rosenfeld, & Haber, 2003; Nayda & Rankin, 2008).

Positive Effect of IL Instruction

MENILP reported positive findings, supporting the effectiveness of information literacy instruction. These positive findings provided justification for scores of curriculum-integrated nursing IL programs and worked their way into many subsequent studies (Barnard, Nash, & O'Brien, 2005; Brettle, 2003; Brettle & Raynor, 2013; Carter-Templeton, 2011; Craig & Corrall, 2007; Eimas & Barton, 2001; Grant & Brettle, 2006; Jette, Tribble, Gagnon, & Mathieu, 2010; Meneses Placeres, 2008a, 2008b; Morgan, Fogel, Hicks, Wright, & Tyler, 2007; Nodarse Rodríguez, 2005; Rush, 2008; Saranto & Hovenga, 2004; Shorten, Wallace & Crookes, 2001; Tronstad, Phillips, Garcia, & Harlow, 2009; Weinert & Palmer, 2007; Wright & McGurk, 2000). Authors occasionally described the work with words like “seminal” (Craig & Corrall, 2007).

Evaluation of IL Instruction

MENILP’s true focus was the evaluation of the PIL program. In this they were also vanguards; ahead of current library trends like measuring value and assessing outcomes. Many subsequent studies have focused on the evaluation methods specifically (Argüelles, 2012; Baro & Ebhomeya, 2013; Carlock & Anderson, 2007; Carter-Templeton, 2011; Davies, Urquhart, Smith, & Hepworth, 1997; Martin, 2008; Meldrum & Tootell, 2004; Powell & Case-Smith, 2003; Saranto & Hovenga, 2004; Wallace, Shorten, Crookes, McGurk, & Brewer, 1999). Once again, the variety of MENILP assessment methods has spawned a diversity of applications. For some authors the multidimensional approach has been the feature of note (Davies et al., 1997; Wallace et al., 1999), for others it was the librarian/faculty team approach (Martin, 2008). One of the most striking evaluation methods of MENILP was the longitudinal, cohort survey of graduates. Four subsequent studies focus on the selection of this study design (Baro & Ebhomeya, 2013; Brettle, 2003; Eldredge, 2002; Powell & Case-Smith, 2003).

Nursing Faculty/librarian Collaborations

While not as note-worthy within the library literature, several works from the nursing perspective addressed the strength of the collaboration between librarians and nursing faculty described in MENILP (Carter-Templeton, 2011; Honey, 2007; Schloman, 2001). The library literature was much more likely to focus on the aspect of curriculum integration.

Evidence-Based Practice

Obviously the world has changed since 1996. Of tremendous import to the practice of health science librarianship has been the advent of evidence-based practice (EBP). Though EBP was not an explicit concern of Fox et al., their work does eloquently speak of a nurse’s need to be able to acquire and appraise information, which are critical steps in the EBP process. By clearly addressing these concerns their study drew the interest of later EBP investigators (Durando & Oakley, 2005; Jacobs et al., 2003; Rush, 2008; Shorten, Wallace & Crookes, 2001; Urquhart, 1998).

A multidimensional evaluation of a nursing information-literacy program could have been sprawling mess of an article. It has the clear aim of a program evaluation but presents five different evaluation strategies, spanning a five year period. The evaluation strategies themselves vary from simple informal feedback to a complex cohort design. It tries to measure both affective and cognitive changes. With such a broad focus in terms of time, methodology, and approach it certainly had the potential to fail. Instead it is a powerful, enduring article with a global impact. Its forward-thinking focus on evaluation and evidence continues to serve the needs of scholars both inside and outside of librarianship. It is well-deserving of the descriptor “classic”.

References

Argüelles, C. (2012). Program-integrated information literacy (PIIL) in a hospital's nursing department: A practical model. Journal of Hospital Librarianship, 12(2), 97-111. doi: 10.1080/15323269.2012.665717

Barnard, A., Nash, R., & O'Brien, M. (2005). Information literacy: Developing lifelong skills through nursing education. Journal of Nursing Education, 44(11), 505-510.

Baro, E. E., & Ebhomeya, L. (2013). Information needs and seeking behaviours of nurses: A survey of two hospitals in Bayelsa State, Nigeria. Health Education, 113(3), 183-195.

Brettle, A. (2003). Information skills training: A systematic review of the literature. Health Information & Libraries Journal, 20(Suppl. 1), 3-9. doi: 10.1046/j.1365-2532.20.s1.3.x

Brettle, A., & Raynor, M. (2013). Developing information literacy skills in pre-registration nurses: An experimental study of teaching methods. Nurse Education Today, 33(2), 103-109. doi: 10.1016/j.nedt.2011.12.003

Carlock, D., & Anderson, J. (2007). Teaching and assessing the database searching skills of student nurses. Nurse Educator, 32(6), 251-255. doi: 10.1097/01.NNE.0000299477.57185.ba

Carter-Templeton, H. D. (2011). Nurses’ information appraisal within the clinical setting. (Doctoral dissertation The University of Tennessee). Retrieved 4 Dec. 2013 from http://etd.uthsc.edu/WORLD-ACCESS/Carter-Templeton/2011-040-Carter-Templeton.pdf

Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. MMWR: Recommendations and Reports, 48(RR. 11), 1-40.

Cobus, L. (2008). Integrating information literacy into the education of public health professionals: Roles for librarians and the library. Journal of the Medical Library Association, 96(1), 28-33. doi: 10.3163/1536-5050.96.1.28

Craig, A., & Corrall, S. (2007). Making a difference? Measuring the impact of an information literacy programme for pre-registration nursing students in the UK. Health Information and Libraries Journal, 24(2), 118-127. doi: 10.1111/j.1471-1842.2007.00688.x

Davies, R., Urquhart, C. J., Smith, J., & Hepworth, J. (1997). Establishing the value of information to nursing continuing education: Report of the EVINCE project. (44). West Yorkshire: British Library Research and Innovation Centre.

Durando, P., & Oakley, P. (2005). Developing information literacy skills in nursing and rehabilitation therapy students. Journal of the Canadian Health Libraries Association, 26(1), 7-11. doi: 10.5596/c05-007

Eimas, R., & Barton, H. (2001). Web-based instruction for undergraduate nurses. In B. I. Dewey (Ed.), Library user education: Powerful learning, powerful partnerships (pp. 283-287). Lanham, MD: Scarecrow Press.

Eldredge, J. (2002). Cohort studies in health sciences librarianship. Journal of the Medical Library Association, 90(4), 380-392. Retrieved 4 Dec. 2013 from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC128954/

Fox, L. M., Richter, J. M., & White, N. (1989). Pathways to information literacy. Journal of Nursing Education, 28(9), 422-425.

Fox, L. M., Richter, J. M., & White, N. E. (1996). A multidimensional evaluation of a nursing information-literacy program. Bulletin of the Medical Library Association, 84(2), 182-190. Retrieved 4 Dec. 2013 from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC299399/

Frier, M. D. (2009). Comments on information literacy interventions in nursing and engineering courses at the Cape Peninsula University of Technology, South Africa. Retrieved 4 Dec. 2013 from http://athlonenursingcollege.wikispaces.com/

Grant, M. J., & Brettle, A. J. (2006). Developing and evaluating an interactive information skills tutorial. Health Information and Libraries Journal, 23(2), 79-88. doi: 10.1111/j.1471-1842.2006.00655.x

Greer, A. (1991). Assessment of learning outcomes: A measure of progress in library literacy. College and Research Libraries, 52(6), 549-557. Retrieved 4 Dec. 2013 from http://hdl.handle.net/2142/41432

Honey, M. L. (2007). Teaching and learning with technology as enabler: a case study on flexible learning for postgraduate nurses. (Doctoral dissertation). The University of Auckland, Auckland, New Zealand. Retrieved 4 Dec. 2013 from https://researchspace.auckland.ac.nz/bitstream/handle/2292/2383/02whole.pdf?sequence=4

Hopkins, B., Callister, L. C., Mandleco, B., Lassetter, J., & Astill, M. (2011). Librarians as partners of the faculty in teaching of scholarly inquiry in nursing to undergraduates at Brigham Young University. Science & Technology Libraries, 30(3), 267-276. doi: 10.1080/0194262X.2011.593416

Jacobs, S. K., Rosenfeld, P., & Haber, J. (2003). Information literacy as the foundation for evidence-based practice in graduate nursing education: A curriculum-integrated approach. Journal of Professional Nursing, 19(5), 320-328. doi: 10.1016/s8755-7223(03)00097-8

Jette, S., Tribble, D.S., Gagnon, J., & Mathieu, L. (2010). Nursing students' perceptions of their resources toward the development of competencies in nursing informatics. Nurse Education Today, 30(8), 742-746. doi: 10.1016/j.nedt.2010.01.016

Kleibel, V., & Mayer, H. (2005). Literaturrecherche für Gesundheitsberufe. Wien: Facultas.

MacKibbon, A., Wilczynski, N., & Eady, A. (2009). PDQ: Evidence-based principles and practice. Shelton, CT: People's Medical Publishing House.

Martin, S. (2008). Reflections on a user education session with nursing students. Health Libraries Review, 15(2), 111-116. doi: 10.1046/j.1365-2532.1998.1520111.x

Meldrum, A. M., & Tootell, H. (2004). Integrating information literacy into curriculum assessment practice: An informatics case study. Journal of University Teaching & Learning Practice, 1(2), 49-58. Retrieved 4 Dec. 2013 from http://ro.uow.edu.au/jutlp/vol1/iss2/2

Meneses Placeres, G. (2008a). Aproximaciones teóricas a la evaluación de la alfabetización informacional en la educación superior. Acimed, 18(1). Retrieved 4 Dec. 2103 from http://hdl.handle.net/10760/12314

Meneses Placeres, G. (2008b). La evaluación en la alfabetización informacional en el contexto de la educación superior: Aproximación teórica. Biblios, (31), 1-11. Retrieved 4 Dec. 2013 from http://eprints.rclis.org/12314/

Morgan, P. D., Fogel, J., Hicks, P., Wright, L., & Tyler, I. (2007). Strategic enhancement of nursing students information literacy skills: Interdisciplinary perspectives. ABNF Journal, 18(2), 40-45.

Nayda, R., & Rankin, E. (2008). Information literacy skill development and life long learning: Exploring nursing students' and academics' understandings. Australian Journal of Advanced Nursing, 26(2), 27-33. Retrieved 4 Dec. 2013 from http://pandora.nla.gov.au/pan/81890/20090327-0014/www.ajan.com.au/Vol26/26-2_Nayda.pdf

Nodarse Rodríguez, M. (2005). La enseñanza de las ciencias de la información en el currículum de los estudiantes de medicina y de otras especialidades afines. ACIMED, 13(6). Retrieved 4 Dec. 2103 from http://eprints.rclis.org/7963/

Powell, C. A., & Case-Smith, J. (2003). Information literacy skills of occupational therapy graduates: A survey of learning outcomes. Journal of the Medical Library Association, 91(4), 468-477. Retrieved 4 Dec. 2103 from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC209513/

Qun, W. (2011). [Information literacy education involved in research and innovation activities]. Medical Information (China), 21. Retrieved 4 Dec. 2013 from http://www.cqvip.com/qk/98226b/201121/39906379.html

Rush, K. L. (2008). Connecting practice to evidence using laptop computers in the classroom. CIN-Computers Informatics Nursing, 26(4), 190-196.

doi: 10.1097/01.NCN.0000304803.38746.ad

Saranto, K., & Hovenga, E. J. S. (2004). Information literacy - what it is about? Literature review of the concept and the context. International Journal of Medical Informatics, 73(6), 503-513. doi: 10.1016/j.ijmedinf.2004.03.002

Schloman, B. F. (2001). Information literacy: the benefits of partnership. Online Journal of Issues in Nursing, 6p. Retrieved 4 Dec. 2013 from www.nursingworld.org/MainMenuCategories/ANAMarketplace/ANAPeriodicals/OJIN/Columns/InformationResources/InformationLiteracy.aspx

Shorten, A., Wallace, C. & Crookes, P. (2001). Developing information literacy: A key to evidence‐based nursing. International Nursing Review, 48(2), 86-92.

doi: 10.1046/j.1466-7657.2001.00045.x

Staropoli, C., & Waltz, C. F. (1978). Developing and evaluating educational programs for health care providers. Philadelphia, PA: F.A. Davis.

Sundin, O. (2003). Informationssökning i övergången från utbildnings-till yrkespraktik En kunskapsöversikt. Borås, Sweden: Department of Library and Information Science, University of Borås.

Tronstad, B., Phillips, L., Garcia, J., & Harlow, M. A. (2009). Assessing the TIP online information literacy tutorial. Reference Services Review, 37(1), 54-64.

doi: 10.1108/00907320910934995

Urquhart, C. (1998). Personal knowledge: A clinical perspective from the VALUE and EVINCE projects in health library and information services. Journal of Documentation, 54(4), 420-442. doi: 10.1108/eum0000000007176

Wallace, M. C., Shorten, A., Crookes, P. A., McGurk, C., & Brewer, C. (1999). Integrating information literacies into an undergraduate nursing programme. Nurse Education Today, 19(2), 136-141. doi: 10.1054/nedt.1999.0621

Weinert, D. J., & Palmer, E. M. (2007). Influence of an information literacy course on students' information search behavior. Journal of Allied Health, 36(1), E1-E12.

Wright, L., & McGurk, C. (2000). Integrating information literacy: University of Wollongong experience. In Bruce, C. & Candy, P. (Eds), Information literacy around the world: advances in programs and research.(pp. 83-97). Wagga Wagga, New South Wales: Centre for Information Studies, Charles Sturt University.