Research Article

 

Assessing the Impact of Reference Assistance and Library Instruction on Retention and Grades Using Student Tracking Technology

 

Dennis Krieb

Director, Institutional Research and Library Services

Reid Memorial Library

Lewis & Clark Community College

Godfrey, Illinois, United States of America

Email: dkrieb@lc.edu

 

Received: 1 Jan. 2018                                                                       Accepted: 12 Apr. 2018  

 

 

cc-ca_logo_xl 2018 Krieb. This is an Open Access article distributed under the terms of the Creative CommonsAttributionNoncommercialShare Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

 

 

DOI: 10.18438/eblip29402

 

 

Abstract

 

Objective – To assess the impact of community college academic librarians upon student retention and grades through reference desk visits and attendance in library instruction classes.  

 

Methods – Student ID data used for this research was collected from students that visited the reference desk to consult about a course-related question or attended a library instruction class for a specific course. After consenting to share their student ID number, the students’ IDs were scanned and uploaded to a Blackboard Analytics data warehouse. A Pyramid Analytics reporting tool was used to query and extract student-level retention and grade data based upon whether the student had visited the reference desk or attended a library instruction class. Chi-square and Fisher’s exact tests were used to discern any statistical difference in retention rates and grades between students that engaged a librarian through reference or instruction and the general student population.

 

Results – When comparing fall-to-fall retention for all degree-seeking students, students that visited the reference desk or attended a library instruction class had a statistically higher rate of retention. When comparing fall-to-fall retention within low-retention student cohorts, students that visited the reference desk or attended a library instruction class had higher rates of retention. Rates of retention in 8 of 10 cohorts were statistically higher for library instruction and in 6 of 10 cohorts were statistically higher for reference visits. With respect to course grades, only one of five high enrollment courses showed a higher grade average for students that attended a library instruction class. None of the differences in average grades between students that attended a library instruction class and all students in the five courses were statistically significant. For the impact of a reference visit upon a course grade, all five courses showed a higher average grade average for students that visited the reference desk for a question related to their course than for all students in the course. Four of the five differences were statistically significant.

 

Conclusions – The data collected by systematically tracking students that interact with community college librarians suggests that reference desk visits and attendance of library instruction classes both have a positive statistically significant impact upon student retention. When looking at course grades, the data does not indicate a statistically significant positive or negative impact for library instruction. The impact of visiting the reference desk upon course grades does suggest a strong statistically significant positive correlation.

 

 


 


 

Introduction

 

Lewis & Clark Community College is a two-year higher education institution located in Godfrey, Illinois. Lewis & Clark has multiple campuses, a river research center, a humanities center, a training center, and Community Education Centers located throughout the more than 220,000-person college district that reaches into 7 counties in Southwestern Illinois. Unduplicated, degree-seeking enrollment for academic year 2016-2017 was 7,673 students.

 

The confluence of reductions in state-level funding and declining student enrollments has generated a sense of urgency upon student retention efforts at Lewis & Clark Community College. In the years from 2006 to 2011, fall-to-fall retention for full-time students dropped from 57% to 52%, and from 42% to 39% for part-time students. These data mirror the low retention rates of all two-year community colleges, where nearly 50% of students leave by the end of their first year of enrollment (Hongwei, 2015). Within this challenging environment, there began a new emphasis by state-level education agencies and higher education accreditors for evidence based initiatives supporting student success.

 

To address the demand for more evidence of success, a new approach to leverage data was decided upon by administrators in Academic Affairs, Enrollment Services, and Institutional Research. A campus culture would be cultivated that relied heavily upon quantitative student assessment of innovative practices using predefined measures of success. This approach would also explore student tracking of support services on campus as a means to better understand the impact of these services upon student success measures.

 

In 2012, the Student Success Team was established to address success initiatives related to grades and retention. Members of the Student Success Team included senior level academic administrators and members of the Institutional Research department. The Student Success Team would act as an academic think tank to investigate, pilot, and assess trends in higher education associated with evidence based practices to improve student success.

 

The Student Integration Model developed by Vincent Tinto suggests that supportive social and educational communities outside of the classroom have a positive impact upon student retention (Tinto, 2012). It was upon this theoretical framework that the Student Success Team began to investigate the impact of student support services at Lewis & Clark upon grades and retention.

 

The first student support service selected by the Student Success Team to investigate was academic tutoring. Branded as the Student Success Center, tutoring at Lewis & Clark is decentralized among various campus locations. Reid Library also hosts a Student Success Center location that provides assistance for students seeking tutoring in writing and study skills. In 2013, students that were tutored at any Student Success Center location were tracked to discern the impact of tutoring upon retention. The fall-to-fall retention rate for degree-seeking students enrolled in Fall 2013 that were tutored was found to be 65.6% (N=640), as compared to the overall retention rate for all degree-seeking students of 51.5% (N=5085).

 

The Student Success Team decided to expand the research of Tinto’s Student Integration Model to Reid Library in 2014. This decision was supported by research connecting the services and collections of academic libraries to Tinto’s Student Integration Model (Oakleaf, 2010). Correlational evidence linking student retention and academic success with academic libraries published by the University of Minnesota (Soria, Fransen, & Nackerud, 2013) was also instrumental in the Student Success Team’s decision to investigate the impact of Reid Library upon student grades and retention.                             

Another aspect of the Student Success Team work would be its emphasis on evidence based research using Lewis & Clark’s technology infrastructure. A data warehouse had recently been implemented, providing the ability to quickly identify calculated success measures such as grades and retention for specific student cohorts. A list of ten student cohorts with retention rates below the overall student retention rate would be used to assess the impact of Tinto’s Student Integration Model within Reid Library.

 

 

Table 1

Student Demographics - Lewis & Clark Community College, Fall 2017

Student Type

Percentage

White

80.2%

African-American

9.8%

Hispanic

2.3%

Female

60.5%

18-19 Years of Age

36.1%

20-24 Years of Age

33.2%

First Generation College

24.7%

Developmental Math or English Placement

53.0%

Accepted Pell Grant

39.2%

Veteran

3.7%


 

 


Literature Review

 

There exists a crisis in retention and completion that is unique to community colleges. Approximately 51% of all entering community college students will have dropped out within their first year (National Student Clearinghouse Research Center, 2016), and only 20% seeking to transfer to a four-year institution will eventually do so (Lloyd & Eckhardt, 2010). For minority and part-time students, retention and completion are often lower when compared to other community college students (Strayhorn, 2012).

 

In his classic work Leaving College, Vincent Tinto (2012) suggests that student attrition in postsecondary education is related to a student’s immersion within the greater campus community. “Institutions of higher education are not unlike other human communities, and the process of educational departure is not substantially different from the other processes of leaving which occur among human communities generally” (p. 204). Student support services help foster a campus community of belonging by creating social relationships, clarifying aspirations and enhancing commitment, developing college know-how, and making college life feasible (Karp, 2011).

 

Research applying Tinto’s theoretical framework of student integration to various student support services has yielded positive correlational relationships between these services and student retention. Derby (2006) discovered a significant positive relationship between student club participation and retention and degree completion. The research of Grillo and Leist (2013) with student academic tutoring also discovered a positive relationship between tutoring and student retention and completion.

 

Research seeking to assess the academic library as a factor for increasing student success metrics such as retention and completion is still a relatively new field of study. Early studies often looked at aggregated data sets to discern any correlational relationships between library input and output measures with institutional retention and completion rates. Mezick’s (2007) study found a positive relationship between total library expenditures and student retention for postsecondary institutions offering a baccalaureate degree. A positive relationship between library professional staff and student retention was also found in the research of Emmons and Wilkinson (2011) when analyzing data sets taken from the 2005-2006 Annual Survey of ARL Statistics.

 

With the recent introduction of predictive and learning analytics within higher education, institutions are now seeking more nuanced data to forecast student behaviour to proactively engage students to improve student success measures (Lourens & Bleazard, 2016). For academic libraries, this new emphasis upon predictive and learning analytics represents a need to rethink how data is collected and how librarians can connect academic library outcomes to institutional outcomes such as retention and graduation (Oakleaf, 2010).

 

One of the first major studies looking into student-level interactions with academic library services and collections was conducted at the University of Minnesota. This research involved collecting student-level data from students that interacted with or used a library service or collection and connecting these data to the students’ subsequent enrollment and grade point averages. Findings from this research suggested that first-year students that used the library had a higher grade point average and fall-to-spring retention rate than their peers that did not use the library (Soria, Fransen, & Nackerud, 2013). An additional study at the University of Minnesota discovered that first-year students that used electronic resources and books had higher odds of graduating over withdrawing (Soria, Fransen, & Nackerud, 2013).

 

The research shared in this paper applies the student-level approach to tracking student engagement with the library, much like the work published by the University of Minnesota. It is hoped that the findings in this research will add to the literature regarding predictive analytics within academic libraries, the technology infrastructure needed to systematically track students that use the library, and the impact of library services – specifically, reference desk encounters and library instruction classes – upon retention and grades.

 

Methods

 

As previously mentioned in the literature review section, the research of Soria, Fransen, and Nackerud (2013) at the University of Minnesota was one of the first published articles to apply student-level tracking data from an academic library to investigate the impact of librarians, services, and collections upon student success measures. This seminal research served as the model for establishing the methods and measures for this paper. By applying the methodology used in the University of Minnesota study, a comparison of findings can be made between a two-year community college and major research university.

 

Independent and Dependent Variables

 

In 2014, Reid Library began systematically tracking student use in the library. Two library service areas would serve as independent variables: 1) attendance of a library instruction class and 2) visiting the reference desk for assistance. A threshold was established that only reference questions associated with an enrolled course would be tracked; directional and other non-course related questions would not be measured for their impact upon retention and grades. There were two dependent variables used in this research: 1) fall-to-fall retention and 2) student grades for five courses having the highest association with reference desk questions and library instruction. All students in this research had a degree-seeking status. 

 

Data Collection

 

A fundamental component of Lewis & Clark’s evidence based research is its technology infrastructure. Central to this architecture is the Blackboard Analytics data warehouse. The data warehouse serves as a data repository, housing data tables related to student characteristics, enrollments, grades, and completions from the Ellucian Student Information System (SIS). The Pyramid Analytics reporting tool provides the ability to query the data warehouse for calculated student success measures based upon treatments or services the student may have received.

 

The technology infrastructure showing how student tracking data is merged with student data retrieved from the SIS is depicted in Figure 1. After a library instruction class or reference visit was completed, the librarian asked for permission from the student to track his or her attendance or visit. This is commonly known as a verbal informed consent. The librarian explained to the student that no content-level information would be recorded, only that they have either attended a library instruction class or visited the reference desk. The student was informed that the data would only be used for research purposes – including the possibility of sharing publicly – to better understand student success metrics, and that no personally identifiable information would ever be shared. Since the inception of this pilot in 2014, no student has ever declined to be tracked.

 

 

 

Figure 1

Technology infrastructure for tracking reference desk visits and library instruction attendance.

 


 

 

If the student agreed to share his or her student ID and course information, the librarian used a barcode scanner or manually entered the student ID in the tracking software. After entering the student ID, a list of the student’s currently enrolled courses was provided by the tracking software.

 

The librarian then selected the appropriate course with which the library instruction class or reference visit was associated. The tracking software platform used for this initial phase is called SARS TRAK. 

 

After scanning the student ID and selecting the associated course, these data were sent to a Blackboard Analytics data warehouse. Student data regarding grades, enrollment, demographics, and other student-level data from the SIS were merged with the tracking data imported from SARS TRAK within the data warehouse. A Pyramid Analytics reporting tool was then used to query the data warehouse for calculated student success measures based upon whether the student had visited the reference desk or attended a library instruction course.

 

Results

 

Tables 2 and 3 compare the fall-to-fall retention rate for all degree-seeking students for academic years 2014/15-2016/17 with the fall-to-fall retention rates of degree-seeking students that attended a library instruction class or visited the reference desk for the same time period. Students that attended a library instruction class had a fall-to-fall retention rate of 60.9% (N=1,304), which was higher than the overall retention rate of 48.8% (N=7,319) for all degree-seeking students. Students that visited the reference desk had a retention rate of 66.2% (N=215).

 

To discern the impact of library instruction and reference assistance for students having characteristics associated with lower retention rates, ten student cohorts were identified as having lower retention rates than the retention rate of 48.8% (N=7,319) for all degree-seeking students.

 

Table 4 shows the overall retention rate for each student cohort and the retention rate for the students within each cohort that attended a library instruction class or visited the reference desk. All 10 low retention student cohorts had a higher rate of retention when attending a library instruction class or visiting the reference desk, with 8 cohorts having a statistically significant difference for library instruction and 6 cohorts statistically higher for reference.

 

Tables 5 and 6 compare the course grade success rates with library instruction and reference desk visits. Success rates are defined at Lewis & Clark as a grade of A, B, or C, and failure is a grade of D, F, or W. 

 

The impact of library instruction on grades was minimal, with only one of the five courses having a higher success rate than the overall course success rate. Courses selected in Table 4 had the highest association of requiring attendance of a library instruction class as part of the course. There was no statistically significant difference in any of the success rates for the five courses.

 

Courses selected in Table 6 had the highest association with a reference question relevant to the course. Unlike library instruction, students in all five courses that visited the reference desk had a higher success rate than the overall course
success rate. Four of the five courses had a statistically significant higher success rate for those students that visited the reference desk for assistance with their coursework.

 

Discussion

 

Testing Tinto’s Student Integration Model in the context of librarian interactions with students has provided Lewis & Clark Community College with correlational evidence that relationships developed with college personnel outside of the classroom are impactful for student success. With respect to the two independent library variables tested in this research, both library instruction and reference assistance were shown to have a positive statistically significant correlational relationship with student retention. The correlational relationship between library instruction and grades was not established in this research; however, the data did reveal a positive statistically significant correlation between reference assistance and grades.

 


 

 

Table 2

Fall-to-Fall Retention Rate for All Degree-Seeking Students, Academic Years 2014/15-2016/17

 

N

Fall-to-Fall

Retention Rate

All Degree-Seeking Students

 

7,319

 

48.8%

N represents a distinct student count.

 

 

Table 3

Fall-to-Fall Retention Rates for Students Attending a Library Instruction Class or Visiting the Reference Desk, Academic Years 2014/15-2016/17

N

Fall-to-Fall

Retention Rate

Attended a Library Instruction Class

 

1,304

 

60.9%**

Visited the

Reference Desk

 

215

 

66.2%**

N represents a distinct student count.

*   P<.05

** P<.01

 

 

Table 4

Fall-to-Fall Retention Rates for Student Cohorts with Low Retention Rates that Attended Library Instruction or Visited the Reference Desk, Academic Years 2014/15-2016/17

 

 

Low Retention Cohort

 

 

N

Retention Rate-        All Degree-Seeking Students

 

 

N

Retention

Rate-

Attended Library Instruction

 

 

N

Retention Rate-

Visited the Reference Desk

Cumulative GPA below 2.0

1,508

29.6%

120

34.1%

29

48.3%*

African- American

787

38.0%

150

46.4%*

37

59.5%**

Cumulative

GPA 2.0 - 2.29

3,509

40.2%

129

51.9%**

16

68.8%

Male

3,292

44.7%

543

57.7%**

70

65.7%**

Part-Time

4,963

45.1%

589

52.9%**

102

63.1%**

Age 20-24

2,569

45.4%

323

54.9%**

46

56.5%

First Generation

2,164

45.4%

217

54.4%**

49

62.0%

Developmental English Placement

327

45.6%

60

55.0%

16

58.8%

Pell Accepted

3,287

46.6%

660

54.1%**

123

59.7%**

Developmental Math Placement

2,337

47.0%

426

58.1%**

79

67.1%**

N represents a distinct student count.

*   P<.05

** P<.01

 

 

Table 5

Course Success Rates for the Highest Courses Associated with a Library Instruction Class, Academic Years 2014/15-2016/17

 

 

N

Success Rate -

All Degree-Seeking Students

 

 

N

Success Rate -

Attended Library Instruction

Second Semester College English

 

1,568

70.8%

440

74.2%

First Semester College English

2,054

71.4%

396

69.6%

Public and Private Communication

1,692

80.3%

282

80.1%

Public Speaking

732

81.8%

127

78.9%

College Reading

(Developmental)

399

80.3%

93

80.2%

N represents an enrolled course count. 

*   P<.05

** P<.01

 

 

Table 6

Course Success Rates for the Highest Courses Associated with a Reference Visit,

Academic Years 2014/15-2016/17

 

 

N

Success Rate -

All Degree-Seeking Students

 

 

N

Success Rate -

Visited the Reference Desk

English 131

2,054

71.4%

102

85.3%**

English 132

1,568

70.8%

99

85.9%**

Reading 125

399

80.3%

32

96.9%*

Psychology 131

1,626

63.8%

31

77.4%

Art 130

597

72.7%

30

90.0%*

N represents an enrolled course count.

*   P<.05

** P<.01

 

 

 


In comparison to similar studies that tracked student use of academic libraries to retention and grades, Soria, Fransen, and Nackerud’s (2013) research at the University of Minnesota serves as the best example of a study using similar methods when comparing library instruction and reference visits, though it should be noted that the University of Minnesota is a selective admissions institution, unlike Lewis & Clark Community College, which is an open admissions institution.

 

When comparing the impact of library instruction upon grades, findings from the University of Minnesota were similar to those discovered at Lewis & Clark, with both showing a modest positive impact, though neither study found the impact to be statistically significant. With respect to reference visits, the University of Minnesota showed a positive correlation with grades, but without statistical significance. Findings from Lewis & Clark showed statistically significant higher grades in four of the five courses measured for students that visited the reference desk.

 

Retention comparisons for both studies found positive statistically significant correlations between library instruction and student retention. Reference visits in the University of Minnesota study showed a slightly negative relationship with retention, though the data were not statistically significant. Reference visits at Lewis & Clark were found to be positively correlated with retention for all 10 cohorts studied, with 6 cohorts having a statistically higher retention rate.

 

It is interesting to note that a reference visit had a more significant impact upon students at Lewis & Clark than for the students in the University of Minnesota study. Because the setting of a reference visit is a one-on-one encounter, an opportunity exists for the student to establish a relationship with the librarian. For community college students at Lewis & Clark, 53% of which are not prepared for college-level math or English courses, the need to develop relationships with college personnel outside of the classroom may be more impactful than for those students at a selective admissions university like the University of Minnesota.

 

Another observation taken from the findings is the higher impact of a reference visit in comparison to attending a library instruction class for both grades and retention at Lewis & Clark. Though a relationship may be developed between a student and a librarian that teaches a library instruction class of 20 or 30 students, the likelihood of this occurrence is smaller than the opportunity for a student to develop a relationship with a librarian through a reference visit. Findings from this research would suggest that student proximity to a librarian is correlated with grades and retention.

 

In spite of the positive findings discovered in this study, there are limitations. Students that visited the reference desk in this study represent a self-selected sample. These students may be more academically motivated to achieve higher grades and graduate than their classroom peers that did not visit the reference desk. Future research into the impact of reference visits upon grades and retention should consider propensity score matching of students to reduce the potential for bias associated with student motivation. 

 

Another limitation of this study is the presumption that all reference desk visits are equally weighted. The length of time spent during a reference desk visit may also have a correlational relationship with grades and retention. Future research should consider grouping reference desk visits by the length of the interview.

 

Conclusion

 

Building on the findings of this research, Lewis & Clark Community College has expanded its tracking of students that interact with support services and co-curricular activities to over 20 points of service, including tutoring, advising, and participation in student life clubs and activities. The library has also expanded its data collection by tracking students that check out items from the collection. As with the current tracking system being used for library instruction and reference assistance, student IDs will be used to identify those students that circulate an item from the library. Data assessing the correlational relationship between student use of the library’s non-digital collection with grades and retention will be available in the fall semester of 2018. Moving forward, a campaign to proactively share the findings of this research with faculty, students, and administrators at Lewis & Clark is currently being planned in hopes of increasing overall student usage of the library.

 

Incorporating the library as a data resource for institutional research has been a goal for the author of this paper. As a result of the research presented in this article, the library has become a partner with peer divisions and departments on campus with retention initiatives. A recent example is Lewis & Clark’s requirement for accreditation to complete a four-year Quality Initiative to improve retention for the Higher Learning Commission. Library findings associated with the research in this paper will serve as a data source in the Quality Initiative that seeks to explore Tinto’s Student Integration Theory through student tracking of support services. Quality Initiative findings for Lewis & Clark will be presented in 2020 at the Higher Learning Commission’s Persistence and Completion Academy Results Forum.

 

It should also be noted that the long-standing legacy of library patron privacy has not been compromised in this research. No personally identifiable information has been disclosed for any student tracked. All data are secured within the institution’s Ellucian student information system and are only accessible by the Institutional Research office at Lewis & Clark Community College.

 

References

 

Derby, D. C. (2006). Student involvement in clubs and organizations: An exploratory study at a community college. Journal of Applied Research in the Community College, 14(1), 39-45.

 

Emmons, M., & Wilkinson, F. C. (2011). The academic library impact on student persistence. College & Research Libraries, 72(2), 128-149. https://doi.org/10.5860/crl-74r1

 

Grillo, M. C., & Leist, C. W. (2013). Academic support as a predictor of retention to graduation: New insights on the role of tutoring, learning assistance, and supplemental instruction. Journal of College Student Retention: Research, Theory & Practice, 15(3), 387-408. http://dx.doi.org/10.2190/CS.15.3.e

 

Hongwei, Y. (2015). Student retention at two-year community colleges: A structural equation modeling approach. International Journal of Continuing Education & Lifelong Learning, 8(1), 85-101.

 

Karp, M. M. (2011). How non-academic supports work: Four mechanisms for improving student outcomes. CCRC Brief. Number 54. Retrieved from https://ccrc.tc.columbia.edu/media/k2/attachments/how-non-academic-supports-work-brief.pdf

Lloyd, P. M., & Eckhardt, R. A. (2010). Strategies for improving retention of community college students in the sciences. Science Educator, 19(1), 33-41. Retrieved from https://files.eric.ed.gov/fulltext/EJ874152.pdf

Lourens, A., & Bleazard, D. (2016). Applying predictive analytics in identifying students at risk: A case study. South African Journal of Higher Education, 30(2), 129-142. http://dx.doi.org/10.20853/30-2-583

 

Mezick, E. M. (2007). Return on investment: Libraries and student retention. Journal of Academic Librarianship, 33(5), 561-566. https://doi.org/10.1016/j.acalib.2007.05.002

National Student Clearinghouse Research Center. (2016). The role of community colleges in postsecondary success: Community colleges outcomes report. Retrieved from https://studentclearinghouse.info/onestop/wp-content/uploads/Comm-Colleges-Outcomes-Report.pdf

Oakleaf, M. (2010). The value of academic libraries: A comprehensive research review and report. Chicago: Association of College and Research Libraries. Retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf

Soria, K. M., Fransen, J., & Nackerud, S. (2013). Library use and undergraduate student outcomes: New evidence for students’ retention and academic success. portal: Libraries    and the Academy, 13(2), 147-164. https://doi.org/10.1353/pla.2013.0010

 

Soria, K. M., Fransen, J., & Nackerud, S. (2017). The impact of academic library resources on undergraduates' degree completion. College & Research Libraries, 78(6), 812-823. https://doi.org/10.5860/crl.78.6.812

 

Strayhorn, T. L. (2012). Satisfaction and retention among African American men at two-year community colleges. Community College Journal of Research and Practice, 36(5), 358-375.  https://doi.org/10.1080/10668920902782508

 

Tinto, V. (2012). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago, IL: University of Chicago Press.