115 A Faceted Taxonomy for Rating Student Bibliographies in an Online Information Literacy Game Chris Leeder, Karen Markey, and Elizabeth Yakel Chris Leeder is a Doctoral Student, Karen Markey is a Professor, and Elizabeth Yakel is a Professor in the School of Information at the University of Michigan; e-mail: cleeder@umich.edu, ylime@umich.edu, yakel@umich.edu. The researchers wish to thank the members of the BiblioBouts team: Victor Rosenberg, Beth St. Jean, Fritz Swanson, Brian Jennings, Greg Peters, Andrew Calvetti, and three university faculty members who participated in the study. Special thanks to the Institute for Library and Museum Studies for their support of this project through a National Leadership Grant. © Chris Leeder, Karen Markey, and Elizabeth Yakel This study measured the quality of student bibliographies through creation of a faceted taxonomy flexible and fine-grained enough to encompass the variety of online sources cited by today’s students. The taxonomy was developed via interviews with faculty, iterative refinement of categories and scoring, and testing on example student bibliographies. It was then applied to evaluate the final bibliographies created in BiblioBouts, an online social game created to teach undergraduates information literacy skills. The scores of players and nonplayers were compared and showed a positive impact from the game. Findings of the evaluations of these student bibliographies are discussed. s online searching of digital libraries becomes the primary avenue for student research and digital information sourc- es proliferate, Information Literacy (IL) competence has become fundamentally important for incoming college students. Important IL skills include: mastering the ability to effectively search for informa- tion, critically evaluating how evidence is presented in information sources and properly integrating them into research assignments. However, many undergrad- uate students do not receive this crucial education when they enter college; the majority of institutions that provide first- year experience programs do not require any information literacy content.1 Aca- demic librarians attempt to reach as many students as they can through a wide range of venues, such as for-credit courses, in- class (often “one shot”) training sessions, walk-up assistance at reference desks, and online chat. Yet these outreach methods still reach only a fraction of the student population. The BiblioBouts project uses social gaming to teach information literacy skills to undergraduates alongside their regular coursework. The game is an online ac- companiment to the traditional in-class research paper assignment and engages students in each stage of the research process through competitive gameplay. BiblioBouts introduces students to a step-by-step research process in which crl-223 116 College & Research Libraries March 2012 each minigame or “bout” embodies a discrete stage of research, walking stu- dents through the steps of searching for and collecting sources, evaluating sources for credibility and relevance, narrowing down the scope of the research topic, selecting the best sources for that topic, and creating a bibliography of quality sources that support the research topic. (For more information on BiblioBouts, see the project Web site at: http://bibliobouts. si.umich.edu/index.html.) To evaluate the game’s effectiveness in achieving its goals, a measurement of learning outcomes needed to be devel- oped. Since the game’s final result is a research paper bibliography, the research- ers chose to measure the effectiveness of the game by measuring the difference in quality of student bibliographies between players (treatment group) and nonplayers (control group). A standardized assess- ment tool was designed to quantitatively assess differences in the quality of the bibliographies produced by students in the same pilot class who did, and did not, play BiblioBouts. Since the majority of the sources cited by players were online, the rating tool needed to be multifaceted, fine-grained, and format-neutral. This study outlines the development of this tool for assessing quality of student bib- liographies to measure the game’s impact. Research Questions Three fundamental research questions motivated this study: 1. How does one operationalize “quality” of student bibliogra- phies? 2. Can a standardized assessment tool be developed that is flexible enough to encompass the variety of online sources cited by today’s students? 3. Does playing the BiblioBouts game improve the quality of stu- dent bibliographies? These questions were addressed through the use of both qualitative and quantitative research techniques during the process of pilot testing and refinement of the game and were measured through the development of a faceted taxonomy for classifying and rating online sources. This research topic is important be- cause developing a standardized bib- liographic assessment tool that is effec- tive in measuring the quality of online citations is potentially useful to a wide audience of librarians, information lit- eracy instructors, and faculty members. In addition, this measurement standard would contribute to the development of the BiblioBouts project and help promote the effectiveness of a new information literacy skills development tool. Thus, this research will be important for librar- ians, information literacy instructors, and any faculty who assign undergraduate student research papers. Literature Review The literature of information literacy as- sessment is large. A recent overview of 91 case studies of information literacy assessment found that citation analysis of bibliographies was the second most com- mon technique (17%), behind multiple choice questionnaires (34%) (Walsh, 2009). However, even within the bibliographic analysis examples, the purpose of the in- formation literacy assessment varied from the overall counting of resource types to a specific focus on Web-based sources. The standards and methodologies used in each of these case studies were developed and used by individual librarians with specific needs and interests. Walsh cites concern over “the lack of rigorously de- signed assessment of information literacy in higher education”2 as demonstrated by the lack of reliability and validity measures in many of these case studies. Citation analysis of student bibliogra- phies has been frequently used by librar- ians as a quantified assessment tool. Cita- tion analysis can measure what sources students are actually using and offers “a flexible, non-invasive, time-efficient as- sessment forum for the documentation of student library use.”3 However, while A Faceted Taxonomy for Rating Student Bibliographies 117 review of the literature reveals that librar- ians have frequently employed citation analysis as an assessment tool, it also shows that they have used varying sets of criteria with varying assessment goals in employing this technique. Criteria fre- quently used include currency, relevance, correctness of citation format, quality, and scholarliness. However, the definitions of terms are not standard and vary from study to study, as do the methods of mea- surement. Currency could be determined by a set number of years from publication date or with a more subjective assessment of current issues in the field. Relevance is notoriously difficult to define,4 and a variety of definitions and criteria are employed in the literature, each of which TABLE 1 Reviewed Bibliography Evaluation Criteria Author(s) Variety Currency Relevance Correctness Scholarliness Kirk 1971 X X X Dykeman & King 1983 X X Gratch 1985 X X X X Kohl & Wilson 1986 X X St. Clair & Magrill 1990 X Young & Ackerson 1995 X X Malone & Videon 1997 X Hovde 2000 Davis & Cohen 2001 X X Oppenheim & Smith 2001 X Davis 2002 X Heller-Ross 2002 X Hinchcliffe, et al. 2002 X X Emmons & Martin 2002 X X Beile 2003 X X X Robinson & Schlegel 2004 X Ursin et al. 2004 Leiding 2005 Long & Shrikhande 2005 X X X Mohler 2005 X Middleton 2005 X Carlson 2006 Clarke & Oppenheim 2006 X X Knight 2006 X X Tuñón & Brydges 2006 X X X Wang 2006 X X Yu 2006 X X Edzan 2008 X X X Knight-Davis & Sung 2008 X X 118 College & Research Libraries March 2012 requires careful evaluation. Correctness of citation format clearly relies upon the specific format (if any) required by the instructor and also requires careful indi- vidual inspection of each citation. Quality of citations is also difficult to quantify, given the many factors that contribute to such a judgment.5 This review classifies the main criteria used in the reviewed literature to assess student bibliographies. Table 1 lists the studies reviewed herein in chronological order, showing the criteria categories along the top row, with Xs indicating the authors who discussed each criterion. One method not included in this review is numeric counts of the total number of citations and the total number of cited sources by type (such as book, journal, magazine, or Web site). While these techniques are employed in almost all of the studies reviewed herein, because they are strictly numeric and do not involve evaluation criteria they are not included in this review. Influential early studies by Kirk,6 Dykeman & King,7 Gratch,8 Kohl & Wilson,9 and Young & Ackerson10 set the groundwork for the evaluation of bibliographic instruction through rating bibliographies. Kirk scored bibliographies according to criteria for variety, relevance, and scholarliness, then compared result- ing scores from two types of bibliographic instruction. Dykeman & King used a similar methodology, including other criteria such as writing, organization, and content and comparing the results between a treatment group that received bibliographic instruction and a control group. Gratch reviewed four research studies, including Kirk and Dykeman & King and pointed out that, while a direct cause-and-effect relationship between instruction and bibliography quality is often assumed, many intervening vari- ables may also affect the outcome. Kohl & Wilson followed the same methodology as Kirk and Dykeman & King but used criteria of appropriateness, currency, and quality. Young & Ackerson used Kohl & Wilson’s criteria in a similar study and proposed suggestions for increasing the effectiveness of evaluation criteria used in evaluating bibliographic instruction. Another common use of citation analy- sis by librarians is as a measurement tool of library ownership of materials11 and for collection development.12 In this model, student citations are counted and used as a measurement of which journals and other library materials are most and least used by students, and the results are used to guide purchase or discontinuation of materials. Other studies focus on a particular disciplinary area and analyze the usage of journals in that field (such as education,13 mathematics,14 composi- tion studies,15 agriculture and biology,16 chemistry,17 or workforce development18). These studies help librarians analyze student usage patterns and assist library administrators in making collection and budgetary decisions. There are numerous citation analysis case studies in the library literature. These studies have a variety of specific evalu- ation goals: determining which library databases were used by students to access materials,19 frequency of use of online sources,20 persistence of URLs cited,21 citation of electronic journals,22 and use of items in a subject-specific resource guide.23 These studies provide precise detail on specific student behaviors in regard to library research and resource usage. Case studies have also demon- strated the positive impact on the quality of student bibliographies of instructor guidelines for citations,24 inquiry-based library instruction,25 and a credit-bearing library instruction course.26 Multiple factors that may influence student bibliography outcomes have been studied, including comparisons of citation practices by student class year, discipline and source types;27 between students liv- ing on-campus and off-campus;28 between class year and number and variety of sources cited;29 and between student work at traditional vs. nontraditional institu- tions30 and multiple campuses.31 Studies A Faceted Taxonomy for Rating Student Bibliographies 119 have also investigated any correlation between number of citations and word count of the paper32 and between number of citations and final grade.33 Some studies developed customized assessment rubrics, often based on the Association of College and Research Libraries (ACRL) Information Literacy Competency Standards for Higher Edu- cation (www.ala.org/ala/mgrps/divs/acrl/ standards/informationliteracycompe- tency.cfm). Rubrics establish criteria for judging performance, identify a range in the quality of performance, determine a scoring standard, and label levels of qual- ity.34 Tuñón discusses the frequent use of rubrics as an assessment tool to document library research skills and notes that “us- ing rubrics in conjunction with citation analysis is not a new concept for librar- ians.”35 Dykeman and King used a scor- ing guide for citations,36 while Long and Shrikhande used an information literacy grading scale.37 One drawback of these rubrics, however, is that they often rely on individual assessments by instructors of attributes such as “variety,” “utilization,” and “quality of resources,” which lack an objective measurement criteria. Rubrics based on the ACRL’s standards rely on interpretation of the IL broad learning goals that also lack an objective standard for measurement. Emmett and Emde used a “backward design method”38 to map specific learning objectives onto the ACRL standards; precisely how these questions were mapped is left unclear, however. The quality of bibliographies has been used as a criterion in citation analysis studies by Davis and Cohen,39 Beile, Boote & Killingsworth,40 Edzan,41 and others. In these studies, “quality” is generally defined as a measure of the student’s use of scholarly sources, which is obtained by grading each citation against a set of standard scoring criteria. For example, Tuñón and Brydges’s rubric scored a citation on five criteria (breadth of re- sources, depth of understanding, level of scholarliness or “quality,” currency, and relevance) on a scale of 1 to 4 points.42 Middleton developed a rating system in which these types of measurements are used to generate a “scholarly index” (SI) to quantify the proportion of scholarly vs. nonscholarly sources used in the bibliog- raphy.43 One limitation of this system is the broad binary choice of “scholarly vs. nonscholarly,” which does not leave much room for more fine-grained analysis of the level of scholarliness in different types of sources. However, the quantitative rating scale of Middleton’s “scholarly index” was one of the initial inspirations for the taxonomy system ultimately developed for this study. While assessment of the scholarliness of sources is an important element of overall quality of sources in a bibliogra- phy, it is not the only criterion. Other im- portant factors that can be considered in assessing quality are currency of sources (Heller-Ross,44 Hinchcliffe et al.,45 Oppen- heim and Smith,46 Young and Ackerson47), relevance of sources to the research topic (Gratch,48 Kohl and Wilson49), and cor- rectness or completeness of citation style (Malone and Videon,50 Hinchcliffe et al.,51 Mohler,52 Wang53). Each of these assess- ment techniques requires substantial time investment by the researcher. Emmett & Emde note “the need to make assessment projects manageable given the time con- straints of most library instructors,”54 and this is certainly a reasonable concern for researchers when considering the adop- tion of a citation analysis methodology. The variety and diversity of the meth- odologies used to measure information literacy skills reviewed, however, sug- gests a lack of agreed-upon standards for assessing student bibliographies, rooted in the divergent goals and methodologies employed. Many of these studies measure certain specific criteria, but most do not measure the overall quality of student work as exhibited in the bibliography; nor do they apply an explicit coding and rating system. Thus, while the studies reviewed informed the development of the taxonomy system, the individual methodologies reviewed above were not 120 College & Research Libraries March 2012 directly adopted for this study because of the unique qualities and diversity of online sources used by student research- ers today. It is difficult and potentially inaccurate to judge online sources by only one binary choice (such as “scholarly vs. nonscholarly”). There are gradations be- tween an online magazine and an online journal, for instance, and sources such as blogs, forums, and digital repositories do not fit neatly into the traditional biblio- graphic format choices. Simply categoriz- ing an online site as a Web page, as many of the above studies have done, does not provide enough information to judge its quality as an information source. Multiple other criteria must also be considered. The faceted classification methodology attempts to integrate multiple criteria into one standard assessment tool. Cool & Belkin developed a classification system for categorizing information-seeking be- haviors, built on four facets with “dichoto- mous values”55 that could be combined across multiple contexts to explicitly char- acterize a wide variety of related types. Crowston & Kwasnik created a faceted classification system for categorizing on- line genres, describing it as a “multidimen- sional description” that can “view the ob- ject from all its angles.”56 Neither of these examples, however, applied the faceted taxonomy model to citation analysis. The faceted classification system also allows for explicit scoring of each facet, as well as combining these individual scores into a comprehensive citation score that can be compared against others on a single scale. Along with this capacity to assess multiple source formats and criteria, the faceted taxonomy can also accommodate new and emerging genres and phenomena. This approach to bibliographic classifica- tion seems particularly appropriate to the Internet era, with its blurring of forms and genres. The faceted taxonomy system does not rely on traditional hierarchies with a single organizing principle, which may not be useful or appropriate for all cases but is open and adaptable to multiple variables and criteria. Methodology Based on the literature review, the researchers determined that a single- category rating criteria (such as scholarly or nonscholarly) was not sufficient to capture enough detail about the sources used by today’s students. The wide range of information formats and content avail- able through online research calls for the development of a more fine-grained rating system that describes the multiple facets of these information sources. While in the early days of the Internet, coding a source as “a Web page” may have been appropriate, today this is not informa- tive. A Web page can contain anything from a scholarly journal database to an individual’s blog; it is fundamentally im- portant to differentiate between the two. These factors led the researchers to cre- ate a rating taxonomy that is “format neu- tral.” A scholarly journal article in print is effectively equivalent to a scholarly journal article online, and both should be considered functionally identical. The physical nature of the source (print or online) does not tell us much about its content and is not a useful measure of the quality of the source. While rating sources simply as “books” or “journals” may be useful for collection development purposes, it only captures one aspect of an information source. A thorough rating system should also capture the type of content contained in the source, the au- thorship, the editorial process employed in creating the information, and the pur- pose of the publication. For the present study, a faceted taxonomy was adopted because it can combine multiple criteria of classification (such as type of source, type of content, or type of authorship) into a single classification of a citation. Thus, the goal of this study was to measure the overall quality of student work as exhibited in the bibliography and to ap- ply an explicit coding and rating system to quantify the results. The faceted taxonomy was developed through a three-stage process. In the first step of developing the taxonomy, A Faceted Taxonomy for Rating Student Bibliographies 121 an initial draft taxonomy was developed and refined iteratively by the researchers. Second, the faculty instructors of three BiblioBouts pilot classes were interviewed regarding their criteria for assessing bibliographies. Third, a scoring scale for the taxonomy was developed by the researchers and reviewed by the faculty members, who independently scored the items in the taxonomy, and the four sets of scores (three faculty and one researcher) were analyzed to establish inter-rater reli- ability. Fourth, the taxonomy and scoring system were applied to bibliographies produced by students in the pilot class, and the scoring results were compared between students who played BiblioBouts and those who did not. Faculty Interviews To gain a better understanding of how bib- liographies are integrated with research assignments, we interviewed the faculty in charge of each pilot class regarding their class learning objectives, research paper requirements, and grading criteria. We asked the following interview questions: “When you assign papers, what oral and written instructions do you give to stu- dents, e.g., length, format, content, grad- ing criteria? What rules and guidelines do you provide that could help students to select literature for their papers? What do you tell students about your expecta- tions for their bibliographic entries, e.g., number, selection, format, variety of such entries, their scholarly nature? What is an average bibliographic entry, an exemplary entry, and a poor entry?” Some interesting findings emerged from these interviews. Two instructors indicated that theirs was “not a writing class” and that they did not specifically grade citations. One commented “it is not our task to teach writing in these classes,” and another stated “I don’t dig back into the sources, typically there’s not time.” These instructors reported that they grade papers mainly on the argument and the use of evidence in the papers. Grading was based on argumentation and understand- ing of concepts but did not incorporate an evaluation of the citations, except as a numerical requirement. The instructors also reported that they generally do not provide specific guidance on what types of sources to cite or what basis students should use for evaluating sources. These results suggest that instructors may view citation practices as a purely “writing” skill distinct from the content of a class and its learning objectives and not in the purview of their teaching expertise. They seem to see the quality of a student’s writing and citing as being the domain of “writing intensive” classes, rather than incorporated as part of the research paper assignments in their courses. In addition, the instructors commented that students did not seem to have much, if any, library research experience before entering these classes and may not be familiar with what elements constitute a quality research bibliography. Thus, the BiblioBouts game holds the promise of integrating critical information literacy skills into the classroom setting in conjunction with actual assignments, so that students see the utility of the skills and can apply them to their practical, real-world lives. Rather than relying on a segregated, formal instruction session on information literacy, which may seem ab- stract to students and not directly related to their coursework, the necessary skills are learned during game play in a use- ful and practical context related directly to their current assignment. Since these faculty do not currently grade citations or bibliographies, BiblioBouts offers an ad- ditional learning opportunity for students as well as a concrete assessment tool for instructors. Draft Taxonomy To develop the taxonomy, the researchers developed an initial list of possible facets, each with a basic set of broad subtypes. The five main facets were: Information Format, Literary Content, Author Iden- tity, Editorial Process, and Publication Purpose. Information Format was defined 122 College & Research Libraries March 2012 as the basic classification of the type of document, such as newspaper or schol- arly journal. The Literary Content was separated from the Information Format to explicitly characterize the informational nature of the material cited, apart from the generic format it is contained in: for instance, a newspaper may contain edi- torials or essays, and a scholarly journal may contain literature reviews or research articles. As with all faceted taxonomies, the categories are not hierarchical but combinatory, as each facet is independent and can be combined with any other facet to capture the unique characteristics of any individual source. No one facet was considered the primary and overriding factor to be used in evaluation. After review and discussion, the team developed the list of subtypes of Informa- tion Format by consulting the categories of “document type” employed by Ulrich’s Periodical Directory (www.ulrichsweb. com), a widely recognized authority on periodical resources. Based on Ulrich’s definitions of “newspaper,” “magazine,” “journal,” and “monographic series,” and their classifications of “consumer” and “trade” versions, twenty-two subtypes of the Information Format facet were enumerated. Other types of information sources used by college students, such as encyclopedias, reports, and conference proceedings, were also added, as well as additional formats applicable to the online environment, including blog and digital repository. The development of this taxonomy was an iterative process. As the initial draft was developed, it was applied to sample cita- tions from student bibliographies. When citations were discovered that could not be classified by the draft taxonomy, new elements were added. The taxonomy was continually revised through this process. Initially, an “intended audience” facet was considered, but it was ultimately dis- carded because the researchers felt that too much subjective judgment was required to apply it to many online sources. Since intended audience is rarely described explicitly and one source may have many intended audiences, this facet was found not to be useful and was removed. This refinement process continued until the researchers felt that the taxonomy was robust enough to encompass the variety of types of sources found on the sample bibliographies. The researchers consulted with faculty colleagues, who reviewed the list and gave feedback, and the taxonomy was finalized. Taxonomy Scoring Once an initial taxonomy was developed, scores were applied by the researchers to each category, based on the researcher’s judgment of how useful and appropriate each type of element would be to an un- dergraduate student researching sources for a class assignment. Values for the scores were based on the reliability, rel- evance, and scholarliness of each assessed item, as well as the likelihood of the item being vetted by an objective reviewer. Ulrich’s was consulted for their classifica- tion of refereed or peer-reviewed status. A scale of 1 to 4 was used, with 1 being low and 4 being high. The score for each of the five facets would be combined to create a total score for each citation. Using this system, the most highly rated citation could earn a total of 20 points, while the lowest rated citation would earn 5 points. First, the researchers developed an initial numeric rating score for each of the taxonomy categories on a scale of 1–4, TABLE 2 Cohen’s Kappa Scores for Paired Raters Paired Raters Kappa Scores 95% Confidence Interval 1-2 .4808 0.3148 – 0.6468 1-3 .2012 0.0515 – 0.3508 1-4 .3553 0.1877 – 0.5229 2-3 .0374 –0.1084 – 0.1832 2-4 .3648 0.1968 – 0.5328 3-4 .1131 –0.0281 – 0.2543 A Faceted Taxonomy for Rating Student Bibliographies 123 1 being low and 4 being high. Then, the faculty members for each of the three pilot classes were asked to independently score the taxonomy to verify the generalizabil- ity of the scoring. Faculty were provided with the draft taxonomy and asked: “For each of the elements listed under the facets below, please circle your rating of the quality of the element on a scale from 1–4, based on how you would evaluate it as a component of a cited source in a college undergraduate research paper in your class.” The scores from all raters were subjected to statistical analysis to TABLE 3 Faceted Taxonomy and Scoring System Facet 1: Information Format Score Facet 2: Literary Content Score Facet 3: Author Identity Score 1A: Blog 1.3 2A: FAQ 1.5 3A: Unknown Authorship 1.0 1B: Promotional Material 1.5 2B: Posting 1.5 3B: Layman 1.3 1C: Public Sharing Site 2.0 2C: Public Relations 1.5 3C: Corporate Authorship 1.8 1D: Policy Statement 2.0 2D: Column 2.0 3D: Professional Amateur 2.3 1E: Consumer Magazine 2.3 2E: Manual 2.0 3E: Applied Professional 3.5 1F: Consumer Newspaper 2.3 2F: Editorial 2.3 3F: Academic Professional 4.0 1G: Directory 2.3 2G: Review 2.3 1H: Encyclopedia 2.3 2H: Class Reading 2.5 Facet 4: Editorial Process Score 1I: News Story 2.3 2I: Entry 2.5 4A: Self-Published 1.3 1J: Informational Video 2.3 2J: Article/News 2.5 4B: Vanity Press 1.3 1K: Course Material 2.5 2K: Article/Essay 2.8 4C: Collaborative Editing 2.5 1L: Trade Newspaper 2.5 2L: Biography 3.0 4D: Moderated Submissions 2.8 1M: Institutional Repository 2.8 2M: Preprint/ Postprint 3.0 4E: Editor and Editorial Staff 3.5 1N: Trade Magazine 2.8 2N: Thesis 3.0 4F: Peer-Reviewed 4.0 1O: Database 3.0 2O: Working Paper 3.0 1P: Digital Repository 3.0 2P: Article/Lit Review 3.3 Facet 5: Publication Purpose Score 1Q: Trade Journal 3.0 2Q: Article/ Research 3.8 5A: Personal 1.3 1R: Report 3.0 2R: Article/ Synthesis 3.8 5B: Commercial 2.0 1S: Conference Proceedings 3.5 2S: Dissertation 3.8 5C: Nonprofit 3.3 1T: Monograph 3.8 2T: Treatise 3.8 5D: K-12 Education 3.3 1U: Monographic Serial 3.8 5E: Government 3.5 1V: Scholarly Journal 4.0 5F: Higher Education 4.0 124 College & Research Libraries March 2012 establish inter-rater reliability. Cohen’s kappa scores were calculated for each pair of raters, as well as a confidence interval at the 95% level (see table 2). While Rater 3 generally rated items slightly higher than the average, and Rater 4 generally rated slightly lower than the average, in all cases the interval does not include zero; thus, the null hypothesis is rejected. The confidence interval of the overall kappa coefficients at the 95% level was 0.1752–0.3019. The chi-square for the kappa scores was 22.8038 with a p-value of 0.0004, again demonstrating that the null hypothesis was rejected. In addition, Kendall’s Coefficient of Concordance for the four raters produced a p value of 0.000, showing significant correlation among the raters. Given this high level of correlation, the researchers felt confident in the overall generalizability of the scor- ing system. Table 2 shows the statistical results of this analysis. The average score for each item among the four raters was used as the final nu- merical value for that element. Scores were consistently higher for scholarly, academic, and peer-reviewed sources, while scores were consistently lower for anonymous, self-published, and nonre- viewed sources. For example, a scholarly journal received an average Information Format score of 4.0, while a blog received a score of 1.3. For Editorial Process, a peer- reviewed source received a 4.0 score, while a self-published source received a 1.3 score. Table 3 shows the resulting averages used for scoring the taxonomy elements. After the individual scores for each of the five facets were generated, they were combined into a total numerical score for the citation. Each citation’s total score is the sum of the Information Format, Lit- erary Content, Author Identity, Editorial Process, and Publication Purpose scores. For example, suppose two different cita- tions were to be rated: a research article by a PhD in a peer-reviewed scholarly journal and an anonymous online Wiki- pedia entry. Table 4 shows the total scores for these two example citations. Coding of Pilot Bibliographies The final bibliographies from one pilot- tested undergraduate class were ana- lyzed using the faceted taxonomy. The class consisted of 100 students who were assigned a research paper as part of the class requirements. The class was an entry-level introductory course cross- listed between the Schools of Information and Sociology with an assigned research topic of “Business Models for Web 2.0 Social Networking Sites & Platforms.” Students volunteered to play BiblioBouts, incentivized by receiving extra credit on TABLE 4 Examples of Coded and Scored Citations Research article in a peer-reviewed journal Anonymous Wikipedia entry Facet Score Facet Score Information format: Scholarly Journal 4.0 Information format: Encyclopedia 2.3 Literary content: Article/Research 3.8 Literary content: Entry 2.5 Author identity: Academic Professional 4.0 Author identity: Unknown 1.0 Editorial process: Peer-Reviewed 4.0 Editorial process: Collaborative Editing 2.5 Publication purpose: Higher Education 4.0 Publication purpose: Nonprofit 3.3 Total score 19.8 Total score 11.6 A Faceted Taxonomy for Rating Student Bibliographies 125 coding. Differences of judgment were discussed and resolved, and agreement was reached on how the sample sources were to be coded. The researchers randomly chose 30 bibliographies were from the entire pool of the class, with 15 chosen from students who played BiblioBouts (treatment group) and 15 from students who did not play (control group). These anonymized bibliographies were distributed equally to the two coders, who then applied the taxonomy and scoring system to the citations in each bibliography. For electronic sources, the coders checked the original URL to verify the source and code it. In some cases, facets were unidentifiable due to broken or incorrect URLs or lack of information on the site. In their research paper. Out of the entire class, 30 students played the game. The 70 remaining nonplayers formed the control group. All students were given an in-class introduction to the BiblioBouts game by the researchers, who also dem- onstrated gameplay. Students received bibliographic instruction from a visiting librarian, which consisted of a review of databases available through the university library and basic search techniques. This session focused only on introducing students to schol- arly resources and query for- mulation. The librarian was not asked to tailor the instruction session to include evaluation strategies, because the game’s Tagging & Rating bout intro- duced students to an evaluation strategy and gave them hands- on practice evaluating sources using the strategy. Final papers from both play- ers and nonplayers were col- lected and anonymized. Bibli- ographies were extracted from the papers, and anonymous codenames were applied. Two independent coders (master ’s degree students in the School of Information) were then hired and given three sample student bibliographies to code. After an initial round of coding, results were compared and discussed to reach a consensus on the criteria to be used in TABLE 5 Means of Agreement between Coders Facet Matches % Agreement Information Format 6/9 0.67 Literary Content 8/9 0.89 Author Identity 9/9 1.00 Editorial Process 7/9 0.78 Publication Purpose 6/9 0.67 Overall 7.2/9 0.80 TABLE 6 Information Format Frequencies Players Nonplayers Total % Total % Blog 33 25.8 10 10.2 Promotional Material 18 14.1 12 12.2 Public Sharing Site 1 0.8 1 1.0 Policy Statement 1 0.8 0 0.0 Consumer Magazine 7 5.5 7 7.1 Consumer Newspaper 11 8.6 17 17.3 News Story 6 4.7 7 7.1 Course Material 3 2.3 1 1.0 Trade Newspaper 6 4.7 2 2.0 Institutional Repository 2 1.6 1 1.0 Trade Magazine 10 7.8 14 14.3 Database 4 3.1 3 3.1 Digital Repository 1 0.8 1 1.0 Report 10 7.8 5 5.1 Conference Proceedings 2 1.6 3 3.1 Monograph 6 4.7 10 10.2 Monographic Serial 1 0.8 1 1.0 Scholarly Journal 6 4.7 3 3.1 Total 128 100 98 100 126 College & Research Libraries March 2012 these cases, the facet was coded as a zero. When the facets were coded, the coders produced total scores for each individual citation as well as overall combined scores for each bibliography. Applying inter-rater reliability analysis to this coding is difficult due to the small number of raters and the larger number of possible categories, which are nominal and independent of each other. Cohen’s kappa statistics would not account for the margin of variance in the coding. Instead, a sample bibli- ography coded by both coders was analyzed for consistency of coding. This sample bibliography con- tained nine citations, and the codes were compared for each of the taxonomy facets. Total percentage of agreement (mean) was calculated. See table 5. The resulting overall mean of agreement was 0.80, or 80 percent agreement between the raters. Given the large number of possible codes for each facet, the researchers felt confident in the overall level of agreement between the coders. Findings The frequency of use for each subtype within a facet was count- ed and the percentage of use by both players and nonplayers was calculated. This analysis revealed which were the most and least used facet subtypes in the coded citations. Results for each of the five facets by players and non- players are presented in tables 6 through 10. Table 6 shows that the most- used Information Format for players was Blog (25.8%), while for nonplayers the most-used format was Consumer Newspa- per (17.7%). In contrast, Blogs were used by only 10.4 percent of nonplayers, while Consumer News- papers were used by only 8.6 percent of players. Overall, Scholarly Journals were used by only 4.7 percent of players and 3.1 percent of nonplayers. Table 7 shows that players used Post- ings as the most frequent Literary Content (21.8%), which follows from the use of Blogs as most-used Information Format. TABLE 7 Literary Content Frequencies Players Nonplayers Total % Total % Posting 27 21.1 2 2.1 Article/Essay 24 18.8 12 12.8 Article/News 18 14.1 26 27.7 Public Relations/ Promotional 16 12.5 11 11.7 Article/Synthesis 8 6.3 7 7.4 Article/Research 7 5.5 5 5.3 Review 6 4.7 4 4.3 Class Reading 4 3.1 1 1.1 Treatise 4 3.1 10 10.6 FAQ 3 2.3 2 2.1 Column 3 2.3 2 2.1 Editorial 3 2.3 3 3.2 Entry 2 1.6 7 7.4 Thesis 2 1.6 0 0.0 Working Paper 1 0.8 2 2.1 Total 128 100 94 100 TABLE 8 Author Identity Frequencies Players Nonplayers Total % Total % Applied professional 63 49.2 59 56.7 Corporate authorship 21 16.4 15 14.4 Academic professional 19 14.8 14 13.5 Professional-amateur 17 13.3 5 4.8 Unknown authorship 8 6.3 11 10.6 Total 128 100 104 100 A Faceted Taxonomy for Rating Student Bibliographies 127 Likewise, following their use of consumer newspapers as the most-used Information Format, nonplayers used Article/News as the most frequent Literary Content (27.1%). Higher-quality sources such as Article/Research or Treatise were used by only a small percentage of partici- pants (3%–6% of players and 5%–10% of nonplayers). Table 8 shows that the most-used Author Identity for both players (52.1%) and nonplayers (61.5%) was Applied Professional, which follows from the reli- ance on blogs and consumer newspapers as Information Formats. Although the authors of blogs in particular may not always be considered professionals, in the case of technology blogs the credentials of the authors often relate to the topic be- ing discussed and give the author somewhat more credibility than a layman. When available, the biographical description of blog authors was checked by the cod- ers for evidence of professional experience. Sources with Aca- demic Professional as the Author Identity were used by 15.7 percent of players and 14.6 percent of nonplayers. Table 9 shows that the most- used Editorial Process for both players (59.5%) and nonplayers (78.1%) was Editor and Editorial Staff, which follows from the reliance on blogs and consumer newspapers. While this is cer- tainly true of newspapers but much less certain with blogs, the blogs used were checked by the coders for evidence of an editorial process to review the content of postings. Peer- reviewed sources were the third choice for both players (10.7%) and nonplayers (9.4%). Table 10 shows that both players (76.6%) and nonplayers (90.6%) relied primarily on Com- mercial publications for their ci- tations. While Higher Education was the second most-used facet for both players (15.3%) and nonplayers (14.6%), it still lags far behind the other categories. Some categories in the taxonomy were not used in any of the coded bibliogra- phies. For Facet 1: Information Format, neither players nor nonplayers used Direc- tory, Encyclopedia, Informational Video, or Trade Journal. In addition, nonplayers did not use Policy Statement. For Facet 2: Literary Content, neither used Manual, Biography, Preprint/Postprint, Article/ Lit Review, or Dissertation. In addition, nonplayers did not use Thesis. For Facet 3: Author Identity, neither players nor nonplayers used Layman. All categories were used for Facet 4: Editorial Process. For Facet 5: Publication Purpose, nonplay- ers did not use the Government category. TABLE 9 Editorial Process Frequencies Players Nonplayers Total % Total % Editor and editorial staff 72 56.3 75 72.1 Self-published 20 15.6 5 4.8 Peer-reviewed 13 10.2 9 8.7 Vanity press 11 8.6 3 2.9 Moderated submissions 11 8.6 10 9.6 Collaborative editing 1 0.8 2 1.9 Total 128 100 104 100 TABLE 10 Publication Purpose Frequencies Players Nonplayers Total % Total % Commercial 95 74.8 87 80.6 Higher education 19 15.0 14 13.0 Personal 5 3.9 1 0.9 Nonprofit 4 3.1 1 0.9 K-12 education 3 2.4 5 4.6 Government 1 0.8 0 0.0 Total 127 100 108 100 128 College & Research Libraries March 2012 Once the individual citation coding was completed, the researchers analyzed the usage of scholarly databases by players and nonplayers. Bibliographic citations were checked for reference to a database used to find the source; these were indicated mainly by the inclusion of a URL in the citation or the mention of a database in the citation without a URL (that is, “retrieved from ProQuest”). Spe- cific databases owned by larger services such as Gale and ProQuest were counted under the larger umbrella. Table 11 shows that players used more scholarly databases (31) than nonplay- ers (25) and that both groups relied on ProQuest in particular for the majority of the identified citations (33 out of 56, or 59%). This suggests that awareness of the ProQuest database is fairly high among these undergraduates, but familiarity with other databases may be low. Usage of refereed journals in the cita- tions was also analyzed. Where citations gave clear citations to journals, Ulrich’s Periodical Directory was consulted to es- tablish if the journal was categorized as “refereed.” Seven articles from refereed journals were identified in the sample bibliographies, with a total of six cita- tions from players and four citations from nonplayers. Table 12 shows that two refereed journals were cited by both players and nonplayers: Decision Support Systems and Journal of Cases on Information Technology. Both players used ProQuest to find the Decision Support Systems article, while one player used ProQuest and one player used Gale to find the Journal of Cases on Information Technology article. For both players and nonplayers, one bibliography contained two scholarly citations, sug- gesting that the particular student may have been a more experienced researcher. Finally, the overall results for players and nonplayers were analyzed. Average number of citations per bibliography, av- erage citation score, and average bibliog- raphy score were compared. The overall citation score is the average of all citations in all the coded bibliographies, and the overall bibliography score is the average of the total bibliography scores. These av- erages present a generalized picture of the final bibliographies of both players and nonplayers. Table 13 shows that, on average, player bibliographies contained more citations (8.6) than nonplayer bibliographies (7.2). This may sug- gest that players of BiblioBouts are exposed to more sources and thus include more citations in their bibliographies. The average cita- tion scores were very close for both groups, with nonplayers scoring slightly higher (13.7) than players (13.5). Why the nonplayer cita- tion score is higher is not clear, as both players and nonplayers relied primarily on nonscholarly online sources. As shown previously in table 6, players most frequently used Blogs and Promotional Materials, while nonplayers most frequently used Consumer Newspapers and Trade Magazines. This may be re- TABLE 11 Databases Used Databases (based on URLs) Players Nonplayers Business & Company Resource Center 0 2 EBSCO 1 0 Gale (including Academic OneFile) 0 2 Gartner 3 0 General Business File 0 1 LexisNexis Academic 0 6 OCLC First Search 1 0 ProQuest (including ABI/Inform) 21 12 Sage 0 1 ScienceDirect 1 0 SpringerLink 4 1 Total 31 25 A Faceted Taxonomy for Rating Student Bibliographies 129 lated to the assignment topic, “Business Models for Web 2.0 Social Networking Sites & Platforms,” which lent itself to a reliance on up-to-date commercial sources. The higher average bibliography score for players (110.1) vs. nonplayers (95.5) reflects the higher average number of citations per bibliography, as well as the fact that players used more scholarly databases and cited slightly more refereed journals than nonplayers. Discussion In this study, a faceted taxonomy for rat- ing student bibliographies was developed to evaluate the overall quality of student work as exhibited in the research bibliog- raphy and to apply a standardized coding and rating system to quantify the results. The goal of this taxonomy was to opera- tionalize “quality” in a manner relevant to the variety of online resources used by today’s students. The use of a faceted classification allowed for a fine-grained analysis of varying sources with vary- ing levels of quality. The results of this study suggest that this faceted taxonomy provides a standardized assessment tool that is flexible enough to encompass the variety of online sources available to students. The proposed taxonomy is adaptable and can be enhanced to meet the changing nature of online informa- tion sources. Additional categories can be added to any facet without changing the overall system. The scoring may be modified by instructors if they wish. The faceted classification system is flexible enough to be modified in many small ways without losing its overall integrity. Analysis of the sample bibliographies showed that most students did not use scholarly resources for the citations in their bibliographies. In part, this may be attrib- utable to the assignment topic (“Business Models for Web 2.0 Social Networking Sites & Platforms”), which lent itself to online sources that might reasonably have more up-to-date information. Since the paper as- signment did not require the students to cite scholarly sources, they were free to select whichever sources they wanted. Relevant scholarly sources did exist, as noted by the use of the refereed journals Decision Sup- port Systems and Journal of Cases on Information Technology. Many more scholarly sources were donated to the BiblioBouts group collection than were included in the sample bibliographies analyzed in this study. Overall, a small percentage of sources (21%) in the coded bibliog- TABLE 12 Citations from Refereed Journals Journal Players Nonplayers Decision Support Systems 1 1 European Management Journal 2 0 International Journal of Advertising 1 0 Journal of Adolescent Research 0 1 Journal of Cases on Information Technology 1 1 Journal of Information Systems Education 1 0 J. of Theoretical and Applied Electronic Commerce Research 0 1 Total 6 4 TABLE 13 Results for Players and Nonplayers Average Players Nonplayers Citations per bibliography 8.6 7.2 Citation score 13.5 13.7 Bibliography score 110.1 95.5 130 College & Research Libraries March 2012 raphies came from the BiblioBouts game, meaning that most students opted for sources they found on their own outside the game. The reasons for this are not clear. There were some technical difficul- ties during the pilot test that may have discouraged players from participating fully and prompted them to complete to their papers on their own. Students may also have relied on force of habit in finding their own sources rather than putting time and effort into the game. If players had used a higher percentage of sources from the game, which were rated and evaluated by other players to establish a consensus assessment of quality, it is likely that the average citation and overall bibliography scores would have been higher. Results of the coding and scoring dem- onstrated that game players used more scholarly databases, cited more refereed journals, included more citations per bib- liography, and achieved higher overall bib- liography scores than nonplayers. While the percentage differences are not large, these initial findings suggest that the game had a positive impact on players’ research behavior. Thus, the researchers feel that the BiblioBouts game holds out the promise of directly integrating information literacy skills into the classroom setting, so that students see the utility of the skills and can apply them to actual assignments. Rather than relying on a segregated, formal in- struction session on information literacy, which may seem abstract to students and not directly related to their coursework, skills are learned during game play in a useful and practical context related di- rectly to their class requirements. These findings have provided very useful data for the project team and have driven the improvements being made to the next version of the game. Among these is a much greater focus on directing stu- dents to use online scholarly databases, through the addition of in-game recom- mendations from the class instructor as well as repeated reinforcement through displaying the databases used by other players who have made donations. In addition, the improved game will provide suggestions for keywords to be used in searching and display keywords attached to documents other players have donated. It is hoped that, through the implementa- tion of these refinements, the BiblioBouts research team will see continued im- provement in the quality of bibliographies of students who play the game. Possible Limitations This pilot test is by definition preliminary, and these initial results require further verification. The proposed taxonomy was only applied in one class, to a small sample of student bibliographies, which limits the generalizability of the results. Other limita- tions include the fact that only three faculty members were involved in creating the scoring values. A wider sample of faculty may produce different scores. Overall, the researchers feel that these initial results are still useful and suggest that online gaming to teach information literacy skills may be useful to librarians, information literacy instructors, and faculty who assign under- graduate research papers. Future Research There are several directions for future research involving the faceted taxon- omy, now that it has been successfully deployed. As the BiblioBouts game is refined and its ease of use improves, we expect to see greater active participation by both students and instructors, which we expect will improve results further. When instructors are actively involved in the game, their donations of high-quality sources will act as models for their stu- dents. The impact of this increased par- ticipation can be measured by using the taxonomy to assess all donated sources in the game, not just a sample. This will also allow the researchers to evaluate the dif- ferences in quality between those sources donated, but not chosen for the final bibliographies, with the chosen sources, to assess the overall improvement in qual- ity as a result of the evaluation strategies introduced during the game. A Faceted Taxonomy for Rating Student Bibliographies 131 Notes 1. Colleen Boff and Kristin Johnson, “The Library and First-Year Experience Courses: A Nationwide Study,” Reference Services Review 30, no. 4 (2002): 277–87. 2. Andrew Walsh, “Information Literacy Assessment: Where Do We Start?” Journal of Librari- anship and Information Science 41 (2009): 19. 3. Karen Hovde, “Check The Citation: Library Instruction and Student Paper Bibliographies,” Research Strategies 17, no. 1 (2000): 3–9. 4. Tefko Saracevic, “Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science, Part III: Behavior and Effects of Relevance,” Journal of the American Society for Information Science and Technology 58, no. 13 (2007): 1915–33. 5. Anne Middleton, “An Attempt to Quantify the Quality of Student Bibliographies,” Perfor- mance Measurement and Metrics 6, no. 1 (2005): 7–18. 6. Thomas Kirk, “A Comparison of Two Methods of Library Instruction for Students in Introductory Biology,” College & Research Libraries 32, no. 6 (1971): 465–74. 7. Amy Dykeman and Barbara King, “Term Paper Analysis: A Proposal for Evaluating Bib- liographic Instruction,” Research Strategies 1, no. 1 (1983): 14–21. 8. Bonnie Gratch, “Toward a Methodology for Evaluating Research Paper Bibliographies,” Research Strategies 3, no. 4 (1985): 170–77. 9. David F. Kohl and Lizabeth A. Wilson, “Effectiveness of Course-Integrated Bibliographic Instruction in Improving Coursework,” RQ 27, no. 2 (1986): 206–11. 10. Virginia E. Young and Linda G. Ackerson, “Evaluation of Student Research Paper Bibli- ographies: Refining Evaluation Criteria,” Research Strategies 13, no. 2 (1995): 80–93. 11. Stacey Knight-Davis and Jan S. Sung, “Analysis of Citations in Undergraduate Papers,” College & Research Libraries 69, no. 5 (2008): 447–58. 12. Reba Leiding, “Using Citation Checking of Undergraduate Honors Thesis Bibliographies to Evaluate Library Collections,” College & Research Libraries 66, no. 5 (2005): 417–29. 13. Laurel A. Haycock, “Citation Analysis of Education Dissertations for Collection Develop- ment,” Library Resources and Technical Services 48 no. 2 (2004): 102–06. 132 College & Research Libraries March 2012 14. Robin N. Sinn, “A Local Citation Analysis of Mathematical and Statistical Dissertations,” Science & Technology Libraries 25, no. 4 (2005): 25–37. 15. Daniel P. Coffey, “A Discipline’s Composition: A Citation Analysis of Composition Studies,” The Journal of Academic Librarianship 32, no. 2 (2006): 155–65. 16. Pali U. Kuruppu and Debra C. Moore, “Information Use by PhD Students in Agriculture and Biology: A Dissertation Citation Analysis,” portal: Libraries and the Academy 8, no. 4 (2008): 387–405. 17. Núria Vallmitjana and L.G. Sabaté, “Citation Analysis of Ph.D. Dissertation References as a Tool for Collection Management in an Academic Chemistry Library,” College & Research Libraries 69, no. 1 (2008): 72–82. 18. C. Keith Waugh and Margie Ruppel, “Citation Analysis of Dissertation, Thesis, and Research Paper References in Workforce Education and Development,” The Journal of Academic Librarianship 30, no. 4 (July 2004): 276–84. 19. Hovde, “Check the Citation.” 20. Debbie Malone and Carol Videon, “Assessing Undergraduate Use of Electronic Resources: A Quantitative Analysis of Works Cited.” Research Strategies 15, no. 3 (1997): 151–58. 21. Philip M. Davis, “The Effect of the Web on Undergraduate Citation Behavior: A 2000 Update,” College & Research Libraries 63, no. 1 (2002): 53–60. 22. Maria Elizabeth Clarke and Charles Oppenheim, “Citation Behaviour of Information Sci- ence Students II: Postgraduate Students,” Education for Information 24, no. 1 (2006): 1–30. 23. Lara Ursin, Elizabeth Blakesley Lindsay, and Corey M. Johnson, “Assessing Library In- struction in the Freshman Seminar: A Citation Analysis Study,” Reference Services Review 32, no. 3 (2004): 284–92. 24. Andrew Robinson and Karen Schlegel, “Student Bibliographies Improve When Professors Provide Enforceable Guidelines for Citations,” portal: Libraries and the Academy 4, no. 2 (2004): 275–90. 25. Mark Emmons and Wanda Martin, “Engaging Conversation: Evaluating the Contribution of Library Instruction to the Quality of Student Research,” College and Research Libraries 63, no. 6 (2002): 545–60. 26. Rui Wang, “The Lasting Impact of a Library Credit Course,” portal: Libraries and the Academy 6, no. 1 (2006): 79–92. 27. Jake Carlson, “An Examination of Undergraduate Student Citation Behavior,” The Journal of Academic Librarianship 32, no. 1 (2006): 14–22. 28. Holly Heller-Ross, “Assessing Outcomes with Nursing Research Assignments and Citation Analysis Of Student Bibliographies,” Reference Librarian 77 (2002): 121–40. 29. Fei Yu, Jan Sullivan, and Leith Woodall, “What Can Students’ Bibliographies Tell Us? Evidence Based Information Skills Teaching for Engineering Students,” Evidence Based Library and Information Practice 1, no. 2 (2006). Available online at http://ejournals.library.ualberta.ca/index. php/EBLIP/article/view/8 [accessed 30 November 2011]. 30. Johanna Tuñón and Bruce Brydges, “A Study on Using Rubrics and Citation Analysis to Measure the Quality of Doctoral Dissertation Reference Lists from Traditional and Nontraditional Institutions,” Journal of Library Administration 45, no. 3 (2006): 459–81. 31. Gloriana St. Clair and Rose Mary Magrill, “Incomplete Citations in Undergraduate Term Papers from Four Campuses. RQ 30, no. 1 (1990): 75–81. 32. Knight-Davis and Sung, “Analysis of Citations in Undergraduate Papers.” 33. Robinson and Schlegel, “Student Bibliographies Improve When Professors Provide En- forceable Guidelines for Citations.” 34. Lorrie A. Knight, “Using Rubrics to Assess Information Literacy,” Reference Services Review 34, no. 1 (2006): 43–55. 35. Tuñón and Brydges, “A Study on Using Rubrics and Citation Analysis,” 464. 36. Dykeman and King, “Term Paper Analysis.” 37. Casey M. Long and Milind M. Shrikhande, “Improving Information-seeking Behavior Among Business Majors,” Research Strategies 20, no. 4 (2005): 357–69. 38. Ada Emmett and Judith Emde, “Assessing Information Literacy Skills Using the ACRL Standards as a Guide,” Reference Services Review 35, no. 2 (2007): 218. 39. Philip M. Davis and Suzanne A. Cohen, “The Effect of the Web on Undergraduate Citation Behavior 1996–1999,” Journal of the American Society for Information Science and Technology 52, no. 4 (2001): 309–14. 40. Penny M. Beile, David N. Boote, and Elizabeth K. Killingsworth, “Characteristics of Edu- cation Doctoral Dissertation References: An Inter-Institutional Analysis of Review of Literature Citations” (paper presented at the annual meeting of the American Educational Research As- sociation, Chicago, Illinois, April 21–25, 2003). 41. N.N. Edzan, “Analysing the References of Final Year Project Reports,” Journal of Educational A Faceted Taxonomy for Rating Student Bibliographies 133 Media & Library Sciences 46, no. 2 (2008): 211–31. 42. Tuñón and Brydges, “A Study on Using Rubrics and Citation Analysis.” 43. Middleton, “An Attempt to Quantify the Quality of Student Bibliographies.” 44. Heller-Ross, “Assessing Outcomes with Nursing Research Assignments.” 45. Lisa Janicke Hinchliffe, Chrstine Kubiak, Stephen Hunt, and Cheri Simonds, “What Students Really Cite: Findings from a Content Analysis of First-Year Student Bibliographies,” in Integrating Information Literacy into the College Experience, Papers Presented at the Thirtieth National LOEX Library Instruction Conference, May 10–11, 2002, eds. Julia K. Nims, Randal Baier, Rita Bullard, Eric Owen (Ann Arbor, MI: Pierian Press, 2003): 69-74. 46. Charles Oppenheim and Richard Smith, “Student Citation Practices in an Information Science Department,” Education for Information 19, no. 4 (2001): 299–323. 47. Young and Ackerson, “Evaluation of Student Research Paper Bibliographies.” 48. Gratch, “Toward a Methodology for Evaluating Research Paper Bibliographies.” 49. Kohl and Wilson, “Effectiveness of Course-Integrated Bibliographic Instruction.” 50. Malone and Videon, “Assessing Undergraduate Use of Electronic Resources.” 51. Hinchcliffe et al., “What Students Really Cite.” 52. Beth A. Mohler, “Citation Analysis as an Assessment Tool,” Science & Technology Libraries 25, no. 4 (2005): 57–64. 53. Wang, “The Lasting Impact of a Library Credit Course.” 54. Emmett and Emde, “Assessing Information Literacy Skills,” 214. 55. Colleen Cool and Nicholas J. Belkin, “A Classification of Interactions with Information,” in Emerging Frameworks and Methods, Proceedings of the Fourth International Conference on Conceptions of Library and Information Science (COLIS4) 2002, eds. Harry Bruce, Ray Fidel, Peter Ingwersen, and Pertti Vakkari (Greenwood Village, CO: Libraries Unlimited, 2002): pp. 1–15. 56. Kevin Crowston and B.H. Kwasnik, “A Framework for Creating a Faceted Classification for Genres: Addressing Issues of Multidimensionality,” in Proceedings of the 37th Hawaii International Conference on System Sciences (HICSS-37), Big Island, HI (Los Alamitos, CA: IEEE Press, 2004): 5-7.