Microsoft Word - 2016ASIST_PosterPaper_294 (1).docx Investigating Perceptions and Support for Transparency and Openness in Research: Using Card Sorting in a Pilot Study with Academic Librarians Liz Lyon School of Information Sciences University of Pittsburgh 135 N. Bellefield Ave Pittsburgh, PA 15250 elyon@pitt.edu Eleanor Mattern School of Information Sciences; University Library System University of Pittsburgh 135 N. Bellefield Ave Pittsburgh, PA 15250 emm100@pitt.edu Wei Jeng School of Information Sciences University of Pittsburgh 135 N. Bellefield Ave Pittsburgh, PA 15250 wej9@pitt.edu Daqing He School of Information Sciences University of Pittsburgh 135 N. Bellefield Ave Pittsburgh, PA 15250 dah44@pitt.edu ABSTRACT This paper explores the role of academic librarians as advocates for research transparency and open research. We describe the design and piloting of a qualitative card-sorting research protocol that investigates academic librarians’ attitudes, awareness and practices related to research transparency. We report on preliminary results from interviews with librarians, presenting their conceptualizations of research transparency and open research, existing library services that support and advocate for both concepts, and potential services that would augment this support and advocacy. Library activities they feel are most important to the advancement of transparency and openness are identified and perceptions of disciplinary differences are noted. Keywords Research transparency; open research; academic libraries; data curation INTRODUCTION In the last decade, there has been a significant policy shift towards articulating and consolidating an open research agenda by governments and research funding. The open agenda has in part been constructed around a desire for a more democratic dissemination of the discoveries and knowledge contained within the scholarly record funded by public monies. There is also a need to ensure the quality, integrity and rigor of the scholarly process and to facilitate accountability and trust in the outcomes of research. The concept of transparency has been included in these descriptions of openness, however the engagement and perceptions of library and information professionals, who are key stakeholders in digital scholarship and open research, are less understood and form the subject of this study. This paper describes the preliminary findings from a pilot exercise that applied a qualitative card-sorting methodology to probe the following research questions: 1. What are the attitudes, understandings and practices of librarians and information professionals to the concepts of transparency and open research? 2. What are the practical implications of these attitudes, understandings, and practices for the development and delivery of innovative research services provided by academic libraries? BACKGROUND Research transparency is a recurrent theme within policy statements developed in recent years. Transparency is one of the principles listed in the Organization for Economic Co-operation and Development (OECD) Guidelines for access to research data from public funding (OECD, 2007). In the United States, the Obama Administration released a memorandum, setting three specific actions for departments and government agencies: transparency, participation and collaboration (Holdren et al., 2009). Transparency as a value has been discussed in ideological terms by Etzioni (2010), who describes the strong variant relating to regulatory contexts and disclosure. The Royal Society Report (2012) references “transparent policies for custodianship, data quality and access” in outlining a set of principles of stewardship that should be shared by custodians of scientific work (p. 63). Other terms are used in the literature to describe concepts related to transparency: reproducibility (Peng, 2011), repeatability (Easterbrook, 2014), and verifiability (Gezelter, n.d.). {This is the space reserved for copyright notices.] ASIST 2016, October 14-18, 2016, Copenhagen, Denmark. The dimensions of openness are investigated by Lyon (2009) who describes openness as a continuum with two orthogonal axes: ‘Access’ and ‘Participation’. Corrall and Pinfield (2014) explore the first of these dimensions and construct a typology of ‘open’. Lyon and Beaton (2015) address the second dimension of participation in the context of libraries, reviewing citizen science initiatives, education and skills development. A 3-dimensional model of open science, extending this prior work was introduced by Lyon (2016), and includes the additional dimension of ‘Transparency’. She suggests transparency is linked to the research lifecycle and points to broad opportunities for library and information sciences professionals to engage with and support transparency in research, including through policy, education and infrastructure. METHODS Frequently used in Human-Computer Interaction (HCI) and design studies, card sorting is a qualitative data collection methodology that probes how individuals categorize and draw relationships among concepts. There are examples in the library and information science literature of researchers who implement card sorting as a methodology, particularly in studies involving usability testing of online resources (e.g., Faiks & Hyland, 2000; Whang, 2008; Lewis & Hepburn, 2010). Morville and Rosenfeld (2007) differentiate between two different approaches to card sorting: open and closed sorting: “In totally open card sorts, users write their own card and category labels. Totally closed sorts allow only pre-labeled cards and categories. Open sorts are used for discovery. Closed sorts are used for validation. There’s a lot of room in the middle” (p. 256). For usability studies of websites in a library setting, there are examples of both open (Lewis & Hepburn, 2010) and closed sorting (Faiks & Hyland, 2000). This study applies both of these approaches. For this study, we developed a semi-structured protocol using card sorting as a method. We piloted the instrument with seven librarians at a research university [name removed for review]. Our sampling method was driven by an interest in interviewing librarians who support a range of disciplinary communities on campus. In doing so, we tested the instrument with librarians who work with scientists, social scientists and humanists. Four sessions were conducted with individual librarians with one to two research team members present. Two researchers conducted a focus group with three librarians. We chose to deliver the instrument both in a one-on-one and a focus group setting to gauge whether one approach was more effective than the other. Table 1 presents a detailed instrument. We provided participants with cards with pre-defined library activities supporting open research and transparency (i.e. assistance with locating a data archive for depositing data, advocacy for open access publishing, training on tools like Open Science Framework) and with blank cards to capture additional activities that they identified. We asked them to categorize them according to their status: activities currently done; activities that not currently done but that can be done immediately; activities done but not requested; activities that can be offered in the future; and not Study Part Activity I. Setting the Stage The research team reviewed study objectives and verbal consent script; obtained permissions for use of audio recorder II. About You We asked librarians about the disciplinary background that informs their work, the disciplinary communities they serve, their research experiences, and their support for research in their roles as librarians III. Concept Construction Participants described what two terms mean to them: open research and research transparency. We asked the participants to keep their understandings in mind as we moved forward in the study. IV. Card Sorting of Library Activities We provided participants with pre-defined library activities that support open research and research transparency and with blank cards to capture additional activities that they identified. We asked participants to categorize the cards according to status of library support: ● Activities that the library is currently doing ● Activities that the library is not currently doing but that you think can be done immediately (quick wins) ● Activities that the library is ready to provide but that patrons have not or have rarely requested ● Activities that you think the library can offer in the future (longer-term wins) ● The rest of activities (i.e. irrelevant, unsure) We asked participants to provide verbal insight into their sorting decisions. V. Benefits and Barriers Participants were invited to comment on their perceptions of the benefits of the services to research and to the library. We asked that they comment on barriers that do or could challenge the library’s development and provision of the open research/research transparency support activities. VI. Importance of Library Activities We gauged librarians’ attitudes concerning the importance of library activities to the advancement of research transparency and open research. The librarians arranged the activities on a numbered scale, with 1 being the least important and 5 being the most important. VII. Debriefing We invited feedback on the pilot instrument from participants, asking if anything was unclear or, from their perspective, could be improved. Table 1. Study Instrument Piloted with Librarians applicable (irrelevant, unsure or unnecessary). We asked the participants speak to benefits and barriers of offering existing and potential services and their thoughts on those services most critical to open research and transparency. We met with librarians in their offices or in a designated place in the main campus library. The length of the interviews with individual librarians ranged from 25 to 50 minutes, while the focus group reached an hour; it was necessary for us to modify the instrument with the focus group because of the timing and we chose to move past the discussion on barriers and benefits. We recorded and transcribed sessions and photographed the results of the participants’ card sorting. PRELIMINARY FINDINGS Definitions of Open Research and Research Transparency When being asked about the meanings of open research and research transparency, librarians interpreted open research as closely connected to other, perhaps more familiar terms in librarianship, such as open access. Describing open research, Librarian 3 related “...it means sharing and being transparent with your research throughout the entire process from a grant proposal to the very end, with publishing. And I think a lot about open access and being able to access final products or information even during the process as well”. Some participants conflated the terms, seeing them as one in the same. For others, there were more clear boundaries. Librarian 4 linked open research with clearly reported protocols, stating: “To me, that [open research] means openness in the disclosure of methods of research”, linking open research with the articulation of process. For at least one librarian, there was a strong connection between transparency and ethics. Librarian 2 explained, “I’m almost feeling like it [research transparency] is more ethical – principles instead of protocol”. Library Activities Supporting Research Transparency We asked participants to sort pre-defined cards that included library activities that may support open research and transparency (closed sorting) and to add new cards to sort (open sorting) (see Table 1, Study Section IV). Few participants added activities to the pre-identified cards. Among those that were added included assistance that would help with later discovery and with greater provision of access: Consult on database and metadata design; Consult on taxonomy development; and Approach community groups to offer assistance with access to information (Librarians 1 and 6). There was limited reference to the research; instead, the participants primarily used the chronology of the academic term to talk about existing and potential services. Three particular topics attracted in-depth commentary from the participants: a recognition of sharing and collaboration as a primary facet of transparency and openness, the library’s role in education and advocacy and the relevance of the disciplinary environment to perceptions of open research. Benefits and Barriers The librarians articulated benefits that library services supporting open research and transparency could have on research developments. Librarian 2, for example, addressed how library assistance with research data stewardship could advance research: “Having well-cared for data that can then be used by other people to discover other things is definitely a good thing for science. It makes me think of all the articles in the news about people who haven’t been able to replicate experiments and stuff” (Librarian 2). Benefits for the library and for librarianship were also highlighted, reflecting current professional challenges: Librarian 4 remarked, “I think it’s a matter of survival. I think it’s a matter of staying where our researchers are and not being left behind. For this librarian, extending research services is necessary for academic libraries to remain relevant. Several participants, however, reported limited experience with independent or collaborative research and suggested that this could introduce barriers to service. Among the barriers described was the challenge of incentivizing researchers to adopt new research practices and to think of the library as a place for support. Librarian 4 questioned whether the academic reward system sufficiently encouraged open research practices among faculty, namely junior scholars on the tenure track. This same librarian questioned whether funders motivate thoughtful attention to open research behaviors. With requirements regarding data management plans, she noted: “I remember from a talk someone saying ‘no one’s ever been denied a research grant because of their data management plan’”. Importance of Library Activities When asked to rank library activities supporting open research and transparency, the librarians considered the importance of the services to the disciplines they themselves served. Librarian 4’s card sorting reflected on the importance of the disciplinary context in making prioritization decisions for services: “You have to know your discipline and what’s important. Quite honestly, if I were to do this for English (see Figure 1), it might look quite differently than for computer science (see Figure 2)”. For academic libraries prioritizing the development and delivery of research services, this librarian suggests that the potentially competing needs of the communities they serve must be considered. One librarian observed that she was prioritizing activities that occurred at the end of the research. She commented: “As I was doing this, I realized that I ranked the most important as the end product – the publishing part. I think that’s because for me, …., this is where I probably would be looking at the most to support research.” This same librarian assigned a lower importance to activities that she viewed as group-specific (benefiting what she perceived as a niche group) and activities that users may be able to accomplish without intervention (i.e. locating books on open research). Librarian 7 articulated the challenge of prioritizing quick wins versus longer term actions, where importance may be measured by what the library can most easily accomplish: “sometimes you prioritize things you can do quickly and get something done but it may not be the biggest most…” The challenges in prioritizing the allocation of resources for new services in terms of quick wins versus most impactful long- term solutions was noted and is illustrated by the comment by Librarian 2: “The consulting activities felt to me very important – maybe because I’ve been thinking about what our role is in terms of education and the things that we can actually provide in the form of consultation. It seems like we can hit the ground running if we do things like help researchers take care of their data…all of that is actually helping from the ground up.” DISCUSSION The card-sorting exercises conducted in interviews and focus groups provided valuable early insight into framing the context of research transparency from the perspective of academic librarians, as well as determining the methodology, vocabulary and process best suited for future investigation of the research questions. The discussion is arranged in two sections reflecting these perspectives. Professional Library Practice The results confirmed the complexity of open research and research transparency concepts, with some participants viewing these terms as synonymous. The narratives highlighted the different facets of research transparency (e.g. collaboration, sharing resources, and documenting metadata). A key finding was the critical importance of establishing the relevance of the research lifecycle at an early stage of the protocol; it provides an essential foundation for examining transparency practices and workflows in some depth. Many of the interviewees did not relate their answers to the research lifecycle and this framing was missing from the conversations. Instead, the librarians were largely conceptualizing research in the context of the academic term. One interpretation of this result is that librarians are positioning their activities as learning opportunities oriented towards students, rather than as advocacy activities oriented towards the research community. This is supported by the emphasis on reference work, training or instruction in research methods and by commentary. A further interpretation is related to service delivery mode: librarians are positioned at a distance from researchers (i.e. located in the library, rather than adopting an immersive mode). Observations from the card-sorting exercises and analysis of the dialogue highlighted key themes requiring further exploration. Firstly, the scope and interpretation of the key terms indicated a predominant focus on access and democratization of information as an important dimension for librarians. This focus is one that can be investigated further. Secondly, there is a need for greater clarity of roles in engagements with the researcher community. The observed tensions between whether the library should ‘advocate’ or ‘educate’ around open researchcaptured the contrasting service values but also strengthened the importance of identifying “transparency verbs” (Lyon, 2016). Implication of Methodology The card-sorting methodology worked effectively in a group and with individuals, although there were time pressures with the focus group. The selection of terms within the instrument requires further consideration to increase clarity. We research lifecycle is required to push librarians to think differently about the timing of their interventions: this is a key conclusion from the pilot. The prioritization of activities revealed a further finding: the perceived differences across disciplinary practice and Figure 1. Card sorting for English Figure 2. Card sorting for Computer Science culture. The application of a single instrument across different disciplines will require careful review to ensure particular domains are not disenfranchised by the choice of transparency concepts, vocabulary, or activities described. CONCLUSIONS AND FUTURE WORK Card sorting provided valuable early insight into framing the context of research transparency from the perspective of academic librarians, as well as determining the methodology, vocabulary and process best suited for future investigation of the research questions. This pilot has provided insightful pointers to inform the refinement of the methodology for a larger-scale study and has teased out librarians’ opinions, understandings and practices related to open research and transparency. The importance of explicitly framing activities within the research lifecycle, the selection of vocabulary used in the instrument (in particular the use of transparency verbs) and differences in disciplinary research practice and culture, have emerged as key points to consider in designing the next stage of this study. REFERENCES Corrall S. & Pinfield, S. (2014). Coherence of “open” initiatives in higher education and research: Framing a policy agenda. Proceedings of iConference 2014, Berlin. Easterbrook, S.M. (2014). Open code for open science? Nature Geoscience 7, 779-781. Etzioni, A. (2010) Is transparency the best disinfectant? Journal of Political Philosophy 18(4), 1-16. Faiks, A. & Hyland, N. (2000). Gaining user insight: A case study illustrating the card sort technique. College & Research Libraries, 61(4), 349-357. Gezelter, J. D. (n.d.) Open science and verifiability. Retrieved from http://web.stanford.edu/~vcs/Nov21/dg- OpenScienceandVerifiability.pdf Holdren, J.P., Orszag, P. & Prouty, P.F. (2009). President’s memorandum on transparency and open government – Interagency collaboration. https://www.whitehouse.gov/sites/default/files/omb/assets/ memoranda_fy2009/m09-12.pdf Lewis, K. M. & Hepburn, P. (2010). Open card sorting and factor analysis: A usability case study. The Electronic Library, 28(3), 401-416. Lyon, L. (2009). Open science at web-scale: Optimising participation and predictive potential. Consultative report. Retrieved from http://opus.bath.ac.uk/30056/1/open- science-report-6nov09-final-sentojisc.pdf Lyon, L. & Beaton, B. (2015). Citizen science, open access, open data, and research inclusivity. Proceedings of ALISE Annual Conference, Chicago. Lyon, L. (2016). Transparency: the emerging third dimension of Open Science and Open Data. LIBER Quarterly. 25(4), 153–171. Morville, P. & Rosenfeld, L. (2007). Information architecture for the World Wide Web. Sebastopol, CA: O'Reilly Media OECD (2007) OECD Principles and guidelines for access to research data from public funding. Retrieved from http://www.oecd.org/sti/sci-tech/38500813.pdf Peng, R.D. (2011). Reproducible research in computational science. Science 334, 1226-1227. Royal Society (2012). Science as an open enterprise: open data for open science. Retrieved from http://royalsociety.org/~/media/Royal_Society_Content/poli cy/projects/sape/2012-06-20-SAOE.pdf Whang, M. (2008). Card-sorting usability tests of the WMU libraries’ web site. Journal of Web Librarianship, 2(2-3), 205-218.