S N 152 Profession © 2011 by The Modern Language Association of America 152 On the Evaluation of Digital Media as Scholarship \ / GEOFFREY ROCKWELL In 2006 the Modern Language Association Task Force on Evaluating Scholarship for Tenure and Promotion issued a recommendation that “[d]epartments and institutions should recognize the legitimacy of schol- arship produced in new media” (Report 5). That a task force of one of the largest and most prestigious scholarly associations in the humanities would recommend digital work be taken seriously was a dramatic move, one whose effects I witnessed going before a tenure and promotion committee prepared to argue for recognition of digital work for a colleague. When I arrived the committee members all had a photocopy of the recommenda- tion, and I discovered that I had prepared the wrong case. The problem was no longer convincing others that digital work could be scholarly; the prob- lem was that colleagues are unsure how to evaluate digital work, whether a peer-reviewed article in an online journal or an interactive research Web site. Colleagues and chairs are willing to entertain the case theoretically, but in practical terms they don’t know how to get started, and that is what this essay is about: getting started—a turning toward the sort of dialogue that could mature into a culture of balanced evaluation.1 Let me start with some definitions. Research, for the purposes of this essay, is the activity that leads to scholarship, which is the outcome that can be shared. Many researchers use digital methods or digital resources but still share their scholarship in print. Others might conduct research in \ / The author is professor of philosophy at the University of Alberta. \ / GEOFFREY ROCKWELL ||| 153 S N 153 archives, never looking at a computer screen, but still publish their schol- arship in digital form. This distinction suggests that a simple approach to evaluation would be to focus on digital scholarship and develop the case for the evaluation of digital outcomes rather than practices. But committees, especially when hiring and tenuring junior faculty members, are also concerned with the research future of the candidate—from what little has been done they are drawing inferences about what will be done in the future. After all, hir- ing and tenuring are about making expensive commitments on behalf of the university to the future. Further, merit increases are often based on activity rather than outcome—otherwise we would unfairly penalize our colleagues who write books and thus don’t have much to show in the years between publications. In short, it is not enough to evaluate only digital outcomes. Evaluators need to consider research activity for digital scholars much as they do for traditional scholars, and that is hard when you don’t have the experience to assess what digital humanists are doing.2 The second and more difficult term to define is digital. For the pur- poses of this essay, what matters is not whether some scholarship is digital at some point in its making. It doesn’t matter if an article in Literary and Linguistic Computing was written with a word processor or if it was written on bookstore receipts I found in my pocket. What matters is that the work is shared with the community in electronic form and, more important, that it is meant to be experienced in electronic form, usually off a computer screen, though some interactive works are presented as installations with- out a screen. This is what our colleagues have trouble evaluating—those works that address their audience differently, that often have no beginning or end and are therefore frustrating to read. Colleagues are being asked to read differently, and thus they are being asked to evaluate something they may not even understand how to access. They are being asked to evaluate a type of scholarship they haven’t had any experience creating, and they can’t therefore imagine the research done to create it. How are colleagues to feel comfortable evaluating when they can’t imagine the making—the poesis of digital work? This in turn raises the question of why colleagues should have to evaluate work in digital form at all. Why not simply sub- contract the work to reviewers familiar with digital research? Why Colleagues Should Evaluate Digital Work Colleagues who want guidance often start with the Guidelines for Evaluating Work with Digital Media in the Modern Languages, by the MLA’s Committee on Information Technology (CIT). These guidelines were developed by 154 ||| ON THE EVALUATION OF DIGITAL MEDIA AS SCHOLARSHIP S N 154 the CIT specifically to help evaluators and candidates. For the evaluators the guidelines recommend you do the following: Delineate and communicate responsibilities Engage qualified reviewers Review work in the medium in which it was produced Seek interdisciplinary advice Stay informed about accessibility issues While all five guidelines are important, I focus on the third one, to “re- view work in the medium in which it was produced.” It is this guideline that creates the most work for evaluators, and it is this one that forces a culture change on us. Ultimately the others follow from this basic collegial responsibility, so let us examine it closely. Responsibility The recommendation calls on evaluators to review the work of their col- leagues and not simply to review the reviews. It seems an obvious point, but reviewing a colleague’s work in whatever form it comes in is what we are supposed to do when formally evaluating for hiring, merit raise, or tenure. Even when there are external reviewers—and it is a good idea to get informed external advice—the final decision rests with the commit- tee, and for that reason committee members should have some familiarity with the work. It is a renunciation of responsibility to not review a candi- date’s work. Experimentation in Form It is common for candidates to prepare cribs to their work for those un- willing to wrestle with CD-ROMs and strange Web sites. When I came up for tenure, I provided the committee with a narrative on my digital work that included screen shots, descriptions, discussions about the nature of my contributions (to coauthored works), and references to associated work that legitimized the digital work. I tried to show how software tools I had worked on that couldn’t be reviewed themselves were reviewed in proxy through grant proposals and conference presentations. I tried to make a “double or nothing” argument that digital work could be treated as scholarship when it was reported back to the research community—so a peer-reviewed conference paper on a Web site that was not peer reviewed legitimized the original digital work and doubled the academic credit (so I would get credit for the conference paper and the digital work). While such narratives are useful to evaluators, and candidates should be encouraged to prepare them, they should never be a substitute for review GEOFFREY ROCKWELL ||| 155 S N 155 of the work in the form it was produced in. The originality of digital work is difficult to assess when all you have is a description. Digital work is often about processes, interactivity, and interface, and no description (even with screen shots) can do the work justice. Many new media works are experi- ments in form, and that experimentation is lost in translation. Digital work to be evaluated as an original scholarly contribution needs to be assessed in such a way that the originality (or lack thereof) is evident. Prose descrip- tions of projects can help the expert imagine the contribution, but they are likely to mislead the evaluator new to new media. Finally, it should be mentioned that some types of digital work, like tools, are about method, and their value comes from how they can be used on different objects. Colleagues really should try tools, even if they are difficult to try, precisely because they instantiate methods and hermeneutical processes.3 Some Types of Digital Work What then are the types of digital work that need assessment, and how are they different from print work? A complete typology of scholarly digital work is beyond this essay, in part because new forms of digital expression seem to emerge yearly, which of course is the point—the digital permits extraordinary experimentation with form. It also permits rapid experimen- tation such that each year there seems to be a new technological fad for which digital humanists demand recognition. Perhaps it is this mashing change that characterizes the digital in form. Still, it is worth starting with a few types of stable work colleagues in the digital humanities have pre- sented to the community and pointing out some of the things evaluators and candidates can discuss to understand the value of the contribution.4 Online Peer-Reviewed Publication The least controversial type of digital work is the peer-reviewed online article in a Web journal like Digital Humanities Quarterly. Where the pro- cesses of peer review are comparable to those of a print journal, it is safe to assume that quality control is equivalent to that of print. Further, an online article can be easily read by internal evaluators without having to learn about a different scholarly medium—just print it out and ignore the venue. There are, however, issues of credibility and persistence of online materials—issues that haunt the venue.5 As long as authors don’t trust online venues or don’t feel the venues have the requisite prestige, then, in a self-fulfilling fashion, the venues will lack the credibility of print journals. This has been addressed partly by attention to the problem of preservation—faculty members (and journals) are encouraged to deposit 156 ||| ON THE EVALUATION OF DIGITAL MEDIA AS SCHOLARSHIP S N 156 their work in institutional repositories (Chan). There is also evidence that online publications get cited more and therefore have more impact, which is why one could argue that junior scholars should be publishing online (Lawrence). Finally, there are some innovative approaches to peer review itself that promise to deal with credibility issues, such as open peer review and editing, where anyone can assess and edit a submission.6 Things to discuss: Why was the online journal chosen? How does it handle peer review? Are there any statistics on access to the article that can give a sense of the impact of the research? Is the work being archived for long-term preservation? Scholarly Electronic Editions One of the most useful contributions of digital humanists has been to create online scholarly electronic editions of resources of interest, from historical documents to literary works. While there are many electronic versions of classic literary texts, often put up in a bout of enthusiasm by students, scholarly electronic editions represent significant and informed research work. The work of the electronic editor, like that of the scholarly print editor, is not trivial. Peter Robinson and Kevin Taylor, in “Publishing an Electronic Textual Edition: The Case of the Wife of Bath’s Prologue on CD-ROM,” describe the series of decisions, informed by knowledge of the context and of the original, about what to show and hide, how to enrich the material, and how to represent it electronically. The oppor- tunities and fluidity of the electronic form mean the editor must master two fields, the intellectual context of the original and current practices in digital representation. There is also now a significant literature around scholarly editing in the electronic age that the digital editor should be aware of and possibly contribute to.7 Ironically, if the editor gets the form right so that the electronic version can be searched and easily read, no one will notice, and such delicate work will be unappreciated in evaluation, but this is true of translation and editorial work, whatever the medium. It could be argued that work on scholarly electronic editions is particularly important at this  juncture since we are in a transformative epoch when new scholarly resources are being designed and built for the next genera- tion to interpret. Things to discuss: How did the editor think through representing the edition electronically? What was the editor able to do differently in elec- tronic form? What was lost? Were others consulted, or was a review so- licited? Who is using the edition, and are there any statistics on usage? Is there a plan for long-term maintenance and preservation? Was the work GEOFFREY ROCKWELL ||| 157 S N 157 deposited in a trusted digital repository? Did the editor also report on the research and decisions behind the edition? Specifications One of the least appreciated contributions is the work of developing guidelines, standards, and specifications. To the untrained eye this looks like service work on a large scale. I prefer to think of it as an Oulipean art—that of designing constraints that encourage controlled innovation. Specifications are, after all, a system of suggestions as to what you should and shouldn’t do. They make possible a potential literature, in this case electronic scholarship. And, in the case of guidelines like those of the Text Encoding Initiative, they present a theory of text in a form that has real consequences.8 If they aren’t confused (and there are poor specifications), then they instantiate and communicate a theory about what the potential for an electronic representation is. The problem with specifications is that they aren’t reviewed the way other work is. In some cases specifications are reviewed by standards bod- ies as they become standards, but the specifications are usually commis- sioned by the standards body, and the politics of standards review are dif- ferent from those of academic peer review. There are, however, ways to tackle standards work. Standards are often published, and these publications are scholarship. A measure of the impact of a standard is its adoption, and some standards can be shown to be widely adopted. Other specifications may be developed as proof of concept or for a particularly innovative project. In those cases we can look at how the innovations were returned to the community through conference papers, clear and useful documentation, and consultations. Things to discuss: What needs and communities do these specifications address? How were the specifications developed? Were there formal pro- cesses that included outside review? How were the specifications returned to the community, and is there evidence of impact? Is there a plan for long-term review and maintenance of the specifications? Research Tools Humanities computing, if you survey its journals, has been as much about the representation of humanities evidence in scholarly electronic editions as about developing tools for preparing, publishing, and studying the new editions. Since the first issue of Computers and the Humanities there have been reviews of tools, articles about tool projects, and laments for the lack of tools that can exploit the new evidence. There is also a history of anxiety 158 ||| ON THE EVALUATION OF DIGITAL MEDIA AS SCHOLARSHIP S N 158 about the recognition of tool work (Sinclair et al.). Software tools that are developed to be used by others, as useful as they may be inside the field, are hard to explain as research outside. How, then, are we to think about such work? One way is to think of tool development as work in applied methodology. If specifications are an implementation of a theory of content, tools are an implementation of a potential method of research. Tools present a theory of the practice of research in a form that others can try. They say something like, “It is useful to do this in this way so we have facilitated the practice in this way.” One of the constraints and opportunities of the digital is that it forces us to be concrete when we imagine potential repre- sentation and method. Everything on the computer is formalized, which is not to say that man-computer processes are formalized. To create a tool is to have to choose a particular theory of practice, think about it, explore its consequences, and formalize parts of it for others. No amount of paper prototyping or imagining is a substitute for actually trying to implement something that works, which is why theorizing about tools is not the same as building them as a research practice. This is new; in the humanities we are not used to having to take a concrete stand on methods that can be tested by others. More to the point, in the humanities we are suspicious of methods and tools and therefore reluctant to stabilize methods in tools for fear that practices will then freeze and be imposed. One of the ways in which we distinguish the humanities from the social sciences is that our practices are themselves at stake, fluid and woven with evidence. Tool work would seem like the Trojan horse of scientism in the humanities. How to evaluate tool work then? Like specifications, tools are not typi- cally peer-reviewed for publication, but they are demonstrated, tested, and used, and evaluators can therefore find all sorts of documentation, from manuals to online comments, about them. In some cases tools are even re- viewed. What would a research review of a tool look like? Alan Galey, Stan Ruecker, and the INKE Team have presented case studies in “How a Pro- totype Argues” of how experimental design prototypes could be evaluated as reified arguments. They look at how a design presents an argument, how it handles objections, how it is an original contribution, and how it is part of a research trajectory. Unfortunately, tools are rarely reviewed in a formal fashion, and this is in part because of how complex it is to review a tool if you are going to go down to the level of a code review. Unlike a monograph, tool development is usually reviewed at the start—it is the grant proposal to build it that is reviewed—on the basis of a description of potential, not the finished implementation. Grants are critical because the cost of a tool lies in its development (and maintenance), not so much its GEOFFREY ROCKWELL ||| 159 S N 159 distribution, so review has to happen earlier. For these reasons, in addition to looking at how the tool has been shared and tested, evaluators should take seriously a history of successful grants as an indication that the peer community is impressed. Things to discuss: What need does the tool meet, and how has it been shown to meet that need? What community is the tool for, and how has the developer engaged the community? How is the tool an improvement on previous tools? Is the interface interesting or the algorithm better? What theories or arguments are borne by the tool? How are those documented or shown? Is there a maintenance plan, or is this a prototype? Was the tool or its development reviewed as a grant proposal or in some other way? Research Blogs and Web 2.0 Activity Cathy Davidson has argued that we are entering a second phase that can be loosely connected to social media technologies, often given the Web 2.0 designation (“Humanities 2.0”).9 Blogs and now Twitter are examples of social media that have been adapted for research work in the academy. Such emergent forms are particularly hard to evaluate since they don’t resemble any traditional academic form and they are more about process and relationships than finished content. A good blogger (or team of blog- gers), however, does a great service to the community by tracking fast- moving issues, linking to new materials, and commenting on those issues. The better blogs will include short reviews, announcements, interesting interventions, and notes about timely matters like exhibits. Blogs, as I have learned, require habits of attention.10 Each post might take half an hour to research and post. Posts may appear to be light and quick, but the good bloggers learn and practice their craft. In some ways running a blog is like moderating a discussion list. How often does Willard McCarty post a pro- vocative note to Humanist to promote discussion?11 The work of facilitat- ing the conversations we value in the humanities should not be dismissed as service; it can be closer to journal editing. Disciplines interested in human expression should take seriously new types of expression. What is really at issue is whether scholars should par- ticipate in experiments or take a critical or judgmental stance and only comment on, review, and theorize about the creative work of others. We have encoded in our departmental divisions views about the values and differences of academic work that separate the creative work of the artist from the critical work of the art historian, or the creative work of the writer of fiction from the theoretical work of the literary scholar who studies her or him. We aren’t entirely sure if the fine, design, and performing arts should be in the academy, as the language of most tenure and promotion 160 ||| ON THE EVALUATION OF DIGITAL MEDIA AS SCHOLARSHIP S N 160 documents shows. Imagine trying to get creative digital work evaluated when you aren’t in the art department. The split between “interpretation” or “theoretical” or “analytical” work on the one hand and, on the other, “archival work” or “editing” falls apart when we consider the theoretical, interpretive choices that go into deci- sions about what will be digitized and how. (Davidson, “Data Mining”) In addition to the problem of assessing new media work, there is the per- ception that at best digital scholarship is essentially community work, edi- torial work, or a form of translation and therefore theoretically light. It needs to be said over and over that there is nothing a priori untheoretical about digital work; it is rather a form of potential theory. I have argued that specifications, for example, instantiate a particular theory of text, and others have argued that prototypes can reify arguments. Every decision of the TEI about how to encode some phenomenon that we take for granted, like a date, is based on a theory of what a date is for the purposes of tex- tual representation. Every research tool bears a theory about the practice of interpretation and the potential for computer-assisted interpretation. Specifications and tools can be done well and be appropriately theorized, or done poorly without a view to the fabric of humanities knowledge. If we don’t recognize and support well-theorized specifications and tools, we will have to live with those that emerge from other groups with needs and questions other than those we care about. Do we really want our tools to be built only by Google and to thus be geared for handling business documentation? Likewise, if we don’t recognize the care and work that goes into maintaining the research commons through editing, blogging, and other social research activities, then our public intellectual space will be managed by others (or simply not be there). I will go further and say that practices of theory that do not, where ap- propriate, take into account their implementation are unethical, especially when consequences are openly discussed. The old way of doing theory is premised on an unexamined view that the way ideas are transmitted is primarily through chains of books by great men. This is simply no longer true, if it ever was. The epidemiology of ideas—the way ideas are trans- mitted, explored, refined, and forgotten—is complex and changing. The Internet is changing the ecology of transmission. A widely read blog can have measurably more readers than a published book. If what we value is appropriate intervention into the flow of conversation we call the hu- manities, then we need to be prepared to measure contributions, no matter what their form, in terms of their effectiveness as interventions. Counting peer-reviewed books and articles just doesn’t cut it as a measurement of GEOFFREY ROCKWELL ||| 161 S N 161 impact, especially with all the problems of peer review and its particular economy. It should be noted that one relevant feature of the digital is that access to information can be logged and measured in ways that were unthinkable before. Viewing statistics are easy to gather for blogs, Web sites, tools, and hypermedia. The statistics we can gather have far more detail than the crude metric of peer-reviewed page counts. While neither page counts nor Web statistics really tell you whether information is having an effect, one can infer a lot more about readers from Google Analytics than one can from sales of a peer-reviewed book. Things to discuss: What are the subject and audience of the blog? What is the contribution to the research community of the work? Are there sta- tistics that show the reach and impact of the blog? What are some exem- plary posts that show the research focus of the blog? Are there plans to archive the blog or to repurpose parts as publications? Hypermedia and New Media Works There is a whole class of new media works that are born digital in the sense that they are authored on and for the computer as creative or expressive works. These works take advantage of the networked computer as an alter- native medium for creative and original expression. Many of these works, especially those on the Web, take advantage of the nonlinear and hyper- textual potential of electronic literature, which is why I am gathering this diverse literature under the rubric hypermedia. Many of these works are experiments in literary interactivity and can only be viewed if you have ac- cess to the right configuration of equipment. Others are playful and game- like. All in all they are a nightmare to review and publish because they are experimental and because they are often technically idiosyncratic. Most are therefore either made available online or self-published since there is no viable publishing and review mechanism.12 Such works are to some extent beyond the scope of this essay since they are not so much humani- ties research as original arts creations that should be assessed as digital or media art. Having complained above about the artificial division of the creative and interpretive, I want to provide ideas about how these can be evaluated. A common way to get research credit for any creative work is to present and publish papers about the making of it.13 There are all sorts of venues, from conferences to media exhibits, where the work can be demonstrated and the research issues around the creative work discussed. As for the evaluator, an important thing to pay attention to is the use of interactivity (Kiousis). The potential for interactivity is what makes the digital work 162 ||| ON THE EVALUATION OF DIGITAL MEDIA AS SCHOLARSHIP S N 162 different from other media. The computer makes it possible to program responses, branching, algorithmic visualizations, and computer-generated sound into the work. In some ways these works are the easiest to evaluate since they are meant to express something that an evaluator could inter- pret. In other words, they are meant to be played by you. You can approach them as a work of art and bring the interpretive traditions of the humani- ties to bear on this new media art. As with other arts you can look for the artist’s statement and ask about the genesis of the work. Finally, you can ask critics familiar with such work to talk you through it. Things to discuss: How should such a work be interpreted? What is the history of this work, and how is the work responding to other works? How does the work use the computing medium and opportunities for interac- tivity? Is it playful, fun, and responsive? How is it documented? How is it exhibited or shared with its audience? Developing a Dialogue The evaluation of digital work is a process that needs to be developed, if possible long before career-changing decisions have to be made. Both candidates and evaluators benefit if there is a dialogue around expecta- tions and evaluation earlier rather than later; candidates will know what they can do to make their case, and evaluators will have a framework for evaluation. Start with the Hiring As the MLA guidelines point out, the dialogue starts with the hiring. The job ad is the first gesture from the institution indicating what it is look- ing for. Job ads should accurately reflect what an institution wants and therefore what it will (should) evaluate. Statements like “the successful candidate will be expected to run a digital humanities lab” are a public sign that the candidate will be expected to manage a lab. Candidates and evaluators should therefore be prepared to assess how well the candidate managed the lab and to take that into account. Job ads that say things like “candidates should submit a portfolio of new media work” are signaling that the institution wants candidates who have created new media and will presumably value that in the future. In short, for the institution the job ad (and other hiring communications) is the first utterance in a dialogue. Departments should not put in the ad things they don’t value and should be prepared to take seriously in evaluation anything that they say they value. Likewise, for candidates the job ad is a first clear indication of what will be evaluated. Things may change in the dialogue, and you may want GEOFFREY ROCKWELL ||| 163 S N 163 them to change, but the job ad is a document that you can use to further the dialogue once hired. Candidates and chairs can also negotiate a memorandum of under- standing about what the expectations are and how work will be evaluated (see Report 11). The more conversation there is at the beginning, when tenure review is in the future, the better.14 Conversations with the Chair and Department It is common for the dialogue to lapse after the thrill of hiring. Chairs need to move on to other issues, and new hires need time to orient themselves to the new job. The dialogue tends to continue through annual reviews, which can end up being the only formal occasions for ongoing dialogue. Needless to say, both parties benefit if the dialogue is pursued more vigor- ously through standard opportunities like • discussing digital opportunities at departmental meetings • presenting digital work to the department and university • preparing grant proposals, both internal and external • developing collaborations with colleagues, and • teaching with technology Broad Engagement The fundamental difference between digital work that is not research and digital work that is research rests in how it contributes to a larger conversation. Original research is responsive to what others have done (and are doing), and it is reported back in ways that inform others in the field. An assessment of the researcher’s contribution to that conversation from his or her own peers is thus especially valuable. While there are few venues for strict peer review of digital work, there are all sorts of ways that researchers can engage and document their engagement in broader academic conversations. They should identify local, national, and interna- tional conferences that will allow them to address a research community of peers. Chairs should help researchers find funding to attend appropri- ate conferences, workshops, and venues. The digital humanities is a field, like computer science, where new knowledge is often shared live at con- ferences because it needs to be shown to be understood. For this reason anyone whose research includes building new media works should find venues to exhibit these works. If there isn’t funding to travel to confer- ences, then researchers should be encouraged to find ways to share results online through blogs, Twitter, discussion lists, and other public forums. If I were an evaluator, I would expect digital work that can’t be peer reviewed and published to then be exhibited or demonstrated in other ways. More 164 ||| ON THE EVALUATION OF DIGITAL MEDIA AS SCHOLARSHIP S N 164 generally, a researcher who is not participating in the conversations of such a fast-changing field is not likely to be doing research-level digital development. Chairs would do well to advise new researchers to start shar- ing their work as soon as possible rather than clinging to it because it could be better. Administrative Conversations Often digital humanists are expected to provide administrative leader- ship in the department around things digital, and this creates special and dangerous circumstances where there is greater need for dialogue. This is especially true when institutions are hiring their first digital humanist and expect that one person to play a “transformative” role. The leadership can take the form of being expected to service the department’s computers, manage the departmental Web site, manage a lab, manage staff, run the online presence for a departmental project, apply for grants to get infra- structure for the department, introduce instructional technology, or get colleagues to use technology. Such expectations can be manifestly unfair when these junior colleagues, instead of being shielded from administra- tion like their peers, have extra service dumped on them while still being expected to live up to traditional research expectations. For this reason it is especially important that there be a dialogue about administrative expecta- tions. Junior faculty members should refuse extraordinary service without reassurances in writing that it will be recognized even if the chair (and the department’s digital agenda) changes. Given the importance of good administration in the light of the expense of computing, it is no surprise that departments and individuals are increasingly looking at alternative academic positions where the mix of responsibilities is differently struc- tured to recognize leadership.15 There are, however, reasons for defining such positions as faculty positions,16 which is why it is useful to imagine structured conversations that can ensure that there are scholarly outcomes associated with the administration. If a conversation around expectations and opportunities takes place early on, leadership projects can be designed to include a research dimension. Instructional technology projects can be designed to have a research dimension, for example. When providing computing support to an important research project like an institute or an online publication that bestows research credit on others, it is particularly important to negotiate meaningful research com- ponents for the digital hire. Negotiations should include discussion about shared credit, opportunities for creative exploration, access to resources for managing the project, recognizing project management as a form of research, and coauthorship of papers about the project. I have always GEOFFREY ROCKWELL ||| 165 S N 165 found that there are opportunities to weave my own research interests into the project if I ask early on. I also try to make sure that all I have to do for the project is the management and that there is sufficient funding to hire others for the text encoding, programming, design, and testing.17 Conversations at Times of Critical Evaluation If there has been a dialogue from the start, then moments of critical evalu- ation, when the evaluators need to make decisions that go on the record, go much smoother. First, there will be documentation about job expecta- tions so that the evaluators can ask, Did the hire do what we discussed? Likewise, candidates can use the accumulated documentation to help them structure their case so that it is recognized by the evaluators. Second, the departmental evaluators should have been exposed to the candidate’s work over the years, and they should have had chances to ask about the work as it was developed, thus giving them a way of evaluating the making. An evaluator who is skeptical of the value of digital work should have had am- ple opportunities to ask the candidate respectfully about the value of works demonstrated, and the candidate should have some idea as to what docu- mentation would satisfy the skeptic. Third, any large and mission-critical projects for the department can be taken into account in the evaluation if there has been explicit discussion of expectations early on. The goal is to develop a consensus as to what documented digital work will count so that both sides can anticipate the final review and agree on outcomes. Some evaluators may feel that too much discussion lessens their ability to make critical judgments because they will get caught in a web of obligations with the candidate. That shouldn’t be the case if there have been frank discus- sions of expectations. Conclusion: The Resistance of Digital Media Every field, especially a new one, finds ways to contribute to the larger scholarly effort of its time in unique ways. Humanities computing is no exception. I would argue it is the difference in the contributions of com- puting humanists that, on the one hand, make the contributions so valu- able taken one by one and that, on the other, make them so hard to classify as scholarship comparable to what other colleagues do. Digital research works resist classification and comparison in so many ways, and that is often their value. This is a period of experimentation with scholarly form, and some of the most useful work will not look like anything else that we recognize as scholarly. And that is the way we want it. Further, failure is to be expected and valued. No one complains that Tim Berners-Lee’s World 166 ||| ON THE EVALUATION OF DIGITAL MEDIA AS SCHOLARSHIP S N 166 Wide Web was a failed hypertext technology because the software he and colleagues created was left behind. It provided the ideas and the specifica- tions, not the particular instantiation. In sum, few digital research contributions can be assessed the way print contributions can, but we can develop a culture of assessment that includes conversations that are in the tradition of the humanities. NOTES | / 1. This paper is also a story woven from a wiki that I started when I was on the MLA Committee on Information Technology. Since my first failures to explain digital work to colleagues I have had a chance to practice talking with chairs about how to assess digital work, and this essay lays out some of what I believe we need to tell our col- leagues as we work together toward inclusive tenure and promotion processes. Above all, I am more than ever convinced that digital work is not “old wine in new bottles” and that we do ourselves a disservice if we try to argue that we are doing the same sort of work, just in digital form. For that reason I started the wiki to experiment with different ways—from fictional cases to a short guide—of introducing colleagues to the difference. The original version of the wiki as it was when I started it is at www .philosophi.ca/pmwiki.php/Main/MLADigitalWork. Although I was responsible for most of it, Ronnie Apter wrote the Links and Bibliography section. The wiki was later reproduced on the MLA site (http://wiki.mla.org) so that it could be openly edited by any member. It is now a community document that can evolve as we all need it to. 2. For a project that documents what digital humanists do, see the Day in the Life of the Digital Humanities project (Day). 3. For that matter, you can get help. If you are trying to assess a learning object in its original form but don’t know how to install it and run it, then ask for help from the university instructional technology unit. If you want to assess a work of new media art, then ask your colleagues in art or design to help you. Getting advice and help from col- leagues across campus allows evaluators to review the work in its original form while getting advice from people familiar with the form. 4. A community-edited version of this discussion of types of digital work is available at the MLA wiki, http://wiki.mla.org. 5. See Siemens et al.: this report from 2001 looks at the issues from different per- spectives and includes a survey of Canadian humanities and social science scholars. In the section “Report on Responses to the Questionnaire,” which I coauthored, we noted that “86% (of respondents) felt that non-electronic outlets were more credible, though 61% felt that peer review ensures similar quality.” 6. Nature had an online “Web debate” on peer review that nicely surveys the alter- natives. See Nature Peer Review. 7. For more, see Siemens and Schreibman. 8. See tei-c.org for more on the Text Encoding Initiative. The TEI guidelines have become a de facto reference point for anyone representing scholarship in electronic form. One needn’t follow the guidelines, but you should at least be able to explain why. 9. For a definition of what Web 2.0 is, see O’Reilly. The term came out of confer- ence brainstorming by O’Reilly and others. GEOFFREY ROCKWELL ||| 167 S N 167 10. I have been maintaining a blog since 2003 called Theoreti.ca at http://theoreti.ca. While I do mention this in annual reports, I don’t expect research credit for it. Other blogs, however, are more substantial works. 11. The Humanist discussion list has been going since 1987 and has probably done more to build community in the field than any other project. See www. digitalhumanities .org/humanist/. 12. Some exceptions are Vectors and Eastgate. Vectors describes itself as a “journal of culture and technology in a dynamic vernacular.” It works with authors to review and publish interactive new media works: see www.vectorsjournal.org/. Eastgate publishes both hypertext editing tools and fiction created with their tools. See www.eastgate.com/. Some traditional publishers have also undertaken digital projects. See Driscoll and Scott. 13. See Boaz and Boaz. While CDs are no longer the favored way of distributing new media work (the Web is), this article tells the “narrative” of the project in a way that could help colleagues understand what it takes. 14. Because of the project orientation of much digital humanities work, it has be- come common in the field to negotiate charters at the beginning that make clear the rights and expectations of all parties, especially vulnerable parties like graduate student research assistants. Many digital humanists will therefore be used to such negotiations and understand their value. For more on charters, see Ruecker and Radzikowska. 15. See Bethany Nowviskie’s blog post on this subject, which has links and announces a forthcoming collection on the issue. 16. Reasons can range from there not being support for alternative academic posi- tions at the university to wanting to have an integrated position where someone teaches using technology, manages the instructional technology, and conducts pedagogical re- search around instructional technology. 17. A general pattern in the academy is that no one respects or budgets for the management of large projects. It is common to be expected to both manage the de- velopment of a digital project and do a lot of the work when, for example, student programmers graduate. I try to make it clear that just managing the digital component is a significant task and that support for the programming and other duties is needed. WORKS CITED | / Boaz, John K., and Mildred M. Boaz. “T. S. Eliot on a CD-ROM: A Narrative of the Production of a CD.” Computers and the Humanities 30 (1996): 131–38. Print. Chan, Leslie. “Supporting and Enhancing Scholarship in the Digital Age: The Role of Open-Access Institutional Repositories.” Canadian Journal of Communication 29.3 (2004): n. pag. Web. 1 Aug. 2011. Davidson, Cathy N. “Data Mining, Collaboration, and Institutional Infrastructure for Transforming Research and Teaching in the Human Sciences and Beyond.” CTWatch Quarterly 3.2 (2007): n. pag. Web. 1 Aug. 2011. ———. “Humanities 2.0: Promise, Perils, Predictions.” PMLA 123.3 (2008): 707–17. Print. Day in the Life of the Digital Humanities. Taporwiki, U of Alberta, 6 Apr. 2011. Web. 19 July 2011. Driscoll, Adrian, and Brad Scott. “Electronic Publishing at Routledge.” Computers and the Humanities 32 (1998): 257–70. Print. 168 ||| ON THE EVALUATION OF DIGITAL MEDIA AS SCHOLARSHIP S N 168 Galey, Alan, Stan Ruecker, and the INKE Team. “How a Prototype Argues.” Literary and Linguistic Computing 25.4 (2010): 405–24. Print. Guidelines for Evaluating Work with Digital Media in the Modern Languages. Modern Lan- guage Association. MLA, 2002. Web. 6 July 2011. Kiousis, Spiro. “Interactivity: A Concept Explication.” New Media and Society 4.3 (2002): 355–83. Print. Lawrence, Steven. “Online or Invisible?” Nature 411.6837 (2001): 521. Print. McCarty, Willard. Humanities Computing. New York: Palgrave, 2005. Print. Nature Pier Review Trial and Debate. Nature. Nature Pub. Group, Dec. 2006. Web. 8 July 2011. Nowviskie, Bethany. #alt-ac: Alternate Academic Careers for Humanities Scholars. Nowviskie, 3 Jan. 2010. Web. 11 July 2011. Blog. O’Reilly, Tim. “What Is Web 2.0: Design Patterns and Business Models for the Next Generation of Software.” O’Reilly. O’Reilly Media, 30 Sept. 2005. Web. 1 Aug. 2011. Report of the MLA Task Force on Evaluating Scholarship for Tenure and Promotion. Modern Language Association. MLA, 2006. Web. 6 July 2011. Robinson, Peter, and Kevin Taylor. “Publishing an Electronic Textual Edition: The Case of the Wife of Bath’s Prologue on CD-ROM.” Computers and the Humanities 32 (1998): 271–84. Print. Ruecker, Stan, and Milena Radzikowska. “The Iterative Design of a Project Charter for Interdisciplinary Research.” Proceedings of the 7th ACM Conference on Designing Interactive Systems DIS 08 (2008): 288–94. Web. 1 Aug. 2011. Siemens, Ray, and Susan Schreibman, eds. A Companion to Digital Literary Studies. Oxford: Blackwell, 2008. Alliance of Digital Humanities Organizations. Web. 4 Aug. 2011. Siemens, Ray, et al. The Credibility of Electronic Publishing. N.p., 2001. Web. 14 Mar. 2011. Sinclair, S., et al. “Peer Review of Humanities Computing Software.” Assn. for Liter- ary and Linguistic Computing and the Assn. for Computers and the Humanities. Athens, Georgia. 2003. Address.