College and Research Libraries Accessing Electronic Journals and Other £-publications: An Empirical Study Stephen P. Harter and Hak Joan Kim This article reports the results of a study of access problems and issues related to scholarly and peer-reviewed electronic journals (e-journals) currently found on the Internet. Data are reported on the accuracy of directory information, the accessibility of e-journal articles, the status of e-journal archives, methods used to retrieve current and back issues, and the variety of data formats used by e-journals. Data also are pre- sented on the accessibility of the texts of electronic publications refer- enced bye-journal articles. The findings illustrate the practical problems that can arise when users attempt to retrieve the texts of electronic pub- lications, and reveal relatively poor accessibility and usability of the e- journals studied. Implications for academic and research libraries are considered. he first scholarly journals, Journal des Sc;avans and the Philosophical Transactions of the Royal Society, began publica- tion in 1665.1 For more than three centu- ries, the journal has played a crucial role in the creation, transmission, and storage of knowledge, as the primary medium of communication among scholars. Despite its successes, however, in the past fifty years the print journal system has been subjected to many criticisms, perhaps the most serious of which are high costs, long delays in the publication process, and failures of peer re- view. 2 The problems of print journals and the concomitant development of computer and communications technology have led to the development of electronic alternatives to print journals-various forms of the electronic jour- nal, or e-joumal. This article reports empirical research conducted in the latter half of 1995 on access problems and issues related to net- worked e-journals, defined as e-journals that are delivered via, or accessed through, the Intemet.3 It is part of a larger study designed to assess the impact of e-jour- nals on scholarship and research by ex- amining references of, and citations to, e- journal '!rticles. 4 The Research Problem Before the introduction of the first graphi- cal World Wide Web (WWW) browser (Mosaic) in 1993, most e-journals were based on the listserv (electronic mailing Stephen P Harter is a Professor in the School of Library and Information Science, Indiana University; e-mail: harter@indiana.edu. Hak Joan Kim is a Doctoral Student at the School of Library and Information Science, Indiana University; e-mail: hajkim@indiana.edu. 440 list) technology operating on the Internet and Bitnet networks and distributing e- journal articles as ASCII text files. The Gopher or FTP protocols were often used as complementary mechanisms providing additional access to listserv-based e-jour- nals. Since 1993, ,the tremendous growth of the WWW has influenced both the number and distribution mechanisms of e-journals. Many e-journals formerly based on listservs and most newly es- tablished e-journals have migrated to the WWW, which allows journals to go far beyond simple ASCII text in the pro- vision of information. Journals pub- lished on the WWW can include hyper- text links connecting references, tables, and other parts of an article, as well as links to other Web files. In addition, they can use powerful new features such as graphics, sound, video, and even mini programs (e.g., Java applets). Web browsers use several Internet pro- tocols, including Gopher, FTP, and http (hypertext transfer protocol). In addi- tion, speed of publication can be greatly enhanced over that of print journals. Thus, although thorny cost, quality, archival, and preservation is- sues have not yet been resolved, there are clearly potential benefits of e-jour- nals over traditional print journals. Some commentators expect e-journals to supplant paper journals in the relatively near future. 5 To this point, vastly more speculation has been written about the potential of e- journals than empirical research con- ducted on existing ones. Although thou- sands of e-journals, newsletters, zines, and other electronic serials now are avail- able on the networks and interest in them has greatly heightened, little empirical research has been conducted.6 Hazel Woodward and Cliff McKnight argue that three levels of access toe-jour- nals must be considered. 7 The first is ac- cess to the bibliographic information on e-journals and publishers, including names and addresses. The second level Accessing Electronic J oumals 441 is access through indexing and abstract- ing services to information about the in- dividual articles published bye-journals. The third level is access to the actual texts of e-journal articles. Several e-journal directories are avail- able in both print and electronic form, 8 and some abstracting and indexing ser- vices have recently begun to index a few e-journals. 9 However, to the authors' knowledge, no empirical research has been published regarding access prob- lems and issues related to the texts of e-journal articles that are currently avail- To the authors' knowledge, no empirical research has been published regarding access problems and issues related to the texts of e-joumal articles that are currently available on the networks. able on the networks. Reliable access and usability seem crucial if e-joumals are to succeed as a new medium of formal schol- arly communication. A related issue is access to the texts of e-journal articles and other kinds of e-publications that have been cited in print or electronic journals. That is, when an author cites an e-publication, what can be said about the accessibility of that item, given the information provided in the ci- tation? Again, to the authors' knowledge, no research has been conducted on this question. The specific research questions ad- dressed in this article include: • To what extent can information provided in e-journal directories be used to retrieve the actual texts of articles? The issues addressed here are accuracy, cur- rency, and completeness of the informa- tion provided, as well as stability of the. locations of these files. • If e-journals cannot be located from the information provided in a directory en- try, can they be found through indepen- dent means (e.g., use of search engines)? · 442 College & Research Libraries • What kinds of problems do users encounter when they try to access e- journals? • To what extent do e-journal ar- chives exist? • How complete are existing e-jour- nal archives? • What methods are used to access current issues and archives of e-journals? • What data formats are used for e-journals? These will have implications for ease of use, printability, and so on. As part of the larger study, references in e-journal articles were checked as a measure of the impact of e-journals on scholarly communication. References to e-journal articles and other types of e-publications also were checked for ac- cessibility and type of resource. Thus, two additional questions are raised: • What are the frequencies and types of e-publications that are references in e-journal articles? • To what extent are references to e-publications in e-journal articles acces- sible to users, based on information pro- vided in the reference? Methodology The primary purpose of the larger study was to assess the impact of scholarly and peer-reviewed (or refereed) e-journals on scholarly communication by studying ci- tations to, and references in, e-journal ar- ticles as of the latter part of 1995. To con- duct the reference study, it was necessary to retrieve and study a sample of e-jour- nal articles. This gave the authors an op- portunity to experience firsthand the kinds of problems that occur when users try to accomplish the same task. Because the larger study dealt with scholarly and peer-reviewed e-journals, the authors needed to exclude other elec- tronic publications such as newsletters and zines. To define their population,· the authors chose to use two print directories that include information on electronic journals: Internet World's on Internet 94, published by Mecklermedia, and Direc- September 1996 tory of Electronic Journals, Newsletters and Academic Discussion Lists, published by the Association of Research Libraries (ARL). 10 All e-journals identified in either directory as peer-reviewed or refereed were included. Procedures followed to se- lect the sample (drawn in June 1995) were: • E-journals selected from the ARL directory were taken from the section en- titled "Electronic journals, magazines, and zines" and identified there as "peer- reviewed." All e-journals meeting these criteria became members of the sample. • E-journals selected from the Mecklermedia directory were taken from the section entitled "Electronic Journals and Newsletters" and identified there as "refereed." All e-journals meeting these criteria became members of the sample. 11 • The two lists were combined and duplicates removed. There were 131 scholarly and peer-reviewed or refereed e-journals in the final list. The authors at- tempted to access all ofthese to retrieve a sample of articles for study. Most of the e-journals in the sample use more than one access mechanism. The authors used all access methods provided by the ARL and Mecklermedia directo- ries to attempt to gain access to thee-jour- nal collections. For example, one e-jour- nal directory entry provided only the editor's e-mail address as an access method, whereas another provided the editor's e-mail address, uniform resource locators (URLs) for FTP, Gopher, and WWW, and a listserv address. The authors checked all the access points provided in each e-journal entry in the two directo- ries for each of the 131 e-journals in the sample. There were several occasions when the authors failed to gain access to the e-jour- nals using any of the access methods pro- vided. In such cases the authors used other methods. First, they used Internet search engines such as Infoseek, Web- crawler, Lycos, and Yahoo. In this way, some of the inaccessible e-journals were located. Second, the authors checked vari- ous e-joumal collection sites such as the one maintained by the Committee on In- stitutional Cooperation (CIC) in its Elec- tronic Journals Collection. 12 Third, the Accessing Electronic J oumals 443 TABLEt Most Frequently Used Combinations of Methods for Accessing E- journals authors attempted to browse (navigate) Combination of No. of in directories of the host server, based Methods E-joumals on partial addresses provided in the di- rectories. Although this technique is in- World Wide Web (WWW) alone 24 tuitive and time-consuming, it was gen- Listserv, FTP, Gopher, and WWW 9 Listserv and WWW 8 erally useful because many of the inac- 7 Listserv, FTP, and Gopher cessible e-joumals had incomplete URLs 7 Listserv, FTP, and WWW in their directory listings. Finally, when 6 Listserv and Gopher the authors failed to access an e-joumal 6 FTP, Gopher, WWW, and e-mail using all the techniques described above, 6 Gopher alone they attempted to contact its editor. Some- '--~---------------_J times this method also was successful. Through their best efforts to access all thee-journals in the sample, the authors identified the extent to which e-journals were accessible, the types of problems that actually occur when users try to ac- cess e-journals, and some possible causes of these problems. In addition, the au- thors investigated whether archives for thee-journal existed and, if so, their state . of completeness. Finally, the authors at- tempted to determine the life status of the e-journals in the sample. An e-joumal was operationally defined as "alive" if its ar- chives existed and contained at least one issue or article published in 1995 (as of September 1, 1995). Otherwise, the authors assumed it had ceased publication. Findings Access Methods and Data Formats Table 1 reports the most frequently used combinations of methods for accessing e- journals. Most e-joumals use two or more access/ distribution methods. Twenty-four e- joumals used the WWW as the sole ac- cess/ distribution method, making it the most popular e-journal distribution mechanism. Six e-journals used Gopher alone. This is not a surprising result when . the explosive growth of the WWW and the simultaneous drop in attention to Go- pher, which itself underwent explosive growth only a few years earlier, are con- sidered. Table 1 also shows the heavy use of list- serv software in combination with other distribution methods. Most e-journal publishers evidently recognize the limi- tations as well as the strengths of listserv- based systems and try to ·complement them with other methods. Alllistserv sys- tems maintain a mailing list of subscrib- ers. The table of contents and abstracts of an issue, or sometimes even the full texts of e-journal articles, are sent through e- mail whenever a new issue or article be- comes available. If an e-journal is avail- able only through the listserv, accessing e-journal issues, especially back issues, is somewhat difficult, particularly for non- subscribers.13 The procedures can be time- consuming and require a fair amount of knowledge on the part of users. On the other hand~ to be notified of current is- sues or articles via e-mail is probably the most convenient of all approaches. Thus, it is not surprising to find listserv soft- ware used in combination with other dis- tribution methods. Accepting the premise that a major func- tion of scholarly e-journals is "to make public the results of original research to the widest possible audience,"14 it follows that several methods should be used to provide access so that users can choose the meth- ods most suitable for them. Nine e-joumals use all four of the most common e-journal distribution methods: listserv, FTP, Gopher, and WWW. 4~6 College & Research Libraries September 1996 TABLE2 and can easily be transmitted Access Methods for Current and Archival Issues of E-journals (N-125)* using the straightforward e- mail and listserv technolo- gies. HTML (Hypertext Mark- Access Method No. of E-journals Using Method Percentage of Total up Language) and PostScript are the second and third most commonly used data formats. HTML is the standard lan- guage used on the WWW for creating and displaying hyper- media documents. PostScript is Adobe's page description language used for delivering and printing complex docu- me_nts that include layout, font descriptions, and graph- ics over the Internet. www Gopher FTP Listserv E-mail Guidon Paper (for current issue only) Telnet to freenet 83 53 52 48 21 7 3 66.4% 42.4 41.6 38.4 16.8 5.6 2.4 0.8 *A given e-journal may provide several access methods. Table 2 shows that about two-thirds of the e-journals in the sample used the WWW as at least one distribution Table 3 also shows that various data formats necessary for mul- timedia presentations, such as DVI, GIF, MPEG, and QuickTime, have begun to be method. Gopher, FTP, and listserv occupied the second, third, and fourth ranks, respectively. In ad- dition, it should be noted that e-mail, which is relatively primi- tive compared to other e-journal distribution methods, is still used as a distribution method by one- sixth of the e-journals studied. Table 3lists the very diverse data formats used by thee-journals in the sample. Among these, the simple ASCII text file was the most commonly used. ASCII is limited in terms of its information representation ability because it cannot handle graphics or special characters such as mathematical symbols. On the other hand, for certain scholarly forms (e.g., es- says) and for the bulk of research and scholarship in many fields (e.g., English and sociology), ASCII is perfectly adequate and perhaps preferable. Plain ASCII texts also have an obvious merit in that they can be easily read and manipulated by all kinds of com- puter hardware and software, · TABLE3 Data Formats Used byE-journals* Data Format No. of E-jouma1s Adobe Acrobat PDF (Portable 6 Document Format) ASCII (American Standard Code for 80 Information Interchange) DVI (Digital Video Interactive) 6 GIF (Graphics Interchange Format) 6 Guidon 7 HTML (Hypertext Markup Language) 62 Hypercard 2 hyperTeXdvi 1 JPEG (Joint Photographic Experts Group) 2 LaTeX 2 Microsoft Word 2 MPEG (Moving Pictures Experts Group) 1 PostScript 24 QuickTime 1 RTF (Rich Text Format) 4 TeX 8 TIF (Tagged Image File) 1 Word Perfect 3 *A given e-joumal may provide data in several data formats . Accessing Electronic Journals 447 TABLE4 problem. If e-journal articles can- no~ be accessed by the target audi- ence when they are needed, their value is diminished. Unstable or Success Rates in Connecting to E- journal Archives* Connection Result Connected first time for all methods given in the directories Failure to connect for one or more methods No. of E-joumals 58 71 Percentage of Total 45 .0% 55.0 limited access toe-journals cannot help but affect their acceptability among scholars as a legitimate medium of formal scholarly com- munication. Table 5 reports the results of the *Two fee-based e-joumals could not be accessed because the authors had no subscription. authors' failures to connect to e-journal archives. More than 40 percent of the e-journals in the sample had one or more identifi- able problems with the data pro- vided in one or both directories. used by e-journals. The kinds and num- bers of data formats used for multimedia presentation indirectly reflect the impact of the WWW on e-journals. With the exception of ASCII text, all the data for- mats listed in table 3 require special soft- ware, and in some cases hardware, to view or use the document or document part. This fact can have a significant im- pact on user accessibility (and on a library's ability to provide access for us- ers to these files). Nearly 10 percent of thee-journals could not be accessed at all through the URLs listed, although the authors tried to con- nect to each on at least three different days and times. Eleven (8.6%) of the e-journals had moved their locations, but only six of these provided a "move no- tice" on their old sites that included a new address (URL). Five e-journals moved but provided no information. In such cases, the authors were able to identify the new location through their own efforts, for ex- ample, by using Internet search engines or via intuitive navigation of host serv-Accessibility of E-journal Articles Table 4 shows the success rates the authors achieved in connecting to thee-jour- nal archives using all access methods listed in the ARL and Mecklermedia directo- ries. Less than half the e-journals in the sample could be accessed on the first try using all the meth- ods and addresses pro- vided. This is a disappoint- ing result, because one of the most important advan- tages claimed for e-jour- nals compared to print journals is speed of access. Such low success rates in accessing e-journals must be considered a serious TABLES Summary of Failures in Attempts to Connect to Archives* Type of Problem No. of Percentage E-joumals of Total Identifiable problems in directory data 55 No response to listserv or e-mail request 9 Failure to connect to Gopher URL 6 on three or more different days Location has moved, new address given 6 Location has moved, no new address given 5 Failure to connect to Web URL on three 3 or more different days Failure to connect to FTP URL on three 3 or more different days 42.6% 7.0 4.7 4.7 3.9 2.3 2.3 Listserv responded but did not accept 0.8 subscription to e-journal *More than one result is possible for a given e-journal. N=71. 448 College & Research Libraries TABLE6 Problems with Data Provided in the Print Directories* Type of Problem Part of URL was missing MissingURL Typographical error No. ofTimes Problem Occurred Incorrect listserv command given URL was entirely incorrect 20 14 9 9 9 3 2 Part of URL was incorrect Incorrect e-mail address September 1996 from relatively easy to find to very difficult. For example, "SUB- SCRIBED" instead of "SUBSCRIBE" was given as a listserv command, a relatively easy error to diagnose (but not, perhaps, for a naive end user). In another case, "utsas" instead of "utas" was given as a part of a URL; this was very difficult to detect. Even though these look like very minor er- rors, access to the e-journals failed because of them. URLs must be cor- rect in every aspect, including case, or the connection will fail. "Minor" * More than one problem may be associated with a typographical errors can have dev- given e-joumal. L.._ _________________ __. astating consequences. ers. In addition, nine (7%) of thee-jour- nals did not respond in any way to the authors' listserv or e-mail requests. For these e-journals the authors can conjec- ture that either they have ceased publica- tion, no longer distribute issues using e-mail, or have a new editor (with the former editor no longer answering his or her e-mail). Nearly half the problems the authors experienced in connecting to the archives of thee-journals in the sample could be traced to inaccurate data in the print di- rectories (see table 6). For twenty e-jour- nals, parts of the URLs were missing. The authors were often able to identify the cor- rect URLs by navigating the sites using the partial information provided in the directory entry. In fourteen cases, the authors discovered a Gopher, FTP, or Web location for an e-journal through In nine directory entries, an incor- rect listserv command was provided. For example, one of the directories gave the subscription command syntax "subscribe " instead of "subscribe " for subscribing to a list controlled by the Listproc listserv. The command is necessary in Listproc, but optional in Listserv. In nine additional cases, the URL pro- vided was entirely incorrect, in many cases probably due to the archive having been moved to a completely different lo- cation. Other less frequently observed problems in the directory listings are given in table 6. Another important issue that should be considered in determining the acces- sibility of e-journals is the completeness TABLE7 Completeness of Archives their own efforts, but this informa- State of Archives No. of Percentage tion was not provided in the rel- evant directory entry. These situa- tions almost certainly resulted from the rapidity of change-from an e- Archives were complete for all methods journal adding a Gopher or Web Archives were incomplete site after the directory information for one or more methods had been gathered and published. Impossible to evaluate The authors found nine typo- completeness of archives E-joumals of Total 87 66.4% 28 21.4 16 12.2 graphical errors in the e-journal Total listings. The types of errors varied ...___ __________________ _. 131 100% of the archives. Many of the e-journals provide several different access methods, and also maintain several different forms of archives based on these methods. Table 7 reports the authors' findings on the completeness of the archives maintained by each e-journal in the sample. For two- thirds of the e-joumals, all archives were complete. More than one-fifth of the e-journals had at least one incomplete ar- chive, and the authors could not deter- mine the completeness of the archives of the remaining sixteen. To publish their articles as quickly as possible, some e-journals have no concept of issue, though they usually retain the concepts of volume and/ or publication year. For these e-journals, articles are individually distributed (published) as they are ac- cepted. Such a mechanism, which is unique to e-journals, speeds up the pub- lication process because there is no need to wait until enough articles are ready before publishing an issue. However, this made it impossible to identify whether articles were missing from the e-joumal archives, because typically no index to published articles was provided. Table 8 reports the life status of the e-joumals in the sample. About a quarter of the e-journals had ceased publication, according to the authors' operational defi- Accessing Electronic J oumals 449 subscription fee or is free can affect the extent to which libraries may be asked for assistance in providing access to it. The authors identified eight pricing models for the e-joumals in the sample (see table 9). In all, only thirteen (10.4%) of the 125 e-journals in the sample for which the fee question could be determined charged a subscription fee. The authors tested the relationship be- tween several access variables and whether a subscription fee was charged, using the chi-square test. A significant relationship was found between the suc- cess the authors had in connecting to the e-journal archives (see table 4) and whether a subscription fee was charged. Free e-journals were significantly more likely to have access problems (x 2= 11.172, p = .002). References to £-publications One of the main purposes of the larger study was to analyze the references from a sample of articles in scholarly and peer- reviewed e-joumals in order to measure the extent to which authors of e-journal articles are currently citing e-joumals and other online sources. Such authors can be assumed to be more knowledgeable about, and sympathetic to, the electronic media than typical authors. Thus, the re- nition of death. Twenty-eight .----------'---------------, TABLES e-joumals had archives but had published no 1995 issue (or ar- ticle). Four e-journals could not Life Status of E-journals in the Sample be accessed at all. In addition, the Life Status No. of E-journals authors could not determine the life status of four e-journals. Two of these were fee-based e-joumals for which the authors had no sub- scription. The remaining two had no concept of volume, issue, or publication year. Although there were articles stored in the ar- chives, it was impossible to tell when they had been placed there. Archives and one issue in 1995 exist [living] Archives exist but no issue published in 1995 [probably ceased publication] 95 28 No connection could be made 4 [probably ceased publication] Could not determine life status 4 E-journal Pricing and Access Total 131 Percentage of Total 72.5% 21.4 3.1 3.1 100% Whether an e-journal charges a L----------------------' 450 College & Research Libraries September 1996 TABLE9 Pricing Arrangements for E- journals If There Is No Print Countemart (N=JO]) Pricing Arrangement No. of Percentage in Category E-joumals All electronic issues are free 91 90.0% 7.9 1.0 Fee for both archival and current issues 8 Fee for current issue; archival issues are free Subscription to electronic version is included with membership in a society 1.0 I[_ There Is a Print CounterJlarl (N=24) Pricing Arrangement No. of Percentage in E-joumals Category Electronic version is free ; 16 66.7% fee for print version Fee for both electronic and print versions 4 16.7 Electronic version is free ; cannot tell 2 8.3 about print version Electronic version is free ; subscription 2 8.3 to print version is included with membership in a society Total (fee charged) Total (free) suits of the reference study should provide an upper bound on the current influence of e-journals on scholarly communication. Among the original sample of 131 e-journals, the authors determined through examination of sample issues that seventy-four of these were schol- arly and peer-reviewed, were acces- sible to them, and published articles with references. The authors retrieved and printed the last four such articles published by each e-journal. 15 Six of the seventy-four e-journals had published fewer than four articles as of Septem- ber 1995, when the data collection took place. In these cases the authors stud- ied all the available articles. From the most recent 279 articles pub- lished in the seventy-four scholarly and peer-reviewed e-journals, there were a to- tal of 4,317 references. The authors clas- sified each reference as to its format 13 10.4% 112 89.6% (book, serial, book chapter, online source, etc.). They further classified the online sources as to subtype (Web page, e-mail, e-journal article, etc.). Finally, the authors tried to retrieve the texts of the referenced online sources. A full discussion of the methodology and findings from the ref- erence study is given in Harter and Kim. 16 Of particular interest here is the result of the authors' attempts to access the online publications that were cited in the sample of e-journal articles. A total of eighty-three online sources (1.9% of the 4,317 references) were cited. Table 10 shows the types and frequencies of references to online sources, arranged by the content of the source (Web page, listserv posting, etc.). Web pages, elec- tronic personal papers, e-mail messages, and e-journal articles were the most com- monly referenced electronic sources. The authors found many of the references to Accessing Electronic Journals 451 TABLElO Types, Number, and Accessibility of Cited Online Sources Type of Online Source No. of References No. and Percentage of Accessible* References Web Page 12 10 (83.3 %) Electronic Personal Paper 10 7 (70.0) E-mail Message 9 2 (22.2) E-joumal Article 9 6 (66.7) Newsgroup Posting 7 0 (0.0) Listserv Posting 5 2 (40.0) Electronic Directory 5 5 (100.0) SGML Encoded Document 5 5 (100.0) Electronic Preprint 3 1 (33.3) Computer Software 2 2 (100.0) Electronic Newspaper 2 2 (100.0) Online Catalog 1 1 (100.0) Local File 1 0 (0.0) Type could not be determined 12 0 (0.0) Total 83 43 (51.8) *A reference is accessible if it leads directly to the text cited given only the information found in the citation. online sources to be inconsistent, incom- plete, and/ or inaccessible. Many did not lead to the text of the cited online source. The authors could not determine the types of twelve online references (14.5%) because bibliographic information was incomplete and because the online sources were not accessible, making fur- ther investigation impossible. For ex- ample, some of the online references pro- vide only a URL and no other biblio- graphic information. In these cases, if the authors could not access the cited refer- ence through the given URL, it was usu- ally not possible to determine the type of online source being cited. The authors studied each of the eighty- three references and tried to access the text of the cited material. One of the great theoretical advantages of references to on- line sources is that the reader can retrieve the full texts of the cited sources very quickly (ideally, by clicking on the refer- ence itself). The third column of table 10 shows the results of our attempts tore- trieve the texts cited by the eighty-three references. Only six of the nine e-journal articles cited were accessible. E-mail mes- sages and newsgroup posl:ings were es- pecially inaccessible, and none of the twelve sources whose types could not be determined could be retrieved. Overall, only about half of the texts of the cited online sources could be accessed. These findings are analyzed in another way in table 11. Online information sources can be obtained using several dif- ferent access protocols. Forty-seven of the cited references included a URL. The au- thors used Netscape, a Web browser, to attempt to access these URLs. However, only two-thirds of the URLs led to the text of the source. For the remaining third, the authors obtained the browser message "The server does not have a DNS entry" or the server error message "404 URL Not Found." Several types of the online ref- erences listed in table 10 (e.g., Web pages, / electronic personal papers, SGML- encoded documents, software programs, and local file references) were usually cited in this way, often only with a URL. 452 College & Research Libraries September 1996 TABLE 11 commercial service (to which Access Protocols and Accessibility for the 83 Online References not all Internet users will have access), none of the methods led to perfect re- Access Protocol No. of Online No. and% of trieval. The authors consider the overall rate of 51.8 per- cent to be extremely low, given that the online sources were apparently considered important enough for an au- thor to cite; and yet in just one or two years, nearly half of the referenced texts were not accessible online. The percentage of successfully ac- References Accessible References URL 47 31 (66.0%) E-mail information 13 4 (30.8) Listserv information 9 6 (66.7) Usenet newsgroup 7 0 (0.0) Electronic newspaper 2 2 (100.0) Incomplete access 5 0 information provided Total 83 43 For online sources in which e-mail information was provided, the authors tried to contact the source using e-mail. Only four of these references led to the online source cited. The authors received the mail server message "User Un- known" in seven cases. In two other in- stances, there was no response to the au- thors' message. Nine of the online references were to · listserv postings. The authors were able to access six of these by using the listserv ac- cess information provided in the reference (e.g., the listserv name and posting date) and by using listserv commands. These required a fair amount of knowledge and effort. The authors' failures were due to the lack of the existence of listserv ar- chives or archives for the time period in question. This also was the explanation for the authors' inability to find the seven cited Usenet newsgroup postings. Two electronic newspaper articles were cited. The authors were fortunate to have ac- cess to Lexis/Nexis, and found the cited articles that way. Finally, table 11 indicates that five on- line references fell in the category "incom- plete access information provided." In these cases, there was insufficient infor- mation for the authors even to attempt to gain access to the source. ·Table 11 shows that, except for elec- tronic newspapers available through a (0.0) (51.8%) cessed material would be even lower if the authors required trans- parent accessibility-that is, if texts found through search engines or through knowledge of arcane listserv commands were considered to be in a different cat- egory than those available simply by clicking on a URL. Table 12 summarizes the accessibility findings when the data are viewed in this way. Table 12 shows that only 37.3 percent of the references in the sample provided transparent (clickable) access to the text of a cited reference. In another 14.5 per- cent of the references, the text could be found, but only with significant effort, as described above. Nearly half of the refer- ences did not lead to the cited text even when significant effort was expended. Direct accessibility to cited e-journal articles and other cited online sources would obviously be extremely convenient to readers and would offer a great advan- tage to e-journals over print. Equally, if readers cannot access original sources of referenced material, references to elec- tronic sources are much less useful than references to print sources. The authors should emphasize that the references studied appeared in recently published e- journal articles-most during the year in which the data gathering took place. The authors might reasonably expect the great majority of such references to be largely correct and complete---at least enough to lead the reader to the text of the refer- enced source. This was not what the au- thors found; only half of the references were accessible, and only somewhat more than a third provided transparent access. When more time has passed, these propor- tions are likely to be much lower. The au- thors conclude that the accessibility of cited online sources is potentially a very serious problem for e-joumals and for the citation of electronic publications more generally. Conclusions The findings of this study illustrate the · practical problems that occur when us- ers attempt to access the texts of articles published in scholarly and peer-reviewed e-joumals that are currently available on the Internet. The problems the authors identified raise several important issues that libraries should consider. If academic and research libraries are to provide users with access to the texts of e-journals, appropriate hardware and software is needed. Many of the e-jour- nals in the sample employ two or more access and distribution methods, and some include data formats such as PostScript, DVI, and TeX, as well as au- dio and video capability, for rich infor- mation representation. Many users will not be able to fully utilize such e-journals on their own computers for lack of their own appropriate hardware Accessing Electronic Journals 453 One of the troubling findings of this study was with the information found in the two (print) e-journal directories the authors used. More than half of the e-journals in the sample could not be ac- cessed by one or more methods given in the directories, despite the recent publi- cation dates of the directories. The most frequent reason for access failure was in- accurate directory information. The au- thors assume that the main cause of inac- curacies is that the directories, which are compiled and published annually, cannot keep up with rapid changes in the rel- evant information. As long as growth and change in e-journals continue to be ram- pant, the print medium may not be well suited for an e-journal directory. In fact, the ARL directory is available in both print and electronic form. However, the electronic version, which is available on a Gopher site, 17 is an abridged version of the annual print e-journal directory and is not continually updated. In this sense, it does not take full advantage of its elec- tronic character. To keep up with rapid changes in e-journals, up-to-date infor- mation should be readily available-and that implies that the directories them- selves should be on the Internet and in a state of continuous revision. Another important issue is the need to create and maintain e-journal ar- chives. A full 55 percent of thee-jour- or software. Just as academic ...-------------------------, libraries subscribe to print journals for those users who are not able to individually subscribe to many journals in TABLE12 Summary of Accessibility Results for References to E-publications their fields, they also will need Result of No. of % of to provide users with appro- _A_cc_e_s_si_bi_li....:ty_ll_e_st_s _______ R_e_fe_re_n_c_es __ R_e_fe_re_n_c_es priate equipment and soft- Transparent (clickable) access ware necessary not only for to text of cited reference accessing e-journals, but also Text of cited reference findable for printing articles and ma- with significant effort, given the information provided in the citation Text of cited reference not findable nipulating related files. In ad- dition, user training will be necessary for many users if they are to effectively use the given the information provided in the citation facilities provided. 31 37.3% 12 14.5 40 48.2 454 College & Research Libraries nals in the sample could not be ac- cessed on the first try by one or more methods listed in the directories. More than half of the access failures were due to the problems of thee-journals them- selves, such as servers being down or server location changes. Moreover, about one-fifth of the e-journals had in- complete archives and almost a quarter had probably ceased publication. The au- thors' data raise serious questions con- cerning the permanence and reliability of e-journal archives. 18 Commercial publishers also will be- come much more involved. This article has dealt with scholarly peer-reviewed e-journals that were in existence as of June 1995 (as given in the ARL and Meckler- media directories used to draw the sample). These did not include any of the major commercial publishers, such as Elsevier Science, Springer Verlag, John Wiley & Sons, Blackwell Science Limited, Academic Press, and Taylor and Francis, who are just now making their presence known. 19 The authors are sure that some of the access problems noted here will be Index to advertisers ACRL 480,492 Archival Products 427 Assoc. of Christian Libs. 483,490 Bios is 414 Bowker 401 EBSCO cover 2 Elsevier Science 458 Greenwood Publishing 491 lSI 478 Libraries Unlimited 404 Library Technologies 457 McGraw-Hill 439 Minolta 444-45 OCLC Online Computer 438,cover3 PAIS 397 Readmore 476 Systems Research 398 Todd Enterprises cover 4 H. W. Wilson 428 September 1996 addressed by these large fee-based pub- lishers, who will probably try to main- tain complete and easily accessible ar- chives. The situation is not so clear for the smaller publishers. Indeed, the authors found significantly more access problems with free e-journals than with fee-based ones. This implies that smaller e-journal publishers need to give more attention to quality control issues, especially those involving access. It also suggests that the idea of free e-journals may be an illusion that cannot be sustained. 20 Academic libraries have traditionally performed archival functions for serial publications that publish academically relevant articles, regardless of format. E-journals should not be an exception. Precisely how and in what formate-jour- nals should be archived by academic and research libraries-paper, microfilm/ fiche, floppy disk, CD-ROM, or on local servers-remains to be determined by the individual libraries or cooperative enti- ties that determine to maintain archives. But without such archives, the unreliable. access toe-journals that the authors have reported makes their evolution into a new medium of scholarly communication somewhat problematic. The traditional scholarly journal sys- tem was not built in a day. The transfor- mation of print scholarly journals to their present form and function has been an ongoing process for more than two cen- turies. The journal has had to continu- ously evolve in order to best serve its role as the primary medium of scholarly com- munication. On the other hand, the schol- arly e-journals that began to be available on the networks in the late 1980s have had a relatively short history, and academic libraries have grappled with them for an even shorter period. E-journals are still in a very early stage of development, and the many issues related to them are in constant change. Nonetheless, e-journals have unique advantages over print jour- nals and offer the potential to revolution- ize the traditional scholarly communica- tion system that heretofore has been dominated by paper journals. The change in the scholarly communication system brought by e-journals has already begun and will probably be profound. Electronic journals are, of course, only one type of e-publication, though certainly one of the most important ones from the perspective of a research library. The study of references to e-publications reported here shows that access to the texts of the cited material is no more reliable than ac- cess to e-journal articles. Potential solutions to this problem are not apparent because there are vastly more citable e-publications on the WWW that are note-journal articles Accessing Electronic J oumals 455 than those that are. Building a framework to approach this knotty problem is extraor- dinarily challenging. Yet traditionally aca- demic libraries have played a crucial role in the scholarly communication process, and cannot ignore the access problems and issues the authors have discussed. It is time to actively attack them. The authors wish to thank Margaret Harter, Debora Shaw, and Taemin Park for their thoughtful comments and questions on an earlier version of this manuscript, and are grateful to OCLC, Inc., for partial funding of the larger study of e-journals of which this report is a part. Notes 1. Charles B. Osburn, "The Place of the Journal in the Scholarly Communications System," Library Resources & Technical Services 28, no. 4 (Oct./Dec. 1984): 315-24. 2. For discussions of the traditional journal system, see Bernard Houghton, Scientific Peri- odicals: Their Historical Development, Characteristics and Control (Hamden, Conn.: Linnet Books, 1975); Fritz Machlup and Kenneth Leeson, Information through the Printed Word: The Dissemina- tion of Scholarly, Scientific, and Intellectual Knowledge, vol. 2, Journals (New York: Praeger, 1978); and Donald W. King, Dennis D. McDonald, and Nancy K. Roderer, Scientific Journals in the United States (Stroudsburg, Penn.: Hutchinson Ross Publishing, 1981). 3. Woodward divides electronic journals into three types: online (available from a commer- cial vendor such as DIALOG or Lexis/Nexis), CD-ROM, and networked (available directly on the Internet or other academic network). See Hazel Woodward, "The Impact of Electronic Infor- mation on Serials Collection Management," IFLA Journal 20, no. 1 (1994): 35-45. However, be- cause DIALOG and other vendors are now searchable on the Internet, these are no longer mutu- ally exclusive categories. Operationally, networked e-journals are defined in this article by the inclusion criteria used by the two directories the authors ].!Sed to draw their samples: Mecklermedia's Internet World's on Internet 94 and ARL's Directory of Electronic Journals, Newslet- ters and Academic Discussion Lists (see note 8 for full citations). Neither directory includes e- journals available through a commercial vendor such as DIALOG. 4. Selected findings from the full study are found in Stephen P. Harter and Hak Joon Kim, "Electronic Journals and Scholarly Communication: A Citation and Reference Study'' (paper presented at midyear meeting of the American Society for Information Science, San Diego, Ca- lif., May 18-22, 1996). The text of the paper is available at URL: . The portion of the study presenting the results of sev- eral citation analyses of selected e-journals is reported in Stephen P. Harter, "The Impact of Electronic Journals on Scholarly Communication: A Citation Analysis," Public-Access Computer Systems Review 7 (1996) [in press]. PACS Review is available at URL: . 5. Andrew M. Odlyzko, "Tragic Loss or Good Riddance? The Impending Demise of Tradi- tional Scholarly Journals," International Journal of Human-Computer Studies 42 (1995): 71-122; Stevan Hamad, "The Postgutenberg Galaxy: How to Get There from Here," Information Society 11 (Oct./Dec. 1995): 285-91. 6. For exceptions, see Blaise Cronin and Kara Overfelt, "£-journals and Tenure," Journal of the American Society for Information Science 46, no. 9 (Oct. 1995): 700-03; Hans Roes, "Electronic Journals: A Survey of the Literature and the Net" (Dec. 1994), available at URL: ; Steve Hitchcock, Leslie Carr, and Wendy Hall, "A Survey of STM Online Journals 1990-95: The Calm before the Storm" (Jan. 1996), available at 456 College & Research Libraries September 1996 URL: ; and F. W. Lancaster, "Attitudes in Academia toward Feasibility and Desirability of Networked Scholarly Publishing," Library Trends 43 (spring 1995) 7 41-52. 7. Hazel Woodward and Cliff McKnight, "Electronic Journals: Issues of Access and Biblio- graphical Control," Serials Review 21, no. 2 (summer 1995): 71-78. A useful analysis of access options is provided by Robert D. Cameron, "To Link or to Copy? Four Principles for Materials Acquisition in Internet Electronic Libraries," available at URL: . 8. Tony Abbott, ed., Internet World's on Internet 94: An International Guide to Electronic Jour- nals, Newsletters, Texts, Discussion Lists, and Other Resources on the Internet (Westport, Conn.: Mecklermedia, 1994); Ann Okerson, ed., Directory of Electronic Journals, Newsletters and Academic Discussion Lists, 5th ed. (Washington, D.C.: ARL, 1995) (a truncated version of this is available at URL: ); "New Jour: Electronic Journals and Newsletters" (the Internet list for new journals and newsletters available on the Internet), avail- able at URL: . 9. For example, the Institute for Scientific Information currently indexes four e-journals in its databases: Chicago Journal of Theoretical Computer Science, Journal of Artificial Intelligence Re- search, Gene-Combis, and Online Journal of Knowledge Synthesis for Nursing. Two additional titles are targeted for inclusion in the near future. Source: E-mail message from Eileen Curtin, Edito- rial Development, Institute for Scientific Information, to Stephen P. Harter, Feb. 12, 1996. 10. As of June 1995, when the study began, the 1995 edition of the Mecklermedia directory had not been published. 11. By inspecting the e-journals themselves, the authors later reduced this list to exclude many e-journals, zines, and newsletters that in their judgment did not report the results of re- search or scholarship, even though they identified themselves in one or both directories as peer- reviewed or refereed. The reference study was conducted on this smaller sample. 12, The CIC Electronic Journals Collection is available at URL: . Other collections are "NewJour: Electronic Journals & Newsletters," available at URL: http:/ I gort.ucsd.edu/newjour />;and "Scholarly Journals Distributed via the World Wide Web," avail- able at URL: . 13. There are severallistserv software packages, including Majordomo, Listproc, Mailbase, and Listserv. Both the available features and commands differ significantly among these pack- ages. Some listserv software does not allow the archiving, searching, or retrieving of files. The commonly used and venerable Listserv software by Eric Thomas is fully featured and powerful, albeit somewhat difficult to use. First, a nonsubscriber must send a subscription command to the Listserv system and then wait (for varying lengths of time) until getting an acceptance mes- sage from the system. Some listserv systems require a confirmation response of the subscription request. After the acceptance message has been received, the newly subscribed user must send an "Index" command to identify the name of the available files. After likely file names are iden- tifiedt he or she must send a "Get" command to obtain the file. In the present article, lowercase "listserv" refers to all the listserv software packages and uppercase "Listserv" refers to Eric Thomas's product. 14. Jill Lambert, Scientific and Technical Journals (London: Clive Bingley, 1985). 15. The authors are grateful to OCLC, Inc., and to the named e-journal publishers for provid- ing temporary access to the following fee-based e-journals for purposes of conducting this study: Applied Physics Letters Online, Current Opinions in Medicine, Current Opinions in Biology, Immunol- ogy Today Online, Online Journal of Current Clinical Trials, and Online Journal of Knowledge Synthe- sis for Nursing. 16. Harter and Kim, "Electronic Journals and Scholarly Communication." 17. See note 8. 18. Citation in a networked information environment and establishing trusted channels of distribution are two of the "integrity issues" discussed by Clifford A. Lynch in "Integrity Issues in Electronic Publishing," in Scholarly Publishing: The Electronic Frontier, Robin P. Peek and Gre- gory B. Newby, eds. (Cambridge, Mass.: MIT Pr., 1996), 133-45. 19. See Gary Taubes, "Science Journals Go Wired," Science 271 (Feb. 9, 1996), 764-67. Also available at URL: . 20. For a thorough discussion of many economic issues related toe-journals, see Ellen Finnie Duranceau, ed. (with contributions by Stevan Hamad, Frank Quinn, Gail McMillan, Fytton Rowland, Robert H. Marks, and Janet H. Fisher), "The Economics of Electronic Publishing," Serials Review 21 (spring 1995): 77-90. A New Service on the Information Superhighway If you have been searching for an easy way to authority control your library's current cataloging, try LTI'sAuthority Express service. With Authority Express, a library uses the Internet to transmit a file of newly cataloged bibliographic records to LTI (via FTP). Overnight, LTI processes the records through its state-of-the-art authority control system. Then, at the library's convenience, it logs into LTI's FTP server to retrieve fully authorized catalog records, along with linked LC name and subject authority records. Authority Express • Keeps authority control current at an affordable price • Integrates easily into existing workflows • Lowers cost by reducing staff time spent on catalog maintenance • Provides next-day turn around for up to 5,000 catalog records • Accepts records for processing even if LTI did not perform the original authority control c'Authority Control for the 21st Century'' • LIBRARY TECHNOI.OGIES, INC. 2300 Computer Avenue, Suite D-19 Willow Grove, PA 19090 (215) 830-9320 Fax: (215) 830-9422 (BOO) 795-9504 email: LTI@LibraryTech.Com Announcing a Nevv Major Reference Work Editors-in-Chief: Alan R. Katritzky, FRS. USA Charles W. Rees. CBE, FRS, Ul Eric F. V. Saini. Due Decentber 1996: This new set has two major aims: to report on the enormous advances in the subject in the last decade and to extend the coverage into areas where CHEC left significant gaps. The latter includes particular- ly saturated and partially saturated compounds, rings containing less common heteroatoms, and many classes of bicyclic and polycyclic systems. Material included in CHEC is, for the most part, not repeated in CHEC II and the two works are designed to be used together. Free Prospectus Available- Send for Your Copy Now! Volume 1: Three-and four-membered rings, with all fused systems containing three and four membered rings Volume 2: Five membered rings with one heteroatom and carbocyclic fused derivatives Volume 3: Five-membered rings with two heteroatoms and carbocyclic derivatives Volume 4: Five-membered rings with more than two heteroatoms and fused carbocyclic derivatives Volume 5: Six-membered rings with one heteroatom and fused carbocyclic derivatives Volume 6: Six membered rings with tow or more heteroatoms and fused carbocyclic derivatives Volume 7: Fused five- and six-membered rings without ring junction heteroatoms Volume 8: Fused five- and six- membered rings with ring junction heteroatoms Volume 9: Seven membered and larger rings and fused derivatives Volume 10: Author and Ring Indexes UK & ALL OTHER COUNTRIES: Elsevier Science Ltd The Boulevard, Langford Lane, Kidlington, Oxford OX5 I GB UK tel. 44 (0) 1865-843685 fax 44 (0) 1865-843946 NORTH AMERICA: Elsevier Science Regional Sales Office 655 Ave of the Americas New York, NY 10010 USA tel. 212-633-3730 fax 212-633-3680 PERGAMON Volume 11: Subject Indexes 1..-------------------------1 An imprint of Elsevier Science