natowitz.p65 322 College & Research Libraries July 1997 322 ook reviews have repeatedly been identified as major deter- minants in the library acquisi- tions process.1 However, despite the widely acknowledged importance of book reviewing, researchers have indicated the need for more critical evaluations of re- views.2 The authors examined this subject by conducting a comparative analysis of Ameri- can history book reviews in three highly regarded journals. Among them, Choice is identified as the primary review me- dium for academic libraries. The other publications, Journal of American History (JAH) and American Historical Review (AHR), are two of the most important journals for specialists in U. S. history. The authors set out to determine the degree to which book reviews in AHR, JAH, and Choice provide collection development specialists with qualitative assessments on which to base their acquisition choices. To achieve this, the contents of 459 book reviews (of 153 American history books reviewed in each of the three journals) were analyzed. In addition to the overall opinion expressed in each review, the authors’ analysis considered the ways in which each of seven evaluative criteria was employed in these reviews. Evaluating Review Content for Book Selection: An Analysis of American History Reviews in Choice, American Historical Review, and Journal of American History Allen Natowitz and Paula Wheeler Carlo This study applies content analysis to compare assessments of 153 titles in American history, each reviewed in Choice, American Historical Review (AHR), and Journal of American History (JAH) between 1988 and 1995. A numerical rating system is used to quantify reviewers’ over- all opinions and their use of specific evaluative criteria. Study findings include: most books were praised in all three journals; the mean overall rating was highest in Choice and lowest in JAH; little agreement existed in identifying either outstanding titles or inferior titles; reviews in Choice contained the fewest evaluative criteria, but appeared 8.5 to 10.6 months prior to reviews of the same titles in JAH and AHR. Allen Natowitz is an Assistant Professor in the Library Department at the College of Staten Island; e- mail: natowitz@postbox.csi.cuny.edu. Paula Wheeler Carlo is an Instructor in the History Department at Nassau Community College. The authors wish to thank Professor Ivan Smodlaka of the College of Staten Island for his generous assistance and expert guidance in the statistical component of this study. Evaluating Review Content for Book Selection 323 Review of Literature Some studies of the general characteris- tics of American history book reviews al- ready have been conducted. In particu- lar, Bilhartz analyzed book reviews published over a thirty-year period in JAH to determine changing perceptions of what constitutes “good history.”3 Casey examined multiple reviews of one hun- dred titles in American history and found that 82 percent of the reviews were favor- able or enthusiastic, and 18 percent were unfavorable or harshly negative.4 To the authors’ knowledge, however, the issue of reviewer consensus regard- ing the same titles in history has not been sufficiently explored. Although some studies have been conducted in other dis- ciplines, they have led to divergent con- clusions. 5 In pursuit of this issue, the present study explores the degree to which book selectors can expect to find reviewer consensus for the same books in three important academic publications. As Fisher has remarked: “Readers have the right to know what to expect from the review section in different journals.”6 Characteristics of the Three Journals With a circulation of 4,000, Choice is pub- lished eleven times a year under the aus- pices of ACRL.7 Since its inception in 1964, its primary purpose has been “to fill an informational need for college libraries. . . .”8 It has been identified as “the most influential review organ for academic publications” and “is a basic selection tool for academic librarians.”9 Unlike subject- specific journals (such as AHR and JAH), Choice publishes reviews of titles in the humanities, science and technology, so- cial and behavioral sciences, as well as reference works. Its reviewers include “college and university faculty who are actively teaching in the subject areas that they review.”10 The typical Choice review of 175 to 200 words is briefer than a re- view in either of the history journals. In this limited space, reviewers are expected to consider a book’s value to an under- graduate collection, analyze qualifica- tions of the author(s), compare a book to similar titles, and assess its strengths and weaknesses.11 Choice publishes reviews of more than 6,000 books annually, or 28 to 29 percent of all books received.12 Ap- proximately 350 of these reviews appear in Choice’s categories of North American history, geography, and area studies. AHR, the official journal of the American Historical Association, has been in existence since 1895. It is published five times a year and has a circulation of 23,000.13 Although AHR contains scholarly articles in all fields of history, “more than half of this important bibliographic source is taken up by reviews.”14 AHR publishes more reviews than any other scholarly journal, more than 1,000 a year from about 4,000 books received by its editors.15 According to William Bishel, AHR’s Assistant Editor for Book Reviews, “most historians would admit that the AHR is the ‘journal of record’ in our discipline.”16 AHR reviewers are required (in most cases) to have published at least one original book-length monograph. Reviews range from 500 to 1,000 words in length (though typically fall between 600 and 650 words).17 The journal provides no spe- cific guidelines for review content other than that the reviewer “should give a clear assess- ment of the book’s content and a critical as- sessment of its contribution to knowledge in the field.”18 JAH appears four times a year and is published under the auspices of the Orga- nization of American Historians, which is the primary scholarly organization for American history specialists.19 From its inception in 1914 until 1964, it was known as the Mississippi Valley Historical Review. Unlike AHR, which covers all areas of his- tory, JAH concentrates on American his- tory. Like AHR, its primary readers are aca- demicians. With a circulation of about 12,000, it has been described as a “substan- Choice publishes reviews of more than 6,000 books annually, 324 College & Research Libraries July 1997 tive bibliographic source and a basic jour- nal for any library collecting in the field.”20 Similar to AHR, reviewers are expected to have major publications in the area(s) they are reviewing. JAH publishes fewer re- views per year than AHR—600 as opposed to 1,000—but because all of them are in American history, its coverage in this area is more thorough than AHR’s. The typical review is 500 words in length, although some are 400 words long and others are longer.21 Like AHR, JAH offers reviewers no specific guidelines. Methodology According to Krippendorff, “Content analy- sis is a research technique for making repli- cable and valid inferences from data in their context.”22 Allen and Reser point out that con- tent analysis has been applied to a variety of library and information science questions, and conclude that this methodology has con- siderable potential for research.23 It was se- lected by the authors as an appropriate tool for quantifying data in order to determine how frequently certain characteristics or cat- egories appear in book reviews. However, de- spite its successful application in a wide vari- ety of research topics, Busha and Harter have advised that in order for content analysis to function effectively as a system of measure- ment, categories must be defined clearly and accurately, and the classification and measure- ment of data must be undertaken with objec- tivity, exactness, and rigor.24,25 A number of book-length monographs provide detailed descriptions of content analysis methodology. 26 The following discussion of the process derives from the work of Budd, Thorp, and Donohew.27 Effective content analysis begins by accurately formulating the research ques- tion. This study asks: What is the com- parative value of the content of reviews published in Choice, AHR, and JAH for purposes of book selection? Selecting a sample is the next step in content analysis. This study initially sampled a total of 375 reviews of Ameri- can history titles published in Choice at five-month intervals, between September 1988 and September 1993. Of these, 153 titles also were reviewed in AHR (be- tween June 1989 and December 1994) and JAH (between December 1988 and March 1995). At this point in the process of execut- ing a content analysis, researchers define their categories. Most of the categories adopted for use in this study were se- lected from those employed in compa- rable projects. The authors drew upon the following categories from Bilhartz: com- pleteness of research, objectivity of author, quality of analysis, and unity of thesis. Bilhartz’s categories of enjoyability of the narrative, literary style, and literary clar- ity are bundled as the single criterion of readability in this study. His separate cri- teria of overall rating and value to the field are combined here as overall opinion. In addition, the authors have adopted quality of editing, which first appeared in Macleod’s study, and created a new cat- egory, placement of events or subject in historical context.28 Coding the content of the data is the next stage in the process of content analy- sis. All the books in this study received an overall opinion, or rating, according to the following scale:29 1 = highly unfavorable 2 = moderately unfavorable 3 = inconclusive (reviewer equivocated or merely summarized content) 4 = favorable 5 = outstanding or significant contri- bution to the field Determination of the overall opinion was influenced by the general tone of a review as well a reviewer ’s opening or concluding comments (e.g., not recom- mended = 1; superlative or major work = 5). The overall opinion did not necessar- ily reflect an accumulation of references to particular evaluative criteria. R e - views also were scrutinized for the pres- ence or absence of the following criteria (mentioned earlier): Evaluating Review Content for Book Selection 325 F IG U R E 1 C om parative D istribution of O verall O pinion (by num ber and percentage)* *R ated on a scale of 1 (low est) to 5 (highest) 326 College & Research Libraries July 1997 l quality or originality of analysis; l quality of narrative, or readability (including style, clarity, lucidity, interest, etc.); l unity of thesis; l quality of editing; l completeness of research (includ- ing factual accuracy); l placement of events or subject in historical context; l objectivity of author (including the absence or presence of bias). These criteria were coded according to the following staggered scale: 0 = criterion not mentioned 1 = negative comments 3 = neutral or equivocal comments 5 = positive comments The authors applied the criteria to a sample of reviews, then compared results in order to standardize and refine the use of these evaluative categories. Once defi- nitions and guidelines were agreed upon, the authors coded the data and concurred as to their accuracy. The next step in content analysis is to scale items. This process is discussed in the following section of this study. The final stage, interpretation, appears in the study’s concluding section. Findings and Results How do the distributions for overall opinion compare among the three jour- nals for the 153 titles reviewed in com- mon? Favorably assessed books (overall opinion = 4) comprised a majority of titles, ranging from 57 to 63 percent across all the journals (see figure 1). Books deemed outstanding or significant contributions to the field (overall opinion = 5) were a distant second in frequency, claiming be- tween 12 and 20 percent of the assess- ments. Together, favorably assessed and outstanding books comprised 74 percent of all reviews in JAH, 77 percent in AHR, and 81 percent in Choice. At the opposite end of the ratings spec- trum, highly unfavorable reviews of books (overall opinion = 1) rarely ap- peared, comprising less than 7 percent of reviews in any one of the journals. Incon- clusive reviews, which were equivocal or merely summarized book content (over- all opinion = 3), amounted to less than 9 percent of reviews in any one of the jour- nals. Hence, the data reveal a pattern com- mon to all three journals: books assessed favorably, followed by those considered outstanding, comprised the overwhelm- ing majority of titles; unfavorably as- sessed books constituted a distinct minor- ity. Certain contrasting reviewing patterns also emerged from the data. JAH review- ers were least likely to characterize books as outstanding (12% of titles, as opposed to 18% in Choice and 20% in AHR). Mod- erately unfavorable (overall opinion = 2) reviews showed up 5 percent of the time in Choice, 12 percent of the time in JAH, and 17 percent of the time in AHR. When percentages of highly unfavorable and moderately unfavorable reviews (overall opinions = 1 and 2) are combined, a no- ticeable difference emerges. Although these reviews comprise only 10 percent of Choice reviews, they account for 21 and 20 percent of AHR and JAH reviews, re- spectively. Thus, it appears that books are less likely to be reviewed negatively in Choice than in the two history journals. On the other hand, Choice reviewers were slightly more likely to equivocate or merely summarize a book’s contents (overall opin- ion = 3) than were AHR reviewers (9% as opposed to 2%, respectively). The highest mean for overall opinion, calculated on a scale of 1 (lowest) to 5 (highest), was found in Choice (3.84). The Favorably assessed and outstanding books comprised 74 percent of all reviews in JAH, 77 percent in AHR, and 81 percent in Choice. Evaluating Review Content for Book Selection 327 lowest mean was found in JAH (3.60), whereas the mean for AHR (3.73) lay be- tween those of the other journals. Statistical analysis found no significant difference in the mean for Choice as com- pared to AHR, nor was there any signifi- cant difference in the mean for AHR as com- pared to JAH. However, there was a statis- tically significant difference beyond the .05 level in the mean overall opinion for Choice when compared to JAH (p = .034). Are the same titles reviewed similarly in all three journals? As demonstrated above, similar per- centages of books were given favorable or outstanding overall evaluations by all three journals. Of the 153 titles, sixty re- ceived an outstanding evaluation in at least one of the three journals. The data were examined further to determine if the same titles received similar overall evalu- ations in all three journals. Table 1 indicates the degree of consen- sus among the three journals for books deemed outstanding (overall opinion = 5). Choice reviewers considered twenty- eight titles (of the original 153) to be out- standing. Of the twenty-eight, only four (14%) were also identified as outstanding in both AHR and JAH. A total of merely eight (29%) of the outstanding titles from Choice received the same rating in one of the other two journals. It follows, surpris- ingly, that twenty of the twenty-eight titles (71%) considered outstanding in Choice did not receive the same high rat- ing in either JAH or AHR. JAH identified nineteen titles as out- standing. Only four (21%) received the same assessment in both Choice and AHR. Of the nineteen titles, a total of eleven (58%) were awarded an outstanding evaluation in at least one of the other two journals; the remaining eight outstanding titles (42%) were not granted an outstand- ing ranking in either Choice or AHR. Of the thirty-one books AHR identified as outstanding, merely four (13%) were also considered outstanding by review- ers in both Choice and JAH. A total of thirteen (42%) of the outstanding titles from AHR re- ceived the same rating in one of the other two journals. Correspondingly, eighteen (58%) of the thirty-one titles received a lower rat- ing in both Choice and JAH. Clearly, the designation of books as outstanding is not unanimous when reviews published in the three journals are compared. However, this lack of unanimity in identifying titles as outstanding does not necessarily mean that reviewer opinions were dramatically opposed to each other. Most of the books deemed outstanding in one of the journals were very likely to be either deemed outstanding or favor- ably assessed in the others (overall opin- ion = 5 or 4). TABLE 1 Degree of Consensus for Books Deemed Outstanding Number and percentage of these outstanding books . . . Number of also deemed outstanding in: deemed outstanding or books favorably assessed by: deemed both of the at least one neither of outstanding other two of the other the other CHOICE JAH AHR journals two journals two journals CHOICE 28 4* (14%) 8 (29%) 20 (71%) X 23 (82%) 23 (82%) JAH 19 4* (21%) 11 (58%) 8 (42%) 15 (79%) X 17 (89%) AHR 31 4* (13%) 13 (42%) 18 (58%) 27 (87%) 26 (84%) X *Only 4 titles were deemed outstanding in all three journals. 328 College & Research Libraries July 1997 For example, of the twenty-eight titles considered outstanding in Choice, twenty- three (82%) were deemed outstanding or favorably assessed in both JAH and AHR. Among the nineteen books awarded an outstanding evaluation in JAH, fifteen from Choice (79%) and seventeen from AHR (89%) were categorized as outstand- ing or favorably assessed. Of the thirty- one titles identified as outstanding in AHR, twenty-seven from Choice (87%) and twenty-six from JAH (84%) were catego- rized as outstanding or favorably assessed. These data indicate the absence of a common definition by reviewers of what constitutes an outstanding American his- tory b ook. Evidently, what reviewers con- sider outstanding scholarship is highly sub- jective and not universally understood. Apparently, if one reviewer believes a book is outstanding, it is improbable that another reviewer will concur. On the other hand, it is extremely likely that if a book has been called outstanding in one of the journals, it will be called either out- standing or favorably assessed in each of the remaining two journals in an over- whelming majority of cases (79 to 89%). Table 2 indicates the degree of reviewer consensus among titles that were either deemed outstanding or favorably as- sessed in all three journals (overall opin- ion = 5 or 4). Choice reviewers considered a total of 124 books to be either outstanding or fa- vorably assessed. Of these titles, eighty- eight (71%) and 101 (81%) were similarly evaluated in JAH and AHR, respectively. JAH reviewers considered a total of 113 books to be outstanding or favorably as- sessed. Eighty-nine (79%) Choice review- ers and ninety-two (81%) AHR reviewers concurred with the JAH assessments. Of the 118 books AHR reviewers labeled out- standing or assessed favorably, Choice re- viewers had the same opinion of 101 (86%) and JAH reviewers of ninety-two (78%). In view of this, a book selector who encounters a favorable or outstanding assessment of a book in Choice, JAH, or AHR can be fairly confident that it will receive a favorable or outstanding assess- ment in the other two journals. However, despite this degree of re- viewer consensus on titles considered outstanding or favorably assessed, a sig- nificant minority of books did not fall into either of these categories. Indeed, re- viewer opinions at times were diametri- cally opposed. For example, 29 percent of the books identified by Choice as outstand- ing or favorably assessed were not deemed likewise by JAH . The highest degree of consensus was found between AHR and Choice. In this case, only 14 per- cent of the 118 titles deemed outstanding or favorably assessed by AHR were not evaluated similarly in Choice. Table 3 indicates the degree of consen- sus for books assessed highly unfavorable or moderately unfavorable (overall opin- ion = 1 or 2). Choice reviewers assessed only fifteen titles in highly unfavorable or moderately TABLE 2 Degree of Consensus for Books That Were Deemed Outstanding or Assessed Favorably Number of books Number and percentage of those books deemed deemed outstanding outstanding or assessed favorably that were also deemed or favorably assessed outstanding or assessed favorably by: CHOICE JAH AHR CHOICE 124 X 88 (71%) 101 (81%) JAH 113 89 (79%) X 92 (81%) AHR 118 101 (86%) 92 (78%) X Evaluating Review Content for Book Selection 329 consensus for poor and mediocre works contrasts with the aforementioned large degree of consensus on titles considered to be outstanding or favorably assessed. Which of the seven evaluative ele- ments are book selectors most likely to encounter in reviews in each of the three journals? All three journals referred to quality of analysis more frequently than any of the other criteria (see figure 2). AHR re- viewers mentioned analysis in 80 percent of the reviews, followed by JAH with 67 percent and Choice with 62 percent. Unity of thesis was the second most frequently mentioned element in AHR and in JAH, appearing in 67 and 62 per- cent of the reviews, respectively. In strik- ing contrast, Choice reviewers commented about unity of thesis in only 17 percent of the examined reviews, making it sixth in order of ranking (see table 4). Q u a l i t y o f r e s e a r c h w a s t h e t h i r d most likely element to appear in reviews in both of the history journals. Refer- ences were found in 54 percent of AHR reviews and in 57 percent of JAH re- views. At 61 percent of the total (slightly higher than the percentages for the his- tory journals), it was the second most likely element to appear in Choice re- v i e w s . A l t h o u g h C h o i c e r e v i e w s a r e briefer than those in AHR and JAH , a book selector will be equally likely to TABLE 3 Degree of Consensus for Books Assessed Highly Unfavorable or Moderately Unfavorable Number of books assessed highly Number and percentage of those books assessed highly unfavorable or unfavorable or moderately unfavorable by: moderately unfavorable CHOICE JAH AHR CHOICE 15 X 3 (20%) 5 (33%) JAH 30 3 (10%) X 13 (43%) AHR 32 5 (16%) 13 (41%) X unfavorable terms. Of these, JAH review- ers had similar opinions of merely three (20%) and AHR reviewers of merely five (33%). JAH reviewers gave highly or mod- erately unfavorable evaluations to thirty books, among which Choice reviewers concurred in only three cases (10%) and AHR reviewers in thirteen (43%). Of the thirty-two books AHR reviewers de- scribed in highly or moderately unfavor- able terms, just five (16%) and thirteen (41%) were similarly described in Choice and JAH, respectively. None of the 153 titles was reviewed in highly unfavorable terms by all three journals. Moreover, only three titles re- ceived either a highly unfavorable or moderately unfavorable evaluation in all three journals. The remaining 150 titles all received at least one favorable or out- standing evaluation. Thus, just as a lack of consensus has emerged among reviewers on what char- acterizes an outstanding book in Ameri- can history, a similar lack of consensus was evident on titles that received ei- ther highly unfavorable or moderately unfavorable evaluations. This lack of Only three titles received either a highly unfavorable or moderately unfavorable evaluation in all three journals. 330 C ollege & R esearch L ibraries July 1997 FIGURE 2 Number and Percentage of 153 Book Reviews in Which Evaluative Elements Were Found Evaluating Review Content for Book Selection 331 find comments about quality of research in Choice reviews. References to an author ’s objectivity were found in 52 percent of AHR reviews and 45 percent of JAH reviews, making it fourth in ranking of mention for both his- tory journals. Choice reviewers, however, mentioned objectivity in only 22 percent of their reviews; hence, it ranks fifth for this journal. Reviewers in both AHR and JAH men- tioned readability fifth in frequency—in 43 and 41 percent of the reviews, respec- tively. Choice reviewers referred to this quality in a similar percentage of reviews, 46 percent, making it third in order of fre- quency. Both AHR and JAH reviewers men- tioned historical context with near-equal frequency, 27 and 24 percent, respectively, ranking it sixth. This element appeared in a similar percentage of Choice reviews, 25 percent, but was fourth in frequency of mention. All three journals referred to quality of editing less frequently than to any of the other criteria. Reviewers in the two history journals deemed it even less im- portant than did Choice reviewers, as it appeared in only 10 percent of the reviews in AHR and in JAH, and in 16 percent of the Choice reviews. Remarkably, the two history journals had precisely the same order in frequency of mention for all seven of the evaluative elements (see table 4). This seems to indi- cate consistency in the way reviewers in the history journals regard the impor- tance of certain criteria. With the exception of two elements, unity of thesis and objectivity, the remain- ing five elements were mentioned in simi- lar numbers and percentages across all three journals. Thesis and objectivity were referred to significantly less often in Choice than in the two history journals. The explanation may lie in Choice’s edi- torial policy of publishing relatively brief reviews. Those reviews in JAH and AHR discussing unity of thesis required at least a paragraph to elaborate on this element. Similarly, when reviewers evaluated ob- jectivity, additional space typically was re- quired to support their assessments. How frequently did reviewers in each of the three journals apply these evalu- ative elements in their reviews? Only two reviews in AHR and one re- view in JAH contained all seven elements; none of the Choice reviews did. Eighteen percent of the Choice reviews contained four or more of the evaluative elements, as compared to 33 percent for JAH and 41 percent for AHR. On average, Choice re- views mentioned 2.49 of the seven ele- ments, JAH mentioned 3.05, and AHR mentioned 3.33.30 Hence, of the three jour- nals, the typical AHR review contained more of the evaluative criteria than either JAH or Choice. Are the evaluative elements of analy- sis, research, and readability employed similarly in all three journals? In addition to noting whether the evaluative elements were mentioned in the reviews, they were coded according to the following scale: 0 = element not mentioned 1 = negative comments 3 = neutral or equivocal comments 5 = positive comments Analysis, research, and readability were the only elements mentioned in suf- TABLE 4 Comparative Ranking for Mention of the Evaluative Elements AHR JAH CHOICE Analysis 1 1 1 Unity of thesis 2 2 6 Research 3 3 2 Objectivity 4 4 5 Readability 5 5 3 Historical context 6 6 4 Editing 7 7 7 332 College & Research Libraries July 1997 ficient numbers to allow determination of significant differences in reviewer usage patterns. As table 5 indicates, AHR re- viewers were significantly less likely to praise a book’s quality of analysis than were reviewers of the same titles in Choice. Reviewers in all three journals were more likely to praise a book’s research or read- ability than its quality of analysis. More- over, when reviewers chose to comment on a book’s quality of research or read- ability, their remarks were positive in 75 percent of the cases. Hence, there appears to be a reviewer tendency in all three jour- nals to praise rather than criticize an author ’s research or readability. To what degree did reviewers apply the evaluative elements to praise, equivocate, or criticize? Of a total of 381 references to the evalu- ative elements in Choice reviews, 216 (57%) were positive, 122 (32%) neutral or equivocal, and 43 (11%) negative. Of 509 references to the evaluative elements in AHR, 258 (51%) were positive, 164 (32%) neutral, and 87 (17%) negative. As to the 467 references in JAH, 238 (51%) were positive, 150 (32%) neutral, and 79 (17%) negative. In light of these data, it appears that Choice reviewers made the lowest percentage of negative comments and the highest percentage of positive comments. However, they made pre- cisely the same percent- age of neutral comments as reviewers in the two history journals. The dis- tributions of negative, neutral, and positive comments in AHR and JAH were identical. How do the three journals compare with regard to the timeliness of their assessments? The time lag for publi- cation of reviews in JAH as compared to Choice ranged from one case of one month prior to Choice to one case of thirty months later than Choice. For AHR, the publication time span varied from one instance of five months prior to Choice to one instance of thirty-seven months after Choice. On av- erage, JAH reviews appeared 8.5 months after reviews of the same titles in Choice, and AHR reviews appeared 10.6 months after reviews of the same titles in Choice.31 Clearly, Choice is the most timely of the journals in its publication of reviews; of the two history journals, JAH is slightly more timely than AHR. Conclusions This study compared the qualitative as- sessments provided by reviews published in American Historical Review, Journal of American History, and Choice. One hun- dred fifty-three books in American his- tory were selected on the basis of being reviewed in all three journals. The con- tents of the 459 reviews were then ana- lyzed, and the analysis revealed the over- all opinion expressed in each review, as well as the extent of reviewer use of seven evaluative elements. Striking similarities, as well as differences, were identified as characteristic of typical reviews appear- ing in the three journals. With regard to similarities, it was ob- served that the preponderance of praised TABLE 5 Comparative Mean Ratings for Analysis, Research, and Readability* Analysis Research Readibility CHOICE 3.78** 4.66 4.61 AHR 3.20** 4.28 4.56 CHOICE 3.74 4.30 4.45 JAH 3.62 4.23 4.66 AHR 3.49 4.02 4.35 JAH 3.45 3.98 4.46 * Mean ratings are based on reviews of the same titles in which a particular evaluative element was mentioned. ** A statistically significant difference exists in the mean for analysis in Choice as compared to the mean for analysis in AHR Evaluating Review Content for Book Selection 333 titles in all three journals make each jour- nal a valuable source for a wide array of recommended books. However, only a fraction of those titles were identified as outstanding, which suggests a certain selectivity among reviewers. Book selec- tors may confidently assume that if one of these journals praises a book (award- ing it either a favorable or outstanding evaluation), the others will do likewise. Moreover, the overwhelming majority of reviews provide book selectors with clearly discernable overall assessments. If frequency of mention reflects the importance of each of the evaluative cri- teria, clearly quality of analysis is most important for reviewers in these three journals. Conversely, because quality of editing was the least-mentioned evalua- tive element, it appears to be the review- ers’ least important assessment tool. Two additional features may be discerned. First, selectors will find quality of re- search mentioned an almost-equal num- ber of times in reviews in each of the three journals. Likewise, readability is men- tioned in similar numbers regardless of the journal; the same holds true for his- torical context. Second, selectors are more likely to find titles praised by all three journals for their research or readability more than for their quality of analysis. Finally, the data indicate that review- ers in all three journals were more likely to praise books for their overall quality than to praise the quality of individual characteristics. Consequently, book selec- tors should not expect positively re- viewed titles to be praiseworthy in every respect. With regard to dissimilarities, it was noted that reviewers of the same books rarely agreed on outstanding titles, indi- cating that selectors should not assume any title will commonly receive an out- standing evaluation in all three journals. This lack of unanimity suggests the highly subjective nature of what differ- ent individuals will consider “outstand- ing.” Likewise, the total absence of re- viewer unanimity in identifying inferior titles (overall opinion = 1 or 2) demon- strates how subjective the negative assess- ments also can be and warns book selec- tors not to expect to find negative reviews of the same title in all three journals. For the most part, AHR reviewers were most likely to label a book “outstanding,” whereas JAH reviewers were least likely to do so. Choice reviewers tended to give fewer negative reviews than either AHR or JAH reviewers did; fewer evaluative elements appeared in Choice than in ei- ther of the other two journals. With regard to dissimilarities in assess- ing individual evaluative criteria, book selectors should know that AHR review- ers considered quality of analysis more often than did reviewers in the other jour- nals, and were more likely to criticize books for poor quality of analysis. Book selectors also will find significantly fewer references to unity of thesis and objectiv- ity in Choice reviews than in reviews ap- pearing in either of the other journals. The data also point out that because Choice reviews appear, on average, 8.5 t o 1 0 . 6 m o n t h s s o o n e r t h a n t h o s e i n JAH or AHR , respectively, Choice re- views enable book selectors to make time- lier decisions. Awareness of variables in reviewing journals should enable book selectors to make better-informed acquisition deci- sions for their respective institutions. Hopefully, the findings of this study will facilitate that process.32 The overwhelming majority of reviews provide book selectors with clearly discernable overall assessments. 334 College & Research Libraries July 1997 Notes 1. An overview of the literature prior to 1989 may be found in Virgil Blake, “The Role of Reviews and Reviewing Media in the Selection Process: An Examination of the Research Record,” Collection Management 11, no. 1/2 (1989): 1–40. For more recent studies, see Dana Watson, “Re- viewing: A Strategic Service,” in A Service Profession, A Service Commitment: A Festschrift in Honor of Charles D. Patterson, eds. Connie Van Fleet and Danny P. Wallace (Metuchen, N.J.: Scarecrow Pr., 1992), 19–41; Judith Serebnick, “Selection and Holdings of Small Publishers’ Books in OCLC Libraries: A Study of the Influence of Reviews, Publishers, and Vendors,” Library Quarterly 62, no. 3 (July 1992): 259–94. 2. For example, Serebnick has maintained that research should examine how well reviews perform their role, in “Selection and Holdings,” 291. Daniel Ream has argued that objective evaluation of book review journals has been relatively neglected in “An Evaluation of Four Book Review Journals,” Research Quarterly 19 (winter 1979): 149. 3. Terry Bilhartz, “In 500 Words or Less: Academic Book Reviewing in American History,” History Teacher 17, no. 4 (Aug. 1984): 526–36. 4. James Benjamin Casey, “Assessment of Quality in Book Selection: An Evaluation of the Effectiveness of Opinions Rendered by Peer Reviews in American History Journals” (Ph.D. diss., Case Western Reserve University, May 22, 1985): 24. 5. Examples of such divergence may be found in William E. Snizek and E. R. Fuhrman, “Some Factors Affecting the Evaluative Content of Book Reviews in Sociology,” American Soci- ologist 14 (May 1979): 108–14; Norval Glenn, “On the Misuse of Book Reviews,” Contemporary Sociology 7, no. 3 (May 1978): 254; James H. Sweetland, “Reference Book Reviewing Tools: How Well Do They Do the Job?” Reference Librarian 15 (fall 1986): 65–74; James Rettig, “Reference Book Reviewing Media: A Critical Analysis,” in Library Science Annual, vol. 2, ed. Bohdan S. Wynar (Littleton, Colo.: Libraries Unlimited, 1986); Duncan Lindsey, The Scientific Publication System in Social Science (San Francisco: Jossey-Bass, 1978); Joseph Palmer, “Agreement and Disagreement among Fiction Reviews in Library Journal, Booklist, and Publishers Weekly,” Library Resources and Technical Services 39, no. 4 (Oct. 1995): 384–90. 6. Susan Fisher, “Book Reviews in Five Library Journals: A Comparative Analysis,” Austra- lian Library Journal 30, no. 1 (Aug. 1981): 96. 7. Bill Katz and Linda Sternberg Katz, Magazines for Libraries, 8th ed. (New Providence, N.J.: Bowker, 1995), 223. 8. Helen MacLam, “About Choice,” Collection Management 16, no. 1 (1992): 73. 9. David Henige, “When Bad Is Good Enough: The Lowest Common Denominator in Refer- ence Publishing and Reviewing,” Reference Services Review 19, no. 1 (1991): 13; Katz and Katz, Magazines for Libraries, 223. 10. “About Choice” (internal document). 11. “Good Books and Gatekeepers,” Choice 32, no. 7 (Mar. 1995): 1024. 12. Marvin D. Leavy, “An Exploration of the Validity of Choice’s ‘Outstanding Academic Books,’” Journal of Academic Librarianship 18, no. 2 (May 1992): 83. The estimate of 28 to 29 percent is as of June 1994. This information was provided by Helen MacLam, Social Sciences Editor at Choice. 13. Ulrich’s International Periodical Directory 1996, 34th ed. (New Providence, R.I.: Bowker, 1996): 3151. 14. Katz and Katz, Magazines for Libraries, 657. 15. William Bishel, “Book Reviewing and Scholarly Communication: The View from the American Historical Review” (paper presented at the annual meeting of the American Historical Association, Jan. 1994). 16. Ibid. 17. Ibid. 18. “Guidelines for Reviewers,” distributed to their reviewers by American Historical Review. 19. Katz and Katz, Magazines for Libraries, 664. 20. Ibid. 21. Steven Stowe, “Thinking about Reviews,” Journal of American History 78, no. 2 (Sept. 1991): 593; Casey Blake, “Book Reviewing and Scholarly Communication” (paper presented at the an- nual meeting of the American Historical Association, Jan. 1994). 22. Klaus Krippendorff, “Content Analysis,” in International Encyclopedia of Communications, vol. 1 (New York: Oxford Univ. Pr., 1989), 403. 23. Bryce Allen and David Reser, “Content Analysis in Library and Information Science Re- Evaluating Review Content for Book Selection 335 search,” Library and Information Science Research 12, no. 3 (July–Sept. 1990): 260. 24. Examples of the use of content analysis may be found in Dean J. Champion and Michael F. Morris, “A Content Analysis of Book Reviews in the AJS, ASR, and Social Forces,” Ameri- can Journal of Sociology 78, no. 5 (Mar. 1993): 1256–65; Judith Fox, “Choice As a Book Selec- tion Tool in Sociology: A Comparison with Contemporary Sociology,” Collection Management 13, no. 1/2 (1990): 135–52; Beth Macleod, “Library Journal and Choice: A Review of Reviews,” Journal of Academic Librarianship 7, no. 1 (Mar. 1981): 23–28; Alexandra Dimitroff, “Research for Special Libraries: A Quantitative Analysis of the Literature,” Special Libraries 86, no. 4 (fall 1995): 256–64; Anne Bridges, “Scholarly Book Reviews and Collection Development: A Case Study in Ameri- can History,” Journal of Academic Librarianship 15, no. 5 (Nov. 1989): 290–93; Bilhartz, “In 500 Words or Less”; Paula Wheeler Carlo and Allen Natowitz, “The Appearance of Praise in Choice Reviews of Outstanding and Favorably Assessed Books in American History, Geography, and Area Studies,” Collection Management 20, no. 3/4 (1996): 97–117. 25. Charles Busha and Stephen Harter, Research Methods in Librarianship Techniques and Inter- pretation (New York: Academic Pr., 1980), 174. 26. Classic studies include Richard Budd, Robert Thorp, and Lewis Donohew, Content Analysis of Communications (New York: Macmillan, 1967); Ole R. Holsti, Content Analysis for the Social Sciences and Humanities (Reading, Mass.: Addison-Wesley, 1969); Klaus Krippendorff, Content Analysis: An Introduction to Its Methodology (Beverly Hills, Calif.: Sage, 1980). 27. Budd, Thorp, and Donohew, Content Analysis of Communications, 6. 28. Bilhartz, “In 500 Words or Less”; Macleod, “Library Journal and Choice.” 29. A similar system of rating was used by Bilhartz, “In 500 Words or Less.” As a basis for their study, Greene and Spornick used the rating system that appears in Periodical Abstracts on Disc, Research II Edition [computer database] (Ann Arbor, Mich.: UMI, 1994). Robert J. Greene and Charles D. Spornick, “Favorable and Unfavorable Book Reviews: A Quantitative Study,” Journal of Academic Librarianship 21, no. 6 (Nov. 1995): 449–53. 30. The average number of evaluative elements for each of the journals is the quotient of the total number of evaluative elements and the number of reviews. Choice 381/153 = 2.49 JAH 467/153 = 3.05 AHR 509/153 = 3.33 31. Ream calculated the mean time lag of Choice reviews to be about four months (127.7 days) after a book was published. Based on Ream’s calculation, JAH reviews appear, on average, 12.5 months after a book is published, whereas AHR reviews appear approximately 14.6 months after a book is published. 32. Data are on file. Inquiries may be addressed to either of the authors.