| URLs in this document have been updated. Links enclosed in {curly brackets} have been changed. If a replacement link was located, the new URL was added and the link is active; if a new site could not be identified, the broken link was removed. |
![[Refereed]](http://www.istl.org/graphics/r2.gif){kind=small}
The Emerging Engineering Scholar: A Citation Analysis of Theses and Dissertations at Western Michigan University
Assistant Professor
University Libraries
Western Michigan University
Kalamazoo, Michigan
edward.eckel@wmich.edu
Abstract
Can one glimpse the development of emerging scholars in the work of engineering graduate students? To answer this question, the author studied the citation patterns in 96 Master's theses and 24 Ph.D. dissertations completed at Western Michigan University's College of Engineering and Applied Sciences between 2002 and 2006. The hypothesis of this study is that an increase in graduate student research competence between the master's and doctoral levels can be seen in their use of scholarly sources such as journal articles and conference papers. For each thesis and dissertation, bibliographic information (title, author, document type, year of publication) was gathered for each individual citation in the reference list(s). The data analysis indicates that doctoral engineering students use a significantly greater number of scholarly journal articles (44.3% to 29.3%) and conference papers (21.9% to 12.5%) than master's students. Also, master's students depend more heavily upon literature available on the web (web sites, government papers, grey literature, trade magazines, and patents). These results give tentative support to the hypothesis. Without knowing how faculty expectations influence the quality of graduate literature reviews, the hypothesis could not be conclusively supported with the data gathered. This study shows that there is a significant difference in the proportions of scholarly and other research sources used by master's and doctoral engineering students. The implications of these citation patterns in the development of the engineering scholar are discussed.
Introduction
The completion of a thesis or dissertation is the required initiation to the scholarly levels of any discipline, including engineering (Lovitts 2007). A significant step in this process is finding and citing scholarly sources in the literature review. This skill should show some development as students move from the master's to the doctoral level. According to Carlson (2006), as students progress academically, sources cited should demonstrate students' "growing sophistication and ability to think critically in their chosen discipline." But is this really the case? One way to get a sense of this process is to look at what types of sources engineering graduate students cite in their theses and dissertations.
The author conducted a citation analysis of 96 Master's theses and 24 Ph.D. dissertations completed at Western Michigan University's College of Engineering and Applied Sciences between 2002 and 2006. In this study, the following questions were asked: Do master's engineering students use the same types of information sources in their research as doctoral students? Is there evidence of improved scholarship between master's and doctoral level work in engineering, based on the percentage of higher quality sources (in particular, journal articles, monographs, and conference papers) used in reference lists? In looking at what engineering graduate students cite, can one glimpse the development of emerging scholars?
Background
Engineering and Applied Sciences at Western Michigan University
Western Michigan University (WMU) in Kalamazoo is Michigan's fourth largest university, with a student enrollment of 25,000. The Carnegie Foundation for the Advancement of Teaching classifies WMU as a "comprehensive doctoral (no medical/veterinary)" research institution with "high research activity" (Carnegie Foundation for the Advancement of Teaching 2007).
The College of Engineering and Applied Sciences serves nearly 2,400 students, with 300 at the master's level and 55 at the Ph.D. level. It offers master's level programs in computer science and engineering management, as well as civil, computer, electrical, industrial, manufacturing, mechanical, and paper/imaging engineering. It offers Ph.D. programs in computer science and electrical, industrial, mechanical, and paper/imaging engineering.
The University Libraries at Western Michigan University
The University Libraries are made up of one Main Campus Library (Waldo Library) and five branch libraries that contain a total of 4.5 million items in print and non-print formats. There is no separate library branch on the College of Engineering campus. All print engineering and applied sciences resources, including all theses and dissertations completed by engineering graduate students, are kept in the Main Library.
Literature Review
According to Parry (1998) "for the doctoral student, the thesis is the representation in language of discipline-specific knowledge shaped by the norms and conventions of a particular disciplinary culture." Through the dissertation, a graduate student demonstrates mastery of the research methods and scholarly discourse of the discipline (Hart 2005; Lovitts 2007). The process by which a graduate student learns how to do a literature review is mostly hidden from the view of librarians, faculty advisors and even the students themselves (Gerholm 1990). One way to make this process more apparent is by analyzing the sources cited, to derive a sense of the development of scholarship between the master's and doctoral level.
Citation analysis is a bibliometric technique that uses citation patterns in documents to trace the relationships between those documents and the original sources and authors. The relationships found provide a picture of the cultures of those disciplines. According to bibliometric pioneer Eugene Garfield, "citation links...provide a quantitative picture of journal utility and relationships that is useful" (Garfield 1979). Garfield (1977a,b) wrote several articles in the 1970s on the citation patterns of engineering journals.
Citation analysis of the science/engineering journal literature has been used by librarians primarily as a means of guiding journal collection development. One of the earliest examples of this type of study was Gross and Gross's seminal 1927 study of citations in a year's worth of the Journal of the American Chemical Society. Other researchers have used the information gathered in citation analyses to create a picture of information use within a specific discipline. For example, Musser and Conkling (1996) looked at citations in major scholarly engineering journals. They found that engineering journal citations averaged 53% journal articles, 19% conference papers, 12% monographs, 9% technical reports, 2% dissertations, and 1% each of theses and standards. Their pre-Internet study gives a good picture of the scholarly use of the engineering literature at that time. Musser (2007) also looked at references in selected mining engineering journals from 1995, finding the document use pattern to be similar to that for general engineering seen in the 1996 study she did with Conkling. Both articles mention the implications of their results to library collection development.
Other researchers have used citation patterns in engineering theses and dissertations to gauge local collection use and strength. Kriz (1977) looked at citations of journal articles in engineering master's theses at West Virginia University as a means to justify cuts in subscriptions. Williams and Fletcher (2006) looked at citations in engineering master's theses at Mississippi State University for the period 2000-2004. Williams' and Fletcher's study makes an interesting comparison with the 1996 Musser and Conkling study in that it gives a picture of the use of engineering documents in several engineering disciplines and charts the rise of web sites as the fourth most cited engineering source.
Few studies have discussed what citation patterns in graduate writing indicate about the research competence of graduate students. Fewer still have compared work done at the master's level with that done at the doctoral level. In a longitudinal study done at Iowa State University, Kushkowski et al. (2003) looked at citations from a broad sample of theses and dissertations completed over a 19-year time frame. They appear to have confined their master's/doctoral comparison to the number of citations per page and number of pages per thesis/dissertation, thereby missing the more interesting comparison of what types of documents were cited and what that indicated about comparative research skills.
Beile et al. (2004) question whether doctoral candidates' citations indicate research expertise or merely reflect the limitations of student research skills. Indeed, studies by both Boote and Beile (2005) and Bruce (1994) show that graduate students' skills at finding and synthesizing the literature in their disciplines are not as advanced as is commonly assumed.
Methods
This study asks two main questions.
- Do doctoral engineering students differ from master's engineering students in the proportions of different research sources used in their thesis/dissertation reference lists?
- Is there evidence of improved research competence between master's and doctoral level work in engineering, based on the percentages of quality scholarly sources used in reference lists? For the purposes of comparison between the two populations, research competency will be defined as the total percentage of scholarly research sources (journal articles, monographs and conference papers) that make up a graduate student's reference list.
The hypothesis of this study is that graduate student use of scholarly sources demonstrates an increase in scholarship between the master's and doctoral levels. To answer these questions, a citation analysis was conducted of 96 master's theses and 25 Ph.D. dissertations completed at WMU's College of Engineering and Applied Sciences during the five-year time frame from 2002 to 2006. The ideal way to test this hypothesis would be to compare percentages of scholarly sources used by a large sample of the same students both at the master's and doctoral levels. For the time frame of this study, there were only three students with whom such a comparison was possible. Hence the method used for this study was a broad comparison of citation patterns between master's and doctoral level engineering students.
This study also examines how the use of generic web documents by graduate students compares to the use of more traditional engineering sources. This study looks at the degree to which the citations given are accurate and complete. Due to time limitations, this study omits any analysis of the quality of the literature reviews themselves. Such an analysis would be important in a more comprehensive study, like the one conducted by Beile et al. in 2004.
Theses and dissertations were identified through a keyword search in the online catalog. Basic bibliographic information was gathered for each thesis and dissertation that was completed during the five-year study time frame. During the study time frame, some engineering theses and dissertations were uploaded to a university electronic server in addition to, or in lieu of, being submitted to the Main Library collection. These additional theses and dissertations were identified through a scan of the University's "Electronic Thesis and Dissertation Collection" (ETD) web site. Three Ph.D. dissertations and seven master's theses were added to the study from this collection. Finally, a search on all engineering subject headings was conducted in the Proquest "Dissertations & Theses: Full Text" database (ProQuest LLC 2007) for any additional College of Engineering and Applied Sciences theses or dissertations.
The final breakdown of the theses and dissertations by department is shown in Table 1 below.
Table 1: Engineering Theses and Dissertations By Department 2002-2006
Department | Number | |
Master's Theses | Civil/Construction Engineering | 18 |
Computer Science | 3 | |
Electrical and Computer Engineering | 8 | |
Industrial and Manufacturing Engineering | 10 | |
Mechanical and Aeronautical Engineering | 19 | |
Materials Science | 6 | |
Paper and Imaging Science and Engineering | 32 | |
Total: | 96 | |
Dissertations | Computer Science | 5 |
Industrial and Manufacturing Engineering | 5 | |
Mechanical and Aeronautical Engineering | 7 | |
Paper and Imaging Science and Engineering | 8 | |
Total: | 25 | |
For each thesis and dissertation, the following information was collected: the academic department, the call number, year of completion, and total number of citations. For each individual citation in the reference lists, the following bibliographic information was gathered: monograph/journal/conference title, article title (for conference papers), author(s)/editor(s), format (journal, conference paper, etc.), URL (when given), and year of publication. URLs were subsequently checked to verify the document type and accessibility.
Format types were coded as follows:
Table 2: Citation Format Codes
Code | Format |
cp | Conference paper/proceeding |
d/t | Dissertation/thesis |
gov | Government document/web site |
grey | Grey literature |
j | Journal |
m | Monograph (including monographic series) |
mag | Popular magazine |
n | Newspaper |
p | Patent (granted) |
st | Standard |
tm | Trade magazine |
tr | Technical report |
web | Web site/web page |
Searches were conducted in WorldCat.org, Ulrichsweb.com or Google to identify unknown document types. When it was not clear if a reference was a scholarly journal or a trade magazine, Ulrichsweb.com was consulted, and the classification in Ulrichsweb.com used. Citations whose format was still unidentifiable after this searching were coded as "Incomplete" (inc). Citations that did not fit the previous categories were coded as "Other" (o). Included in this category would be computer software, personal/private communications, interviews, materials quoted from another source, patent applications, public radio reports, prepublication or not-officially published items, speeches and lectures. No attempt was made to code citations for "quality" or appropriateness for use in a scholarly document.
The Grey Literature Network (2007) defines grey literature as "information produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishing." This definition would normally include technical reports, government documents, standards, patents, and even dissertations and theses. To get a more nuanced view of graduate student sources, these more specific formats were counted separately. Grey literature was more narrowly defined for this study as professional documents such as white papers, working papers, training materials, preprints and eprints. Government publications, grey literature and technical reports found online were coded under these more specific categories rather than as web sites.
Statistics calculated include the age of each citation, number of citations by format, and the average number of citations per thesis for each discipline. In addition, the mean percentage of the total citations for each document type was also calculated for both dissertations and theses. Additional statistical analysis was conducted using SPSS 15.0 for Windows. Statistics calculated using SPSS were comparisons of the mean proportions of different citation formats and a frequency table of the cumulative ages of those formats. The mean percentages were compared using the t-test for significance at the .05 level.
Results
A total of 2,903 citations from master's theses and 2,886 citations from doctoral dissertations were analyzed in this study. The average number of citations per thesis and dissertation by department is shown in Table 3 below.
Table 3: Average Citations per Thesis or Dissertation By Department
Department | Avg cites/thesis | Avg cites/dissertation |
Civil and Construction | 42 | n/a |
Computer Science | n/a | 83 |
Electrical/Computer | 31 | n/a |
Industrial/Manufacturing | 23 | 101 |
Mechanical/Aeronautical | 25 | 133 |
Materials Science | 41 | n/a |
Paper Chemistry/Imaging | 27 | 129 |
Note: Departments not represented in theses or dissertations, are indicated as "n/a."
The breakdown of document types and percentages is shown in Table 4 below.
Table 4: Percentages by Document Type – Theses vs. Dissertations 2002-2006
Theses | Dissertations | ||||
Document Type | Number | Percent of Total | Number | Percent of Total | Significant Difference |
Scholarly Journal | 851 | 29.3 | 1278 | 44.3 | yes |
Monograph | 595 | 20.5 | 499 | 17.3 | yes |
Conference Paper | 362 | 12.5 | 633 | 21.9 | yes |
Web Document | 266 | 9.2 | 101 | 3.5 | yes |
Grey Literature | 220 | 7.6 | 48 | 1.7 | yes |
Trade Magazine | 195 | 6.7 | 53 | 1.8 | yes |
Patent | 74 | 2.5 | 42 | 1.5 | no |
Government Document | 72 | 2.5 | 21 | 0.7 | yes |
Dissertation/Thesis | 63 | 2.2 | 55 | 1.9 | no |
Technical Report | 63 | 2.2 | 38 | 1.3 | no |
Incomplete Citation | 45 | 1.6 | 13 | 0.5 | yes |
Other | 43 | 1.5 | 60 | 2.1 | not calculated |
Standard | 37 | 1.3 | 45 | 1.6 | no |
Newspaper | 12 | 0.4 | 0 | 0.0 | not calculated |
Popular Magazine | 5 | 0.2 | 0 | 0.0 | not calculated |
Total | 2903 | 100.2* | 2886 | 100.1* | |
*Totals do not add up to 100.0% due to rounding.
While scholarly journals were the most commonly cited source for both theses and dissertations, the percentages of journal citations at WMU are lower than the 59% average obtained for engineering by Kushkowski et al. (2003) in their study of theses and dissertations at a large mid-western university and the 53% average obtained by Musser and Conkling in their 1996 study of citations in sixteen different engineering journals. The 29.3% of scholarly journal citations in theses is lower than the 38% seen in the 2006 Williams and Fletcher study of engineering theses at Mississippi State University.
In decreasing order of percentage, scholarly journal citations in master's theses were followed by monographs, conference papers, and web documents, and in the Ph.D. dissertations by conference papers, monographs, and web documents. The top four document formats in this study match the top four formats cited in the 2006 Williams and Fletcher study.
Nearly 19% of master's students cited 0 (11%) or 1 (7%) scholarly journals. These percentages are lower than the ones Kriz uncovered in his 1977 analysis of master's theses at West Virginia University. The lowest number of citations to journals in an individual dissertation was 12.
As can be seen in Table 4, doctoral students used a significantly greater percentage of both scholarly journals and conference papers than did master's students. Master's students, while using a slightly higher percentage of monographs, also used a significantly greater percentage of web documents than did doctoral students. While scholarly journals, monographs and conference papers accounted for a cumulative total of 83.5% of sources cited by doctoral students, these categories only accounted for 62.3% of total sources cited by master's students. In addition, master's students used a significantly higher percentage of grey literature, trade magazines, and government documents than doctoral students. The difference in master's and doctoral student use of the other source types (patents, dissertations/theses, technical reports, and standards) was not found to be significant.
Figures 1 and 2 give the percentages of selected document types cited by graduate students within specific departments. In the master's theses, scholarly journal citation percentages ranged from a low of 9.0% in industrial/manufacturing engineering to a high of 58.1% in electrical/computer engineering. Monographs were found to be especially important in the industrial/manufacturing engineering area. Grey literature was more highly cited in the civil engineering theses than in the other disciplines. Of all the disciplines represented in the master's theses examined, paper chemistry showed the greatest dispersion among cited document types, with its sources spread more evenly among conference papers, journal articles, monographs, trade magazines, and web documents.
Among the dissertations, scholarly journal citations ranged from a low of 33.3% for computer science to a high of 54.9% for mechanical/aeronautical engineering. Conference papers were far less cited by doctoral students in industrial/manufacturing engineering than for the other departments. In addition, for computer engineering students, conference papers slightly edged out journal articles (36.5% to 33.3%) in terms of frequency of citation. Monographs were much more likely to be cited by industrial/manufacturing engineering students than for the other disciplines. Finally, while almost no patents were cited in dissertations by computer, industrial/manufacturing or mechanical/aeronautical engineering students, they made up 4% of all documents cited in paper chemistry dissertations. However, there were so few doctoral dissertations in each department for the study period, and variations in the total number of citations between individual dissertations were so extreme (ranging from 23 to 242 in industrial/manufacturing engineering dissertations alone) that no real conclusions can be drawn from the disciplinary breakdown.
Looking at Tables 5 and 6, which chart the time span or aging of the top three most commonly cited document types, it can be seen that sources less than five years old made up a higher percentage of master's student references than doctoral student references. The average age of scholarly journals and monographs was higher for dissertations than for theses. From these figures, it appears that doctoral engineering students are using older sources as a whole. This may be because they are more thorough in following up citations in article reference lists. It may also reflect the greater comprehensiveness of the dissertation reference lists.
Document Type | Average Age (in years) | Range (in years) | 5 years old or less (percent) | 10 years old or less (percent) |
Conference Papers | 9.7 | 73 | 46 | 72 |
Scholarly Journals | 11.8 | 135 | 40 | 63 |
Monographs | 11.9 | 75 | 29 | 60 |
|
Document Type | Average Age (in years) | Range (in years) | 5 years old or less (percent) | 10 years old or less (percent) |
Conference Papers | 9.2 | 94 | 42 | 72 |
Scholarly Journals | 16.6 | 151 | 22 | 47 |
Monographs | 15.7 | 73 | 24 | 45 |
Discussion
Differences in citation patterns between master's and doctoral students
The sources used by doctoral students tended to be more focused upon scholarly sources (scholarly journals, monographs, and conference papers) than were those used by master's students (83.5% to 62.3%). In addition, sources used by master's students showed a greater dispersion among a range of other sources, such as web documents, grey literature, patents, trade magazines, and technical reports, with significantly fewer scholarly journals and conference papers cited. This finding suggests that master's students are less discriminating in their use of sources. The slightly higher percentage of monographs cited by master's students may reflect a stronger need for basic background information. The emphasis on scholarly journals and conference papers in the doctoral dissertations might result from doctoral students needing to demonstrate a broader familiarity with the relevant scholarly literature than that expected of master's students. This would also help explain why the scholarly journals and monographs cited by the doctoral students were, on average, older and covered a greater time span than the same source types as cited by master's students.
Generic web sites have now become the fourth most commonly cited document by both master's and doctoral engineering students at this university. This supports the results obtained by Williams and Fletcher in 2006. This new emphasis on web sites is problematic for two reasons. First, engineering documents are consulted for very specific kinds of information for which there are usually few substitutes. Technical reports present grant-funded research results, patents give descriptions of new technologies, standards give descriptions of commonly used industry-wide standards that must be adhered to, government documents provide statistics or regulations, handbooks are consulted for materials or structural data, etc. Unlike these sources, generic web sites have no particular use that easily differentiates them from other engineering document types other than convenience of access. In addition, the fleeting nature of web sites poses a problem for scholarly work that is meant to be of lasting value (Lawrence et al. 2001; Davis 2003; Spinnellis 2003; Thomas 2004; Hovde 2007).
What are engineering graduate students getting from these web sites that they are not getting from other documents? Has convenience of access become a more important factor than validity, authority, and stability? Do engineering graduate students now depend upon web sites to provide background information that they used to get from specialized handbooks and encyclopedias? The citations in this study were coded as generic web sites only when they did not fit one of the other engineering document types. Furthermore, a significant number of the government documents, grey literature documents and technical reports (as well as some of the trade magazine articles) in this study were originally found online via the web, as indicated by the URLs given in their citations. So the total number and percentage of sources that originated via the open Internet is probably significantly higher than the figures given in Table 4. On the positive side, certain resources such as government documents and grey literature may show up more frequently in graduate student reference lists now than ten years ago due to their being more readily accessible via the Internet (Davis 2003). Also, some engineering information, particularly with regards to industrial products and computer science, may only be available on the web.
The variation seen in the citation patterns between the different engineering departments at this particular institution, while not conclusive, do indicate that there are differences in what types of sources are emphasized in those departments. These emphases may originate from faculty thesis or dissertation advisors. Faculty advisors probably have a significant influence on the types of research materials their graduate students cite. In addition, attitudes and biases toward or against certain types of research materials may also reflect deficiencies in students' undergraduate training.
An additional characteristic of the citations used by graduate students in this study was the over-reliance on certain sources by some authors. For example, in the doctoral dissertations examined, one student was responsible for 29 out of 30 citations to Society of Automotive Engineers (SAE) conference papers. In addition, there were numerous examples of both master's and doctoral students citing the same journal over and over again. All 34 references to the journal Ergonomics were by the same doctoral student. All 29 citations to the International Journal of Fatigue were by 2 master's students.
There are a number of explanations for this pattern. Familiarity with a particular journal may lead a researcher to tend to cite that journal more frequently. In addition, certain journals or conference proceedings may provide the bulk of relevant citations in a particular discipline. Faculty advisors may emphasize certain types of sources to the exclusion of others. Finally the particular strengths or weaknesses of a library collection could make itself felt in the reference lists of theses and dissertations at that school by the ready availability of certain journal or conference titles and the lack of others. (For many graduate students, even a two- to three-day turn-around for an article from interlibrary loan may be too long to be worth the trouble.)
Finally, this study provides an interesting contrast between master's and doctoral students in terms of the accuracy of citations. While the number of incomplete citations for both theses and dissertations was comparatively minor, a significant difference was observed in the proportion of incomplete citations in each group (1.6% to 0.5% respectively). From this, it appears that master's students are indeed less familiar with citation format styles and conventions, particularly the need to include all relevant bibliographic information.
Limitations of the study
As mentioned earlier, no attempt was made to analyze the quality of the literature reviews in the theses or dissertations, nor were citations coded for quality like the Beile et al. (2004) study. While this would have provided more information regarding the research and synthesis skills of the graduate students, it would have required far more time than was available to this researcher.
Figure 3 compares the use of information sources by three engineering graduate students (A, B, and C) at WMU for whom a master's thesis and a subsequent doctoral dissertation were available. It illustrates the limitations of comparing journal, monograph, and conference paper percentages. In this figure, one can see how citation patterns changed between the master's and doctoral level. While the use of scholarly journals went up considerably between the master's and doctoral work for all three students, the use of monographs and conference papers varied considerably. The total scholarly source use (journals plus monographs plus conference papers) actually went down in the case of student A. In the cases of students A and B, usage of web documents went up between the master's and doctoral level work. It is not clear why this is so. It may be that these two students went further afield in their web searching, finding more material that they considered relevant. There was no obvious relation with the topics of their dissertations.
No information was gathered from graduate students or faculty regarding what implicit or explicit expectations there are for the use of research sources. Given the nature of master's versus doctoral study, the expectations for master's versus doctoral literature reviews are probably different. Master's thesis bibliographies and literature reviews may not be as closely scrutinized, critiqued, and revised as those of doctoral students. The possibility exists that the significant differences noted between master's and doctoral citation patterns may in part be simply a matter of meeting faculty expectations rather than reflecting the research skill of the individual student. Given this possibility, the hypothesis can not be supported with the data gathered without additional research into how faculty expectations affect the quality of graduate research.
Finally, data and results from this study are probably not generalizable to other engineering colleges due to the variation between different engineering departments in terms of research focus, mentoring, and research instruction offered.
Implications for graduate education
Given that many master's students subsequently go on to get a Ph.D., it is important that they develop skills of critical evaluation that are as rigorous as those expected of doctoral students. Barry (1997) has recommended formal training in literature searching for doctoral students as part of standard graduate training in research methods. Boote and Beile (2005) suggest that a more effective approach would be to integrate the literature review training and process into the whole graduate program, making use of the experience of not only disciplinary faculty, but writing instructors and librarians.
There is little formal training in literature searching for graduate engineering students at WMU. In the University Libraries, librarians have tried to be proactive in meeting this need with voluntary literature review workshops, tailored to specific disciplines and held several times a semester. The engineering workshop offered by this researcher in Fall 2007 resulted in only three participants. Several electrical engineering graduate students have commented to this researcher that they were totally unaware of library resources and services before meeting the engineering librarian. Obviously, more needs to be done. However, given this reality and the results of this study, it appears that WMU engineering graduate students do remarkably well, given the paucity of formal training they receive.
While faculty are no doubt aware of the fact that their graduate students are finding more of their sources via Internet search engines, it appears from the evidence in this study that few of them are steering their graduate advisees away from this practice. To support quality graduate research, thesis/dissertation committees and graduate colleges may need to set guidelines for the use of Internet resources. A good example of the type of statement that could be included in thesis and dissertation guidelines and manuals is the following quote from Spinellis' 2003 article in the Communications of the ACM:
Researchers should appreciate the limitations of Web citations regarding their probable lifespan and use them sparingly rather than gratuitously, keeping in mind the Web is not an organized library. Where possible, they should prefer citing the published version of a work to its online version, and citing materials in organized collections over material in corporate or personal web pages.
Suggestions for future research
Before more solid conclusions can be drawn about the comparative research abilities of master's and doctoral engineering students, further study is needed to evaluate the quality of cited resources in master's and doctoral research, perhaps using criteria like that mentioned in Beile et al. (2004). Ideally, one should compare a large enough sample of theses and subsequent dissertations by the same students to draw a direct correlation between research competency and graduate level. It was not possible to do such a robust comparison at Western Michigan University. Such a study would probably involve obtaining theses and dissertations from a number of institutions in order to gather a large enough sample.
In addition, few studies have analyzed the types of web sites used by engineering students, to characterize them in a functional way (i.e., in terms of what types of information is being sought). Such a study would be very valuable indeed, given the increasing importance of web sites in engineering research.
Also, it would be useful to interview engineering faculty to probe what explicit guidelines or instruction in research skills they give their graduate students, and whether or not the standards or expectations for doctoral versus master's students are indeed all that different.
Conclusion
It is difficult to separate the graduate engineering scholar from the milieu in which he or she operates. This study of engineering master's theses and doctoral dissertations at Western Michigan University, while not proving the hypothesis, shows that there is a significant difference in the proportions of scholarly and other research sources used by master's and doctoral students. What this difference indicates about absolute research competence, however, is unclear at this time, given the lack of information on how faculty expectations influence graduate research work.
Boote and Beile (2005) have made the case that "doctoral students must be scholars before they are researchers." It can be argued furthermore that this should hold true for master's students as well. Citing relevant literature in engineering places the thesis or dissertation research in context, and as such is just as essential in master's work as it is in doctoral work.
Some questions remain after this study. What are the expectations for graduate engineering students in terms of scholarship? Is the bar for doctoral students set higher than that for master's students, given that the Ph.D. is considered the terminal degree for most teaching faculty? The answers to these questions are a direct result of the priorities of the engineering faculty and the institution. It is hoped that future research will uncover the differences in faculty expectations for master's and doctoral engineering students.
Perhaps more importantly, where and how are graduate students expected to learn these skills? There is little formal instruction in literature searching at the graduate level in engineering at WMU. Parry (1998) states that "little is known about the nature of writing norms in doctoral theses, or about how doctoral students learn to master these norms." In her interviews with faculty dissertation advisors, Barry (1997) documents how faculty do not see training of their students in literature searching skills as their responsibility. They would rather say "Go see the librarian" than sit down at a computer with an advisee and show the student how to start a literature review. However, such unspoken or implicit knowledge has a big influence on the development of graduate students into scholars. "Most of it will be acquired slowly through the interaction with others...without anyone ever making a deliberate effort to teach the newcomer the rules of the game" (Gerholm 1990). As always, engineering librarians continue to struggle with the issue of how to effectively assist engineering dissertation and thesis advisors in explicitly teaching their students the "rules of the game."
References
Barry, C.A. 1997. Information skills for an electronic world: training doctoral research students. Journal of Information Science 23(3):225-238.
Beile, P.M., Boote, D.N. & Killingsworth, E.K. 2004. A microscope or a mirror?: a question of study validity regarding the use of dissertation citation analysis for evaluating research collections. Journal of Academic Librarianship 30(5):347-353.
Boote, D.N. & Beile, P. 2005. Scholars before researchers: on the centrality of the dissertation literature review in research preparation. Educational Researcher 34(6): 3-15.
Bruce, C.S. 1994. Research students' early experiences of the dissertation literature review. Studies in Higher Education 19(2):217-229.
Carlson, J. 2006. An examination of undergraduate student citation behavior. The Journal of Academic Librarianship 32(1):14-22.
Carnegie Foundation for the Advancement of Teaching. 2007. Carnegie Classification of Institutions of Higher Education: Western Michigan University. [Online]. Available: {http://classifications.carnegiefoundation.org/} [December 3, 2007].
Davis, P.M. 2003. Effect of the web on undergraduate citation behavior: guiding student scholarship in a networked age. portal: Libraries and the Academy 3(1):41-51.
Garfield, E. 1977a. Characteristics of highly cited publications in the engineering sciences. In: Essays of an Information Scientist, pp. 441-446. Philadelphia: ISI Press.
Garfield, E. 1977b. Journal Citation Studies. 21. Engineering Journals. In: Essays of an Information Scientist, pp. 304-309. Philadelphia: ISI Press.
Garfield, E. 1979. Citation Indexing -Its Theory and Application in Science, Technology, and Humanities. New York: Wiley.
Gerholm, T. 1990. On tacit knowledge in academia. European Journal of Education 25(3):263-271.
Grey Literature Network Service. 2007. Welcome to GreyNet. [Online]. Available: http://www.greynet.org/ [February 8, 2007].
Gross, P.L.K. & Gross, E.M. 1927. College libraries and chemical education. Science 66(1713):385-389.
Hart, C. 2005. Doing Your Masters Dissertation : Realizing Your Potential as a Social Scientist. London; Thousand Oaks, Calif.: Sage.
Hovde, K. 2007. You can't get there from here: student citations in an ephemeral electronic environment. College & Research Libraries 68(4):312-321.
Kriz, H.M. 1977. Citation counting and the future of engineering libraries. Engineering Education 67(7):707-710.
Kushkowski, J.D., Parsons, K.A. & Wiese, W.H. 2003. Master's and doctoral thesis citations: analysis and trends of a longitudinal study. portal: Libraries and the Academy 3(3):459-479.
Lawrence, S., Pennock, D.M., Flake, G.W., Krovetz, R., Coetzee, F.M., Glover, E., et al. 2001. Persistence of web resources in scientific research. Computer 34(2):26-31.
Lovitts, B.E. 2007. Making the Implicit Explicit : Creating Performance Expectations for the Dissertation. Sterling, Va.: Stylus.
Musser, L. 2007. A study of references in mining engineering publications. Issues in Science and Technology Librarianship [Online]. Available: http://www.istl.org/07-winter/article3.html [March 1, 2007].
Musser, L.R. & Conkling, T.W. 1996. Characteristics of engineering citations. Science & Technology Libraries 15(4):41-49.
Parry, S. 1998. Disciplinary discourse in doctoral theses. Higher Education 36(3):273-299.
ProQuest LLC. 2007. Proquest Dissertations & Theses: Full Text. [Online]. Available: {http://www.proquest.com/en-US/} [March 5, 2007].
Spinellis, D. 2003. Decay and failures of web references. Communications of the ACM 46(1):71-77.
Thomas, J. 2004. Web site demise and graduate research: persistence of web pages cited in social work theses. Behavioral & Social Sciences Librarian 22(2):67-77.
Williams, V.K. & Fletcher, C.L. 2006. Materials used by master's students in engineering and implications for collection development: a citation analysis. Issues in Science and Technology Librarianship [Online]. Available: http://www.istl.org/06-winter/refereed1.html [March 1, 2007].
