CP OJ Whole No. 2 Serie 1, No. 2 HARVARD MONOGRAPHS IN EDUCATION THE MARKING SYSTEM OF THE COLLEGE ENTRANCE EXAMINATION BOARD BY L. THOMAS HOPKINS Graduate School of Education Harvard University Series 1 No. 2 " STUDIES IN EDUCATIONAL PSYCHOLOGY AND EDUCATIONAL MEASUREMENT Edited by WALTER F. DEARBORN OCTOBER, 1921 Published by THE GRADUATE SCHOOL OF EDUCATION HARVARD UNIVERSITY, CAMBRIDGE 38, MASS. HARVARD MONOGRAPHS IN EDUCATION SERIES I Studies in Educational Psychology and Educational Measurement. Manuscripts for Series I should be addressed to Professor WALT KB F. DF.AK- BORN, Psycho-Educational Clinic, Palfrey House, Oxford Street, Cambridge 38, Mass. Remittances should be made by check or money order to The Graduate School of Education, Harvard University, Cambridge 38, Mass. Series I of the Harvard Monographs in Education has been established for publishing the results of statistical an,d experimental studies and of educational tests in the general fields of educational psychology and educational measurements. The numbers are as follows: 1. A Comparison of the Intelligence and Training of School Children in a Massachusetts Town. E. A. SHAW and E. A. LINCOLN. IN PRESS. o The Marking System of the College Entrance Examination Board. L. THOMAS HOPKINS. Postage prepaid, 40 cents. Whole No. 2 Series 1, No. 2 HARVARD MONOGRAPHS IN EDUCATION THE MARKING SYSTEM OF THE COLLEGE ENTRANCE EXAMINATION BOARD ^ BY . THOMAS HOPKINS v Graduate School of Education Harvard University Series 1 No. 2 STUDIES IN EDUCATIONAL PSYCHOLOGY AND EDUCATIONAL MEASUREMENT Edited by WALTER F. DEARBORN FROM GRADUATE SCHOOL OF EDUCATION HARVARD UNIVERSITY OCTOBER, 1921 Published by THE GRADUATE SCHOOL OF EDUCATION HARVARD UNIVERSITY, CAMBRIDGE 38, MASS. This study was undertaken at the suggestion of Professor Walter F. Dearborn of the Harvard Graduate School of Education. The writer is greatly indebted to him for assistance and counsel during the progress of the investigation. COPYRIGHT 1921 By L. THOMAS HOPKINS The Marking System of the College Entrance Examination Board This study represents an investigation into the distribution of the marks of the College Entrance Examination Board for the years 1902 to 1920 inclusive. It was made in order to discover if there were any grounds for the strong criticism of the college entrance examinations by New England educators, more especially secondary school principals and teachers. It is published at this time because the Board in its Twentieth Annual Report recognized the existence of sudden and violent fluctuations, from year to year, in the results of the examinations, in many subjects, and voted to employ expert assistance to aid in determin- ing the specific causes. SCOPE OF THE STUDY. The subjects selected were English Readings, Elementary French, Elementary Algebra and Plane Geometry for the reason that they were offered by nearly all candidates, thus involving a relatively large num- ber of cases. The arrangement of marks has been altered somewhat. A sample distribution as published by the board is as follows : Solid Geometry 90-100 75-89 60-74 50-59 40-49 0-39 1916/1152* 1.8% 6.1% 18.2% 12.8% 14.1% 47% Most of the larger colleges and universities admit on a mark of 60 or above while some of the smaller institutions will accept as low as 50. Assuming that the distribution ought to approximate the normal, for reasons which will be established later, and that anyone rated below 50 has failed to pass, the data in each case have been corrected from the above to read as follows: Solid Geometry 1916/1152 90-100 75-89 60-74 50-59 0-49 1.8% 6.1% 18.2% 12.8% 61.1% The highest number of cases involved in any distribution was Ele- mentary Algebra 1920/5249 and the lowest Elementary French 1902/509 with only 13 out of the 76 instances when the number fell below 1000. FACTS BROUGHT TO LIGHT. The following significant facts were discovered: (a) Out of 76 distributions graphed every one is bimodal with the exceptions of : English Readings 1902/800, 1906/1380, 1907/1661, 1908/1698, 1912/1731. In this and all similar cases the numerator of the fraction represents the year and the denominator the number of persons taking the examination. M17581S 4 The Marking System of the In every instance the second mode in the distribution occurs in the assignment of the lowest marks and very often contains a greater percentage of cases than the one in the middle. (&) Every distribution is skewed negatively or toward the lower end of the distribution of marks except : Elementary Algebra 1906/1180, 1913/1916, 1918/3826. Elementary French 1909/1196, 1916/2872. English Readings 1903/996. (c) The order in which the subjects approximate the normal dis- tribution is as follows : English Readings, Elementary French, Elemen- tary Algebra, Plane Geometry. In Figs. I and II are reproduced twenty selected graphs, five for each of the above subjects respectively. EFFECT OF YEARLY INCREASE. Various reasons suggested themselves as to why the results are so far from those expected. Bimodal distributions usually indicate a poor selection of cases. As the second mode in every instance is in the lower end or failure group, this might be caused by the influx of a large num- ber of unprepared persons in the hope of slipping by. This explanation is discarded, however, for (a) the data show that this does not occur at intervals but appears regularly in all subjects, (&) the yearly increase in the number of candidates, with the exception of 1916, has been rela- tively constant as is shown in Table I. RECOMMENDED CANDIDATES. If all candidates of doubtful preparation could be eliminated a different result might be obtained. Consequently graphs were made for the years 1912-1916 inclusive for "only those candidates who were recommended for examinations on the ground of full and satisfactory preparation. ' '* It was found, however, that (a) In Elementary Algebra and Plane Geometry, every distribu- tion is bimodal, seven out of every ten are skewed negatively or toward the lowest grades, while the other three are skewed positively or toward the highest grades. (&) Of the five in Elementary French, four are bimodal and three are skewed positively. (c) In English Readings only one, 1916/2431, is bimodal, all the others tending roughly toward the normal. * Further study of the group could not be made, as only these limited data are pub- lished by the Board. College Entrance Examination Board 2.0 40 14 24 21 .0 37 832 A 382127 8.4 O5 7.6 341 131 26 2.8 20 14 4 r^ 34 13 27 22 29 31 13 31 22 4.7 r^ 35 9.9 30 20 .7 24 425 Fig. I Graphs in the first column represent English Readings, the second Elementary French. The different divisions are as follows : 90-100, 75-89, 60-74. 50 59 0-49 The figures show the percentage of cases. The Marking System of the \O 3 4.6 7.6 46 1423 1244 24 45 M 3.1 241448 )2 23[|4|46 29 1347 3,6 I I 199.756 545 Z.1 241746 12201352 Fig. II Graphs in the first column represent Elementary Algebra^ the second Plane Geometry. Divisions as in Fig. I. College Entrance Examination Board g ? OS Ol ^f 00 fl ^ rH I 1 rH t- .4^ L, o Ol U5 IO ^O t* ^C ^ CO tO OO OO CO t* ^5 OS .* fl CXI rH rH O3 iH WC3 COOiCOtM'^ Ol S I ^2 > s CO C j to CN3 rH OS '^ OS CO OS CO OS CO ^* ^ C s rH "* IO to !> OOoqeqojOcxicocot-t^rH-^co '"S M exi cxicxi eqfMN-^CfB-* * d OQ !> FH Q % * * * * * DC kO ^_ Tf Ol "*"* S >> I- ,j to 10 01 rH cocofsco-*' 't^osocoosio'-tco O ^ V) ^ 3 rH &l ^~- C g c w .2 o M rH O 4-1 rt o a 3 3 C 2 o rH 01 t^ & o Cl trHlraOtO"3cOCOOflCSCXICXIt~ B 's "o 00 03 01 OS 01 01 Wr^^hcoS^^CoS^^CO^CQ rt rH EH rH 1-HrH COCOCO^kO _cj * *2 X O o3 w QJ 'C W) .S K M ,0 C 03 g co rH 00 rH 1 1 * * * co So *" _ 15 !? | I-H rH OS ^ h b rH ll ^^^J - 5 *^ ^ f*^ (U rt | CO OS rHr}(mCXIkOOOnSOO5r- ItOrHOS . Q. rt 03 -t-j ^ 00 e o CO rH NCO^^S^SclcO^COCOrHW gig H J EH u 1 ^ s *a | -c! 4-1 U O> o5 S (O CO 1 1 00 01 i 1 rH * rH ' rHrHOJrH -*IOOSt^OS *? & i-H . " S g rH HJ > C PH y t, a> - H r< ~j 73 ^ O S rj ' H .5 (U m .^ Q , a> rt U CJ "S o> a i ' -S OS kO 01 rH Cl t~ X ISslSr1i3lioiii O ^ >H CP p>| 1"^ c ^^ o EH o 13 'a ^ * * * * fi P*^ 1 1 01 rH t- eo rH Cl CO rH rHt-ODCXItOCO^'COOSOStOCXlcOOS CXI i 1 OJ ^t< 00 rH t~ rf 30 IH OQ 1 u 'ri oa rj 03 aj "1 p3 *-" 00 c co ^ O rHOOOCO^rHlOcO^COt-OlCMcO aJ D=M .S 'be 1 o GC 01 Ol 01 OC co SSS^St2St2SwcO*?2 OQ *H O -f-* ~ c3 W H 1 & Q EH "w o * *- 03 c M ^. M ^ w CO b-OOOiOrHCQCOT^lOCDt^OOOiO S cr. 01 c-. 5. 01 O5OiO^OiOiOiOiOiOiOiO5CiOiO5 H * The Marking System of the It is very evident from this that there is slight improvement in the ratings of the recommended candidates in English Readings and Ele- mentary French but none in Elementary Algebra and Plane Geometry. The difference, however, is not marked enough to conclude that it is due to better preparation. TOTAL YEARLY BANKS. Theoretically, as the number of cases increases the nearer the dis- tribution should correspond to the normal. Graphs were prepared show- ing the distribution of the total number of marks given for all subjects from 1902 to 1920 inclusive for all candidates, and from 1912 to 1916 for recommended candidates only. These show that in every case, (a) the distribution is bimodal, (&) it is skewed toward the lower end. Fig. Ill gives a selected list of graphical representations for totals of different years. If all of the marks assigned in all subjects from 1902 to 1920 in- clusive were combined into one grand total average distribution it would be as follows : Grand Total 90-100 75-89 60-74 50-59 0-49 445,620 4.78% 18.34% 31.14% 13.78% 31.96% In other words out of 445,620 cases only 4.78% received the highest grade while 31.96% failed. How many of the latter tried over again and succeeded there are no data to show. A grand total average distribution for only those candidates recom- mended on the ground of full and satisfactory preparation as published for 1912 to 1916 inclusive is Grand Total 90-100 75-89 60-74 50-59 0-49 87,642 6.35% 22.32% 32.28% 13.69% 25.36% This is slightly better than the one given above, but considering the fact that the individuals involved here were highly selected, a failure of one-fourth, or 21,910 cases out of 87,642, places upon the Board the re- sponsibility for a condition which is far reaching in its social and eco- nomic effects. SELECTED DISTRIBUTIONS. That the reader may have some samplings of extreme variations as a basis of comparison a selected list of graphs is given in Fig. IV. These are taken from different subjects and different years. The lowest num- ber of cases involved is 641 while the highest is 2063. WHAT WAS EXPECTED. As was said at the beginning of this article, it was expected that the results would approximate the normal distribution. Briefly the evi- College Entrance Examination Board 4.S 31 14 4.1 37 15 3.4 32 36 3.6 8 14 33 5.6 8 14 31 4.1 31 /5 33 17 30 15 32 19 4.1 r^ 29 IS 15 35 5.2 31 17 5*2 2.9 31 35 Fig. Ill Totals for different years. Number of marks assigned will be found in Table 1. Divisions as in Fig. I. 10 21 5? 8Jj &2 klT IS 67 7.4 73 57 0,1 The Marking System of the '?, mil U^ |I3 I.Sr l.i Fig. TV Selected distribution in different subjects and years. The range of cases involved is from 641 to 2063. Divisions as in Fig. I.* College Entrance Examination Board 11 dence supporting this is as follows: (a) Physical differences approximate the normal curve* as do mental characteristics, f (b) Marks, represent- ing, as they do, estimates of mental abilities, are themselves distributed according to the same frequencies as the abilities they are designed to represent,! (c) The normal distribution of marks is the one usually found when a fairly large number of students are graded. Concluding then that the assignment of any relatively large number of grades ought to approximate the normal distribution and steadily so as the number increases over 500, this further question remains: What is the best method of dividing this distribution into groups for translat- ing standing into a scale of marks? After a careful examination of all possible schemes we have concluded that the five division one is best. This is based on the orientation of a large number of cases around a central group whose accomplishment is considered median or average. Above and below lie groups of smaller size containing superior and in- ferior students in relation to the average and above and below these the still smaller groups of exceptions or failures. The method of dividing our theoretical distributions into the five divisions which we will represent by tb,e letters A, B, C, D, E, would be as follows : Find the median of the distribution and lay off on the base, on either side, the distance of 1 P. E. Within the area embraced by this P. E. there will fall 50% of the total number of cases. This would rep- resent the center or average or C group. Now lay off on either side of =h P. E. a distance equal to 2 P. E. Each one of the areas thus designated will contain 23% of these cases,|| and would be represented by the let- ters B and D respectively. Again laying off the distance of 2 P. E. on either side we will reach the limits of the normal curve as for all practical purposes the ordinate may be taken as zero when the abscissa is 5 P. E. The last two divisions just made would each contain 2% of the total number of cases and would be represented by the letters A and E. The relationship between the cases represented by the five divisions of our normal probability integral and our marking system would now be as follows :fi A B C D E 2% 23% 50% 23% 2% * Brooks: The Foundation of Zoology, pp. 156-157, and Yule: An Introduction to the Theory of Statistics, p. 84. t See the distribution of the IQ's of 905 unseleeted children 5-14 years of age in Terman: The Measurement of Intelligence, p. 66. t Dearborn: School and University Grades. University of Wisconsin Bulletin No. 368. $ Dearborn, Ibid, also Foster : The Administration of the College Curriculum, pp. 250-300. H A table of the values of P. E. of the normal probability integral will be found in Kugg: Statistical Methods Applied to Education, p. 391. 1f This was the division used by Buckingham in the standardization of the Bucking- ham Spelling Scale. 12 The Marking System of the In like manner if we should lay off on either side of the mean the distance of A. D. we would find the following distribution: ABODE 2% 20% 56% 20% 2% or if we should take for our unit .5