MEMORANDUM

TO: Members of the Academic Committee on English

FROM: Kathleen Burk

SUBJECT: Rater Performance Report

Enclosed is the annual Regents' Testing Program Rater Performance Report for the essay raters from your institution. The data are presented for fall through summer. System averages are provided for use in interpreting the statistics for individual raters.

It is important that you closely inspect the performance of raters representing your institution. Pay particular attention to the percentage of failing ratings (ones) given by each rater in comparison with the system average percentage failing ratings. It is extremely unlikely, for example, that raters who failed more than 40% or fewer than 15% of the essays are rating according to the standards specified for the Regents' Test. Unless these raters rated a very small number of essays, they are most likely using different standards. Also, raters with percentage agreement statistics as low as 80% are probably not using the same standards for rating as other raters.

Please identify any raters who appear to be using a different set of standards and require that they review the Essay Scoring Manual before returning to a rating session. Raters who use their own standards because they do not agree with the Regents' Testing Program standards should not be sent to future rating sessions.

Please contact the Regents' Testing Program office if you have any questions about this report.

Enclosures

Reported for each rater are the number of days rated each quarter, the total number of days rated for four quarters, the total number of essays rated, the number of essays rated per day, the agreement percentage, and the percentage of ones, twos, threes, and fours given over the four-quarter period. (The agreement percentage is the percentage of essays for which the pass/fail rating was the same as the final pass/fail score the essay received. It is not reported for raters who gave fewer than 15% ones. Although this statistic is often high for these raters, it may be misleading for raters with such high passing rates.)

The following table of system averages is provided to assist you in interpreting the information for individual raters.

                                    X Y Z  COLLEGE

Name     Days/Qtr       Number   Total  Rate   Percent      DISTRIBUTION OF RATINGS   
       Participated      Days   Essays (E/DY)  Agrment                              
        F   Sp  Su                                          1       2       3      4   
______________________________________________________________________________
__________
        0   1   0         1       69    69.0    81.0      27.54   46.38   24.64   1.45
        0   1   0         1       84    84.0    82.0      34.52   46.43   19.05   0.00
        0   1   0         1       68    68.0    97.0      13.24   67.65   17.65   1.47
        0   1   0         1       77    77.0    96.0      14.29   58.44   25.97   1.30
        0   0   0         1       96    96.0    89.0      38.54   45.83   13.54   2.08
        0   1   0         1       84    84.0    87.0      27.38   47.62   22.62   2.38
        0   0   1         1       96    96.0    78.0      38.54   51.04    9.38   1.04
        0   1   0         1       74    74.0    86.0      27.03   47.30   22.97   2.70
        0   0   1         1       54    54.0    87.0      24.07   51.85   24.07   0.00
        0   1   0         1       92    92.0    80.0      36.96   47.83   13.04   2.17
        0   1   1         2      317   158.5    87.0      26.81   64.35    7.89   0.95
        0   1   0         1       91    91.0    93.0      14.29   62.64   23.08   0.00
        0   2   1         3      268    89.3    88.0      19.78   59.70   19.78   0.75
        0   1   1         2      149    74.5    89.0      16.11   58.39   22.82   2.68

Last updated: August 28, 1998