Proposal for Computerized Administration of the Regents' Essay Test

(This is a draft, working document presented for discussion only. Please email reactions or suggestions to Kathleen Burk (email: kburk@gsu.edu)

A proposal to computerize the administration of the Regents' Essay Test was initiated in 1993 and has received strong support from a variety of groups. In 1999, it was recommended that the Regents' Test be used to ensure that University System graduates have technological competence by requiring all students to compose their Regents' Test essays through computerized administration. Computerized administration of the Regents' Essay Test, whether optional or required for all students, has the potential for having a major impact on students and instruction throughout the System.

By requiring students to use pen and paper to write their essays, the current system may be doing students a disservice and creating unnecessary constraints on instruction. A number of institutions rely on the use of word processing rather than handwriting for freshmen English composition classes and require students to handwrite some essays only for practice for the Regents' Test. Instructors of these courses have requested that students be tested in a manner more similar to the way composition is taught. A revised process would accommodate not only the students who have become accustomed to using computers for writing assignments in high school and college but the many older non-traditional students who have come to depend on word processing in their work settings.

Computerized administration would also allow a more flexible administration period. The Regents' Test is currently given during only one week each semester. A more flexible schedule for test administration and rating would help to provide students with sufficient opportunities to take the test and remediation. Because computerized essay administration would allow electronic transmission of essays and on-line rating of essays, the test could be administered and rated throughout the semester depending on student need rather than being limited to a one-week period.

Although the primary reason for proposing computerized testing is to benefit students and instruction, the essay rating process could also be improved through a computerized system. Word-processed essays would not be printed but made available on-line for faculty members to rate on their office computers. An on-line rating system could incorporate a number of features that are not now practical when a large number of raters must meet in one place at one time to grade handwritten essays. For example, an on-line system would be very efficient for monitoring the performance of raters to identify those who are not using the appropriate standards and providing immediate retraining and rater calibration as needed. Also, the increasing number of essays is almost exceeding the capacity of the current rating system. It is becoming unwieldy to have approximately 40,000 essays per year rated by three raters on only two Saturdays each semester. An on-line system also would eliminate the paper-shuffling process. Currently, each essay is shipped a total of six times, with extensive handling in between each shipment.

The possibility of allowing students the option of taking the Regents' Testing Program essay test with a word processor was discussed initially at the 1993 meeting of the Academic Committee on English. A committee of test coordinators and faculty members, members of the Testing Subcommittee of the Academic Committee on English, Scoring Coordinators, and Regents' Test Coordinators have considered various issues related to the implementation of computerized essay testing. The Regents' Testing Program Office has also conducted studies on the rating of handwritten and typed essays and found no reason not to allow the option of word-processed essays.

A survey was conducted in 1998 of Vice Presidents for Academic Affairs, Members of the Academic Committee on English, Regents' Test Coordinators, Vice Presidents for Student Affairs, Registrars & Admissions Directors, Members of the Academic Committee on Learning Support and other interested University System personnel. Over 75% agreed that students should have the option of taking the essay test through a computerized system, while only 6% disagreed.

In 1999, the following recommendation was made by the Academic Committee on English and unanimously approved by the Administrative Committee on Academic Affairs:

The English Academic Committee recommends that in order to ensure that all University System of Georgia graduates are technologically proficient all students take the Regents' Test on the computer, with a phase-in period of 3-5 years.

Following is a summary of some of the details that have been discussed and recommendations made about the proposed system for essay test administration and scoring.

Proposed Features of the Regents' Test Computerized Essay Administration

• In order to ensure test security and standardization, all students would use the same simple word processing or editing software. Features would be limited to basic editing commands (delete, backspace delete, block move, possibly underlining), a spelling checker, and a word count. Simple instructions and a help feature would be available. Students would have the opportunity to practice writing an essay using the system before the date of the test administration and would have a short amount of time (e.g., 10 minutes) to practice at the beginning of the test administration session. The package would be simple enough for students familiar with any word processor to use for the first time during the test administration.



• The system must be designed to ensure test security and standardization while minimizing the involvement of lab personnel or proctors in the test administration. The following features of the system would be useful or essential:

presentation of topics and a method for student identification



timing of test -- showing time left, providing warning near the end of the test, and terminating when time is up



prevention of electronic access to other documents through hard drive or other media



timed backup and save


• Proctors would have to ensure that student identification is in order, that students do not receive assistance from other students, and that students do not have access to materials not on the computer.



• Essays will not be printed but transmitted and rated on-line by faculty at all institutions.



• There appears to be consensus that students using the word processor should have the same time limit as those taking the handwritten version of the essay test.



• An expanded period of test administration would be necessary and desirable. An on-line rating system would allow students to take the test and receive their results anytime during the semester rather than during a one-week period.

Proposed Features of the Essay Rating System

The on-line system would allow raters to work on a flexible schedule. The process would begin by providing the rater with instructions and a sample of calibrated essays for training and practice. When the rater is rating the practice essays appropriately, actual essays would be randomly selected (based on a set of rules) from the pool of essays that are to be rated. Only the topic and essay would be shown to the rater, and the identification information would be maintained separately. The rater would record the rating for each essay, and periodic monitoring would take place. For example, if a rater is passing or failing an unusual number of essays, is not agreeing with ratings given by other raters, or is taking an unusually short or long time on each essay, retraining could be provided. In addition, all raters might be given practice essays occasionally to ensure that they are using the appropriate standards. Raters who do not use the appropriate standards even after retraining would not be given further essays to rate.

A method of recording the ratings, matching them with the student identification information, and reporting the results would be necessary. In addition, consistent with current practice, a periodic report of rater performance would be provided to the rater, the Regents' Testing Program Office, and the English department head at the rater's institution.

The Academic Committee on English and its Testing Subcommittee would be consulted further about the specific process used for rating the essays. There is general agreement that handwritten and word-processed essays be rated separately but that the same standard be applied.