A Summary Of Research With The Console-Operations Tutor: LOADER
John D. Farquhar
LOADER is a complex problem-solving computer-based environment requiring the operation of a simulated control-panel console. Three separate studies have used LOADER to analyze the acquisition of procedural skill utilizing both visual and instructional feedback. This brief paper summarizes capabilities of the LOADER tutor and presents results from each of the three studies.
Description of the Software
Analogous to many military and industrial tasks that call for knowledge of various procedures, LOADER is a laboratory research tool developed to assess instructional strategies in the acquisition of console-operation skill (Farquhar, 1993a). The task simulated by the LOADER environment is the operation of a remote crane control arm to load various canisters from a set of storage bins to one or more railroad cars (see Figure 1 for the simulation interface). Simple tasks include the loading of a single canister whereas complex tasks require the operator to analyze the availability and capacity of railroad cars in order to appropriately position and load a number of different-sized canisters. Complexity of the problem depends upon variables such as the number and size of canisters to load and the position and capacity of rail cars. Simple tasks may be performed with as few as 5 steps while complex procedures may require "solving" several subgoals and consist of as many as 150 steps.
Located on the upper half of the LOADER interface is the dynamic graphical model. This model provides a representation of the loading task from a birds-eye-view perspective. Positions of the rail cars, storage bins, canisters, doors, the overhead crane, and thentially fromdicated with a number of graphical elements. Colors aid the identification of canister sizes and rail car capacities.
The dynamic model responds to control panel interactions with animated sequences representing their outcomes. For example, engaging a selected rail car south with the button labeled "South," results in the animated motion of the rail car moving down toward the rail docks. In addition to the motion of rail cars, car doors and bin doors open and close, the crane moves above the rail cars and storage bins, and the crane lowers, grasps and lifts canisters.
In addition to displaying visual feedback in the form of a dynamic model, LOADER is capable of delivering a variety of instructional feedback messages. These messages represent levels of information including notification and explanation of the error, general to specific descriptions of the subgoal procedure, as well as indicating the location and operation of the expected next step. An expert module solves LOADER problems given a scenario bounded by the position and capacity of rail cars, the size and location of canisters, the position of the crane, and all current knob and switch settings. Thousands of scenarios can be easily defined for the system by stating the position of the various elements. Each problem scenario can be automatically solved by the expert system. The type and timing of instructional feedback is determined by the tutor's instructional module which makes decisions based upon the type or error made and the experience of the user. Considering the six feedback levels and the variations of messages available within each level, it is possible to encounter hundreds of different feedback messages.
Software and Hardware Specifics
LOADER is programmed entirely in OpenScript using the ToolBook software development environment from Asymetrix Corporation. Software requirements include MicroSoft Windows as well as ToolBook run-time files. The program is specifically designed for '486 compatible computers with monitors of 800 X 600 screen resolution.
Study 1: Visual Feedback
The first study investigated the use of the dynamic model which represents the loading task in an interactive animation (Farquhar, 1993b). Twenty-nine subjects were divided between two treatment groups. One group received instruction with the tutor which included the dynamic model. The second group completed the same tasks without the model.
After completing a series of learning trials, subjects were tested with a final procedure on the tutor. In this final session, both groups were given the same procedure without the dynamic model present. While the practice trials revealed no significant differences between groups, the test trial revealed that the group receiving the dynamic model during instruction completed the test faster and with fewer errors.
Study 2: Instructional Feedback Performance
The second study included one hundred twenty-three subjects (Farquhar & Regian, 1994). Independent variables of the study consisted of the type of feedback and the timing of feedback for non-critical errors. Each variable had two levels, forming a 2 by 2 matrix. The levels of type of feedback were elaborative and corrective feedback, whereas levels of timing were immediate or delayed for non-critical errors. The resulting subjects per treatment condition is shown in Figure 2.
Elaborative feedback includes one or all of the following items: 1) a notification of an error; 2) a reason why the response was incorrect; 3) a brief description of the appropriate subgoal; 4) a detailed list of steps to complete the subgoal; 5) an indication of progress toward completion of the subgoal; and, 6) a direct identification of the very next step. Corrective feedback, on the other hand, simply gave notification of an error while identifying the correct answer.
The immediate feedback condition delivered feedback (either elaborative or corrective) upon the execution of any type of error. Conversely, the delayed feedback condition makes a decision on when to provide feedback. If the error was determined to be critical (or crucial to the completion of the problem), immediate feedback was given. If the error was determined to be non-critical, then the error was recorded, but not immediately reported to the student. Under this condition, all errors were reported in an end-of-problem information screen.
Dependent variables of the study included the speed and accuracy of two achievement tests. The two achievement tests each consisted of a single procedural problem equivalent but not identical to the procedures experienced during practice. Two training sessions were conducted. The first test was conducted after the first training session, while the second test was conducted after the second.
Results show that when immediate feedback was delivered, subjects receiving elaborative feedback performed fewer errors on the achievement measure than subjects receiving corrective feedback. When corrective feedback was given, the delay of feedback for non-critical errors resulted in improved performance over the use of immediate feedback for all errors. This difference was shown for both speed and accuracy of performance. Figure 3 shows the interaction for the time to complete the achievement test, while Figure 4 shows the interaction for number of errors across the achievement tests
Study 3: Instructional Feedback Preferences
In a continuation of Study 2, preferences for various feedback were sought. (Farquhar, 1994) Using a questionnaire, subjects evaluated the four different types of feedback delivery as indicated by Figure 2. An analysis of responses showed a general preference for elaborative feedback over simple corrective feedback. Subjects preferences for the timing of feedback was mixed indicating this preference may depend upon individual differences such as skill proficiency.
Farquhar, J.D. (1993a). The design, development, and preliminary evaluation of LOADER: A tool for investigating the acquisition of console-operation skill. Technical report submitted to the Air Force Office of Scientific Research Summer Research Program. (submitted 9/15/93)
Farquhar, J.D. (1993b). Acquisition of console-operation skill through a dynamic graphical model. Insight: The Visual Performance Technical Group Newsletter. 15(2), 5-7
Farquhar, J.D. (1994). Effects of type and timing of feedback for non-critical errors within an intelligent tutoring system. Doctoral dissertation defended July 1994.
Farquhar, J.D. & Regian, J.W. (1994).The type and timing of feedback within an intelligent console-operations tutor. Paper presented at the 1994 Conference of the Human Factors and Ergonomics Society.
Cite this document as: