Design considerations for teaching as research
Aspect
Explanation
Teaching as research
  1. Empirical work in education research adequate for publication usually requires adhering to the best research design that is attainable given the configuration of learners.
  2. Assessing learning outcomes for accrediting bodies may require comparisons of learning under different experimental conditions.
Learning outcome (performance) choices

Selecting learning outcomes that embody professionals' authentic performances (the tasks that professionals perform and the deliverables they create) eliminates memorization as a successful learning tactic and increases the appeal of the project to research journals because of the growing importance of accelerating the development of job-enabling expertise in organizations.

Intact sections as the context The naturally occurring context for generating and evaluating evidence of learning is a set of intact sections of courses that are scheduled by departments. The existence of intact sections facilitates treating all students in the section in the same way, e.g., as a control or treatment condition.

Potential design tradeoffs engendered by intact sections

  1. Although it precludes randomization of participants across conditions within a section, treating all students in a section the same way with respect to learning activities reduces the likelihood of student objection to differential treatment.
  2. Although it may reduce student objection to differential treatment, randomizing by term in which all sections in a term experience the same condition introduces the threat of maturation (differences over time) to the research design.
Covariate availability

Some useful covariates may not be available, e.g., institutional records do not contain SAT scores or high school GPAs for transfer students, which may make it desirable to create appropriate pre-measures within the course.

Minefield of expectations about instruction
  1. Some students may object to experimentation as a sign that they may not be receiving the best possible education (nevermind that "the best possible education" would be subject to validation by research). Experiments may prompt students to believe that they may:
    1. Receive instruction of unproven validity
    2. Have to do more work when the condition is new
    3. Receive instruction that would disadvantage them
  2. Some administrators may not not want to permit experimentation because that might imply something deficient about current or prior instruction or because it is time consuming to respond to student comments about it.
Institutional Review Board (IRB) approval

Research on learning in courses requires IRB approval, but IRB promises to respond in 48 hours for this kind of research, and approval for classroom research lasts as long as there are no changes. Forms required by the Institutional Review Board are available online from the GSU Office of Research Compliance.

Sample designs

Business process modeling as a predictor of:

  1. Audit performance
  2. Database querying and audit performance

An example with a design error. Can you find it?

Updated November 16, 2005