A summary of ‘Improved Learning in a Large-Enrollment Physics Class’ by Deslauriers, Schelew & Wieman

Deslauriers et al. (2011) measured learning outcomes from research-based instruction in a university introductory level physics class. The outcomes were compared to outcomes from tutoring the same material in traditional lectures by an experienced and highly rated instructor. That is, the latter was used as control in a purportedly controlled experiment.

science332Deslauriers et al. (2011) found that students subject to research-based instruction scored more than twice as well as students subject to traditional lecture tutoring (74% vs. 41%). Both attendance and engagement reportedly went up. The authors were seemingly puzzled by the increased attendance (‘reason not known’), while it seems obviously a case of rubbernecking or something similar but more sophisticated. (Google awaits!)

The research-based instruction in question (it could be almost anything) was designed to have students engage in deliberate practice at thinking scientifically. Thinking scientifically was taken as making and testing predictions and arguments about relevant topics, solving problems, and questioning their own reasoning and that of others, presumably after the appropriate reasoning was revealed. Multiple (unspecified) ‘best instructional practices’ were incorporated, but the educational benefit was believed to derive from the overall, integrated deliberate practice framework rather than any particular practice. Class schedules were mostly an alternating sequence of discussions in small student groups, clicker questions, and instructor feedback. That the measured engagement in such classes surpasses the measured engagement in traditional lectures comes as no surprise.

While I am personally convinced that active learning and deliberate practice better provide learning than traditional lectures, I have doubts regarding size of the reported effect and both internal and external validity of the experiment.

Could other factors influence the measured learning outcomes and reduce the real effect of research-based instrution? Two obvious factors are that the research-based instruction was more resource intensive (the instructor had an assistant in class throughout the experiment; materials were pilot-tested before used in class) and contrbution of the Hawthorne effect (‘where any change in conditions is said to result in improved performance’). Regarding the latter, one may add temporarily. The Hawthorne effect is, interestingly enough, dismissed by Deslauriers et al. (2011) because the effect could not be detected in the original Hawthorne data. But that the effect was not present in some data does not mean that the effect does not exist.

How well was the experiment controlled? The experiment group (subjected to research-based instruction) was instructed by two of the authors of the study, and not the usual instructor. This is presented as something that should reduce the measured effect because the authors had less teaching experience and background knowledge in physics. Clearly, the experiment would be better controlled if the usual instructor was trained and assisted (outside of class) in research-based instruction. The experiment would also be better controlled if the experiment and control group had had the same instructor prior to the experiment. Alas, they did not. Both of these improvements would have been readily achievable.

Can the findings be generalized outside the experiment, like tutoring in other subjects, and at all university level tutoring? The experiment lasted for one week, so the foremost concern is whether the research-based instrution would provide a similar effect on the full-scale course level. I am sure it would, but research-based instruction, as argued above, does require more resources than the traditional lectures. A ‘twice as well’ effect is probably unlikely, however (Hawthorne again, and temporality).

In a subsequent survey of the students subjected to the research-based instruction, students express satisfaction with the alternative form of instruction. This finding is taken as evidence against concerns of student opposition towards changes in the instruction. But not all students answered the survey, and measures were taken within the experiment group to avoid resistance against the instructional approach. The survey evidence is thus questionable if not useless.

It amazes me that a study with so many obvious weaknesses is published in one of the most prestiguous scientific journals.

Deslauriers, L, E. Schelew, C. Wieman (2011), Improved Learning in a Large-Enrollment Physics Class. Science 332, 862 – 864.


Tags: , ,

%d bloggers like this: