Batting averages change a lot from year to year because player performance changes a lot from year to year – injury, age, steroids, and confidence make a big difference from year to year. Value added scores should not change a lot from year to year because teaching performance does not change a lot from year to year. I am pretty much the same teacher this year as I was last year. Batting averages reflect actual changes in performance. Value added scores mostly reflect random factors, not teaching performance.

On the SAT, a student will generally get very similar scores on multiple retakes of the SAT, so it is a consistent test. It is a crude, but consistent measure of a narrow sort of ability. That is why the colleges use it. It is true that it doesn’t predict GPA very well, but that isn’t what it is used for. Value added will be used as a measure of teacher effectiveness even though it is not reliable. SAT is not used to predict student GPA in college, so it doesn’t matter that it does a poor job of it.

Hattie, in Visible Learning, points to a study showing increase in “self-directed learning (d=0.54) and attitude toward learning (d=0.52).

The catch is its an unpublished study.

Smith, R. A. (2003) Problem-based versus lecture-based medical teaching and learning: A meta-analysis of cognitive and non-cognitive outcomes. Unpublished Ph.D., University of Florida

‘We do not advocate using value-added measures alone when making decisions about hiring, firing, tenure, compensation, placement, or developing teachers, but surely value-added information ought to be in the mix given the empirical evidence that it predicts more about what students will learn from the teachers to which they are assigned than any other source of information.’

This is a striking conclusion, and one that should reframe the debate about this measure amongst teachers. The stigma attached to value-added- a heartless measure that doesn’t take into account the ‘art’ of teaching- is methodically (almost politely) dismantled by Glazerman and his people. One of those sharp and lean arguments where you can go in thinking one thing and come out shaking your head.

It appeals to those interested in policy, for obvious reasons, but also to fans of statistics. The comparison between SAT and batting averages was fascinating.

Coupled with the ‘moral conversation’ reading (the only two I’ve gotten to thus far), this is already an outstanding list. DM make this a bi-weekly or monthly feature, if you have time. Thanks man.

How do you manage to motivate students over a year to submit to that kind of instruction, which Sweller et al. admit they’ll probably find miserable?

Sadly, in my experience, students subjected to that kind of instruction for long enough eventually acclimate to such a degree that any other approach to learning mathematics (to use those two words loosely) is unthinkable. While they may not enjoy that kind of instruction, they become so comfortable with it that they vehemently object to anything else.

And then there are the students who actually do enjoy that kind of instruction, and do well under it, and believing they are “good at math,” while they may really just be good at cataloging and following directions.

Alex: It would be interesting to see a study comparing student engagement or long-term results in PBL versus some kind of worked-problem curriculum.

I agree. That kind of study is unlikely to be randomized or controlled, though, so don’t expect it from Sweller et al, no matter how essential it is to the matter at hand.

It would be interesting to see a study comparing student engagement or long-term results in PBL versus some kind of worked-problem curriculum.

Ayep. I would still qualify Kirschner and Sweller’s “awareness” of PBL variations in air quotes.

I will be there with all the giddiness of a screaming fan at a rock concert, where you are the rockstar. Thank you for your blog and challenging the status quo. You inspire me with you dedication and innovation.

Keep up the great work.

Jason: This can lead to (for example) the simplification that all problem-based learning is conceptually the same, when there are multiple approaches and contexts with significant differences where some criticisms may apply and some may not.

That’s the objection raised by Hmelo-Silver et al but Kirschner et al specifically address it:

Furthermore, while scaffolding, like all guidance for novices, is better than no scaffolding, the ultimate scaffold, providing learners with all information needed including a complete problem solution–either prior to a task or just-in-time during a task–is better still (2007, p. 117)

Basically, they acknowledge there are many different forms and degrees of PBL but then note that all of them are the same in offering something less than 100% guided instruction through worked examples. Which is most effective. Therefore all forms of PBL are less effective.

This can lead to (for example) the simplification that all problem-based learning is conceptually the same, when there are multiple approaches and contexts with significant differences where some criticisms may apply and some may not. (Specific example: the Kirchner et. al. mentions the criticism that PBL methods can lead to a difficulty in generalizing and that students trying to explain science were only able to explain them in the context of specific cases. How does this account for, say a PBL approach to geometry proofs where each specific problem/proof *is* the general thing being taught?)