First, Michael Goldstein:

Khan Academy alone gives the following information: time spent per day on each skill, total time spent each day, time spent on each video, time spent on each practice module, level of mastery of each skill, which ‘badges’ have been earned, a graph of skills completed over number of days working on the site, and a graphic showing the total percentage of time spent by video and by skill.

Second, Jose Ferreira, CEO of Knewton:

So Knewton and any platform built on Knewton can figure out things like, “You learn math best in the morning between 8:32 and 9:14 AM. You learn science best in 40-minute bite-sizes. At the 42-minute mark your clickrate always begins to decline. We should pull that and move you to something else to keep you engaged. That thirty-five minute burst you do at lunch every day? You’re not retaining any of that. Just hang out with your friends and do that stuff in the afternoon instead when you learn better.”

I don’t have a lot of hope for a system that sees learning largely as a function of time or time of day, rather than as a function of good instruction and rich tasks. It isn’t useless. But it’s the wrong diagnosis. For instance, if a student’s clickrate on multiple-choice items declines at 9:14 AM, one option is to tell her to click multiple-choice items later. Another is to give her more to do than click multiple-choice items.

These systems report so much time data because time is easy for them to measure. But what’s easy to measure and what’s useful to a learner aren’t necessarily the same thing. What the learner would really like to know is, “What do I know and what don’t I know about what I’m trying to learn here?” And adaptive math systems have contributed very little to our understanding of that question.

For example, a student solves “x/2 + x/6 = 2” and answers “48,” incorrectly. How does your system help that student, apart from a) recommending another time in the day for her to do the same question or b) recommending a lecture video for her to watch, pause, and rewind?

Meanwhile, these trained meatsacks have accurately diagnosed the student’s misunderstanding and proposed specific follow-ups. That’s the kind of adaptive learning that interests me most.

Featured Comments

Chris Lusto:

But then we’d need like an entire army of trained meatsticks, each assigned to a manageably small group of students, possibly even personally invested in their success, with real-time access to their brains and associated thoughts, perhaps with a bank of research-based strategies to help guide those students toward a deeper understanding of…something.

That seems an awful lot like a world without clickrates, and I’m not sure it’s a world I want to live in. Or maybe I’m just cynical between 11:30 and 12:00, on average, and should think about it later.

Dan Anderson:

A big advantage with meatsacks over computers is the ability of a human to look at the work. Computers can only indirectly evaluate where the student went wrong; they can only look at the shadow on the ground to tell where the flyball is going. Meatsacks can evaluate directly where the student is going awry.

Emma Brown on Teach to One, a rebranding of School of One, which attempts to teach math by warehousing students in front of laptops:

Results so far have been mixed. Two of the initial three New York schools dropped the program at the end of that first year.

In what universe are those results “mixed”?

Annie Murphy Paul, describing systems that attempt to adapt to what you know and don’t know about math:

Tyler breezed through the first part of his homework, but 10 questions in he hit a rough patch. “Write the equation in function form: 3x-y=5,” read the problem on the screen. Tyler worked the problem out in pencil first and then typed “5-3x” into the box. The response was instantaneous: “Sorry, wrong answer.” Tyler’s shoulders slumped. He tried again, his pencil scratching the paper. Another answer – “5/3x” – yielded another error message, but a third try, with “3x-5,” worked better. “Correct!” the computer proclaimed.

S.H. Erlwanger [pdf] forty years ago:

Through using IPI, learning mathematics has become a “wild goose chase” in which [Benny] is chasing particular answers. Mathematics is not a rational and logical subject in which he can verify his answers by an independent process.

See if this describes your adaptive learning startup:

A basic assumption in [your startup’s name here] is that pupils can make progress in individualized learning most effectively if they proceed through sequences of objectives that are arranged in a hierarchical order so that what a student studies in any given lesson is based on prerequisite abilities that he has mastered in preceding lessons.

I don’t have anything against personalization per se. But the technology that enables that personalization defines and constrains the math we can personalize. Currently it defines that math very, very narrowly.

Individualization in [your startup’s name here] implies permitting him to cover the prescribed mathematics curriculum at his own rate. But since the objectives in mathematics must be defined in precise behavioral terms, important educational outcomes, such as learning how to think mathematically, appreciating the power and beauty of mathematics, and developing mathematical intuition are excluded.

Look, if you’re building one of these systems, you have to read and understand Benny’s Conception of Rules and Answers in IPI Mathematics. Ask questions here. Let’s figure this out. We’d all love for you to make some interesting new mistakes. Right now you’re just repeating mistakes that are forty years old.

BTW. In Education’s Digital Future last night, I said I felt the next Kasparov v. Deep Blue competition would be between a grandmaster teacher and an adaptive learning engine. Give them both some written student work. Which one can accurately identify what the student knows and doesn’t know and what to do next?

I said this in a small group and a couple of technologists razzed me. One said that he doesn’t even get that kind of feedback in the lecture halls at Stanford, which is totally fair, though that isn’t the model I’m defending.

Another said, “Actually, computers are already better.” He told me that adaptive systems can tell you the best time of the day for you to study, how much time you spend on problems, the answer you choose most often when you’re stuck, and a bunch of other metrics that are simple enough to parse from a student’s clickstream. Of course, not one of them addresses the student’s most pressing question, “Why am I getting this answer wrong?” So, like Benny, the student clicks a different answer and the wild goose chase begins again.

Scott Elias:

[Flipping your classroom] carries a load of assumptions, including (minimally) the fact that students (1) have access, (2) will bother to watch it, and (3) have the skills to process and make meaning of what they’re watching (note-taking, summarizing, and the ability to ask good questions about what they don’t understand for starters). In my experience, these skills often need to be explicitly taught and scaffolded for students.

Brian Stockus:

What is with the insistence on the lecture (direct instruction) model? Teachers appear to be loving the ability to offer more engaging, open-ended activities in class now that students are watching lectures at home. What was stopping these teachers from offering these kinds of activities before? Why do teachers think students have to be told what to do before they actually do any math?

In case you missed it, Justin Reich and I are co-sponsoring the #mtt2k prize and the eligibility window for applications closes August 15. Upload some commentary on a Khan Academy video to YouTube and tag it #mtt2k. You could win a few hundred dollars to take the missus or the mister to the boardwalk before school starts.

Here is my submission, playing out of competition.

If you couldn’t make it through the setup (a Khan-style explanation of Angry Birds) here is the punchline:

Okay, wait. Obviously, Khan Academy would never lecture about Angry Birds. But what makes Angry Birds different from math and science? Angry Birds makes it easy to play, experiment, get feedback, and learn. I’m not saying lectures and explanations are never necessary in math and science – or in Angry Birds, for that matter. When I couldn’t get past that one really tricky level, I went online and found a walkthrough. But the walkthrough – the explanation – wasn’t the first thing I did when I experienced Angry Birds. So why does Khan Academy make an explanation the very first thing a student experiences with a new topic in math. When we put the explanation first, we get lousy learning and bored students.

Comments open until I come to my senses.

17 hours later. Comments closed. I couldn’t handle it. Sorry.