Of course, when training Matsuda’s program, you need to feed it LOTS of examples, so it can generalize successfully. As you say, isolated examples won’t work. But that’s just a problem of scale.

I guess I don’t really understand your point when you talk about computers not being able to have concepts. If you taught a computer the rule, “When you divide both sides of an equation by something, you must divide each term by that quantity”, is that not a concept? That essentially is the concept of distribution.

It really doesn’t matter whether the concept was “discovered” by a learning program such as Matsuda’s, or was programmed in (which is how most cognitive tutors acquire their production rules).

Again, I’m not at all saying that programs like CTAT can’t be useful. But they’re still light-years away from having anything like real _concepts_. And _misconceptions_ are mistakes involving concepts.

Dan @ddmeyer recognizes that he himself obsesses with motivation. Ideas about motivation, especially those of intrinsic and extrinsic motivation, live in the world of behaviorist learning theory that western culture knows so well we have a hard time knowing/thinking outside of of it (like fish & water).

The present constructivist theory of knowing and learning, superseding behaviorism, really messes up the idea of motivation. At first, it creates a changed definition for learning. It is not a definition that relies on a “what” to be learned (what overwhelms us as math teachers), but instead focuses on hypothetical models for knowing and defines learning changes to those knowing structures modeled. So “what” is to be learned is recognized as an idea of the teacher, and something they want to “see” replicated in the learner. Now motivation has become more of a problem OF the teacher, not a lacking in the learner.

I continue this rumination at http://blaw0013.blogspot.com/2014/03/on-constructivism-and-motivation.html and won’t fill Dan’s space with my meandering thoughts.

Last comment: I was intrigued to return here by @Don Byrd’s comments, especially that “Does this mean that cognitive tutors can’t identify misconceptions? No, but only if they’ve been programmed in, and therefore only if they’ve been recognized by someone.” What struck me was the reminder that an observer must exist in any statement that is made.

You say “Any correct or incorrect step taken by a student, if it has been successfully generalized into a production rule, can be used in feedback with future students.” Can the program use it properly in _different problems_ where in fact the same rule applies? If the answer is no, I’d say it’s definitely not discovering a misconception in a reasonable sense. If yes, it might count as discovering a misconception — though even then, what it actually discovered would be a special case of what we would see as the real misconception. I don’t see how any “production rule” could really capture the essence of a misconception!

Still, this does sound like it could be a useful tool, especially with a very large class.

Take a problem, such as an equation to solve: 2x+4=10. Matsuda’s program has a parser with which to read the problem (the program identifies features such as an x, a coefficient of x, a constant that’s being added, and a right-hand side).

First let a bunch of students work through the problem, some correctly, some incorrectly. Then activate the learning component of the Matsuda’s program, which allows it to learn from examples. Play back one student’s work at a time, flagging each step as correct or incorrect. For example, if a student’s trying to solve 2x+4=10 started by writing x+4=5, you would mark it incorrect.

At this point, Matsuda’s program would try to generate a rule that would have gotten the student from the start state to their first line. Here the rule would be something like “In Ax + B = C”, replace Ax with x, and divide C by A. Since you’ve marked this step incorrect, the computer would learn that that’s not a good rule to follow. But it would have created the rule in it’s memory, so if it saw other students doing the same thing on other problems, it would recognize the mistake. If you’ve programmed a feedback message or little video tutorial about that misconception, it can be shown to any future students. Each rule (called a production rule. in ACT-R, is coded in Jess.

This does require human tagging of each step of a large sample of student work, but it doesn’t require a human to code the production rule in Jess. It also allows the computer to follow multiple correct solution paths. Any correct or incorrect step taken by a student, if it has been successfully generalized into a production rule, can be used in feedback with future students.

Matsuda’s paper linked in comment #21 describes correct and incorrect parsers in more detail–sometimes the incorrect parsers are what students are using.

Dan, sorry if my replies are too long. I can take the conversation over to my blog if you’d prefer.

I took a quick look at the ACT-R and fMRI paper. It’s interesting and impressive research. And the video you describe sounds very impressive, though of course the fact their program did so well _once_ doesn’t mean that much. More to the point, I don’t see how this works suggests they’re anywhere near being able to identify misconceptions that haven’t been programmed in. I’m not a classroom teacher anymore, but I’m still tutoring, and still being blindsided by students misunderstanding something that never occurred to me at all. I’m sure that becomes less and less common as you get more experienced, but there are so many ways to misunderstand!