As teachers we OFTEN forget we look at students’ understanding from our own point of view; we assume there is this basic knowledge, so of course what we present will make sense. Then we teach a procedure; students get a correct response (algorithms work, after all), and we move on. This works pretty well for most multiple choice tests…but then when we ask for an explanation, critical thinking, or a logic based problem is presented…kaboom. Then we cry, “These students don’t have number sense!” Nope, they don’t because we never taught it at a deep level. Why? Too many reasons to list here. I pray Common Core can help alleviate some of this lock-step memorizing the procedure thinking. I am afraid it won’t because the curriculum to teach this way isn’t out there in full force (yet!).

I don’t know where we are headed in technology in education, specifically in math. (By the way, technology is a really irritating word to me…a pencil is a piece of technology.) I believe there is great hope with adaptive AI cognitive tutors based on the learning sciences like perceptual control theory, flow, skill theory, and Bayesian knowledge tracing. God bless the developers who want to take this on in a world of crow-sourced apps that have all the bells and whistles crying out like the lights of Vegas. My cry is for VCs to invest in companies producing this type of work. Gates has given $$$$ to Kahn…and the truth is, I think if they will be honest and self-reflective, they have the ability to design such products.

Dan, I like your work with gaming and instruction…it takes a skilled teacher to scaffold the dialogue that will take a student from merely playing a game to dissecting the learning.

You might want to check out this blog (yes, I responded there too!) as well as this EARLY number sense, narrow scope of instruction, math app. I have used it and have conducted action research in several classrooms of my students. Waiting for developers to fund follow on apps (NO, I do not work for them or represent them…just giving it my teacher EXPERT stamp of approval!) (Blog is two parts)

http://theeducationscientist.blogspot.ch/2014/02/khan-academy-and-illusion-of.html#.VLSxZivF-Sp

http://www.nativebrain.com/

Another German entrepreneur converted his creative skills of engraving
toothpicks, to start an uncommon business selling
engraved toothpick souvenirs.

you are calling for a tremendous amount of work in that statement, and are basically calling for the KA to turn into an entirely different kind of product.

True.

But I have to disagree with you at one point:”In my mind, Khan Academy could do one simple thing to improve itself several times over:

Ask questions that computers don’t grade.”

In fact, I think that would severly make the product worse. It’s easy to just say “Link students to coaches to do non-machine grading!” but you are calling for a tremendous amount of work in that statement, and are basically calling for the KA to turn into an entirely different kind of product. Additionally, if we agree coaching is a good thing, is virtual coaching through a website the best way to do it? Maybe an in-person coach would be even better? So maybe the KA should provide those- oh, yeah, those are teachers and they already exist. And that’s also why Khan has called for teachers to supplement KA with teaching, and not use it on its own. SBAC can make non-machine graded items (i.e. writing in an answer with words), because they have huge resources that they are able to put towards human grading those items, not become they have better technology.

To come back to the original point- I think technology products should always focus most on what technology products are made to do well. They shouldn’t try to focus on doing replicating an interaction or on higher-order skills that might best occur face-to-face. But we as educators should also not be solely relying on technology to provide all of education’s needs. There’s no silver bullet here- even CC itself argues for an integrative approach of tackling a problem from different angles.

I think your claim:
What platforms like Khan Academy do effectively is assess students’ basic understanding of foundational concepts
>>>
is what Dan and others here are questioning.

Are you making the claim that “basic understanding of foundational concepts” is the same thing as ‘being able to reliably and correctly compute?

It is quite obvious that what KA does is assess procedural fluency. Can the student correctly and reliably compute …..? Yes/no?

But, for me, it’s not at all obvious that what KA does is assess understanding. I generally think of understanding of as being able to address the ‘why’ questions that Dan has noted are missing. This is true of even the most foundational concepts [counting, addition, …]

Rich problem-solving as described above is not yet assessable in an automated way, which positions these platforms as a supplemental learning tool; an enabler of deeper and more meaningful learning experiences.

I welcome the suggestion of making students’ thinking visible as far as possible, even – indeed, especially – if it can not be automatically evaluated. This serves the widely stated purpose of educational technologies as fostering deeper connections between teacher and learner.

(Full disclosure: I work for Whizz Education; an online adaptive math tutor for 5-14 year-old students. Alongside the tutor we provide a suite of Teachers’ Resources as a way of recognising and supporting the need to push beyond the core knowledge and skills we are able to capture through the automated tutoring model.)

===
A computer graded my responses to every single question in eighth grade.

That means I was never asked, “Why?” or “How do you know?” Those are seriously important questions but computers can’t grade them and Khan Academy didn’t ask them.
===

If Khan or anyone else were to use “hinge point questions” [http://goo.gl/3j2hnh], would this not provide a way (of sorts) of machine-marking multiple choice questions that are of the “Why?” or “How do you know?” type?

Seb Schmoller