A Scathing Review of Khan Academy

I really hate to do this.  Khan Academy, when it was first founded, was painstakingly built around the best of ideals.  In attempting to help some relatives understand subjects their teachers at school were failing to teach effectively, Sal Khan discovered that he could fill a huge gap in U.S. education.  Sal had no background in teaching, which turned out to be a huge advantage.  In failing to follow traditional U.S. teaching pedagogy, Sal discovered a better way.  Large numbers of students flocked to his videos, for a superior way to become educated.  Sal recognized the enormous contribution he could make to society, not just in the U.S. but in the world, and he quit his job to found Khan Academy, a website dedicated to using this fresh new take on education to make a world class education available to anyone in the entire world with access to the internet, for free.  Over time, Khan Academy evolved from a video hosting site for Sal's videos into a full learning platform with exercises and evaulations.  Sal collaborated with some U.S. schools to come up with new teaching pedagogies that put Khan Academy at the core, with great success.  As time went on though, Khan Academy began to hire experienced educators onto its staff, with the goal of bringing more experience into the process of improving Khan Academy.  What Sal perhaps did not expect was that this would turn out to be a trojan horse for the traditional U.S. educational values that made Khan Academy necessary in the first place, to sneak in and corrupt the platform, turning it into little more than another online school.  It is nice that Khan Academy is free, but I am not even sure that is a good thing, because it has become an engine for spreading a low quality teaching pedagogy to less developed countries.  I love what Khan Academy wants to be...or at least wanted to be, but it has not become that.  Worse, there was a time when it was actually closer, but it has backslid since.

My biggest gripe with Khan Academy, and the one that finally pushed me over the edge into writing this, is in how skill evaluation is handled.  A couple days ago, I was working through the basic arithmetic section (maybe I will write about why I was doing that in another post).  I recently bought a new mouse and keyboard for my desktop, because my old ones were wearing out (letters worn off, scroll wheel not working...), and the only decently priced options of good quality were wireless.  Unfortunately, once in a while the keyboard fails to correctly send a character (a common problem with wireless devices).  The most common issue is spaces, which may partially be caused by the switches not being sensitive enough, but as I was doing a Khan Academy test for multiplication, it also missed a number now and then.  Now, Khan Academy gives achievements for speed, and on top of that, I am currently working through math I am already quite skilled at, so I do not want to take a lot of time.  Unfortunately, once in a while, I end up missing a problem, due to keyboard error.  More unfortunately, this means I will not be able to score 100% on the test.  Even more unfortunately than this, Khan Academy provides no means for resetting a test, meaning I am forced to complete up to 30 problems before I can even start over and retake the test.  From a software development perspective, this is a very serious user interface failure, but for educational skill evaluation, it kind of makes sense.  At least, it makes sense in the context of traditional American dogmatic education pedagogy, a pedagogy that is more interested in teaching values than actually educating students, and a pedagogy that has a long history of criticism for its poor performance compared to more serious education.  It does not make sense in the context of providing a world class education.  In fact, evaluation by testing in general has mountains of evidence proving it to provide poor quality skill assessment, and further, testing oriented evaluation discourages legitimate teaching and learning and instead encourages rote memorization without understanding as well as cheating.  Traditional test based evaluation teaches students that the test is more important than the learning and understanding, and the easiest way to score well on a test is cheating.

Inability to reset tests is not the underlying problem though.  The underlying problem is the fundamental pedagogy behind this poor design and the fact that it runs directly counter to Khan Academy's goals and Sal's own understanding of education.  Posted in Khan Academy's, official forum, this post excerpt from a Khan Academy product manager says it all.

I understand how frustrating it can be to have to retake an entire practice set or unit test because of one or a few mistakes, and how useful a restart button might seem to alleviate this. This is something that we have debated internally since it was initially brought up and with a wide range of team members, and after much discussion - we have decided against implementing a restart button because it goes against our pedagogical perspective.

We want to encourage growth mindset and learning on our platform, and a “Restart Button” would not encourage that. We want to encourage all learners to see that getting problems wrong or not getting 100% on the first try is one of the most important parts of learning--even if that error was a simple calculation mistake, it teaches us to be more diligent. In order to master something, you will first have to practice with material that you don't quite know. The current system is intended to identify the sections in which you need the most help by analyzing your initial answers and adjusting your practice to fit your specific needs.

First, she makes it clear that this is about pedagogy, not about doing what is best for the students, not about doing what the users feel will best facilitate their own learning, and not about Khan Academy's prime directive of providing a free, world class education.  The very fact that Khan Academy even has a "pedagogical perspective" is evidence of its corruption.  The use of language like this is evidence that Khan Academy has been infiltrated by traditional education influences that are corrupting it from the inside.  I have spent significant time studying what is and is not effective in education, and the only place I find the word "pedagogy" used is as a justification for ignoring evidence.  Khan Academy showed that taking the lectures out of the classroom (a technique I have successfully used in my own teaching as a college professor) improved the general effectiveness of education.  The most common excuse for not adopting this practice is that it does not fit the school's or teacher's "pedagogy".  In my experience, when educators use the term "pedagogy", what they really mean is "pedagogical dogma".  Dogma is religion not science.  It is unsubstantiated belief not empirical evidence.  It has no place in secular education!

Second, she claims that including a reset button would not encourage a growth mindset.  I see two problems here.  The first is the idea that encouraging a growth mindset is an important part of all educational pursuits.  Say I want to learn math.  Khan Academy believes that instead of teaching me math, they need to teach me a growth mindset, with some math on the side.  Attempting to control how people think is a fascist pursuit that is unfortunately common in the U.S. education system.  It is part of the "pedagogy" aka dogma.  What is the purpose of the course, to teach math or to teach a "growth mindset"?  If the purpose is to teach a growth mindset, then make it a "growth mindset" course and teach the math in its own course.  Unfortunately, traditional U.S. educational pedagogy is full of ideas that educators are responsible for teaching students things other than the subjects at hand, and the result is curriculum and course design that is full of things that hinder learning and make students hate learning.  The second problem is the belief that this does promote a growth mindset.  The fact is, it does not.  In the forum thread that this excerpt comes from, around a hundred Khan Academy users have described their response to failing a question as randomly answering the remaining questions to get through the test, so they can start over.  This is not a growth mindset.  This the same mindset we see in public school students who are taught that test performance is the most important part of education.  According to Khan Academy's pedagogy, it is also cheating, because deliberately excluding a way to reset the test is a tacit assertion that resetting would be cheating.  This is the kind of cheating that is not strictly wrong though.  It is only cheating because Khan Academy's pedagogy treats it as such.  It does not interfere in learning, aside from the fact that skipping questions this way wastes time that could be spent learning, the solution for which is a reset option.  The fact is, this does not encouraging a learning mindset.  Instead, it encourages a test oriented mindset, it frustrates students, and it punishes students for making mistakes.  The instance of a test I personally was taking when I finally became frustrated enough to start looking for a solution is especially telling.  It was my second attempt.  My first attempt, I had legitimately missed a question, because I had not read the instructions well.  My second attempt, I missed the second question of 31 questions, because my keyboard failed to register the final zero, when I was trying to type "3000".  I was left with 29 remaining questions.  To get a perfect score on the test, I would now have to answer 60 questions, 29 of which were purely punishment for my keyboard missing a zero.  Not only is this completely unjust for a mistake that was not even my fault, it is also completely inappropriate for encouraging a growth mindset, even if I had legitimately gotten the problem wrong.  Sure, being able to retake the test to get a better score is superior to how it works in U.S. schools, but that only solves part of the problem with using testing for evaluation.  It still makes students hate learning (the opposite of a growth mindset), and it still encourages cheating over learning and wastes time that should be used for learning.

Third, she suggests that this helps students become more comfortable with failure.

We want to encourage all learners to see that getting problems wrong or not getting 100% on the first try is one of the most important parts of learning--even if that error was a simple calculation mistake, it teaches us to be more diligent.

The problem is that this is not what this accomplishes.  If getting problems wrong is so important to learning (and it is), then why punish students for failure?  What I get out of the lack of a reset button is that getting 100% on the first try is the only important thing, because if I do not, I will be forced to spend inordinate amounts of time answering questions I am already skilled at, before I am allowed to prove that I have improved my skill on the one I missed.  The lack of a reset button actually accomplishes the exact opposite of what this product manager claims it does.  The right way to help students become comfortable with failure as a critical part of learning is to make turnaround as fast as possible.  If I miss a problem on the test, let me give up on the test immediately, so I can go back and study what I missed, and then let me start the test over from the beginning, so I do not feel like I am being punished for not getting it perfect on the first try (ironic, isn't it?).

Fourth, she says, "it teaches us to be more diligent".  Aside from the fact that the reported behavior (skipping questions by answering them randomly) proves this is a false assertion, this comes back to the same thing as "learning mindset".  Is the primary goal of a math course to teach diligence or math?  If the goal is to learn math, than attempting to teach diligence interferes with the goal of the class.  My position on this is the same: If you want to teach diligence, offer a course on diligence.  It is inappropriate to allow your personal views on how people should behave get in the way of effective learning.  This idea that diligence should be part of a math course is part of the pedagogical dogma in U.S. education.  The truth however, is that attempting to teach diligence, a growth mindset, or anything other than math in a math course burdens the student with things that get in the way of learning math.  Thus, the feature (or lack thereof) being defended not only does not teach diligence, teaching diligence has no place in a math course in the first place.

These are the most egregious problems with Khan Academy right now, and they are all based on corruption of the original goal of providing a fee, world class education.  This is not the only feature affected; it is merely the most obvious.  There was a time when Khan Academy's practice lessons would continue until you got a certain number in a row correct.  This was a generally good system, as it would give students practice until they had achieved a certain degree of mastery.  I don't know the justification, but this seems to have been reverted back to making practices only have a set number of problems.  I suspect the reason is more of this pedagogical dogma, because a person that reasons the same way the person who wrote the above quoted post reasoned might argue that going until you get, say, four in a row right, is less evidence of mastery than doing four at a time, because doing only four at a time could allow you to get up to 6 in a row right, without earning credit for mastery (the first of the first four is wrong, the last of the second four is wrong).  The problem is that first, this is treating practice as evaluation (being this concerned about performance of practice is inappropriate for effective education), and second, there is a much easier way to fix it, if students truly do need more practice.  If you want students to have to get at least 7 in a row correct to complete practice, then change the threshold to 7, instead of leaving it at four, and now you can keep the serial practice instead of moving to chunks.  I found the serial practice to be quite effective for my own learning, and I am severely disappointed that they eliminated that.

The fact is, Khan Academy is no longer a place to get a world class education.  Yes, the videos are still there, and they are still good, but a world class education needs more than just quality lecture.  A world class education should also optimize learning effectiveness, and that means removing all burdens to effective education, including the minimization of wasted time.  The pedagogical dogma that has been adopted by Khan Academy in recent years excludes this and limits Khan Academy to only being able to provide an American class education, which is decidedly worse than many countries.  The U.S. ranks 13th in PISA score, which ranks countries by the collaborative problem solving skills of their students.  The U.S. is above the average of all participating countries, but only by a narrow margin.  If Khan Academy wants to provide a world class education, using pedagogically driven teaching methods, it needs to replace its design team with educators from Singapore, Japan, and China.  Alternatively, it could abandon the concept of pedagogy entirely and instead opt for a dynamic, evidence driven approach to education, which it advocated for at one time but has since clearly abandoned, and which has the potential to beat even these top three countries.

I watched a video several years ago, where Sal Khan talked about the critical role of failure in education.  He explained how he was very careful not to punish his son for failure but rather to glory in it and encourage him to try again.  We need an online learning platform based on this, not on American dogmatic education pedagogy.  We need a system where failure is never punished, and where the next iteration can begin instantly upon failure.  Unfortunately, this platform is not Khan Academy, and at this point, Khan Academy is so steeped in pedagogical dogma that requests for improvement are ignored or justified away, instead of receiving any serious consideration.  It is clear to me that the current education team at Khan Academy can no longer be trusted to provide a world class education.  I hope Sal will use whatever influence he has to replace this team with people who are willing to give up their outdated and disproved beliefs, in favor of an evidence based approach to learning.  In other words, it is not Khan Academy's students who need a learning mindset and diligence.  It is Khan Academy's own staff that needs to develop a learning mindset and diligence in their stated goal of bringing everyone who wants it a free, world class education.