Monday, March 23, 2020

The Solution to Khan Academy

I just wrote an article criticizing some decisions made by Khan Academy staff, as well as the underlying motives for making those decisions.  I explained in detail why the decisions were bad, and how the decisions actually undermined Khan Academy's stated goals, a well as failing to accomplish their intended purposes.  I did not offer any solutions to these problems, nor did I make any suggestions on how to improve things.  As a companion to that article, I am now writing this, to be published at the same time, just in case Khan Academy wants to improve.  Of course, anyone may use these ideas.  If Khan Academy staff is not willing to accept constructive criticism, adopt a learning mindset, and be more diligent in their stated goal of providing a free, world class education, perhaps someone else will.

First, to be clear, I really like Khan Academy.  I want it to succeed.  I would very much like to see Khan Academy improve its mindset and get back to doing what it was made to do than see someone else take up the torch, but if Khan Academy will not take advantage of modern technology to revolutionize learning, I sincerely hope someone will, because we need it.

The main problem I discussed in the previous article was the evaluation system, specifically unit tests.  Unit test can be over 30 questions long.  The questions are graded, one by one, as soon as they are submitted.  The problem is that if you miss a question, there is no way to restart the test.  If you exit and reenter, it will just start you right where you left off.  There is no reset button.  So, if you miss say the second question in a 31 question test (as I did), you cannot pass the test without doing 60 more problems.  That is the remaining 29 problems and then the 31 problems in your next attempt.  I am not going to recap all of the ways this is horrifically wrong and detrimental both to students and to Khan Academy's goal of providing a world class education, but it is really bad design.  I have come up with several solutions to this problem, from the easiest, to the one that requires the smallest changes in pedagogical dogma, to the one that will provide the best learning experience.

Obviously the easiest solution is to just add a reset button that lets students start over, without having to complete the test.  And to be clear, this is already better than the current situation, regardless of your pedagogical dogma, because most people quickly discover that they can just enter random (wrong) answers for the rest of the questions to get through fast.  This is a waste of their time, which they could be using to study the topic they missed, and even if they do not, skipping the rest of the test faster is not making things any worse for them, though it does interfere with Khan Academy's ability to keep track of where students need more work.  The fact is, there is no way adding a reset button can make things worse for anyone, and for many students it will improve their learning.

Perhaps the best solution that maintains the pedagogical dogma as much as possible, would merely be to leave things as they are by default, but giving students and teachers options they can change in the settings to change the behavior.  By default, there is no way to restart and students will just be forced to put in random answers to get through and start over.  If this is what Khan Academy's pedagogical perspective prefers, then go ahead and make it the default.  However, also add an option so students or their coaches can have a reset option.  This will allow teachers to control the behavior of the platform for their students, and it will allow adults working independently to chose what works best for them, if they become frustrated enough to go looking for a solution.

The best solution, however, is to lose the dogma and make students restart when they fail a problem!  Perhaps this sounds a little extreme, but allow me to explain.  Part of the whole testing thing is to figure out where students need more work.  The way tests currently work, they force students to complete them before trying again, allowing the platform to determine all of the places they need work, all in one shot.  This is unnecessary though.  We can do this far more dynamically, in a way that is also more natural.  This will result not only in less frustration but also in better learning, because the best time to learn is right after failure has primed your brain to want to learn.  (Actually, sometimes the best solution for repeated failure is taking a nap, so your brain can have some time to build new connections, but this is not about repeated failure.)  So, imagine you start the test.  You get a few problems in, and you make a mistake.  Instead of forcing you to go on, the test immediately terminates and presents you with the suggestion that now is a good time to review the video or do a practice set.  We do not care about the rest of the topics right now.  We can deal with them later.  The student may then ignore the recommendation and start a new test, or the student can heed the recommendation, practice the subject again or watch the video, and then test again.  Learning coaches like teachers or parents should probably have an option to require additional practice instead of making it optional, but this probably should not be the default.  This will encourage students to approach failure by putting some more time into learning, before trying again.  It will remove the barrier to effective learning, by reducing the repetition on subjects students are already proficient at and by reducing the frustration of having to waste time on problems they already know, before they are allowed to test again.

That solution could still work reasonably well watered down a bit.  What if instead, students are allowed to complete the test, however the act of practicing will reset a pending test.  In this scenario, when I miss a problem, the test gives me the option to continue or quit.  If I chose to quit, the test will resume where I left off if I get back into it, but, if I practice all of the topics I missed questions on, the test will automatically reset (or at least give me the option to reset).  This is not as good for people who have made mistakes that are not their fault (for example, I ended up having to work through 60 problems, because my wireless keyboard missed a keystroke), but it would still be better than the current situation (I would have been able to restart after 4 practice problems instead of 29 test problems).

The ideal solution has one thing the others do not: They encourage students to strive for perfection.  Without a reset button, students are encourage to give up, once they get a passing grade, because trying to get a higher grade results in significant amounts of punishment, frustration, and misery.  A reset button would largely eliminate that discouragement, making perfection more accessible but not actually encouraging it.  Encouraging students to go back and study as soon as they miss a problem will encourage them to stop testing as soon as a weakness is identified, to fix that weakness.  It will provide some motivation for students to fix gaps in their skill immediately, instead of rewarding them for aiming for "good enough".  My personal opinion is that no education program has any excuse for graduating students without high proficiency, given our current technology and teaching capacity.  There is no reason that Khan Academy could not just keep teaching students until they get 100%.  In public schools, this is impossible, because you have one teacher and many students, and the teacher just does not have the time to strive for perfect proficiency.  Khan Academy is scalable though.  It can afford to at least encourage students to strive for perfection.  75% does not need to be good enough anymore, because if one student needs a few more hours of practice than another, Khan Academy does not have to split its time between that student and others.  Every student who is struggling can go back, review the lecture, and practice, and it will not cut into lesson time for anyone else.  But, we cannot do this with the current system, because it harshly punishes students for failure.  The only way teaching effectiveness can be maximized is by eliminating punishment as much as possible and providing a positive response to failure as immediately as possible.  Ideally that means building resetting right into the learning flow, so that when students fail they are immediately given the opportunity to improve and allowed to restart evaluation as soon as they have taken the opportunity to improve.

Khan Academy is already pretty good, but as I have said before, the very fact that the team has a pedagogical perspective suggests they are locked into a dogma that cannot provide the flexibility required to produce a good, evidence based (or world class) learning system.  Above are some good options for improving some specific features, within whatever pedagogical dogma the team may subscribe to, but the better long term solution would be to leave the pedagogy and dogma behind and instead take a more scientific, evidence based approach to learning.  Evidence based learning is not well suited to pedagogy in general, because it needs more flexibility than strictly defined pedagogy can provide or allow, to rapidly adapt to new discoveries in education research.  Pedagogy is inherently dogmatic, and dogma has no place in evidence based anything.

I hope these ideas will find some good use in helping Khan Academy become what it desires to be, but if they do not, I hope someone eventually uses them to produce an electronic learning system that is truly world class and ideally free.

A Scathing Review of Khan Academy

I really hate to do this.  Khan Academy, when it was first founded, was painstakingly built around the best of ideals.  In attempting to help some relatives understand subjects their teachers at school were failing to teach effectively, Sal Khan discovered that he could fill a huge gap in U.S. education.  Sal had no background in teaching, which turned out to be a huge advantage.  In failing to follow traditional U.S. teaching pedagogy, Sal discovered a better way.  Large numbers of students flocked to his videos, for a superior way to become educated.  Sal recognized the enormous contribution he could make to society, not just in the U.S. but in the world, and he quit his job to found Khan Academy, a website dedicated to using this fresh new take on education to make a world class education available to anyone in the entire world with access to the internet, for free.  Over time, Khan Academy evolved from a video hosting site for Sal's videos into a full learning platform with exercises and evaulations.  Sal collaborated with some U.S. schools to come up with new teaching pedagogies that put Khan Academy at the core, with great success.  As time went on though, Khan Academy began to hire experienced educators onto its staff, with the goal of bringing more experience into the process of improving Khan Academy.  What Sal perhaps did not expect was that this would turn out to be a trojan horse for the traditional U.S. educational values that made Khan Academy necessary in the first place, to sneak in and corrupt the platform, turning it into little more than another online school.  It is nice that Khan Academy is free, but I am not even sure that is a good thing, because it has become an engine for spreading a low quality teaching pedagogy to less developed countries.  I love what Khan Academy wants to be...or at least wanted to be, but it has not become that.  Worse, there was a time when it was actually closer, but it has backslid since.

My biggest gripe with Khan Academy, and the one that finally pushed me over the edge into writing this, is in how skill evaluation is handled.  A couple days ago, I was working through the basic arithmetic section (maybe I will write about why I was doing that in another post).  I recently bought a new mouse and keyboard for my desktop, because my old ones were wearing out (letters worn off, scroll wheel not working...), and the only decently priced options of good quality were wireless.  Unfortunately, once in a while the keyboard fails to correctly send a character (a common problem with wireless devices).  The most common issue is spaces, which may partially be caused by the switches not being sensitive enough, but as I was doing a Khan Academy test for multiplication, it also missed a number now and then.  Now, Khan Academy gives achievements for speed, and on top of that, I am currently working through math I am already quite skilled at, so I do not want to take a lot of time.  Unfortunately, once in a while, I end up missing a problem, due to keyboard error.  More unfortunately, this means I will not be able to score 100% on the test.  Even more unfortunately than this, Khan Academy provides no means for resetting a test, meaning I am forced to complete up to 30 problems before I can even start over and retake the test.  From a software development perspective, this is a very serious user interface failure, but for educational skill evaluation, it kind of makes sense.  At least, it makes sense in the context of traditional American dogmatic education pedagogy, a pedagogy that is more interested in teaching values than actually educating students, and a pedagogy that has a long history of criticism for its poor performance compared to more serious education.  It does not make sense in the context of providing a world class education.  In fact, evaluation by testing in general has mountains of evidence proving it to provide poor quality skill assessment, and further, testing oriented evaluation discourages legitimate teaching and learning and instead encourages rote memorization without understanding as well as cheating.  Traditional test based evaluation teaches students that the test is more important than the learning and understanding, and the easiest way to score well on a test is cheating.

Inability to reset tests is not the underlying problem though.  The underlying problem is the fundamental pedagogy behind this poor design and the fact that it runs directly counter to Khan Academy's goals and Sal's own understanding of education.  Posted in Khan Academy's, official forum, this post excerpt from a Khan Academy product manager says it all.
I understand how frustrating it can be to have to retake an entire practice set or unit test because of one or a few mistakes, and how useful a restart button might seem to alleviate this. This is something that we have debated internally since it was initially brought up and with a wide range of team members, and after much discussion - we have decided against implementing a restart button because it goes against our pedagogical perspective.
We want to encourage growth mindset and learning on our platform, and a “Restart Button” would not encourage that. We want to encourage all learners to see that getting problems wrong or not getting 100% on the first try is one of the most important parts of learning--even if that error was a simple calculation mistake, it teaches us to be more diligent. In order to master something, you will first have to practice with material that you don't quite know. The current system is intended to identify the sections in which you need the most help by analyzing your initial answers and adjusting your practice to fit your specific needs.
First, she makes it clear that this is about pedagogy, not about doing what is best for the students, not about doing what the users feel will best facilitate their own learning, and not about Khan Academy's prime directive of providing a free, world class education.  The very fact that Khan Academy even has a "pedagogical perspective" is evidence of its corruption.  The use of language like this is evidence that Khan Academy has been infiltrated by traditional education influences that are corrupting it from the inside.  I have spent significant time studying what is and is not effective in education, and the only place I find the word "pedagogy" used is as a justification for ignoring evidence.  Khan Academy showed that taking the lectures out of the classroom (a technique I have successfully used in my own teaching as a college professor) improved the general effectiveness of education.  The most common excuse for not adopting this practice is that it does not fit the school's or teacher's "pedagogy".  In my experience, when educators use the term "pedagogy", what they really mean is "pedagogical dogma".  Dogma is religion not science.  It is unsubstantiated belief not empirical evidence.  It has no place in secular education!

Second, she claims that including a reset button would not encourage a growth mindset.  I see two problems here.  The first is the idea that encouraging a growth mindset is an important part of all educational pursuits.  Say I want to learn math.  Khan Academy believes that instead of teaching me math, they need to teach me a growth mindset, with some math on the side.  Attempting to control how people think is a fascist pursuit that is unfortunately common in the U.S. education system.  It is part of the "pedagogy" aka dogma.  What is the purpose of the course, to teach math or to teach a "growth mindset"?  If the purpose is to teach a growth mindset, then make it a "growth mindset" course and teach the math in its own course.  Unfortunately, traditional U.S. educational pedagogy is full of ideas that educators are responsible for teaching students things other than the subjects at hand, and the result is curriculum and course design that is full of things that hinder learning and make students hate learning.  The second problem is the belief that this does promote a growth mindset.  The fact is, it does not.  In the forum thread that this excerpt comes from, around a hundred Khan Academy users have described their response to failing a question as randomly answering the remaining questions to get through the test, so they can start over.  This is not a growth mindset.  This the same mindset we see in public school students who are taught that test performance is the most important part of education.  According to Khan Academy's pedagogy, it is also cheating, because deliberately excluding a way to reset the test is a tacit assertion that resetting would be cheating.  This is the kind of cheating that is not strictly wrong though.  It is only cheating because Khan Academy's pedagogy treats it as such.  It does not interfere in learning, aside from the fact that skipping questions this way wastes time that could be spent learning, the solution for which is a reset option.  The fact is, this does not encouraging a learning mindset.  Instead, it encourages a test oriented mindset, it frustrates students, and it punishes students for making mistakes.  The instance of a test I personally was taking when I finally became frustrated enough to start looking for a solution is especially telling.  It was my second attempt.  My first attempt, I had legitimately missed a question, because I had not read the instructions well.  My second attempt, I missed the second question of 31 questions, because my keyboard failed to register the final zero, when I was trying to type "3000".  I was left with 29 remaining questions.  To get a perfect score on the test, I would now have to answer 60 questions, 29 of which were purely punishment for my keyboard missing a zero.  Not only is this completely unjust for a mistake that was not even my fault, it is also completely inappropriate for encouraging a growth mindset, even if I had legitimately gotten the problem wrong.  Sure, being able to retake the test to get a better score is superior to how it works in U.S. schools, but that only solves part of the problem with using testing for evaluation.  It still makes students hate learning (the opposite of a growth mindset), and it still encourages cheating over learning and wastes time that should be used for learning.

Third, she suggests that this helps students become more comfortable with failure.
We want to encourage all learners to see that getting problems wrong or not getting 100% on the first try is one of the most important parts of learning--even if that error was a simple calculation mistake, it teaches us to be more diligent.
The problem is that this is not what this accomplishes.  If getting problems wrong is so important to learning (and it is), then why punish students for failure?  What I get out of the lack of a reset button is that getting 100% on the first try is the only important thing, because if I do not, I will be forced to spend inordinate amounts of time answering questions I am already skilled at, before I am allowed to prove that I have improved my skill on the one I missed.  The lack of a reset button actually accomplishes the exact opposite of what this product manager claims it does.  The right way to help students become comfortable with failure as a critical part of learning is to make turnaround as fast as possible.  If I miss a problem on the test, let me give up on the test immediately, so I can go back and study what I missed, and then let me start the test over from the beginning, so I do not feel like I am being punished for not getting it perfect on the first try (ironic, isn't it?).

Fourth, she says, "it teaches us to be more diligent".  Aside from the fact that the reported behavior (skipping questions by answering them randomly) proves this is a false assertion, this comes back to the same thing as "learning mindset".  Is the primary goal of a math course to teach diligence or math?  If the goal is to learn math, than attempting to teach diligence interferes with the goal of the class.  My position on this is the same: If you want to teach diligence, offer a course on diligence.  It is inappropriate to allow your personal views on how people should behave get in the way of effective learning.  This idea that diligence should be part of a math course is part of the pedagogical dogma in U.S. education.  The truth however, is that attempting to teach diligence, a growth mindset, or anything other than math in a math course burdens the student with things that get in the way of learning math.  Thus, the feature (or lack thereof) being defended not only does not teach diligence, teaching diligence has no place in a math course in the first place.

These are the most egregious problems with Khan Academy right now, and they are all based on corruption of the original goal of providing a fee, world class education.  This is not the only feature affected; it is merely the most obvious.  There was a time when Khan Academy's practice lessons would continue until you got a certain number in a row correct.  This was a generally good system, as it would give students practice until they had achieved a certain degree of mastery.  I don't know the justification, but this seems to have been reverted back to making practices only have a set number of problems.  I suspect the reason is more of this pedagogical dogma, because a person that reasons the same way the person who wrote the above quoted post reasoned might argue that going until you get, say, four in a row right, is less evidence of mastery than doing four at a time, because doing only four at a time could allow you to get up to 6 in a row right, without earning credit for mastery (the first of the first four is wrong, the last of the second four is wrong).  The problem is that first, this is treating practice as evaluation (being this concerned about performance of practice is inappropriate for effective education), and second, there is a much easier way to fix it, if students truly do need more practice.  If you want students to have to get at least 7 in a row correct to complete practice, then change the threshold to 7, instead of leaving it at four, and now you can keep the serial practice instead of moving to chunks.  I found the serial practice to be quite effective for my own learning, and I am severely disappointed that they eliminated that.

The fact is, Khan Academy is no longer a place to get a world class education.  Yes, the videos are still there, and they are still good, but a world class education needs more than just quality lecture.  A world class education should also optimize learning effectiveness, and that means removing all burdens to effective education, including the minimization of wasted time.  The pedagogical dogma that has been adopted by Khan Academy in recent years excludes this and limits Khan Academy to only being able to provide an American class education, which is decidedly worse than many countries.  The U.S. ranks 13th in PISA score, which ranks countries by the collaborative problem solving skills of their students.  The U.S. is above the average of all participating countries, but only by a narrow margin.  If Khan Academy wants to provide a world class education, using pedagogically driven teaching methods, it needs to replace its design team with educators from Singapore, Japan, and China.  Alternatively, it could abandon the concept of pedagogy entirely and instead opt for a dynamic, evidence driven approach to education, which it advocated for at one time but has since clearly abandoned, and which has the potential to beat even these top three countries.

I watched a video several years ago, where Sal Khan talked about the critical role of failure in education.  He explained how he was very careful not to punish his son for failure but rather to glory in it and encourage him to try again.  We need an online learning platform based on this, not on American dogmatic education pedagogy.  We need a system where failure is never punished, and where the next iteration can begin instantly upon failure.  Unfortunately, this platform is not Khan Academy, and at this point, Khan Academy is so steeped in pedagogical dogma that requests for improvement are ignored or justified away, instead of receiving any serious consideration.  It is clear to me that the current education team at Khan Academy can no longer be trusted to provide a world class education.  I hope Sal will use whatever influence he has to replace this team with people who are willing to give up their outdated and disproved beliefs, in favor of an evidence based approach to learning.  In other words, it is not Khan Academy's students who need a learning mindset and diligence.  It is Khan Academy's own staff that needs to develop a learning mindset and diligence in their stated goal of bringing everyone who wants it a free, world class education.