I am beginning to learn more about the state of research and progress in education, and it is disappointing me more and more. A few weeks ago, I refreshed my knowledge of the Montessori and Waldorf methods. Now, I have complained about lack of empirical research in education pedagogy, but I am starting to see a common trend that seems to be hiding this problem within education communities, and that is the role of personal experience in the development of teaching methods.
The Montessori method was developed by Maria Montessori over 100 years ago. According to resources on the topic, she developed the system through observation. This is often treated as a claim that the Montessori method is based on scientific study, however, this is completely false. Observation is used in scientific studies, but it is universally considered a poor substitute for controlled studies. Montessori's observational approach to pedagogical development was even less though. All teachers learn something about what works and what does not, through personal experience, which can also be called "observation". Watching other educators teach (and the resources suggest Montessori did, in developing her system) can also reveal valuable things. What observation fails to do, however, is account for uncontrolled factors and exhaustively test the options. Another way of putting this is that teachers learn a lot about what works and what does not, by paying attention to their own experience and by observing the experiences of others, however, this is not in any sense rigorous science. It cannot be used to produce a quality, evidence based learning system. In the case on Montessori, she observes that certain teaching techniques were incredibly success, but personal bias led her to miss all of the students these techniques were not working for. As a result, the Montessori system she developed based on her observations works incredibly well but only for a moderate percentage of students. Modern Montessori schools boast impressive results, but that is actually merely survivor bias. Students who do not thrive in Montessori schools are typically removed from those schools by their parents, thus Montessori schools rarely have students who perform poorly. This does not mean Montessori schools have no value. It merely means that the Montessori method is not a very broadly applicable method. It is certainly not the "best" method, by any stretch. Using Montessori broadly in public schools would result in a much higher rate of failure. As private schools or charter schools, where parents can decide whether or not to keep kids in them based on performance, Montessori schools are an excellent idea, because they work very well for many students.
This is not a unique problem. It turns out nearly every pedagogy is based on either personal observation and experience or on personal ideals and beliefs. There are no common pedagogies, at least not in the U.S., that are based on rigorous research and evidence based science. The closest we have is milestone based learning, but it is hardly common, and it often focuses more heavily on evaluation and progression rate than actual teaching and learning.
The fact is, observation based decision making is a terrible way to handle something as important as learning. Few teachers experiment with multiple methods at the same time, with sample sizes big enough to obtain quality data. Even if the teaching method is changed every school term and the class sizes are 50 or more students, a lifetime of teaching is not sufficient to collect the data necessary to decide what method will perform best in general. Students from wealthier families will likely respond differently to the same method as students from poor or middle class families. Students from one region may respond differently from students in another. Limited class sizes are not inherently bad for this, but for a single teacher, class size is sample size. This does mean that in schools with smaller class sizes, the value of observation and experience in developing good teaching methods is even lower.
What we need is scientifically rigorous, empirical studies. We need researchers who assign teachers to use certain methods. We need thousands or tens of thousands of students learning using each method. We need actual data to be collected, rather than just the impressions and ideas one or two individuals get from personal observations of very small samples of students. For education to advance significantly we need real, scientifically rigorous research, and we need it in large quantities.
Thursday, April 2, 2020
Monday, March 23, 2020
The Solution to Khan Academy
I just wrote an article criticizing some decisions made by Khan Academy staff, as well as the underlying motives for making those decisions. I explained in detail why the decisions were bad, and how the decisions actually undermined Khan Academy's stated goals, a well as failing to accomplish their intended purposes. I did not offer any solutions to these problems, nor did I make any suggestions on how to improve things. As a companion to that article, I am now writing this, to be published at the same time, just in case Khan Academy wants to improve. Of course, anyone may use these ideas. If Khan Academy staff is not willing to accept constructive criticism, adopt a learning mindset, and be more diligent in their stated goal of providing a free, world class education, perhaps someone else will.
First, to be clear, I really like Khan Academy. I want it to succeed. I would very much like to see Khan Academy improve its mindset and get back to doing what it was made to do than see someone else take up the torch, but if Khan Academy will not take advantage of modern technology to revolutionize learning, I sincerely hope someone will, because we need it.
The main problem I discussed in the previous article was the evaluation system, specifically unit tests. Unit test can be over 30 questions long. The questions are graded, one by one, as soon as they are submitted. The problem is that if you miss a question, there is no way to restart the test. If you exit and reenter, it will just start you right where you left off. There is no reset button. So, if you miss say the second question in a 31 question test (as I did), you cannot pass the test without doing 60 more problems. That is the remaining 29 problems and then the 31 problems in your next attempt. I am not going to recap all of the ways this is horrifically wrong and detrimental both to students and to Khan Academy's goal of providing a world class education, but it is really bad design. I have come up with several solutions to this problem, from the easiest, to the one that requires the smallest changes in pedagogical dogma, to the one that will provide the best learning experience.
Obviously the easiest solution is to just add a reset button that lets students start over, without having to complete the test. And to be clear, this is already better than the current situation, regardless of your pedagogical dogma, because most people quickly discover that they can just enter random (wrong) answers for the rest of the questions to get through fast. This is a waste of their time, which they could be using to study the topic they missed, and even if they do not, skipping the rest of the test faster is not making things any worse for them, though it does interfere with Khan Academy's ability to keep track of where students need more work. The fact is, there is no way adding a reset button can make things worse for anyone, and for many students it will improve their learning.
Perhaps the best solution that maintains the pedagogical dogma as much as possible, would merely be to leave things as they are by default, but giving students and teachers options they can change in the settings to change the behavior. By default, there is no way to restart and students will just be forced to put in random answers to get through and start over. If this is what Khan Academy's pedagogical perspective prefers, then go ahead and make it the default. However, also add an option so students or their coaches can have a reset option. This will allow teachers to control the behavior of the platform for their students, and it will allow adults working independently to chose what works best for them, if they become frustrated enough to go looking for a solution.
The best solution, however, is to lose the dogma and make students restart when they fail a problem! Perhaps this sounds a little extreme, but allow me to explain. Part of the whole testing thing is to figure out where students need more work. The way tests currently work, they force students to complete them before trying again, allowing the platform to determine all of the places they need work, all in one shot. This is unnecessary though. We can do this far more dynamically, in a way that is also more natural. This will result not only in less frustration but also in better learning, because the best time to learn is right after failure has primed your brain to want to learn. (Actually, sometimes the best solution for repeated failure is taking a nap, so your brain can have some time to build new connections, but this is not about repeated failure.) So, imagine you start the test. You get a few problems in, and you make a mistake. Instead of forcing you to go on, the test immediately terminates and presents you with the suggestion that now is a good time to review the video or do a practice set. We do not care about the rest of the topics right now. We can deal with them later. The student may then ignore the recommendation and start a new test, or the student can heed the recommendation, practice the subject again or watch the video, and then test again. Learning coaches like teachers or parents should probably have an option to require additional practice instead of making it optional, but this probably should not be the default. This will encourage students to approach failure by putting some more time into learning, before trying again. It will remove the barrier to effective learning, by reducing the repetition on subjects students are already proficient at and by reducing the frustration of having to waste time on problems they already know, before they are allowed to test again.
That solution could still work reasonably well watered down a bit. What if instead, students are allowed to complete the test, however the act of practicing will reset a pending test. In this scenario, when I miss a problem, the test gives me the option to continue or quit. If I chose to quit, the test will resume where I left off if I get back into it, but, if I practice all of the topics I missed questions on, the test will automatically reset (or at least give me the option to reset). This is not as good for people who have made mistakes that are not their fault (for example, I ended up having to work through 60 problems, because my wireless keyboard missed a keystroke), but it would still be better than the current situation (I would have been able to restart after 4 practice problems instead of 29 test problems).
The ideal solution has one thing the others do not: They encourage students to strive for perfection. Without a reset button, students are encourage to give up, once they get a passing grade, because trying to get a higher grade results in significant amounts of punishment, frustration, and misery. A reset button would largely eliminate that discouragement, making perfection more accessible but not actually encouraging it. Encouraging students to go back and study as soon as they miss a problem will encourage them to stop testing as soon as a weakness is identified, to fix that weakness. It will provide some motivation for students to fix gaps in their skill immediately, instead of rewarding them for aiming for "good enough". My personal opinion is that no education program has any excuse for graduating students without high proficiency, given our current technology and teaching capacity. There is no reason that Khan Academy could not just keep teaching students until they get 100%. In public schools, this is impossible, because you have one teacher and many students, and the teacher just does not have the time to strive for perfect proficiency. Khan Academy is scalable though. It can afford to at least encourage students to strive for perfection. 75% does not need to be good enough anymore, because if one student needs a few more hours of practice than another, Khan Academy does not have to split its time between that student and others. Every student who is struggling can go back, review the lecture, and practice, and it will not cut into lesson time for anyone else. But, we cannot do this with the current system, because it harshly punishes students for failure. The only way teaching effectiveness can be maximized is by eliminating punishment as much as possible and providing a positive response to failure as immediately as possible. Ideally that means building resetting right into the learning flow, so that when students fail they are immediately given the opportunity to improve and allowed to restart evaluation as soon as they have taken the opportunity to improve.
Khan Academy is already pretty good, but as I have said before, the very fact that the team has a pedagogical perspective suggests they are locked into a dogma that cannot provide the flexibility required to produce a good, evidence based (or world class) learning system. Above are some good options for improving some specific features, within whatever pedagogical dogma the team may subscribe to, but the better long term solution would be to leave the pedagogy and dogma behind and instead take a more scientific, evidence based approach to learning. Evidence based learning is not well suited to pedagogy in general, because it needs more flexibility than strictly defined pedagogy can provide or allow, to rapidly adapt to new discoveries in education research. Pedagogy is inherently dogmatic, and dogma has no place in evidence based anything.
I hope these ideas will find some good use in helping Khan Academy become what it desires to be, but if they do not, I hope someone eventually uses them to produce an electronic learning system that is truly world class and ideally free.
First, to be clear, I really like Khan Academy. I want it to succeed. I would very much like to see Khan Academy improve its mindset and get back to doing what it was made to do than see someone else take up the torch, but if Khan Academy will not take advantage of modern technology to revolutionize learning, I sincerely hope someone will, because we need it.
The main problem I discussed in the previous article was the evaluation system, specifically unit tests. Unit test can be over 30 questions long. The questions are graded, one by one, as soon as they are submitted. The problem is that if you miss a question, there is no way to restart the test. If you exit and reenter, it will just start you right where you left off. There is no reset button. So, if you miss say the second question in a 31 question test (as I did), you cannot pass the test without doing 60 more problems. That is the remaining 29 problems and then the 31 problems in your next attempt. I am not going to recap all of the ways this is horrifically wrong and detrimental both to students and to Khan Academy's goal of providing a world class education, but it is really bad design. I have come up with several solutions to this problem, from the easiest, to the one that requires the smallest changes in pedagogical dogma, to the one that will provide the best learning experience.
Obviously the easiest solution is to just add a reset button that lets students start over, without having to complete the test. And to be clear, this is already better than the current situation, regardless of your pedagogical dogma, because most people quickly discover that they can just enter random (wrong) answers for the rest of the questions to get through fast. This is a waste of their time, which they could be using to study the topic they missed, and even if they do not, skipping the rest of the test faster is not making things any worse for them, though it does interfere with Khan Academy's ability to keep track of where students need more work. The fact is, there is no way adding a reset button can make things worse for anyone, and for many students it will improve their learning.
Perhaps the best solution that maintains the pedagogical dogma as much as possible, would merely be to leave things as they are by default, but giving students and teachers options they can change in the settings to change the behavior. By default, there is no way to restart and students will just be forced to put in random answers to get through and start over. If this is what Khan Academy's pedagogical perspective prefers, then go ahead and make it the default. However, also add an option so students or their coaches can have a reset option. This will allow teachers to control the behavior of the platform for their students, and it will allow adults working independently to chose what works best for them, if they become frustrated enough to go looking for a solution.
The best solution, however, is to lose the dogma and make students restart when they fail a problem! Perhaps this sounds a little extreme, but allow me to explain. Part of the whole testing thing is to figure out where students need more work. The way tests currently work, they force students to complete them before trying again, allowing the platform to determine all of the places they need work, all in one shot. This is unnecessary though. We can do this far more dynamically, in a way that is also more natural. This will result not only in less frustration but also in better learning, because the best time to learn is right after failure has primed your brain to want to learn. (Actually, sometimes the best solution for repeated failure is taking a nap, so your brain can have some time to build new connections, but this is not about repeated failure.) So, imagine you start the test. You get a few problems in, and you make a mistake. Instead of forcing you to go on, the test immediately terminates and presents you with the suggestion that now is a good time to review the video or do a practice set. We do not care about the rest of the topics right now. We can deal with them later. The student may then ignore the recommendation and start a new test, or the student can heed the recommendation, practice the subject again or watch the video, and then test again. Learning coaches like teachers or parents should probably have an option to require additional practice instead of making it optional, but this probably should not be the default. This will encourage students to approach failure by putting some more time into learning, before trying again. It will remove the barrier to effective learning, by reducing the repetition on subjects students are already proficient at and by reducing the frustration of having to waste time on problems they already know, before they are allowed to test again.
That solution could still work reasonably well watered down a bit. What if instead, students are allowed to complete the test, however the act of practicing will reset a pending test. In this scenario, when I miss a problem, the test gives me the option to continue or quit. If I chose to quit, the test will resume where I left off if I get back into it, but, if I practice all of the topics I missed questions on, the test will automatically reset (or at least give me the option to reset). This is not as good for people who have made mistakes that are not their fault (for example, I ended up having to work through 60 problems, because my wireless keyboard missed a keystroke), but it would still be better than the current situation (I would have been able to restart after 4 practice problems instead of 29 test problems).
The ideal solution has one thing the others do not: They encourage students to strive for perfection. Without a reset button, students are encourage to give up, once they get a passing grade, because trying to get a higher grade results in significant amounts of punishment, frustration, and misery. A reset button would largely eliminate that discouragement, making perfection more accessible but not actually encouraging it. Encouraging students to go back and study as soon as they miss a problem will encourage them to stop testing as soon as a weakness is identified, to fix that weakness. It will provide some motivation for students to fix gaps in their skill immediately, instead of rewarding them for aiming for "good enough". My personal opinion is that no education program has any excuse for graduating students without high proficiency, given our current technology and teaching capacity. There is no reason that Khan Academy could not just keep teaching students until they get 100%. In public schools, this is impossible, because you have one teacher and many students, and the teacher just does not have the time to strive for perfect proficiency. Khan Academy is scalable though. It can afford to at least encourage students to strive for perfection. 75% does not need to be good enough anymore, because if one student needs a few more hours of practice than another, Khan Academy does not have to split its time between that student and others. Every student who is struggling can go back, review the lecture, and practice, and it will not cut into lesson time for anyone else. But, we cannot do this with the current system, because it harshly punishes students for failure. The only way teaching effectiveness can be maximized is by eliminating punishment as much as possible and providing a positive response to failure as immediately as possible. Ideally that means building resetting right into the learning flow, so that when students fail they are immediately given the opportunity to improve and allowed to restart evaluation as soon as they have taken the opportunity to improve.
Khan Academy is already pretty good, but as I have said before, the very fact that the team has a pedagogical perspective suggests they are locked into a dogma that cannot provide the flexibility required to produce a good, evidence based (or world class) learning system. Above are some good options for improving some specific features, within whatever pedagogical dogma the team may subscribe to, but the better long term solution would be to leave the pedagogy and dogma behind and instead take a more scientific, evidence based approach to learning. Evidence based learning is not well suited to pedagogy in general, because it needs more flexibility than strictly defined pedagogy can provide or allow, to rapidly adapt to new discoveries in education research. Pedagogy is inherently dogmatic, and dogma has no place in evidence based anything.
I hope these ideas will find some good use in helping Khan Academy become what it desires to be, but if they do not, I hope someone eventually uses them to produce an electronic learning system that is truly world class and ideally free.
A Scathing Review of Khan Academy
I really hate to do this. Khan Academy, when it was first founded, was painstakingly built around the best of ideals. In attempting to help some relatives understand subjects their teachers at school were failing to teach effectively, Sal Khan discovered that he could fill a huge gap in U.S. education. Sal had no background in teaching, which turned out to be a huge advantage. In failing to follow traditional U.S. teaching pedagogy, Sal discovered a better way. Large numbers of students flocked to his videos, for a superior way to become educated. Sal recognized the enormous contribution he could make to society, not just in the U.S. but in the world, and he quit his job to found Khan Academy, a website dedicated to using this fresh new take on education to make a world class education available to anyone in the entire world with access to the internet, for free. Over time, Khan Academy evolved from a video hosting site for Sal's videos into a full learning platform with exercises and evaulations. Sal collaborated with some U.S. schools to come up with new teaching pedagogies that put Khan Academy at the core, with great success. As time went on though, Khan Academy began to hire experienced educators onto its staff, with the goal of bringing more experience into the process of improving Khan Academy. What Sal perhaps did not expect was that this would turn out to be a trojan horse for the traditional U.S. educational values that made Khan Academy necessary in the first place, to sneak in and corrupt the platform, turning it into little more than another online school. It is nice that Khan Academy is free, but I am not even sure that is a good thing, because it has become an engine for spreading a low quality teaching pedagogy to less developed countries. I love what Khan Academy wants to be...or at least wanted to be, but it has not become that. Worse, there was a time when it was actually closer, but it has backslid since.
My biggest gripe with Khan Academy, and the one that finally pushed me over the edge into writing this, is in how skill evaluation is handled. A couple days ago, I was working through the basic arithmetic section (maybe I will write about why I was doing that in another post). I recently bought a new mouse and keyboard for my desktop, because my old ones were wearing out (letters worn off, scroll wheel not working...), and the only decently priced options of good quality were wireless. Unfortunately, once in a while the keyboard fails to correctly send a character (a common problem with wireless devices). The most common issue is spaces, which may partially be caused by the switches not being sensitive enough, but as I was doing a Khan Academy test for multiplication, it also missed a number now and then. Now, Khan Academy gives achievements for speed, and on top of that, I am currently working through math I am already quite skilled at, so I do not want to take a lot of time. Unfortunately, once in a while, I end up missing a problem, due to keyboard error. More unfortunately, this means I will not be able to score 100% on the test. Even more unfortunately than this, Khan Academy provides no means for resetting a test, meaning I am forced to complete up to 30 problems before I can even start over and retake the test. From a software development perspective, this is a very serious user interface failure, but for educational skill evaluation, it kind of makes sense. At least, it makes sense in the context of traditional American dogmatic education pedagogy, a pedagogy that is more interested in teaching values than actually educating students, and a pedagogy that has a long history of criticism for its poor performance compared to more serious education. It does not make sense in the context of providing a world class education. In fact, evaluation by testing in general has mountains of evidence proving it to provide poor quality skill assessment, and further, testing oriented evaluation discourages legitimate teaching and learning and instead encourages rote memorization without understanding as well as cheating. Traditional test based evaluation teaches students that the test is more important than the learning and understanding, and the easiest way to score well on a test is cheating.
Inability to reset tests is not the underlying problem though. The underlying problem is the fundamental pedagogy behind this poor design and the fact that it runs directly counter to Khan Academy's goals and Sal's own understanding of education. Posted in Khan Academy's, official forum, this post excerpt from a Khan Academy product manager says it all.
Second, she claims that including a reset button would not encourage a growth mindset. I see two problems here. The first is the idea that encouraging a growth mindset is an important part of all educational pursuits. Say I want to learn math. Khan Academy believes that instead of teaching me math, they need to teach me a growth mindset, with some math on the side. Attempting to control how people think is a fascist pursuit that is unfortunately common in the U.S. education system. It is part of the "pedagogy" aka dogma. What is the purpose of the course, to teach math or to teach a "growth mindset"? If the purpose is to teach a growth mindset, then make it a "growth mindset" course and teach the math in its own course. Unfortunately, traditional U.S. educational pedagogy is full of ideas that educators are responsible for teaching students things other than the subjects at hand, and the result is curriculum and course design that is full of things that hinder learning and make students hate learning. The second problem is the belief that this does promote a growth mindset. The fact is, it does not. In the forum thread that this excerpt comes from, around a hundred Khan Academy users have described their response to failing a question as randomly answering the remaining questions to get through the test, so they can start over. This is not a growth mindset. This the same mindset we see in public school students who are taught that test performance is the most important part of education. According to Khan Academy's pedagogy, it is also cheating, because deliberately excluding a way to reset the test is a tacit assertion that resetting would be cheating. This is the kind of cheating that is not strictly wrong though. It is only cheating because Khan Academy's pedagogy treats it as such. It does not interfere in learning, aside from the fact that skipping questions this way wastes time that could be spent learning, the solution for which is a reset option. The fact is, this does not encouraging a learning mindset. Instead, it encourages a test oriented mindset, it frustrates students, and it punishes students for making mistakes. The instance of a test I personally was taking when I finally became frustrated enough to start looking for a solution is especially telling. It was my second attempt. My first attempt, I had legitimately missed a question, because I had not read the instructions well. My second attempt, I missed the second question of 31 questions, because my keyboard failed to register the final zero, when I was trying to type "3000". I was left with 29 remaining questions. To get a perfect score on the test, I would now have to answer 60 questions, 29 of which were purely punishment for my keyboard missing a zero. Not only is this completely unjust for a mistake that was not even my fault, it is also completely inappropriate for encouraging a growth mindset, even if I had legitimately gotten the problem wrong. Sure, being able to retake the test to get a better score is superior to how it works in U.S. schools, but that only solves part of the problem with using testing for evaluation. It still makes students hate learning (the opposite of a growth mindset), and it still encourages cheating over learning and wastes time that should be used for learning.
Third, she suggests that this helps students become more comfortable with failure.
Fourth, she says, "it teaches us to be more diligent". Aside from the fact that the reported behavior (skipping questions by answering them randomly) proves this is a false assertion, this comes back to the same thing as "learning mindset". Is the primary goal of a math course to teach diligence or math? If the goal is to learn math, than attempting to teach diligence interferes with the goal of the class. My position on this is the same: If you want to teach diligence, offer a course on diligence. It is inappropriate to allow your personal views on how people should behave get in the way of effective learning. This idea that diligence should be part of a math course is part of the pedagogical dogma in U.S. education. The truth however, is that attempting to teach diligence, a growth mindset, or anything other than math in a math course burdens the student with things that get in the way of learning math. Thus, the feature (or lack thereof) being defended not only does not teach diligence, teaching diligence has no place in a math course in the first place.
These are the most egregious problems with Khan Academy right now, and they are all based on corruption of the original goal of providing a fee, world class education. This is not the only feature affected; it is merely the most obvious. There was a time when Khan Academy's practice lessons would continue until you got a certain number in a row correct. This was a generally good system, as it would give students practice until they had achieved a certain degree of mastery. I don't know the justification, but this seems to have been reverted back to making practices only have a set number of problems. I suspect the reason is more of this pedagogical dogma, because a person that reasons the same way the person who wrote the above quoted post reasoned might argue that going until you get, say, four in a row right, is less evidence of mastery than doing four at a time, because doing only four at a time could allow you to get up to 6 in a row right, without earning credit for mastery (the first of the first four is wrong, the last of the second four is wrong). The problem is that first, this is treating practice as evaluation (being this concerned about performance of practice is inappropriate for effective education), and second, there is a much easier way to fix it, if students truly do need more practice. If you want students to have to get at least 7 in a row correct to complete practice, then change the threshold to 7, instead of leaving it at four, and now you can keep the serial practice instead of moving to chunks. I found the serial practice to be quite effective for my own learning, and I am severely disappointed that they eliminated that.
The fact is, Khan Academy is no longer a place to get a world class education. Yes, the videos are still there, and they are still good, but a world class education needs more than just quality lecture. A world class education should also optimize learning effectiveness, and that means removing all burdens to effective education, including the minimization of wasted time. The pedagogical dogma that has been adopted by Khan Academy in recent years excludes this and limits Khan Academy to only being able to provide an American class education, which is decidedly worse than many countries. The U.S. ranks 13th in PISA score, which ranks countries by the collaborative problem solving skills of their students. The U.S. is above the average of all participating countries, but only by a narrow margin. If Khan Academy wants to provide a world class education, using pedagogically driven teaching methods, it needs to replace its design team with educators from Singapore, Japan, and China. Alternatively, it could abandon the concept of pedagogy entirely and instead opt for a dynamic, evidence driven approach to education, which it advocated for at one time but has since clearly abandoned, and which has the potential to beat even these top three countries.
I watched a video several years ago, where Sal Khan talked about the critical role of failure in education. He explained how he was very careful not to punish his son for failure but rather to glory in it and encourage him to try again. We need an online learning platform based on this, not on American dogmatic education pedagogy. We need a system where failure is never punished, and where the next iteration can begin instantly upon failure. Unfortunately, this platform is not Khan Academy, and at this point, Khan Academy is so steeped in pedagogical dogma that requests for improvement are ignored or justified away, instead of receiving any serious consideration. It is clear to me that the current education team at Khan Academy can no longer be trusted to provide a world class education. I hope Sal will use whatever influence he has to replace this team with people who are willing to give up their outdated and disproved beliefs, in favor of an evidence based approach to learning. In other words, it is not Khan Academy's students who need a learning mindset and diligence. It is Khan Academy's own staff that needs to develop a learning mindset and diligence in their stated goal of bringing everyone who wants it a free, world class education.
My biggest gripe with Khan Academy, and the one that finally pushed me over the edge into writing this, is in how skill evaluation is handled. A couple days ago, I was working through the basic arithmetic section (maybe I will write about why I was doing that in another post). I recently bought a new mouse and keyboard for my desktop, because my old ones were wearing out (letters worn off, scroll wheel not working...), and the only decently priced options of good quality were wireless. Unfortunately, once in a while the keyboard fails to correctly send a character (a common problem with wireless devices). The most common issue is spaces, which may partially be caused by the switches not being sensitive enough, but as I was doing a Khan Academy test for multiplication, it also missed a number now and then. Now, Khan Academy gives achievements for speed, and on top of that, I am currently working through math I am already quite skilled at, so I do not want to take a lot of time. Unfortunately, once in a while, I end up missing a problem, due to keyboard error. More unfortunately, this means I will not be able to score 100% on the test. Even more unfortunately than this, Khan Academy provides no means for resetting a test, meaning I am forced to complete up to 30 problems before I can even start over and retake the test. From a software development perspective, this is a very serious user interface failure, but for educational skill evaluation, it kind of makes sense. At least, it makes sense in the context of traditional American dogmatic education pedagogy, a pedagogy that is more interested in teaching values than actually educating students, and a pedagogy that has a long history of criticism for its poor performance compared to more serious education. It does not make sense in the context of providing a world class education. In fact, evaluation by testing in general has mountains of evidence proving it to provide poor quality skill assessment, and further, testing oriented evaluation discourages legitimate teaching and learning and instead encourages rote memorization without understanding as well as cheating. Traditional test based evaluation teaches students that the test is more important than the learning and understanding, and the easiest way to score well on a test is cheating.
Inability to reset tests is not the underlying problem though. The underlying problem is the fundamental pedagogy behind this poor design and the fact that it runs directly counter to Khan Academy's goals and Sal's own understanding of education. Posted in Khan Academy's, official forum, this post excerpt from a Khan Academy product manager says it all.
I understand how frustrating it can be to have to retake an entire practice set or unit test because of one or a few mistakes, and how useful a restart button might seem to alleviate this. This is something that we have debated internally since it was initially brought up and with a wide range of team members, and after much discussion - we have decided against implementing a restart button because it goes against our pedagogical perspective.
We want to encourage growth mindset and learning on our platform, and a “Restart Button” would not encourage that. We want to encourage all learners to see that getting problems wrong or not getting 100% on the first try is one of the most important parts of learning--even if that error was a simple calculation mistake, it teaches us to be more diligent. In order to master something, you will first have to practice with material that you don't quite know. The current system is intended to identify the sections in which you need the most help by analyzing your initial answers and adjusting your practice to fit your specific needs.First, she makes it clear that this is about pedagogy, not about doing what is best for the students, not about doing what the users feel will best facilitate their own learning, and not about Khan Academy's prime directive of providing a free, world class education. The very fact that Khan Academy even has a "pedagogical perspective" is evidence of its corruption. The use of language like this is evidence that Khan Academy has been infiltrated by traditional education influences that are corrupting it from the inside. I have spent significant time studying what is and is not effective in education, and the only place I find the word "pedagogy" used is as a justification for ignoring evidence. Khan Academy showed that taking the lectures out of the classroom (a technique I have successfully used in my own teaching as a college professor) improved the general effectiveness of education. The most common excuse for not adopting this practice is that it does not fit the school's or teacher's "pedagogy". In my experience, when educators use the term "pedagogy", what they really mean is "pedagogical dogma". Dogma is religion not science. It is unsubstantiated belief not empirical evidence. It has no place in secular education!
Second, she claims that including a reset button would not encourage a growth mindset. I see two problems here. The first is the idea that encouraging a growth mindset is an important part of all educational pursuits. Say I want to learn math. Khan Academy believes that instead of teaching me math, they need to teach me a growth mindset, with some math on the side. Attempting to control how people think is a fascist pursuit that is unfortunately common in the U.S. education system. It is part of the "pedagogy" aka dogma. What is the purpose of the course, to teach math or to teach a "growth mindset"? If the purpose is to teach a growth mindset, then make it a "growth mindset" course and teach the math in its own course. Unfortunately, traditional U.S. educational pedagogy is full of ideas that educators are responsible for teaching students things other than the subjects at hand, and the result is curriculum and course design that is full of things that hinder learning and make students hate learning. The second problem is the belief that this does promote a growth mindset. The fact is, it does not. In the forum thread that this excerpt comes from, around a hundred Khan Academy users have described their response to failing a question as randomly answering the remaining questions to get through the test, so they can start over. This is not a growth mindset. This the same mindset we see in public school students who are taught that test performance is the most important part of education. According to Khan Academy's pedagogy, it is also cheating, because deliberately excluding a way to reset the test is a tacit assertion that resetting would be cheating. This is the kind of cheating that is not strictly wrong though. It is only cheating because Khan Academy's pedagogy treats it as such. It does not interfere in learning, aside from the fact that skipping questions this way wastes time that could be spent learning, the solution for which is a reset option. The fact is, this does not encouraging a learning mindset. Instead, it encourages a test oriented mindset, it frustrates students, and it punishes students for making mistakes. The instance of a test I personally was taking when I finally became frustrated enough to start looking for a solution is especially telling. It was my second attempt. My first attempt, I had legitimately missed a question, because I had not read the instructions well. My second attempt, I missed the second question of 31 questions, because my keyboard failed to register the final zero, when I was trying to type "3000". I was left with 29 remaining questions. To get a perfect score on the test, I would now have to answer 60 questions, 29 of which were purely punishment for my keyboard missing a zero. Not only is this completely unjust for a mistake that was not even my fault, it is also completely inappropriate for encouraging a growth mindset, even if I had legitimately gotten the problem wrong. Sure, being able to retake the test to get a better score is superior to how it works in U.S. schools, but that only solves part of the problem with using testing for evaluation. It still makes students hate learning (the opposite of a growth mindset), and it still encourages cheating over learning and wastes time that should be used for learning.
Third, she suggests that this helps students become more comfortable with failure.
We want to encourage all learners to see that getting problems wrong or not getting 100% on the first try is one of the most important parts of learning--even if that error was a simple calculation mistake, it teaches us to be more diligent.The problem is that this is not what this accomplishes. If getting problems wrong is so important to learning (and it is), then why punish students for failure? What I get out of the lack of a reset button is that getting 100% on the first try is the only important thing, because if I do not, I will be forced to spend inordinate amounts of time answering questions I am already skilled at, before I am allowed to prove that I have improved my skill on the one I missed. The lack of a reset button actually accomplishes the exact opposite of what this product manager claims it does. The right way to help students become comfortable with failure as a critical part of learning is to make turnaround as fast as possible. If I miss a problem on the test, let me give up on the test immediately, so I can go back and study what I missed, and then let me start the test over from the beginning, so I do not feel like I am being punished for not getting it perfect on the first try (ironic, isn't it?).
Fourth, she says, "it teaches us to be more diligent". Aside from the fact that the reported behavior (skipping questions by answering them randomly) proves this is a false assertion, this comes back to the same thing as "learning mindset". Is the primary goal of a math course to teach diligence or math? If the goal is to learn math, than attempting to teach diligence interferes with the goal of the class. My position on this is the same: If you want to teach diligence, offer a course on diligence. It is inappropriate to allow your personal views on how people should behave get in the way of effective learning. This idea that diligence should be part of a math course is part of the pedagogical dogma in U.S. education. The truth however, is that attempting to teach diligence, a growth mindset, or anything other than math in a math course burdens the student with things that get in the way of learning math. Thus, the feature (or lack thereof) being defended not only does not teach diligence, teaching diligence has no place in a math course in the first place.
These are the most egregious problems with Khan Academy right now, and they are all based on corruption of the original goal of providing a fee, world class education. This is not the only feature affected; it is merely the most obvious. There was a time when Khan Academy's practice lessons would continue until you got a certain number in a row correct. This was a generally good system, as it would give students practice until they had achieved a certain degree of mastery. I don't know the justification, but this seems to have been reverted back to making practices only have a set number of problems. I suspect the reason is more of this pedagogical dogma, because a person that reasons the same way the person who wrote the above quoted post reasoned might argue that going until you get, say, four in a row right, is less evidence of mastery than doing four at a time, because doing only four at a time could allow you to get up to 6 in a row right, without earning credit for mastery (the first of the first four is wrong, the last of the second four is wrong). The problem is that first, this is treating practice as evaluation (being this concerned about performance of practice is inappropriate for effective education), and second, there is a much easier way to fix it, if students truly do need more practice. If you want students to have to get at least 7 in a row correct to complete practice, then change the threshold to 7, instead of leaving it at four, and now you can keep the serial practice instead of moving to chunks. I found the serial practice to be quite effective for my own learning, and I am severely disappointed that they eliminated that.
The fact is, Khan Academy is no longer a place to get a world class education. Yes, the videos are still there, and they are still good, but a world class education needs more than just quality lecture. A world class education should also optimize learning effectiveness, and that means removing all burdens to effective education, including the minimization of wasted time. The pedagogical dogma that has been adopted by Khan Academy in recent years excludes this and limits Khan Academy to only being able to provide an American class education, which is decidedly worse than many countries. The U.S. ranks 13th in PISA score, which ranks countries by the collaborative problem solving skills of their students. The U.S. is above the average of all participating countries, but only by a narrow margin. If Khan Academy wants to provide a world class education, using pedagogically driven teaching methods, it needs to replace its design team with educators from Singapore, Japan, and China. Alternatively, it could abandon the concept of pedagogy entirely and instead opt for a dynamic, evidence driven approach to education, which it advocated for at one time but has since clearly abandoned, and which has the potential to beat even these top three countries.
I watched a video several years ago, where Sal Khan talked about the critical role of failure in education. He explained how he was very careful not to punish his son for failure but rather to glory in it and encourage him to try again. We need an online learning platform based on this, not on American dogmatic education pedagogy. We need a system where failure is never punished, and where the next iteration can begin instantly upon failure. Unfortunately, this platform is not Khan Academy, and at this point, Khan Academy is so steeped in pedagogical dogma that requests for improvement are ignored or justified away, instead of receiving any serious consideration. It is clear to me that the current education team at Khan Academy can no longer be trusted to provide a world class education. I hope Sal will use whatever influence he has to replace this team with people who are willing to give up their outdated and disproved beliefs, in favor of an evidence based approach to learning. In other words, it is not Khan Academy's students who need a learning mindset and diligence. It is Khan Academy's own staff that needs to develop a learning mindset and diligence in their stated goal of bringing everyone who wants it a free, world class education.
Tuesday, October 30, 2018
Screen Time
Many Silicon Valley executives, tech industry managers, and workers are cutting down on or even outright banning technology in their homes. Yes, this is beyond ironic, but that is not what I want to discuss today.
Popular wisdom says that screen time is bad for young children. There is some research to back up this belief, however it is far more specific than most people realize. For example, one study found that some children around kindergarten age had significant difficulty using physical books and magazines. The children in the study who had been allowed significant screen time tended to try to swipe the surface of the pages, expecting that to be enough to move to the next page. It was clear they expected books to work like smart phone or tablet touchscreens. There have also been a few studies that found addiction-like behavior associated with significant amounts of screen time, and this is an issue experienced by many of the parents who are banning the technology in the home. Another problem backed by significant research is that children that get a lot of screen time tend to be less social than kids who get very limited screen time. All of these may seem like serious issues, but context is critical in understanding what is actually happening.
The first issue, where children struggle to understand how to use books, might seem like a serious problem that is caused by modern technology use, but technology use has nothing to do with it. The problem is not technology use. The problem is lack of experience with books. Parents that read with their children, a practice that is incredibly highly recommended by child development experts, do not have children that struggle to use books. It does not matter how much or little screen time these children get, if their parents read books with them, the children do not suffer from this problem. It is not the screen time that decides this, it is the book time.
The second issue, addiction-like behavior, is not addiction at all. Even educated people in the U.S. rarely understand addiction. They believe that children throwing fits when the technology is taken away is a sign of addiction. It's not! Neither is children sneaking into the technology behind parents' backs. This is actually normal behavior for children, and attentive parents will see this and use effective discipline to correct the behavior. The fact is, the human brain craves stimulation. Fully developed adult brains typically crave less stimulation than teen and child brains (though, this depends on a lot of factors during early childhood through teen development; this positive trait can and often is partially trained out of us by adulthood). The human brain is excellent at optimization as well. What is the easiest, cheapest way to get stimulation, especially for a child that does not know how to read? Tablets and phones offer endless stimulation, far beyond anything books and television can offer. They offer it for the cost of almost no physical effort. No wonder the brains of children love it. The fact is, their brains are trying to learn, and taking that away may actually be worse for their long term development than letting them have it. This is not addiction at all. This is merely the human brain optimizing it's use of time and effort. If parents want their children to spend less time at a screen, they need to find other forms of stimulation capable of competing. Taking away the stimulation without replacing it will tell the brain that it does not need as much development and resources, and in the long run it will result in teens and adults who have a more difficult time learning new things. The brains of babies and young children are capable of incredible rates of learning. This is necessary, because a new born child's brain is not even developed enough to translate light coming into the eyes as images, language as meaningful communication, or to effectively and precisely control muscle movements. It takes days to weeks for babies to be able to resolve images. It takes months for a baby to start understanding language. It takes years to develop the brain sufficiently to have decent fine muscle control. And this applies in some degree or another to all of the senses. As children get older, this rapid rate of brain development and learning becomes less necessary. As this happens, the brain tends to slowly lose its plasticity. Some of this plasticity can be retained, however, by replacing the learning that is becoming less necessary with more learning. Modern technology provides ample opportunity for the kinds of stimulation that can do this. Denying children this stimulation may actually put them at a disadvantage as adults who are less capable of learning new things than their peers who had the benefit of this stimulation. Of course, children won't benefit from spending all of their time in front of a screen. As with the book issue, if screen time is preventing children from gaining experiences with other important things, then that is a problem. But if parents are struggling with this because their children do not want to do anything without a screen, that is a discipline problem, not an addiction. Learning to use technology responsibly is critical in our culture, and denying children the opportunity to learn this sort of discipline is doing them a serious disservice. Instead of complaining about addictiveness of the technology, parents need to step up and teach their children sufficient self discipline to eventually be able to function as adults that are constantly surrounded with this technology. Waiting until they are teens or adults is likely to result in far more serious problems than having to deal with tantrums for a few weeks or months.
The last problem, lack of social interaction, is also a serious problem, and it is also a problem that has far less to do with screens than people tend to believe. Screens, and especially small screens like those found on phones and tablets, do lend themselves to solo use. The problem is not the technology though. The problem is how it is used. If children are left alone with tablets or phones for hours on end, instead of getting the interpersonal interaction that parents are supposed to provide, of course those children will be less social. And that's not all! Children that do not get a lot of interpersonal time with parents are also more likely to turn to friends or others who are less experienced when making life altering decisions. They are more likely to be rebellious in their teens or even earlier. Children that get too little interpersonal time can even develop attachment disorder, which is a common disorder in orphans and children of drug addicts and is closely tied to lack of discipline, feelings of abandonment, and feelings of insecurity. The fact is, screen time with parents has few if any of the harmful side effects associated with solo screen time. If a child is experiencing issues as a result of screen time, it is very likely that screens are being used to babysit, which is not an appropriate way to use screens. Some solo screen time is not harmful, but if screen time is being used to occupy a child so parents can have more free time, that's probably too much. Involved and attentive parents are unlikely to experience most of the issues associated with excessive screen time.
The trend here should be obvious. Nearly all of the issues parents are blaming on screens are actually parenting problems. None of these problems are hard to avoid. If screen time is causing problems, parents need to make more time for their children. They need to find screen activities that they can participate in with their children. They need to quit using screens as babysitters. Often, part of the solution is spending less time on their own screens. And of course, they need to take responsibility for teaching their children self discipline and for making sure their children get experience with things other than screens. If this is so hard that the only solution is to eliminate screens entirely, maybe it is time to rethink having children in the first place. If you do not have time for your children, then you have far more serious personal problems than screens.
To make this perfectly clear, screen time is not a problem! Screen time is incredibly valuable, and in our culture, children who do not get enough screen time will almost certainly be at a significant disadvantage as teens and adults. Instead of sabotaging your children, turn screen time into a group activity, and spend more time with your children even outside of screen time. A $100,000+ a year job is not worth the harm not being there for your children when they need it will cause. If your money and social class are worth more to you than the well being of your children, you either need to talk to a therapist or an adoption agency.
Popular wisdom says that screen time is bad for young children. There is some research to back up this belief, however it is far more specific than most people realize. For example, one study found that some children around kindergarten age had significant difficulty using physical books and magazines. The children in the study who had been allowed significant screen time tended to try to swipe the surface of the pages, expecting that to be enough to move to the next page. It was clear they expected books to work like smart phone or tablet touchscreens. There have also been a few studies that found addiction-like behavior associated with significant amounts of screen time, and this is an issue experienced by many of the parents who are banning the technology in the home. Another problem backed by significant research is that children that get a lot of screen time tend to be less social than kids who get very limited screen time. All of these may seem like serious issues, but context is critical in understanding what is actually happening.
The first issue, where children struggle to understand how to use books, might seem like a serious problem that is caused by modern technology use, but technology use has nothing to do with it. The problem is not technology use. The problem is lack of experience with books. Parents that read with their children, a practice that is incredibly highly recommended by child development experts, do not have children that struggle to use books. It does not matter how much or little screen time these children get, if their parents read books with them, the children do not suffer from this problem. It is not the screen time that decides this, it is the book time.
The second issue, addiction-like behavior, is not addiction at all. Even educated people in the U.S. rarely understand addiction. They believe that children throwing fits when the technology is taken away is a sign of addiction. It's not! Neither is children sneaking into the technology behind parents' backs. This is actually normal behavior for children, and attentive parents will see this and use effective discipline to correct the behavior. The fact is, the human brain craves stimulation. Fully developed adult brains typically crave less stimulation than teen and child brains (though, this depends on a lot of factors during early childhood through teen development; this positive trait can and often is partially trained out of us by adulthood). The human brain is excellent at optimization as well. What is the easiest, cheapest way to get stimulation, especially for a child that does not know how to read? Tablets and phones offer endless stimulation, far beyond anything books and television can offer. They offer it for the cost of almost no physical effort. No wonder the brains of children love it. The fact is, their brains are trying to learn, and taking that away may actually be worse for their long term development than letting them have it. This is not addiction at all. This is merely the human brain optimizing it's use of time and effort. If parents want their children to spend less time at a screen, they need to find other forms of stimulation capable of competing. Taking away the stimulation without replacing it will tell the brain that it does not need as much development and resources, and in the long run it will result in teens and adults who have a more difficult time learning new things. The brains of babies and young children are capable of incredible rates of learning. This is necessary, because a new born child's brain is not even developed enough to translate light coming into the eyes as images, language as meaningful communication, or to effectively and precisely control muscle movements. It takes days to weeks for babies to be able to resolve images. It takes months for a baby to start understanding language. It takes years to develop the brain sufficiently to have decent fine muscle control. And this applies in some degree or another to all of the senses. As children get older, this rapid rate of brain development and learning becomes less necessary. As this happens, the brain tends to slowly lose its plasticity. Some of this plasticity can be retained, however, by replacing the learning that is becoming less necessary with more learning. Modern technology provides ample opportunity for the kinds of stimulation that can do this. Denying children this stimulation may actually put them at a disadvantage as adults who are less capable of learning new things than their peers who had the benefit of this stimulation. Of course, children won't benefit from spending all of their time in front of a screen. As with the book issue, if screen time is preventing children from gaining experiences with other important things, then that is a problem. But if parents are struggling with this because their children do not want to do anything without a screen, that is a discipline problem, not an addiction. Learning to use technology responsibly is critical in our culture, and denying children the opportunity to learn this sort of discipline is doing them a serious disservice. Instead of complaining about addictiveness of the technology, parents need to step up and teach their children sufficient self discipline to eventually be able to function as adults that are constantly surrounded with this technology. Waiting until they are teens or adults is likely to result in far more serious problems than having to deal with tantrums for a few weeks or months.
The last problem, lack of social interaction, is also a serious problem, and it is also a problem that has far less to do with screens than people tend to believe. Screens, and especially small screens like those found on phones and tablets, do lend themselves to solo use. The problem is not the technology though. The problem is how it is used. If children are left alone with tablets or phones for hours on end, instead of getting the interpersonal interaction that parents are supposed to provide, of course those children will be less social. And that's not all! Children that do not get a lot of interpersonal time with parents are also more likely to turn to friends or others who are less experienced when making life altering decisions. They are more likely to be rebellious in their teens or even earlier. Children that get too little interpersonal time can even develop attachment disorder, which is a common disorder in orphans and children of drug addicts and is closely tied to lack of discipline, feelings of abandonment, and feelings of insecurity. The fact is, screen time with parents has few if any of the harmful side effects associated with solo screen time. If a child is experiencing issues as a result of screen time, it is very likely that screens are being used to babysit, which is not an appropriate way to use screens. Some solo screen time is not harmful, but if screen time is being used to occupy a child so parents can have more free time, that's probably too much. Involved and attentive parents are unlikely to experience most of the issues associated with excessive screen time.
The trend here should be obvious. Nearly all of the issues parents are blaming on screens are actually parenting problems. None of these problems are hard to avoid. If screen time is causing problems, parents need to make more time for their children. They need to find screen activities that they can participate in with their children. They need to quit using screens as babysitters. Often, part of the solution is spending less time on their own screens. And of course, they need to take responsibility for teaching their children self discipline and for making sure their children get experience with things other than screens. If this is so hard that the only solution is to eliminate screens entirely, maybe it is time to rethink having children in the first place. If you do not have time for your children, then you have far more serious personal problems than screens.
To make this perfectly clear, screen time is not a problem! Screen time is incredibly valuable, and in our culture, children who do not get enough screen time will almost certainly be at a significant disadvantage as teens and adults. Instead of sabotaging your children, turn screen time into a group activity, and spend more time with your children even outside of screen time. A $100,000+ a year job is not worth the harm not being there for your children when they need it will cause. If your money and social class are worth more to you than the well being of your children, you either need to talk to a therapist or an adoption agency.
Monday, November 6, 2017
Open Ended Assignments
A few weeks ago, I got an email from a student complaining about how I structure assignments. He was bothered by the fact that I require some decision making that is beyond the scope of the assignments. He complained that he was spending more time trying to decide what to do for the assignments than he was actually doing them. Now, I want to be clear: This student is very intelligent. I teach upper division undergrad Computer Science courses. I don't get students that are not very intelligent, because they would not have passed the prerequisites otherwise. Sometimes I get students that struggle with creativity though.
Why do highly intelligent students struggle with creativity? How can I get students in an upper division Computer Science course, which is part of a field where intelligence and creativity are both critical, who struggle with the creativity element? As with most problems I have discussed, the problem is not with the students. The problem is with our education system. We do a poor job of teaching creativity. Yes, many schools are trying to focus on creativity more now. They are adding art courses, music classes are starting to come back in some places, and there is more emphasis on artistic expression in things like English. Creativity does not mean art though. Computers can make art out of randomness or through complex algorithms, and computers are literally incapable of creativity. They can make art that appears to be creative, but the creativity is artificial. "Creative" classes in public school tend to be the same way. Music classes are all about singing. Singing is not terribly creative (it can be, but you won't find that below the college level outside of private lessons). Kindergarten art tends to be creative, but it also tends to mostly be scribbling. Much past kindergarten, art classes are generally highly structured, squeezing out creativity. There are some reasons for this. Gaining skill takes practice, and purely creative practice tends to miss important skills. Structured assignments have value in improving skill, but when it is almost all structured assignments, creativity is lost. Despite many attempts public schools do a poor job of encouraging creativity. They prefer to focus heavily on technical skills and memorization over creativity. And frankly, colleges are not much better. There is more creativity in liberal arts courses, but when you get to more technical courses, there are a lot of structured assignments with very little creative flexibility. Being required to think for themselves is something students are generally not used to, because creativity is marginalized in most of our education system.
Why do I encourage creativity? In Computer Science, creativity is a critical skill. CS is all about advanced problem solving. Creativity is essential to effective problem solving. Now, most of my students have decent basic problem solving skills. Given a structured programming assignment, they can figure out how to solve the problem presented. This is a great start. So what happens when a boss says, "Hey, someone suggested using this new piece of software. Can you experiment and compare it to what we are currently using?" The first thing the employee has to do is devise an application that the software can be tested against. If the software is a programming language, this is literal. The employee will probably have to write a program twice, once for each language, and it is going to have to highlight the most important aspects of what the company will use the software for. This is precisely the kind of creativity my open ended assignments require. A more likely scenario is that an employee is put on the design team. Solving a programming problem when you have a design document in front of you is totally different from creating the design document in the first place. This kind of creativity cannot be learned from highly structured assignments. It can only be learned with a potentially uncomfortable level of flexibility in assignments.
Here are some examples of assignments I have given:
A major part of the reason I use open ended assignments so heavily is that I cannot design good assignments. I am not suggesting I have a problem or am inferior to other teachers. It is just a fact that teachers cannot design good assignments for 10 to 30 students. There is no good assignment for that many students. Public school curriculum tends to shoot for the lowest reasonable expectations. This strategy allows all students with a reasonable understanding to get a good grade, but it cheats the more skilled and more advanced students out of additional opportunities for growth. In addition to that, I can only design assignments that fit my experience, and I am prone to designing assignments that fit my own interests, because that is what I know the most about. In short, I will end up training my students poorly in topics that are of personal interest if I design assignments completely on my own, and they will miss out on things that might interest them and that will likely be more relevant to them in the long run. So instead, I leave the assignments open ended, so that my students can pick topics that interest them and are relevant to them.
How well does this work out? Honestly, I think it works out great. One fear of open ended assignments is that students will be lazy, doing the minimum possible, and learning the minimum possible. This is a valid fear, but the evidence suggests it is not worth worrying about. Those three assignments I listed above have resulted in some excellent work from students. The first one could be done trivially. A student who pays attention in class could complete the assignment in less than 10 minutes. And I would give the student full credit, because it fulfills the requirements. This semester not a single student did this! Out of 16 students, not one took the easy path. Most wrote programs designed to highlight the issues they ran into, which made for a very valuable discussion when they presented their work. The second one they went even further! Again, not a single student took the easy path. Some students struggled. They could have given up and done it the easy way, but they did not. In my assembly language course, I have seen a broad range of strategies. Some students divide the work into a lot of tiny programs, each demonstrating one thing. Others write larger, more complex programs covering a range of things in one shot. Some students take the easy route first, then they use the time they saved to write larger, more complex programs that don't technically pass off anything, because they want to. And honestly, it does not matter terribly if they do take the easy route, because most of the assignments are only intended to demonstrate one proficiency and then give students the opportunity to learn. The place I expect them to show full ability is in the final project. I love that most of my students are motivated to take advantage of the additional learning opportunities open ended assignments provide, but even if they don't, they are still proving that they can do the things I am teaching them to do.
One of the most valuable parts of open ended assignments is motivation. If I give students a strictly defined assignment, they don't own it. It was not their idea. Odds are they are not terribly interested in the topic. Their commitment to the assignment is only implicit in the fact that they chose to take the course. When they design their own assignments around a small set of requirements though, it is their assignment. In designing it, they are committing to it. They own it. Multiple times this semester I have had students present their work, when it is complete enough to get credit, but there are still things they don't like about it. They go home after class and keep working on it, because they are not satisfied with their work. They have already gotten full credit, but they are committed to the work, and they own the assignment, so they care that it is done right. A few days later, I get an email, or they pull me aside in class, to show me what they have accomplished. These students are not just designing the minimal assignment, and they are not just stopping once they have the grade. They are choosing to work hard when they don't have to, because I am allowing and encouraging them create the assignments they want to do, instead of making them do assignments that I like. Not only is this producing better work, but my students are also learning more, gaining more experience, and gaining a love for the topics of my courses. They are also learning to learn on their own, as many choose to go beyond what I am teaching them. Open ended assignments allow and encourage students to do work that they are motivated to do. This results in better work overall, superior learning, and it allows more advanced students to do more challenging work and less advanced students to do work more consistent with their level of experience.
Open ended assignments can result in more grading work. When all assignments are identical, it is easy to produce a rubric that exhaustively defines how various elements of the assignment will affect the grade. It would take an absurdly complex rubric to cover all possibilities for an open ended assignment. Instead, I use the proficiency based method I have discussed in the past. Instead of expecting a perfect program from students, I distill the requirements down to only the proficiencies I want the students to demonstrate in that particular assignment. Each assignment has one, two, or maybe three bullet points. If a student can demonstrate that all of those bullet points have been satisfied, the assignment is given full credit. Otherwise it gets no credit. Multiple attempts at demonstrating proficiency are allowed, within reason, and each failure receives feedback. This is how we learn in real life. This system eliminates the burden of carefully reviewing every assignment for every students to count up a score. I makes it much easier to deal with open ended assignments. This semester, I am trying something new as well. Instead of turning in work for me to grade, and instead of meeting with me one-on-one to demonstrate work, students present their work to the class. This allows me to easily grade work during class time, and it also provides discussion material and examples to other students. If a student makes a common mistake or uses a bad practice, I can (politely and without judgement) point it out to the class and explain how it might be improved upon. This allows all students to learn from each student's mistakes. It also gives students the opportunity to provide other students with feedback, and it allows less advanced students to learn from the work of more advanced students. In addition to all of this, it reduces the grading burden, by putting most of the grading in class. All of this makes open ended assignments have a far more reasonable grading burden, despite the additional complexity it adds. (It also happens to give students some experience with code reviews, which are commonly used in larger companies in industry, as a quality assurance strategy. In artistic courses, it would teach students to give and take constructive criticism better.)
Open ended assignments help to teach creativity by providing more opportunity and encouragement to be creative. Many students will likely struggle with it initially, because they are used to being spoon fed assignments, without any need to exercise creativity. Creativity cannot be taught using normal teaching techniques. It is developed through practice. Open ended assignments provide this practice.
Why do highly intelligent students struggle with creativity? How can I get students in an upper division Computer Science course, which is part of a field where intelligence and creativity are both critical, who struggle with the creativity element? As with most problems I have discussed, the problem is not with the students. The problem is with our education system. We do a poor job of teaching creativity. Yes, many schools are trying to focus on creativity more now. They are adding art courses, music classes are starting to come back in some places, and there is more emphasis on artistic expression in things like English. Creativity does not mean art though. Computers can make art out of randomness or through complex algorithms, and computers are literally incapable of creativity. They can make art that appears to be creative, but the creativity is artificial. "Creative" classes in public school tend to be the same way. Music classes are all about singing. Singing is not terribly creative (it can be, but you won't find that below the college level outside of private lessons). Kindergarten art tends to be creative, but it also tends to mostly be scribbling. Much past kindergarten, art classes are generally highly structured, squeezing out creativity. There are some reasons for this. Gaining skill takes practice, and purely creative practice tends to miss important skills. Structured assignments have value in improving skill, but when it is almost all structured assignments, creativity is lost. Despite many attempts public schools do a poor job of encouraging creativity. They prefer to focus heavily on technical skills and memorization over creativity. And frankly, colleges are not much better. There is more creativity in liberal arts courses, but when you get to more technical courses, there are a lot of structured assignments with very little creative flexibility. Being required to think for themselves is something students are generally not used to, because creativity is marginalized in most of our education system.
Why do I encourage creativity? In Computer Science, creativity is a critical skill. CS is all about advanced problem solving. Creativity is essential to effective problem solving. Now, most of my students have decent basic problem solving skills. Given a structured programming assignment, they can figure out how to solve the problem presented. This is a great start. So what happens when a boss says, "Hey, someone suggested using this new piece of software. Can you experiment and compare it to what we are currently using?" The first thing the employee has to do is devise an application that the software can be tested against. If the software is a programming language, this is literal. The employee will probably have to write a program twice, once for each language, and it is going to have to highlight the most important aspects of what the company will use the software for. This is precisely the kind of creativity my open ended assignments require. A more likely scenario is that an employee is put on the design team. Solving a programming problem when you have a design document in front of you is totally different from creating the design document in the first place. This kind of creativity cannot be learned from highly structured assignments. It can only be learned with a potentially uncomfortable level of flexibility in assignments.
Here are some examples of assignments I have given:
- Create a simple multi-threaded program in Python. Keep track of issues you run into.
- Create a multi-threaded C++11 program that uses mutual exclusion.
- Write a program that demonstrates that you understand [some assembly language principle].
A major part of the reason I use open ended assignments so heavily is that I cannot design good assignments. I am not suggesting I have a problem or am inferior to other teachers. It is just a fact that teachers cannot design good assignments for 10 to 30 students. There is no good assignment for that many students. Public school curriculum tends to shoot for the lowest reasonable expectations. This strategy allows all students with a reasonable understanding to get a good grade, but it cheats the more skilled and more advanced students out of additional opportunities for growth. In addition to that, I can only design assignments that fit my experience, and I am prone to designing assignments that fit my own interests, because that is what I know the most about. In short, I will end up training my students poorly in topics that are of personal interest if I design assignments completely on my own, and they will miss out on things that might interest them and that will likely be more relevant to them in the long run. So instead, I leave the assignments open ended, so that my students can pick topics that interest them and are relevant to them.
How well does this work out? Honestly, I think it works out great. One fear of open ended assignments is that students will be lazy, doing the minimum possible, and learning the minimum possible. This is a valid fear, but the evidence suggests it is not worth worrying about. Those three assignments I listed above have resulted in some excellent work from students. The first one could be done trivially. A student who pays attention in class could complete the assignment in less than 10 minutes. And I would give the student full credit, because it fulfills the requirements. This semester not a single student did this! Out of 16 students, not one took the easy path. Most wrote programs designed to highlight the issues they ran into, which made for a very valuable discussion when they presented their work. The second one they went even further! Again, not a single student took the easy path. Some students struggled. They could have given up and done it the easy way, but they did not. In my assembly language course, I have seen a broad range of strategies. Some students divide the work into a lot of tiny programs, each demonstrating one thing. Others write larger, more complex programs covering a range of things in one shot. Some students take the easy route first, then they use the time they saved to write larger, more complex programs that don't technically pass off anything, because they want to. And honestly, it does not matter terribly if they do take the easy route, because most of the assignments are only intended to demonstrate one proficiency and then give students the opportunity to learn. The place I expect them to show full ability is in the final project. I love that most of my students are motivated to take advantage of the additional learning opportunities open ended assignments provide, but even if they don't, they are still proving that they can do the things I am teaching them to do.
One of the most valuable parts of open ended assignments is motivation. If I give students a strictly defined assignment, they don't own it. It was not their idea. Odds are they are not terribly interested in the topic. Their commitment to the assignment is only implicit in the fact that they chose to take the course. When they design their own assignments around a small set of requirements though, it is their assignment. In designing it, they are committing to it. They own it. Multiple times this semester I have had students present their work, when it is complete enough to get credit, but there are still things they don't like about it. They go home after class and keep working on it, because they are not satisfied with their work. They have already gotten full credit, but they are committed to the work, and they own the assignment, so they care that it is done right. A few days later, I get an email, or they pull me aside in class, to show me what they have accomplished. These students are not just designing the minimal assignment, and they are not just stopping once they have the grade. They are choosing to work hard when they don't have to, because I am allowing and encouraging them create the assignments they want to do, instead of making them do assignments that I like. Not only is this producing better work, but my students are also learning more, gaining more experience, and gaining a love for the topics of my courses. They are also learning to learn on their own, as many choose to go beyond what I am teaching them. Open ended assignments allow and encourage students to do work that they are motivated to do. This results in better work overall, superior learning, and it allows more advanced students to do more challenging work and less advanced students to do work more consistent with their level of experience.
Open ended assignments can result in more grading work. When all assignments are identical, it is easy to produce a rubric that exhaustively defines how various elements of the assignment will affect the grade. It would take an absurdly complex rubric to cover all possibilities for an open ended assignment. Instead, I use the proficiency based method I have discussed in the past. Instead of expecting a perfect program from students, I distill the requirements down to only the proficiencies I want the students to demonstrate in that particular assignment. Each assignment has one, two, or maybe three bullet points. If a student can demonstrate that all of those bullet points have been satisfied, the assignment is given full credit. Otherwise it gets no credit. Multiple attempts at demonstrating proficiency are allowed, within reason, and each failure receives feedback. This is how we learn in real life. This system eliminates the burden of carefully reviewing every assignment for every students to count up a score. I makes it much easier to deal with open ended assignments. This semester, I am trying something new as well. Instead of turning in work for me to grade, and instead of meeting with me one-on-one to demonstrate work, students present their work to the class. This allows me to easily grade work during class time, and it also provides discussion material and examples to other students. If a student makes a common mistake or uses a bad practice, I can (politely and without judgement) point it out to the class and explain how it might be improved upon. This allows all students to learn from each student's mistakes. It also gives students the opportunity to provide other students with feedback, and it allows less advanced students to learn from the work of more advanced students. In addition to all of this, it reduces the grading burden, by putting most of the grading in class. All of this makes open ended assignments have a far more reasonable grading burden, despite the additional complexity it adds. (It also happens to give students some experience with code reviews, which are commonly used in larger companies in industry, as a quality assurance strategy. In artistic courses, it would teach students to give and take constructive criticism better.)
Open ended assignments help to teach creativity by providing more opportunity and encouragement to be creative. Many students will likely struggle with it initially, because they are used to being spoon fed assignments, without any need to exercise creativity. Creativity cannot be taught using normal teaching techniques. It is developed through practice. Open ended assignments provide this practice.
Thursday, March 9, 2017
Ineffective Teaching Methods
Earlier today, I decided two write two articles. One was on mandatory group work, and the other was on controlling the pace of learning. Now I have a third topic, which is withholding feedback. So, I decided to just write a single article on commonly used ineffective teaching methods.
Mandatory Group Work
I was once a student, and I was required to work in groups in some of my classes. I am something of an introvert, I am a very fast learner, and thus far, I have taken only a few classes where I did not already have a strong background in the subject being taught. (Welcome to an education system where the paper matters more than the actual learning.) In a few classes, the mandatory group work was legitimately necessary due to the subject matter. In most, however, it was an artificial construct based on the misapplication of school guidelines. The college I graduated from and now teach at has formal guidelines teachers are very highly encouraged to follow. One of those guidelines is that students should teach each other. This is based on very solid research indicating that when one person teaches something to another, the teacher will learn the subject better as well as the learner. In other words, when students teach each other, everyone benefits. In addition, this leaves more time for professors to spend helping students that need help specifically from the professor. In other words, the school's policy is backed by solid science.
Most classes I have taken with mandatory group work did not actually need the group work to be effective. In all, I recall only one where the group work was actually beneficial to me, and that was partially because there was more work involved than was reasonable for a single person to do, and the other person was a fast enough learner to catch up with me (the other person was a woman; I have not had a similar experience with male students). There were maybe two other instances where working as a group helped the other members of the group, but, ironically, I knew the material so well that teaching them did not actually benefit me significantly. All of the other instances of mandatory group work I was required to do for my college classes actually made things more difficult for everyone. In one case, where we got to choose our own topics, the other members of my group thought they understood the topic I suggested, and in the end, because they did not, one member of the group left in a fit a rage, because myself and the third member would not let him present incorrect information in the group presentation. The group ultimately fell apart, because neither of the other members understood the topic that I found interesting, but they did not realize that until it was too late. I could easily have gone solo on both the research paper and the presentation, but that was not allowed, due to a misapplication of the school's teaching policy.
The big problem with mandatory group work is that it is almost impossible to group people in a way that works well for everyone. Some people just work better on their own (I am one of these people). Some people need to work with people who have complimentary personalities. Some people work best when they are ahead of others in the group, and some work better with those who are ahead of them. And of course, some people work better with those close to the same level as them. This does not even account for differences in personality or those students who are lazy and freeload off of the rest of the group. The fact is, for most topics, group work is not the best arrangement for everyone. Yes, I understand that most work in the real world is group work, and I even point this out to my students when I encourage them to work in groups. Don't forget though, school is not the real world workplace, and it is not appropriate to treat it like it is. Students recognize this, which is why many complain about mandatory group work. When group work interferes with effective learning, it is a bad idea, no matter how common group work is in the workplace. School is not the place to learn to get along with others. To be totally frank, I think the workplace is a better place to learn this skill, for those who have not already learned it.
There are some places where group work is valuable or even necessary. Certain types of creative design benefit significantly from collaboration. I teach video game design, which is one of these types of creative design. The value of collaboration is so high that collaboration is an essential skill for good video game design. Even then though, I don't require group work. I strongly encourage it, and I require students to present their ideas and listen to feedback, even if they are not part of a group. This is not always a good solution, but it is a way of getting some of the benefits of groups without having formal groups. Another case I have seen where group work is appropriate is when the class size is too big to be reasonable to grade students individually for all of the work. Ideally, this should be a temporary situation, where additional sections will be added to distribute some of the grading burden. The last place where group work is appropriate is when a project is necessarily too large for a single student (interestingly, this is the reason group work is necessary in the real world as well).
The takeaway here is that group work is not necessary for students to teach each other, and it is often a hindrance to effective learning. When considering requiring students to work in groups, serious thought should be given to the value of doing this, as the cost to students is almost certainly going to be high. There are places where it is appropriate, but there are perhaps more where it is not. My personal solution is to encourage students to self organize into groups but allow them to work solo if they prefer. I find this works better on two levels. One is that when students self organize, they are more likely to form groups that work better. The other is that some of the students who work solo do so because they learn so much faster than everyone else. The benefit to this is that no one is holding them back, and they are not pulling anyone else in deeper than they can handle. I also find that these solo students are more likely to and more capable of helping people in other groups. Mandatory group work not only hinders effective learning, it also tends to hinder the ability of students to teach each other, because they feel partitioned.
Pace of Learning
It was recently suggested to me that allowing students to work at their own pace hinders their ability to teach each other. Unfortunately, this is an attitude I have seen a lot. Most classes I have taken or otherwise been involved with limit the dispensation of knowledge, in the attempt to enforce a particular pace of learning. The theory is, if everyone is at the same place, they will be more able to help each other. To me, this is totally counter-intuitive. It does not make any sense to me that a collection of people who all have the same level of knowledge would be able to effectively teach each other, because if they all know the same stuff, what is there to teach? There are, of course, some teaching opportunities, as some students catch on to the subject at hand faster and are able to help those who catch on slower, but in a traditional setting, there is not much time for this, because the class just keeps progressing. If a fast student for a topic spends time teaching a slower one, they both get behind on whatever is taught next.
In my experience, a collection of students where some are further ahead than others is a good thing, because those who are ahead are better equipped to help those who need it. In fact, before learning was industrialized into an assembly line system (and let me point out that it got significantly worse when this happened), heterogeneous classrooms were the norm, and it was not unusual for more advanced (and often older) students to help less advanced ones. In my mind, this is what teaching each other in a learning environment is all about, and as I mentioned, my experience supports this.
Artificially controlling the pace of learning comes with a plethora of other problems. It destroys motivation. It is hard to stay motivated to learn something when it is dispensed at a slow rate over a long period of time. The best time to teach someone something is when they want to learn it, and that is generally right after they have learned what comes directly before it. Some people can handle the "weekly 1 hour episode" model, but the reason things like Netflix and Hulu are becoming so much more popular than regular television is that people can binge watch. More and more data is showing that people prefer to watch an entire season of a show, one episode right after another, over the course of a few days to a week, and many people have cited this specifically as a reason for dumping cable TV in favor of these online media providers. This is a form of learning, though not as useful, as the learning is almost exclusively about fictional people in varying degrees of fictional settings. Imagine what would happen if we just let students binge learn. Interestingly, it turns out that natural learning favors binge learning over regulated learning. It is common knowledge where I teach that motivated students learn more between semesters than they do during semesters from their classes.
Regulating the pace of learning destroys motivation to learn, it makes learning frustrating, and the end result of that is often that students dislike or even hate learning. Further though, not only is regulating the pace of learning unnecessary, it is actually harmful to the goal of students teaching each other. In both of the courses I teach, students teaching each other happens the most and most effectively when students are not at the same place in the course content. Students who are significantly ahead are almost always better teachers than those who are about at the same level as the students needing help. Allowing students to work at their own pace allows them to stay motivated and help each other more effectively.
Withholding Feedback
I hope this sounds like it is as bad of an idea as I think it is. Withholding feedback from students is a terrible idea. A great way to make sure students cannot learn from their mistakes is to avoid telling them what those mistakes were. If the goal is to teach students, this is a horrible strategy. Just today, I had a discussion with a student on this very topic. She told me that her teacher would not provide test results aside from the final grade of the test. This meant that she would have no idea what questions she got wrong, which further means that she is unable to study and improve her understanding of those things that she is weak on. This is horribly stupid for a course that is supposed to teach students. Now, there is a justification for this. The justification is that this helps to prevent cheating. If test results were provided to the students, they could forward those results to other students to help them cheat on the test. In my opinion, the fact that this is even a concern is a major flaw in the assessment setup for the course. If the course design makes it seem like a good idea to withhold feedback, then the course is poorly designed, and it should not be taught until it is fixed.
When I first took physics, my professor had an excellent policy (which I have mentioned before). If a student scored higher on the comprehensive final than the composite grade for the course, then the grade for the final would be the course grade. Otherwise the composite grade of all assignments would be used. I was the first, and thus far only, student to ever benefit from this policy. The reason I was able to do this, despite years and hundreds of students failing to score better on their final, was that I took advantage of feedback on my tests. When I took the tests, I generally got a good idea of where I was weak, and looking at the test results afterwards (results being what I got right and wrong, not just the grade for the test) gave me an even better feel for where I was weak. Then, I studied those things. By the end of the semester, I had significantly strengthened my understanding in the things I had initially been weak in. The result was that, with a composite grade of B in the course, I was able to earn a solid A on the final, which was my grade for the course. According to a brother who took the same course from the same professor many years later, he now tells students that getting a higher grade on the final has only ever been done once, instead of never. Without good feedback, I could not have done that. Without being able to see where I made mistakes on the tests, I would have gotten a B in the course, because I would not have had the opportunity to learn the material as well as I did. Withholding feedback in the way that was described to me by the student mentioned earlier hinders effective learning!
Good feedback is essential to effective learning. Yes, I understand that teachers have to worry about cheating and making sure that assessments are effective and accurate. If this requires withholding feedback though, there is something wrong. Frankly, there are plenty of better alternatives. The easiest is to have a large pool of questions from which test questions are randomly chosen. This is a good enough strategy for the Ham radio license tests, which have a few hundred questions, where the questions and the answers are public knowledge, because trying to memorize 200 answers for a 30 question test is more work than actually learning the material. Another is to just come up with unique questions each time the class is taught (this is actually a good strategy for building a question pool to use once there are enough). Some kinds of questions are just especially hard to cheat, for example any kind of essay question. For math questions, adding a variable that is randomly changed each semester is an effective way of generating unique questions at very low expense. Alternatively, requiring students to show their work or describe the process they are using can be effective for math and other logical subjects. Personally, I prefer to eliminate tests entirely, and instead grade based on specific proficiencies as well as a demonstration of application. This won't work with every subject, but it will work for a lot.
Courses and assessments should be designed with feedback in mind. Even the final for a class that has one should be made so that it won't compromise security to provide students with the full results, once it is over. Students that fail may choose to retake the class, and knowing where they need to focus their study may be very valuable to them. If good feedback cannot be provided for a class or assessment without compromising the security of the course, then it is poorly designed and should not be used at all. The ability to provide feedback should be a quality metric in course design and teaching. If we cannot provide students with the knowledge required for them to learn and improve, then we have no business telling them that we can teach them.
Mandatory Group Work
I was once a student, and I was required to work in groups in some of my classes. I am something of an introvert, I am a very fast learner, and thus far, I have taken only a few classes where I did not already have a strong background in the subject being taught. (Welcome to an education system where the paper matters more than the actual learning.) In a few classes, the mandatory group work was legitimately necessary due to the subject matter. In most, however, it was an artificial construct based on the misapplication of school guidelines. The college I graduated from and now teach at has formal guidelines teachers are very highly encouraged to follow. One of those guidelines is that students should teach each other. This is based on very solid research indicating that when one person teaches something to another, the teacher will learn the subject better as well as the learner. In other words, when students teach each other, everyone benefits. In addition, this leaves more time for professors to spend helping students that need help specifically from the professor. In other words, the school's policy is backed by solid science.
Most classes I have taken with mandatory group work did not actually need the group work to be effective. In all, I recall only one where the group work was actually beneficial to me, and that was partially because there was more work involved than was reasonable for a single person to do, and the other person was a fast enough learner to catch up with me (the other person was a woman; I have not had a similar experience with male students). There were maybe two other instances where working as a group helped the other members of the group, but, ironically, I knew the material so well that teaching them did not actually benefit me significantly. All of the other instances of mandatory group work I was required to do for my college classes actually made things more difficult for everyone. In one case, where we got to choose our own topics, the other members of my group thought they understood the topic I suggested, and in the end, because they did not, one member of the group left in a fit a rage, because myself and the third member would not let him present incorrect information in the group presentation. The group ultimately fell apart, because neither of the other members understood the topic that I found interesting, but they did not realize that until it was too late. I could easily have gone solo on both the research paper and the presentation, but that was not allowed, due to a misapplication of the school's teaching policy.
The big problem with mandatory group work is that it is almost impossible to group people in a way that works well for everyone. Some people just work better on their own (I am one of these people). Some people need to work with people who have complimentary personalities. Some people work best when they are ahead of others in the group, and some work better with those who are ahead of them. And of course, some people work better with those close to the same level as them. This does not even account for differences in personality or those students who are lazy and freeload off of the rest of the group. The fact is, for most topics, group work is not the best arrangement for everyone. Yes, I understand that most work in the real world is group work, and I even point this out to my students when I encourage them to work in groups. Don't forget though, school is not the real world workplace, and it is not appropriate to treat it like it is. Students recognize this, which is why many complain about mandatory group work. When group work interferes with effective learning, it is a bad idea, no matter how common group work is in the workplace. School is not the place to learn to get along with others. To be totally frank, I think the workplace is a better place to learn this skill, for those who have not already learned it.
There are some places where group work is valuable or even necessary. Certain types of creative design benefit significantly from collaboration. I teach video game design, which is one of these types of creative design. The value of collaboration is so high that collaboration is an essential skill for good video game design. Even then though, I don't require group work. I strongly encourage it, and I require students to present their ideas and listen to feedback, even if they are not part of a group. This is not always a good solution, but it is a way of getting some of the benefits of groups without having formal groups. Another case I have seen where group work is appropriate is when the class size is too big to be reasonable to grade students individually for all of the work. Ideally, this should be a temporary situation, where additional sections will be added to distribute some of the grading burden. The last place where group work is appropriate is when a project is necessarily too large for a single student (interestingly, this is the reason group work is necessary in the real world as well).
The takeaway here is that group work is not necessary for students to teach each other, and it is often a hindrance to effective learning. When considering requiring students to work in groups, serious thought should be given to the value of doing this, as the cost to students is almost certainly going to be high. There are places where it is appropriate, but there are perhaps more where it is not. My personal solution is to encourage students to self organize into groups but allow them to work solo if they prefer. I find this works better on two levels. One is that when students self organize, they are more likely to form groups that work better. The other is that some of the students who work solo do so because they learn so much faster than everyone else. The benefit to this is that no one is holding them back, and they are not pulling anyone else in deeper than they can handle. I also find that these solo students are more likely to and more capable of helping people in other groups. Mandatory group work not only hinders effective learning, it also tends to hinder the ability of students to teach each other, because they feel partitioned.
Pace of Learning
It was recently suggested to me that allowing students to work at their own pace hinders their ability to teach each other. Unfortunately, this is an attitude I have seen a lot. Most classes I have taken or otherwise been involved with limit the dispensation of knowledge, in the attempt to enforce a particular pace of learning. The theory is, if everyone is at the same place, they will be more able to help each other. To me, this is totally counter-intuitive. It does not make any sense to me that a collection of people who all have the same level of knowledge would be able to effectively teach each other, because if they all know the same stuff, what is there to teach? There are, of course, some teaching opportunities, as some students catch on to the subject at hand faster and are able to help those who catch on slower, but in a traditional setting, there is not much time for this, because the class just keeps progressing. If a fast student for a topic spends time teaching a slower one, they both get behind on whatever is taught next.
In my experience, a collection of students where some are further ahead than others is a good thing, because those who are ahead are better equipped to help those who need it. In fact, before learning was industrialized into an assembly line system (and let me point out that it got significantly worse when this happened), heterogeneous classrooms were the norm, and it was not unusual for more advanced (and often older) students to help less advanced ones. In my mind, this is what teaching each other in a learning environment is all about, and as I mentioned, my experience supports this.
Artificially controlling the pace of learning comes with a plethora of other problems. It destroys motivation. It is hard to stay motivated to learn something when it is dispensed at a slow rate over a long period of time. The best time to teach someone something is when they want to learn it, and that is generally right after they have learned what comes directly before it. Some people can handle the "weekly 1 hour episode" model, but the reason things like Netflix and Hulu are becoming so much more popular than regular television is that people can binge watch. More and more data is showing that people prefer to watch an entire season of a show, one episode right after another, over the course of a few days to a week, and many people have cited this specifically as a reason for dumping cable TV in favor of these online media providers. This is a form of learning, though not as useful, as the learning is almost exclusively about fictional people in varying degrees of fictional settings. Imagine what would happen if we just let students binge learn. Interestingly, it turns out that natural learning favors binge learning over regulated learning. It is common knowledge where I teach that motivated students learn more between semesters than they do during semesters from their classes.
Regulating the pace of learning destroys motivation to learn, it makes learning frustrating, and the end result of that is often that students dislike or even hate learning. Further though, not only is regulating the pace of learning unnecessary, it is actually harmful to the goal of students teaching each other. In both of the courses I teach, students teaching each other happens the most and most effectively when students are not at the same place in the course content. Students who are significantly ahead are almost always better teachers than those who are about at the same level as the students needing help. Allowing students to work at their own pace allows them to stay motivated and help each other more effectively.
Withholding Feedback
I hope this sounds like it is as bad of an idea as I think it is. Withholding feedback from students is a terrible idea. A great way to make sure students cannot learn from their mistakes is to avoid telling them what those mistakes were. If the goal is to teach students, this is a horrible strategy. Just today, I had a discussion with a student on this very topic. She told me that her teacher would not provide test results aside from the final grade of the test. This meant that she would have no idea what questions she got wrong, which further means that she is unable to study and improve her understanding of those things that she is weak on. This is horribly stupid for a course that is supposed to teach students. Now, there is a justification for this. The justification is that this helps to prevent cheating. If test results were provided to the students, they could forward those results to other students to help them cheat on the test. In my opinion, the fact that this is even a concern is a major flaw in the assessment setup for the course. If the course design makes it seem like a good idea to withhold feedback, then the course is poorly designed, and it should not be taught until it is fixed.
When I first took physics, my professor had an excellent policy (which I have mentioned before). If a student scored higher on the comprehensive final than the composite grade for the course, then the grade for the final would be the course grade. Otherwise the composite grade of all assignments would be used. I was the first, and thus far only, student to ever benefit from this policy. The reason I was able to do this, despite years and hundreds of students failing to score better on their final, was that I took advantage of feedback on my tests. When I took the tests, I generally got a good idea of where I was weak, and looking at the test results afterwards (results being what I got right and wrong, not just the grade for the test) gave me an even better feel for where I was weak. Then, I studied those things. By the end of the semester, I had significantly strengthened my understanding in the things I had initially been weak in. The result was that, with a composite grade of B in the course, I was able to earn a solid A on the final, which was my grade for the course. According to a brother who took the same course from the same professor many years later, he now tells students that getting a higher grade on the final has only ever been done once, instead of never. Without good feedback, I could not have done that. Without being able to see where I made mistakes on the tests, I would have gotten a B in the course, because I would not have had the opportunity to learn the material as well as I did. Withholding feedback in the way that was described to me by the student mentioned earlier hinders effective learning!
Good feedback is essential to effective learning. Yes, I understand that teachers have to worry about cheating and making sure that assessments are effective and accurate. If this requires withholding feedback though, there is something wrong. Frankly, there are plenty of better alternatives. The easiest is to have a large pool of questions from which test questions are randomly chosen. This is a good enough strategy for the Ham radio license tests, which have a few hundred questions, where the questions and the answers are public knowledge, because trying to memorize 200 answers for a 30 question test is more work than actually learning the material. Another is to just come up with unique questions each time the class is taught (this is actually a good strategy for building a question pool to use once there are enough). Some kinds of questions are just especially hard to cheat, for example any kind of essay question. For math questions, adding a variable that is randomly changed each semester is an effective way of generating unique questions at very low expense. Alternatively, requiring students to show their work or describe the process they are using can be effective for math and other logical subjects. Personally, I prefer to eliminate tests entirely, and instead grade based on specific proficiencies as well as a demonstration of application. This won't work with every subject, but it will work for a lot.
Courses and assessments should be designed with feedback in mind. Even the final for a class that has one should be made so that it won't compromise security to provide students with the full results, once it is over. Students that fail may choose to retake the class, and knowing where they need to focus their study may be very valuable to them. If good feedback cannot be provided for a class or assessment without compromising the security of the course, then it is poorly designed and should not be used at all. The ability to provide feedback should be a quality metric in course design and teaching. If we cannot provide students with the knowledge required for them to learn and improve, then we have no business telling them that we can teach them.
Wednesday, January 11, 2017
Proficiency Based Grading
Sometimes also called Competency Based Grading, Proficiency Based Grading strives to grade students exclusively on how proficient they are in the subject. I started teaching college level computer science courses in Fall 2015. I designed my own courses using many of the techniques I have written about in the past. With over a year of experience (a total of 5 semester long classes taught), I now want to discuss how I taught and graded, the outcomes of my classes, and what I have learned.
From the beginning, I was determined to use a proficiency based grading approach. My first course was Video Game Design. The course focuses on the design aspect of games, but it also must spend some time on the development aspect of video games, since the students are expected to actually produce one. I began the course design by enumerating topics I expected students to learn. The course is an introductory course, so the topics were broad, and I designed the course to provide and assess exposure rather than deep understanding. Each topic could easily be a course by itself. The list was composed of five broad topics, one on general game design (including tabletop games), a couple on video game design and development, and a couple on the value and creation of games beyond mere entertainment. Each of these five topics had two to four subtopics. These subtopics are what I expect students to know about. Of course, any video game course should also include actually making a video game, for the students to show that they can apply the knowledge. I also included a simple table top game project, for learning basic game mechanics design. The first iteration of the course required students to write a short essay or give a presentation on each of the various topics, including some research. In addition to this, each student was expected to design and develop a video game, and groups of up to 4 students was encouraged. About a third of the class time was spent on lectures, another third was spent on ungraded in-class group assignments, and the last third was workdays for students to work on projects. This worked fairly well, but neither myself nor the students liked the time spent on lectures much.
The next few semesters saw significant changes to this course. I reduced the number of lectures, and then I removed them from the classroom entirely. I wrote a series of articles, an informal textbook of sorts, and posted them on my technical blog. Then I provided students with links to the articles and a recommended reading schedule. This freed up an enormous amount of class time, which I filled with in-class coding demonstrations. This became most of the development aspect of the course. Near the end of the first semester doing this, I discussed it with some of my students, and they told me it really helped them. The coding demonstrations were unscripted, which means that I made mistakes and had to find and fix them as the students watched. This helped them learn two incredibly valuable lessons. One is that even seasoned programmers make mistakes. The other was the processes used by seasoned programmers to fix mistakes. In the typical scripted classroom style, students don't see these things, and it makes them underestimate their abilities and struggle with one of the most common activities in software development (debugging). My students expressed their appreciation for exposing them to a real software development process.
The current state of the class, as taught last semester, still needs some work, but it has made some wonderful progress. Instead of assigning point values to each individual assignment, I assign grade percentages to the major topics. The application part of the assessment has been broken up into its own five sections. The grading rubric is divided into two parts: knowledge and application. Each part has five sections, and each section represents one proficiency and 10% of the overall grade for the course. The five knowledge sections each have the same subtopics they had in the original course design. The two sections that have four subtopics only require three to be checked off to get credit (exposure, not depth). The application section is largely composed of things students should be able to demonstrate in the two projects. This fits on one sheet of paper, and each student is given one when the semester starts. This way, students know exactly what they need to learn up front, and with each of the topics discussed in my technical blog, the resources are all readily available. Since the knowledge proficiencies are supposed to be broad and shallow, I encourage students to pass off multiples at a time. Students may choose to pass them off in essays, with up to three topics (which do not have to be from the same sections), or in presentations with up to two topics. The rule here is, one paragraph discussing each topic and one discussing how they are all related. For presentations, students should fit them within 10 to 15 minutes, discuss each subject, and discuss their relationship. I have mentioned cost of failure as having a major influence in quality of learning. To reduce cost of failure, the only formal due date is a week before the end of the semester, to give me time for grading, and if a student submits an essay that does not sufficiently prove an understanding of something, I give the student some feedback and ask the student to answer some specific questions to prove that proficiency. There is no permanent failure in the course until final grades are posted. This means the grades students have at the end of the semester reflect all of the proficiency they have demonstrated to me up to that point. Some students do struggle with things earlier in the course, but if they figure them out by the end, and show me that they have, then their grade reflects that they understand those things.
As I said, some work is still needed. This last semester, I asked my students for their feedback. Several told me that they would have learned better if we had discussed some of the reading (originally the in-class lectures) in class. One student said he would have liked more in-class group activities, which I had reduced significantly that semester. So, next time around, I am going to have a more solid reading schedule, and we are going to start some classes with discussions on the readings. I am also going add back in some of the in-class group activities.
Quality course design requires a level of balance that can be difficult to achieve, and when I first designed this course, I did it with that in mind. In my experience, this is not something a lot of teachers do, which may be part of the problem with our education system. Of course, in my case, an essential element of getting the feedback I need to improve my course is a good relationship with my students. The college I teach at has had student evaluations for my courses pretty much every semester, but I don't get to see the results if too few students respond. I have never gotten enough responses to see the results, but that does not bother me, because my relationship with my students helps encourage them to share their opinions with me frankly and in person. I expect to be making adjustments to this course after every semester I teach it, and I hope it gets better every time.
Last semester I added a new course. The course is ARM Assembly Programming. I won't go into details on the subject, but it is essentially programming on the lowest level that is reasonable for humans to do (one step away from machine language). In some ways it is simpler and in some ways it is more complex than programming in higher level languages. It is widely regarded as significantly harder than programming in higher level languages. It turns out that it is very suitable to a proficiency based approach. As with the game class, I started by enumerating what I expected students to learn from the course. Instead of using these as proficiency categories though, they were used only to determine what proficiencies should go into the class. This is the kind of class that, in the tech industry, an employer might look at specifically (for some context, in the tech and engineering industries, employers do look at college transcripts as well as outcomes for specific courses related to their particular work). I wrote up a list of proficiencies, and then I divided them into lettered sections, A through D. In the D section, I put proficiencies that an employer might reasonably expect a student who got a D in my class to have. In the C section, I put proficiencies that an employer might expect for a C grade, and so on. The proficiency checklist included 10 to 15 proficiencies per section, but most of them were fairly small. I expected students to mostly write programs that would each check off multiple proficiencies. From there, I had to determine what learning materials to provide my students with. Again, I looked at how things are done in industry. When an employee is asked to learn something new in this industry, it does not generally come with a manual or some nice little course. A manager says, "I need you to learn this language," and the employee is expected to figure out how to do it. So, I encouraged my students to find online resources on their own. I did provide a few myself though, including a nice tutorial that covered most of the D and C proficiency sections, and some reference materials. I also provided data sheets, like what might be provided on the off chance that an employer did provide anything. Students with questions were encouraged to ask Google first, and then ask their class mates. Only if these had failed, were they encouraged to ask me (in practice, I did not turn down requests for help though). The last part of the grading was a final project, which would hopefully be of some value as part of a portfolio.
I have only taught this course once so far, but it went very well. I gave a few lectures in class. The only things I spoon fed students were things that were required for their work but which were not integral parts of the topic of the course. I also did live coding demos, as my experience with the other class taught me that those are very valuable. Most classes started with a QA session, where students asked me questions, and we discussed the answers. Around half of the class periods for the course were workdays, since students were encouraged to learn primarily on their own in groups, instead of listening to me talk a lot. During workdays, students would also pass off proficiencies with me. I would sit down with them, with my master sheet with the proficiencies for the entire class, and they would show me their code, explain what it was doing, and then show me that it actually worked. I would check them off for any proficiency their code demonstrated. For a class of 18 students, this worked very well, though as the end of the semester approached, I had to start accepting email submissions due to a lack of time in class. Working with students one on one during this grading time also allowed me to give feedback and provide some advice and help when students failed to correctly demonstrate a proficiency. As with the video game class, the cost of failure was low. If a student failed to demonstrate a proficiency, the only consequence was not getting a check mark on the sheet at that time. The student could demonstrate it correctly later to get the check, with no penalty aside, perhaps, from lost time.
Using a proficiency checklist, instead of regular formal assignments was a significant benefit to students' ability to learn. Some students wrote one small program for every proficiency. Others devised larger assignments for themselves that would cover up to 10 proficiencies all in one program. Students tended to pick the strategy that worked the best for them. The ones that wrote one program per proficiency learned best by doing one thing at a time, in small pieces. The ones that devised larger assignments learned best by applying lots of things at once to a single larger project. Some students worked together on some things, while others went solo. All of this allowed students to learn the different things required at the pace that worked best for them, and aside from a few students that got way ahead and a few that struggled a bit more than the rest, everyone had about the same average pace, which frankly was pretty fast as far as the learning went.
The student feedback on this course was also good. One student who struggled for nearly all of the semester, barely making a passing grade during the last two weeks, told me afterwards that she really liked the design of the course. She said the low cost of failure reduced the stress that she had from struggling and getting behind, allowing her to continue to function and progress in the course even though she was struggling. In any other class, she would likely have given up halfway through the semester, because her grade would have already been established as failing. As it was, she was able to pass the course when another student helped her understand the things she was struggling with, by passing off a large number of proficiencies in the last two weeks of the semester. When the semester ended, she legitimately had a good enough grasp on the topic to pass the course, and she understood it well enough that an employer might want to hire her in a position that requires it. Using a traditional grading system, she would have failed the class, even though she was reasonably proficient by the end. With my grading system, she was able to get the grade that accurately reflected her proficiency at the end of the semester.
I have learned a number of interesting and valuable things from my experience thus far. One is that low cost of failure is better for encouraging students to learn than high cost of failure. High cost of failure might motivate them to do the work, but they will only do what is necessary, and if they get behind, they will give up and stop learning. The outcome of my assembly programming class taught me some things that I did not expect. Most of my students who passed the class went way beyond the scope of the class. One student learned to do networking in assembly, which is not trivial even in higher level languages. Several students learned how to manipulate text colors and position in the terminal, which is not horribly difficult, but which is also not trivial in assembly. A few wrote their own memory managers, entirely for the challenge, and again, this is not an easy task. Every student who got an A in the class went way beyond the scope of the class (this was not required for an A either). I believe the reason for this is that reducing the expectations of the course to a list of proficiencies allowed students to optimize their work, giving them more time to learn things that they found more interesting. Otherwise stated, the course had absolutely no busy work, which freed up time for other things. In addition, the low cost of failure reduced stress, which allowed students to feel more comfortable pursuing knowledge beyond the scope of the class. Lastly, allowing students to select their own final projects (with only a few requirements) let them use their imaginations, and they were able to come up with ideas that required them to learn more to implement.
I noticed an interesting thing in my courses this semester. The grades in my courses are not well distributed. In fact, in my experience, traditional course grades tend to take on something of a bell curve. Often the peak is around C or a low B. Fewer students get As and Fs. This does not happen in my courses. The middle range tends to be fairly empty, while most students get high Bs to As, a few get lower Bs and Cs, very few get Ds, and a few get Fs. I don't think my classes get more Ds and Fs than normal, but they tend to get more As and Bs, and fewer Cs. Now, perhaps this could be explained by the classes being too easy, except that I would expect to see fewer Fs and Ds. In fact, I would expect to see the same bell curve, just shifted upwards. I don't see that, and I can honestly tell you that my classes are not too easy. I am an extremely fast learner. Almost all of the courses I took in college were easy for me (including some that were extremely hard for others). If anything, my classes are harder than average, because I tend to underestimate how hard things are for others. One benefit I have is that my classes are not required, so every student taking one of my classes is doing it voluntarily, because he or she wants to take the class and learn the topic. This combined with my low stress grading system allows motivated students to learn quickly and get good grades. Students who struggle early but then figure it out are not getting penalized for struggling at first, which probably also increase the grade of motivated students. Overall, this teaching and grading style seems to provide motivated students with a better learning environment than more traditional styles.
In closing, I would like to point out a few other interesting things that my experiences support. First, live demonstrations are good. Teaching students theory and then expecting them to figure out how to apply it themselves in assessments is a common teaching method, but walking students through practical application a few times is extremely valuable in helping students learn. Second, when the focus of the class is learning instead of getting the answers right on the test, students actually learn. A course with 4 tests distributed throughout the semester, where test scores are permanent, only encourages students to study for the tests. When evaluation is about learning instead of memorizing some things temporarily, students seem to learn far better. Third, taking the lectures out of the classroom, and putting the homework in the classroom really works. I got this idea from Salman Khan of Khan Academy, when he worked with an Arizona school, using the Khan Academy videos for homework and doing the actual course work in class. When I moved the initial learning out of class and the coursework into the classroom, my students progressed much faster, because they were able to get help when they needed it (while doing the work), instead of having to wait for the next class period.
This has been a great experience for me. I won't be teaching my own course this semester, but I will be working with another professor, so perhaps I will learn some things I might have missed. This will also give me a bit more experience with courses where students may not be as invested, as his courses are required for the major. I do plan on teaching more of my own courses in the future though, and I will try to report back on what new things I learn as I do this.
From the beginning, I was determined to use a proficiency based grading approach. My first course was Video Game Design. The course focuses on the design aspect of games, but it also must spend some time on the development aspect of video games, since the students are expected to actually produce one. I began the course design by enumerating topics I expected students to learn. The course is an introductory course, so the topics were broad, and I designed the course to provide and assess exposure rather than deep understanding. Each topic could easily be a course by itself. The list was composed of five broad topics, one on general game design (including tabletop games), a couple on video game design and development, and a couple on the value and creation of games beyond mere entertainment. Each of these five topics had two to four subtopics. These subtopics are what I expect students to know about. Of course, any video game course should also include actually making a video game, for the students to show that they can apply the knowledge. I also included a simple table top game project, for learning basic game mechanics design. The first iteration of the course required students to write a short essay or give a presentation on each of the various topics, including some research. In addition to this, each student was expected to design and develop a video game, and groups of up to 4 students was encouraged. About a third of the class time was spent on lectures, another third was spent on ungraded in-class group assignments, and the last third was workdays for students to work on projects. This worked fairly well, but neither myself nor the students liked the time spent on lectures much.
The next few semesters saw significant changes to this course. I reduced the number of lectures, and then I removed them from the classroom entirely. I wrote a series of articles, an informal textbook of sorts, and posted them on my technical blog. Then I provided students with links to the articles and a recommended reading schedule. This freed up an enormous amount of class time, which I filled with in-class coding demonstrations. This became most of the development aspect of the course. Near the end of the first semester doing this, I discussed it with some of my students, and they told me it really helped them. The coding demonstrations were unscripted, which means that I made mistakes and had to find and fix them as the students watched. This helped them learn two incredibly valuable lessons. One is that even seasoned programmers make mistakes. The other was the processes used by seasoned programmers to fix mistakes. In the typical scripted classroom style, students don't see these things, and it makes them underestimate their abilities and struggle with one of the most common activities in software development (debugging). My students expressed their appreciation for exposing them to a real software development process.
The current state of the class, as taught last semester, still needs some work, but it has made some wonderful progress. Instead of assigning point values to each individual assignment, I assign grade percentages to the major topics. The application part of the assessment has been broken up into its own five sections. The grading rubric is divided into two parts: knowledge and application. Each part has five sections, and each section represents one proficiency and 10% of the overall grade for the course. The five knowledge sections each have the same subtopics they had in the original course design. The two sections that have four subtopics only require three to be checked off to get credit (exposure, not depth). The application section is largely composed of things students should be able to demonstrate in the two projects. This fits on one sheet of paper, and each student is given one when the semester starts. This way, students know exactly what they need to learn up front, and with each of the topics discussed in my technical blog, the resources are all readily available. Since the knowledge proficiencies are supposed to be broad and shallow, I encourage students to pass off multiples at a time. Students may choose to pass them off in essays, with up to three topics (which do not have to be from the same sections), or in presentations with up to two topics. The rule here is, one paragraph discussing each topic and one discussing how they are all related. For presentations, students should fit them within 10 to 15 minutes, discuss each subject, and discuss their relationship. I have mentioned cost of failure as having a major influence in quality of learning. To reduce cost of failure, the only formal due date is a week before the end of the semester, to give me time for grading, and if a student submits an essay that does not sufficiently prove an understanding of something, I give the student some feedback and ask the student to answer some specific questions to prove that proficiency. There is no permanent failure in the course until final grades are posted. This means the grades students have at the end of the semester reflect all of the proficiency they have demonstrated to me up to that point. Some students do struggle with things earlier in the course, but if they figure them out by the end, and show me that they have, then their grade reflects that they understand those things.
As I said, some work is still needed. This last semester, I asked my students for their feedback. Several told me that they would have learned better if we had discussed some of the reading (originally the in-class lectures) in class. One student said he would have liked more in-class group activities, which I had reduced significantly that semester. So, next time around, I am going to have a more solid reading schedule, and we are going to start some classes with discussions on the readings. I am also going add back in some of the in-class group activities.
Quality course design requires a level of balance that can be difficult to achieve, and when I first designed this course, I did it with that in mind. In my experience, this is not something a lot of teachers do, which may be part of the problem with our education system. Of course, in my case, an essential element of getting the feedback I need to improve my course is a good relationship with my students. The college I teach at has had student evaluations for my courses pretty much every semester, but I don't get to see the results if too few students respond. I have never gotten enough responses to see the results, but that does not bother me, because my relationship with my students helps encourage them to share their opinions with me frankly and in person. I expect to be making adjustments to this course after every semester I teach it, and I hope it gets better every time.
Last semester I added a new course. The course is ARM Assembly Programming. I won't go into details on the subject, but it is essentially programming on the lowest level that is reasonable for humans to do (one step away from machine language). In some ways it is simpler and in some ways it is more complex than programming in higher level languages. It is widely regarded as significantly harder than programming in higher level languages. It turns out that it is very suitable to a proficiency based approach. As with the game class, I started by enumerating what I expected students to learn from the course. Instead of using these as proficiency categories though, they were used only to determine what proficiencies should go into the class. This is the kind of class that, in the tech industry, an employer might look at specifically (for some context, in the tech and engineering industries, employers do look at college transcripts as well as outcomes for specific courses related to their particular work). I wrote up a list of proficiencies, and then I divided them into lettered sections, A through D. In the D section, I put proficiencies that an employer might reasonably expect a student who got a D in my class to have. In the C section, I put proficiencies that an employer might expect for a C grade, and so on. The proficiency checklist included 10 to 15 proficiencies per section, but most of them were fairly small. I expected students to mostly write programs that would each check off multiple proficiencies. From there, I had to determine what learning materials to provide my students with. Again, I looked at how things are done in industry. When an employee is asked to learn something new in this industry, it does not generally come with a manual or some nice little course. A manager says, "I need you to learn this language," and the employee is expected to figure out how to do it. So, I encouraged my students to find online resources on their own. I did provide a few myself though, including a nice tutorial that covered most of the D and C proficiency sections, and some reference materials. I also provided data sheets, like what might be provided on the off chance that an employer did provide anything. Students with questions were encouraged to ask Google first, and then ask their class mates. Only if these had failed, were they encouraged to ask me (in practice, I did not turn down requests for help though). The last part of the grading was a final project, which would hopefully be of some value as part of a portfolio.
I have only taught this course once so far, but it went very well. I gave a few lectures in class. The only things I spoon fed students were things that were required for their work but which were not integral parts of the topic of the course. I also did live coding demos, as my experience with the other class taught me that those are very valuable. Most classes started with a QA session, where students asked me questions, and we discussed the answers. Around half of the class periods for the course were workdays, since students were encouraged to learn primarily on their own in groups, instead of listening to me talk a lot. During workdays, students would also pass off proficiencies with me. I would sit down with them, with my master sheet with the proficiencies for the entire class, and they would show me their code, explain what it was doing, and then show me that it actually worked. I would check them off for any proficiency their code demonstrated. For a class of 18 students, this worked very well, though as the end of the semester approached, I had to start accepting email submissions due to a lack of time in class. Working with students one on one during this grading time also allowed me to give feedback and provide some advice and help when students failed to correctly demonstrate a proficiency. As with the video game class, the cost of failure was low. If a student failed to demonstrate a proficiency, the only consequence was not getting a check mark on the sheet at that time. The student could demonstrate it correctly later to get the check, with no penalty aside, perhaps, from lost time.
Using a proficiency checklist, instead of regular formal assignments was a significant benefit to students' ability to learn. Some students wrote one small program for every proficiency. Others devised larger assignments for themselves that would cover up to 10 proficiencies all in one program. Students tended to pick the strategy that worked the best for them. The ones that wrote one program per proficiency learned best by doing one thing at a time, in small pieces. The ones that devised larger assignments learned best by applying lots of things at once to a single larger project. Some students worked together on some things, while others went solo. All of this allowed students to learn the different things required at the pace that worked best for them, and aside from a few students that got way ahead and a few that struggled a bit more than the rest, everyone had about the same average pace, which frankly was pretty fast as far as the learning went.
The student feedback on this course was also good. One student who struggled for nearly all of the semester, barely making a passing grade during the last two weeks, told me afterwards that she really liked the design of the course. She said the low cost of failure reduced the stress that she had from struggling and getting behind, allowing her to continue to function and progress in the course even though she was struggling. In any other class, she would likely have given up halfway through the semester, because her grade would have already been established as failing. As it was, she was able to pass the course when another student helped her understand the things she was struggling with, by passing off a large number of proficiencies in the last two weeks of the semester. When the semester ended, she legitimately had a good enough grasp on the topic to pass the course, and she understood it well enough that an employer might want to hire her in a position that requires it. Using a traditional grading system, she would have failed the class, even though she was reasonably proficient by the end. With my grading system, she was able to get the grade that accurately reflected her proficiency at the end of the semester.
I have learned a number of interesting and valuable things from my experience thus far. One is that low cost of failure is better for encouraging students to learn than high cost of failure. High cost of failure might motivate them to do the work, but they will only do what is necessary, and if they get behind, they will give up and stop learning. The outcome of my assembly programming class taught me some things that I did not expect. Most of my students who passed the class went way beyond the scope of the class. One student learned to do networking in assembly, which is not trivial even in higher level languages. Several students learned how to manipulate text colors and position in the terminal, which is not horribly difficult, but which is also not trivial in assembly. A few wrote their own memory managers, entirely for the challenge, and again, this is not an easy task. Every student who got an A in the class went way beyond the scope of the class (this was not required for an A either). I believe the reason for this is that reducing the expectations of the course to a list of proficiencies allowed students to optimize their work, giving them more time to learn things that they found more interesting. Otherwise stated, the course had absolutely no busy work, which freed up time for other things. In addition, the low cost of failure reduced stress, which allowed students to feel more comfortable pursuing knowledge beyond the scope of the class. Lastly, allowing students to select their own final projects (with only a few requirements) let them use their imaginations, and they were able to come up with ideas that required them to learn more to implement.
I noticed an interesting thing in my courses this semester. The grades in my courses are not well distributed. In fact, in my experience, traditional course grades tend to take on something of a bell curve. Often the peak is around C or a low B. Fewer students get As and Fs. This does not happen in my courses. The middle range tends to be fairly empty, while most students get high Bs to As, a few get lower Bs and Cs, very few get Ds, and a few get Fs. I don't think my classes get more Ds and Fs than normal, but they tend to get more As and Bs, and fewer Cs. Now, perhaps this could be explained by the classes being too easy, except that I would expect to see fewer Fs and Ds. In fact, I would expect to see the same bell curve, just shifted upwards. I don't see that, and I can honestly tell you that my classes are not too easy. I am an extremely fast learner. Almost all of the courses I took in college were easy for me (including some that were extremely hard for others). If anything, my classes are harder than average, because I tend to underestimate how hard things are for others. One benefit I have is that my classes are not required, so every student taking one of my classes is doing it voluntarily, because he or she wants to take the class and learn the topic. This combined with my low stress grading system allows motivated students to learn quickly and get good grades. Students who struggle early but then figure it out are not getting penalized for struggling at first, which probably also increase the grade of motivated students. Overall, this teaching and grading style seems to provide motivated students with a better learning environment than more traditional styles.
In closing, I would like to point out a few other interesting things that my experiences support. First, live demonstrations are good. Teaching students theory and then expecting them to figure out how to apply it themselves in assessments is a common teaching method, but walking students through practical application a few times is extremely valuable in helping students learn. Second, when the focus of the class is learning instead of getting the answers right on the test, students actually learn. A course with 4 tests distributed throughout the semester, where test scores are permanent, only encourages students to study for the tests. When evaluation is about learning instead of memorizing some things temporarily, students seem to learn far better. Third, taking the lectures out of the classroom, and putting the homework in the classroom really works. I got this idea from Salman Khan of Khan Academy, when he worked with an Arizona school, using the Khan Academy videos for homework and doing the actual course work in class. When I moved the initial learning out of class and the coursework into the classroom, my students progressed much faster, because they were able to get help when they needed it (while doing the work), instead of having to wait for the next class period.
This has been a great experience for me. I won't be teaching my own course this semester, but I will be working with another professor, so perhaps I will learn some things I might have missed. This will also give me a bit more experience with courses where students may not be as invested, as his courses are required for the major. I do plan on teaching more of my own courses in the future though, and I will try to report back on what new things I learn as I do this.
Subscribe to:
Posts (Atom)