Sometimes also called Competency Based Grading, Proficiency Based Grading strives to grade students exclusively on how proficient they are in the subject. I started teaching college level computer science courses in Fall 2015. I designed my own courses using many of the techniques I have written about in the past. With over a year of experience (a total of 5 semester long classes taught), I now want to discuss how I taught and graded, the outcomes of my classes, and what I have learned.
From the beginning, I was determined to use a proficiency based grading approach. My first course was Video Game Design. The course focuses on the design aspect of games, but it also must spend some time on the development aspect of video games, since the students are expected to actually produce one. I began the course design by enumerating topics I expected students to learn. The course is an introductory course, so the topics were broad, and I designed the course to provide and assess exposure rather than deep understanding. Each topic could easily be a course by itself. The list was composed of five broad topics, one on general game design (including tabletop games), a couple on video game design and development, and a couple on the value and creation of games beyond mere entertainment. Each of these five topics had two to four subtopics. These subtopics are what I expect students to know about. Of course, any video game course should also include actually making a video game, for the students to show that they can apply the knowledge. I also included a simple table top game project, for learning basic game mechanics design. The first iteration of the course required students to write a short essay or give a presentation on each of the various topics, including some research. In addition to this, each student was expected to design and develop a video game, and groups of up to 4 students was encouraged. About a third of the class time was spent on lectures, another third was spent on ungraded in-class group assignments, and the last third was workdays for students to work on projects. This worked fairly well, but neither myself nor the students liked the time spent on lectures much.
The next few semesters saw significant changes to this course. I reduced the number of lectures, and then I removed them from the classroom entirely. I wrote a series of articles, an informal textbook of sorts, and posted them on my technical blog. Then I provided students with links to the articles and a recommended reading schedule. This freed up an enormous amount of class time, which I filled with in-class coding demonstrations. This became most of the development aspect of the course. Near the end of the first semester doing this, I discussed it with some of my students, and they told me it really helped them. The coding demonstrations were unscripted, which means that I made mistakes and had to find and fix them as the students watched. This helped them learn two incredibly valuable lessons. One is that even seasoned programmers make mistakes. The other was the processes used by seasoned programmers to fix mistakes. In the typical scripted classroom style, students don't see these things, and it makes them underestimate their abilities and struggle with one of the most common activities in software development (debugging). My students expressed their appreciation for exposing them to a real software development process.
The current state of the class, as taught last semester, still needs some work, but it has made some wonderful progress. Instead of assigning point values to each individual assignment, I assign grade percentages to the major topics. The application part of the assessment has been broken up into its own five sections. The grading rubric is divided into two parts: knowledge and application. Each part has five sections, and each section represents one proficiency and 10% of the overall grade for the course. The five knowledge sections each have the same subtopics they had in the original course design. The two sections that have four subtopics only require three to be checked off to get credit (exposure, not depth). The application section is largely composed of things students should be able to demonstrate in the two projects. This fits on one sheet of paper, and each student is given one when the semester starts. This way, students know exactly what they need to learn up front, and with each of the topics discussed in my technical blog, the resources are all readily available. Since the knowledge proficiencies are supposed to be broad and shallow, I encourage students to pass off multiples at a time. Students may choose to pass them off in essays, with up to three topics (which do not have to be from the same sections), or in presentations with up to two topics. The rule here is, one paragraph discussing each topic and one discussing how they are all related. For presentations, students should fit them within 10 to 15 minutes, discuss each subject, and discuss their relationship. I have mentioned cost of failure as having a major influence in quality of learning. To reduce cost of failure, the only formal due date is a week before the end of the semester, to give me time for grading, and if a student submits an essay that does not sufficiently prove an understanding of something, I give the student some feedback and ask the student to answer some specific questions to prove that proficiency. There is no permanent failure in the course until final grades are posted. This means the grades students have at the end of the semester reflect all of the proficiency they have demonstrated to me up to that point. Some students do struggle with things earlier in the course, but if they figure them out by the end, and show me that they have, then their grade reflects that they understand those things.
As I said, some work is still needed. This last semester, I asked my students for their feedback. Several told me that they would have learned better if we had discussed some of the reading (originally the in-class lectures) in class. One student said he would have liked more in-class group activities, which I had reduced significantly that semester. So, next time around, I am going to have a more solid reading schedule, and we are going to start some classes with discussions on the readings. I am also going add back in some of the in-class group activities.
Quality course design requires a level of balance that can be difficult to achieve, and when I first designed this course, I did it with that in mind. In my experience, this is not something a lot of teachers do, which may be part of the problem with our education system. Of course, in my case, an essential element of getting the feedback I need to improve my course is a good relationship with my students. The college I teach at has had student evaluations for my courses pretty much every semester, but I don't get to see the results if too few students respond. I have never gotten enough responses to see the results, but that does not bother me, because my relationship with my students helps encourage them to share their opinions with me frankly and in person. I expect to be making adjustments to this course after every semester I teach it, and I hope it gets better every time.
Last semester I added a new course. The course is ARM Assembly Programming. I won't go into details on the subject, but it is essentially programming on the lowest level that is reasonable for humans to do (one step away from machine language). In some ways it is simpler and in some ways it is more complex than programming in higher level languages. It is widely regarded as significantly harder than programming in higher level languages. It turns out that it is very suitable to a proficiency based approach. As with the game class, I started by enumerating what I expected students to learn from the course. Instead of using these as proficiency categories though, they were used only to determine what proficiencies should go into the class. This is the kind of class that, in the tech industry, an employer might look at specifically (for some context, in the tech and engineering industries, employers do look at college transcripts as well as outcomes for specific courses related to their particular work). I wrote up a list of proficiencies, and then I divided them into lettered sections, A through D. In the D section, I put proficiencies that an employer might reasonably expect a student who got a D in my class to have. In the C section, I put proficiencies that an employer might expect for a C grade, and so on. The proficiency checklist included 10 to 15 proficiencies per section, but most of them were fairly small. I expected students to mostly write programs that would each check off multiple proficiencies. From there, I had to determine what learning materials to provide my students with. Again, I looked at how things are done in industry. When an employee is asked to learn something new in this industry, it does not generally come with a manual or some nice little course. A manager says, "I need you to learn this language," and the employee is expected to figure out how to do it. So, I encouraged my students to find online resources on their own. I did provide a few myself though, including a nice tutorial that covered most of the D and C proficiency sections, and some reference materials. I also provided data sheets, like what might be provided on the off chance that an employer did provide anything. Students with questions were encouraged to ask Google first, and then ask their class mates. Only if these had failed, were they encouraged to ask me (in practice, I did not turn down requests for help though). The last part of the grading was a final project, which would hopefully be of some value as part of a portfolio.
I have only taught this course once so far, but it went very well. I gave a few lectures in class. The only things I spoon fed students were things that were required for their work but which were not integral parts of the topic of the course. I also did live coding demos, as my experience with the other class taught me that those are very valuable. Most classes started with a QA session, where students asked me questions, and we discussed the answers. Around half of the class periods for the course were workdays, since students were encouraged to learn primarily on their own in groups, instead of listening to me talk a lot. During workdays, students would also pass off proficiencies with me. I would sit down with them, with my master sheet with the proficiencies for the entire class, and they would show me their code, explain what it was doing, and then show me that it actually worked. I would check them off for any proficiency their code demonstrated. For a class of 18 students, this worked very well, though as the end of the semester approached, I had to start accepting email submissions due to a lack of time in class. Working with students one on one during this grading time also allowed me to give feedback and provide some advice and help when students failed to correctly demonstrate a proficiency. As with the video game class, the cost of failure was low. If a student failed to demonstrate a proficiency, the only consequence was not getting a check mark on the sheet at that time. The student could demonstrate it correctly later to get the check, with no penalty aside, perhaps, from lost time.
Using a proficiency checklist, instead of regular formal assignments was a significant benefit to students' ability to learn. Some students wrote one small program for every proficiency. Others devised larger assignments for themselves that would cover up to 10 proficiencies all in one program. Students tended to pick the strategy that worked the best for them. The ones that wrote one program per proficiency learned best by doing one thing at a time, in small pieces. The ones that devised larger assignments learned best by applying lots of things at once to a single larger project. Some students worked together on some things, while others went solo. All of this allowed students to learn the different things required at the pace that worked best for them, and aside from a few students that got way ahead and a few that struggled a bit more than the rest, everyone had about the same average pace, which frankly was pretty fast as far as the learning went.
The student feedback on this course was also good. One student who struggled for nearly all of the semester, barely making a passing grade during the last two weeks, told me afterwards that she really liked the design of the course. She said the low cost of failure reduced the stress that she had from struggling and getting behind, allowing her to continue to function and progress in the course even though she was struggling. In any other class, she would likely have given up halfway through the semester, because her grade would have already been established as failing. As it was, she was able to pass the course when another student helped her understand the things she was struggling with, by passing off a large number of proficiencies in the last two weeks of the semester. When the semester ended, she legitimately had a good enough grasp on the topic to pass the course, and she understood it well enough that an employer might want to hire her in a position that requires it. Using a traditional grading system, she would have failed the class, even though she was reasonably proficient by the end. With my grading system, she was able to get the grade that accurately reflected her proficiency at the end of the semester.
I have learned a number of interesting and valuable things from my experience thus far. One is that low cost of failure is better for encouraging students to learn than high cost of failure. High cost of failure might motivate them to do the work, but they will only do what is necessary, and if they get behind, they will give up and stop learning. The outcome of my assembly programming class taught me some things that I did not expect. Most of my students who passed the class went way beyond the scope of the class. One student learned to do networking in assembly, which is not trivial even in higher level languages. Several students learned how to manipulate text colors and position in the terminal, which is not horribly difficult, but which is also not trivial in assembly. A few wrote their own memory managers, entirely for the challenge, and again, this is not an easy task. Every student who got an A in the class went way beyond the scope of the class (this was not required for an A either). I believe the reason for this is that reducing the expectations of the course to a list of proficiencies allowed students to optimize their work, giving them more time to learn things that they found more interesting. Otherwise stated, the course had absolutely no busy work, which freed up time for other things. In addition, the low cost of failure reduced stress, which allowed students to feel more comfortable pursuing knowledge beyond the scope of the class. Lastly, allowing students to select their own final projects (with only a few requirements) let them use their imaginations, and they were able to come up with ideas that required them to learn more to implement.
I noticed an interesting thing in my courses this semester. The grades in my courses are not well distributed. In fact, in my experience, traditional course grades tend to take on something of a bell curve. Often the peak is around C or a low B. Fewer students get As and Fs. This does not happen in my courses. The middle range tends to be fairly empty, while most students get high Bs to As, a few get lower Bs and Cs, very few get Ds, and a few get Fs. I don't think my classes get more Ds and Fs than normal, but they tend to get more As and Bs, and fewer Cs. Now, perhaps this could be explained by the classes being too easy, except that I would expect to see fewer Fs and Ds. In fact, I would expect to see the same bell curve, just shifted upwards. I don't see that, and I can honestly tell you that my classes are not too easy. I am an extremely fast learner. Almost all of the courses I took in college were easy for me (including some that were extremely hard for others). If anything, my classes are harder than average, because I tend to underestimate how hard things are for others. One benefit I have is that my classes are not required, so every student taking one of my classes is doing it voluntarily, because he or she wants to take the class and learn the topic. This combined with my low stress grading system allows motivated students to learn quickly and get good grades. Students who struggle early but then figure it out are not getting penalized for struggling at first, which probably also increase the grade of motivated students. Overall, this teaching and grading style seems to provide motivated students with a better learning environment than more traditional styles.
In closing, I would like to point out a few other interesting things that my experiences support. First, live demonstrations are good. Teaching students theory and then expecting them to figure out how to apply it themselves in assessments is a common teaching method, but walking students through practical application a few times is extremely valuable in helping students learn. Second, when the focus of the class is learning instead of getting the answers right on the test, students actually learn. A course with 4 tests distributed throughout the semester, where test scores are permanent, only encourages students to study for the tests. When evaluation is about learning instead of memorizing some things temporarily, students seem to learn far better. Third, taking the lectures out of the classroom, and putting the homework in the classroom really works. I got this idea from Salman Khan of Khan Academy, when he worked with an Arizona school, using the Khan Academy videos for homework and doing the actual course work in class. When I moved the initial learning out of class and the coursework into the classroom, my students progressed much faster, because they were able to get help when they needed it (while doing the work), instead of having to wait for the next class period.
This has been a great experience for me. I won't be teaching my own course this semester, but I will be working with another professor, so perhaps I will learn some things I might have missed. This will also give me a bit more experience with courses where students may not be as invested, as his courses are required for the major. I do plan on teaching more of my own courses in the future though, and I will try to report back on what new things I learn as I do this.
