Observation Based Pedagogy

I am beginning to learn more about the state of research and progress in education, and it is disappointing me more and more.  A few weeks ago, I refreshed my knowledge of the Montessori and Waldorf methods.  Now, I have complained about lack of empirical research in education pedagogy, but I am starting to see a common trend that seems to be hiding this problem within education communities, and that is the role of personal experience in the development of teaching methods.

The Montessori method was developed by Maria Montessori over 100 years ago.  According to resources on the topic, she developed the system through observation.  This is often treated as a claim that the Montessori method is based on scientific study, however, this is completely false.  Observation is used in scientific studies, but it is universally considered a poor substitute for controlled studies.  Montessori's observational approach to pedagogical development was even less though.  All teachers learn something about what works and what does not, through personal experience, which can also be called "observation".  Watching other educators teach (and the resources suggest Montessori did, in developing her system) can also reveal valuable things.  What observation fails to do, however, is account for uncontrolled factors and exhaustively test the options.  Another way of putting this is that teachers learn a lot about what works and what does not, by paying attention to their own experience and by observing the experiences of others, however, this is not in any sense rigorous science.  It cannot be used to produce a quality, evidence based learning system.  In the case on Montessori, she observes that certain teaching techniques were incredibly success, but personal bias led her to miss all of the students these techniques were not working for.  As a result, the Montessori system she developed based on her observations works incredibly well but only for a moderate percentage of students.  Modern Montessori schools boast impressive results, but that is actually merely survivor bias.  Students who do not thrive in Montessori schools are typically removed from those schools by their parents, thus Montessori schools rarely have students who perform poorly.  This does not mean Montessori schools have no value.  It merely means that the Montessori method is not a very broadly applicable method.  It is certainly not the "best" method, by any stretch.  Using Montessori broadly in public schools would result in a much higher rate of failure.  As private schools or charter schools, where parents can decide whether or not to keep kids in them based on performance, Montessori schools are an excellent idea, because they work very well for many students.

This is not a unique problem.  It turns out nearly every pedagogy is based on either personal observation and experience or on personal ideals and beliefs.  There are no common pedagogies, at least not in the U.S., that are based on rigorous research and evidence based science.  The closest we have is milestone based learning, but it is hardly common, and it often focuses more heavily on evaluation and progression rate than actual teaching and learning.

The fact is, observation based decision making is a terrible way to handle something as important as learning.  Few teachers experiment with multiple methods at the same time, with sample sizes big enough to obtain quality data.  Even if the teaching method is changed every school term and the class sizes are 50 or more students, a lifetime of teaching is not sufficient to collect the data necessary to decide what method will perform best in general.  Students from wealthier families will likely respond differently to the same method as students from poor or middle class families.  Students from one region may respond differently from students in another.  Limited class sizes are not inherently bad for this, but for a single teacher, class size is sample size.  This does mean that in schools with smaller class sizes, the value of observation and experience in developing good teaching methods is even lower.

What we need is scientifically rigorous, empirical studies.  We need researchers who assign teachers to use certain methods.  We need thousands or tens of thousands of students learning using each method.  We need actual data to be collected, rather than just the impressions and ideas one or two individuals get from personal observations of very small samples of students.  For education to advance significantly we need real, scientifically rigorous research, and we need it in large quantities.