Put a tad sarcastically: would somebody please show me the research that says that the only way to RAISE test scores – and keep them rising – is via mind-numbing bad-teaching ‘test prep’? Would you please point me to any research that says that the best or only way to raise test scores is to teach worse?

I didn’t think so.

Look, accountability is a serious business. And the stakes have gotten much higher over the past 20 years. (Be careful what you wish for: in my day the lament was that nobody cared in the least about education and schools and it was mostly true.) But it is simply false to say that asking students to take tests is essentially wrong. The challenge is to offer kids a fabulous education; then, the test results take care of themselves, as they are supposed to.

A thought experiment. Hold your immediate disagreement; let’s try a simple thought experiment. Close your eyes and visualize the classrooms in the best schools and districts in your state – especially in those schools that are outliers in their demographic. What do you see? Do you see endless and grim test prep regimen, with horrible Gradgrind teachers? Or do you see far better teaching than is found in the low-performing schools, whose only arrow in the quiver is – more worksheets? You know the answer: in the best schools in America, private as well as public, we see more high-level questioning, more intriguing assignments, more constructed response tasks on local tests, and more higher-order instructional approaches than in low-performing schools. And we see high, not low, test scores. How could it be otherwise? That’s validity 101.

In fact, by definition in the best schools in America, local assessments are more rigorous than state tests. The most recent review of the NY State Regents exams only underscores what many of us have known (or should have known) for decades: state tests are not very hard, and there is going to be hell to pay when the new common assessments are rolled out (as NAEP scores have told us for decades.)

Can we consider common sense for a spell, then, without the posturing? The crude response of test prep is understandable in light of local fears, and the harm it does to students and teachers is real. But test prep exists not because there are tests but because educators in weak schools have utterly lost their way. Test prep, in other words, is a compensatory strategy by educators who don’t seem to know what high-quality teaching and learning actually look like, and/or they don’t have faith in the power of a great education to cause good test results.

What we should then be freaking out about is that so many educators still don’t seem to know or believe in ‘best practice’, have no access to it in their work, and are under no accountability pressures to get that expertise in their jobs. (My electrician and carpenter work to higher standards: they have to be re-certified every few years on current code and expert knowledge.) Alas, awareness and use of best practice is optional in almost every school in America. Would those of you who are quick to disagree with me here want your child to be taught by someone who prized their freedom to do whatever, over their obligation to find out and do what works best – when current approaches are not clearly working? Would you go to doctors who had such an attitude?

The tests themselves: nowhere near as bad as people seem to believe. I have written various pieces over the last few years on what I have learned from looking at released test items. I refer you to those pieces as a start, in hopes of having a more rational discussion of the proper role, strengths and weaknesses of external testing in each school. If you do what I did – look at all the released tests in those states that do so, such as Florida and Massachusetts – I think you’ll come away thinking pretty much what I did: most test questions in the core subjects of reading and math are fair and appropriate, given the standards. (I confess I am less enthusiastic about some tests in other subjects, but those haven’t been tied to NCLB and other high stakes in most places.)

Here’s the epiphany that I had after reviewing dozens of the most challenging questions (as judged by the patterns of scores): the questions that are most difficult ironically demand transfer of learning, not rote learning – the very aim we all prize. This becomes especially obvious in ELA: the student gets a brand-new reading (or writing prompt) and has to make sense of it, using whatever strategies and skills they have learned, without being told explicitly what to do or without hints from teachers. The questions that are difficult for our kids in reading are inevitably questions at the heart of instruction: ‘main idea’ and ‘author purpose’. (see gallery, below)

Why would teachers who had taught her kids to read well fear such a test? Why are local administrators so intellectually bankrupt in weak schools that all they can counsel is test prep, in light of what the tests actually demand? Why isn’t there a local plan to hammer away at the big ideas of reading across all grade levels, as a team effort, given that the results on these targets have been dismal for years?

Same in math: I didn’t find one question that I thought was unfair or pointless, given the Standards. The questions may not be brilliant, but after a while, they start to seem pretty straightforward and even predictable, and the poor student results start to seem really weird. For example, when there is a triangle and some missing info about sides and angles, you can almost bet before looking at the numbers and the figure that it is either about 180-degrees in a triangle or the Pythagorean Theorem.  In each case, this is core content. Yet once again, the results are surprisingly poor.

Here are many instructive examples in ELA and math:

The 64-million dollar question, then is: why are our low-performing kids not taking content we know they were taught and tested on and using it to address a test question that demands it – if you think about the question? The difficulty, in other words, is not in the content complexity but in the thinking demand: do the students know which content to apply, when -in a question that is not so dumbed-down as to basically tell you what to recall and use? That is the point of math: to see if, when faced with novel problems, students can solve them.

The weak results speak volumes to the weakness of instruction and (especially) assessment locally: in their classes kids get few real problems, just low-level ‘plug and chug’ as if superficial drill can bypass the need to think.

In over 30 years of work in schools I have rarely seen solid school-wide and district-wide tests. Rather, in typical local common assessments they often mimic the format of state tests without having the rigor of state tests. In the best schools, however, many individual teacher assessments are often creative, funny, challenging. I challenge you, therefore, to audit your local assessments using these audit materials. ELA blank audit assessment  Math blank audit assessment  Audit of Algebra I Test.  Included is my audit of a good district’s exam in Algebra; you may want to compare it to your own. (I will provide an audit of a fully-released state tests in math and ELA in my follow-up post.)

In that light, it’s worth remembering that Jaime Escalante had no interest in suing ETS or the College Board over his students’ scores on the AP being challenged because he felt that the external tests were key to the raising of the bar at Garfield HS (A point I made via sports in a past blog.) It is also worth remembering that once Escalante proved what was possible, AP scores from many other teachers at Garfield also climbed – even in non-math courses.

You don’t have to agree with me or like what I am saying. But please suspend disbelief until you have investigated all the released state test items, gone to the best schools in your state to see what they do, and (especially) audited local assessment. Then, see if you don’t feel as I do: that the problem is not testing and accountability per se but our unthinking response to testing and accountability in low-performing schools.

I am not criticizing the hard work or intentions of teachers in test-prep schools. I am criticizing the fact that though the methods are ineffective and doomed to be so by simple logic, test prep is all they keep doing. it’s wrong for kids and it’s a thoughtless response to the challenge. Someone locally needs to say that the test-prep emperor has no clothes.

Nor am I saying that good scores = good schools; that reverses what I said. I said that in really good schools it happens that test scores tend to be high – as they should be, if the tests are valid. Nor am I saying that any school with only high SES is a good school. There are countless schools in the suburbs that are not very good at all, i.e. they provide little or no value added, and the work is incredibly dreary – kids only persist out of extrinsic motivation. Such districts receive able kids, they graduate able kids, and not much vital learning happens in between.

Good schools and teachers, and our obligation to learn from them. What, then, do I mean by ‘good schools’ that are about something more than test prep, even in the face of tests? Nothing fancy. I mean the obvious: schools that any of us would want our kids to go to. Highly-qualified staff; a caring environment; where they really know your kid, and play to his/her interests/strengths; where they use endlessly engaging ways of getting kids hooked on a subject; where there is a challenging, yet stimulating curriculum, with worthy assignments; where they demand much of students but give much in the way of support; and whose graduates, regardless of GPAs, go on to make it in the world at something they care about – without being shocked and dismayed to learn that school standards were so low that they are unprepared for anything beyond school. In short, staff are mindful of the tests but not in a panic about them.

Many of you might not so much disagree with me as be disappointed that in this political war we find ourselves in that I would provide aid and comfort to the crazies who would destroy public education as we know it.

But I know this deep in my bones: if we who care about public education keep avoiding reasoned challenges to our beliefs from friends, if we keep dragging our heels on reform and accountability, and if we keep falling back on ad hominem attacks against everyone who disagrees with us, then we are no better than our enemies. And we disgrace ourselves as educators who, more than any other group in society, have the obligation to keep learning and questioning – including the questioning of thoughtless or demonstrably ineffective approaches such as ‘coverage’, and ‘test prep’.

PS: A week later this piece in Edutopia proved my point beautifully.

In contrast to common practice, in which teachers explain reading passages and supply background information before students read, “close reading” confines initial study to the text itself. Students make sense of it by probing its words and structure for information and evidence. Through questions and class exercises, teachers guide students back through the reading in a hunt for answers and deeper understanding.

Gathered for a leadership-network meeting facilitated by the Aspen Institute, the chief academic officers of the 14 participating districts expressed praise for the approach, but deep concerns as well, about providing the type of professional development necessary to deliver it well in their districts. To preserve the frank, problem-sharing nature of the meeting, the Aspen Institute asked that Education Week not quote district leaders by name.

“I’m really worried that we haven’t prepared our teachers for this,” one chief academic officer said. “The academic and cognitive demand [on teachers] is quite high.”

Moving teachers toward this way of working will require “some significant professional development” as they learn to refrain from providing quick answers, figure out instead how to formulate new kinds of questions that take them and their students back to the text repeatedly in their search for understanding.

None of the chief academic officers at the Aspen meeting criticized “close reading” as a goal, and most lauded it. But they saw a rocky road ahead in reaching it… How would teachers respond to a “sea change” that reframes their role from provider of information to facilitator of a group inquiry? And where would they get deep, focused lessons and units for such instruction?

“The percentage of my teachers who weren’t ever taught some of the skills you’re talking about here, like the ‘pivot point’ in a paragraph,” said one official, her voice trailing off in a sigh. “The teachers themselves don’t know many of those concepts.”

For some of us, we can only say – huh? This is really not new. One need only hop over to Annapolis or Santa Fe and sit in on classes at St. John’s College, the Great Books college (I am an alumnus). This is the way St. John’s has been doing it for 80 years – close reading of the Great Books, from Plato to Freud; no in-class lecturing, just students trying to figure out the meaning of the text with very deft probing by teachers (called tutors at St. John’s, not ‘profess-ors’).

This approach long-ago spawned the Junior Great Books program (and Touchstones, developed by 2 St. John’s tutors) used widely. So-called Socratic Seminar emerged out of this, trumpeted first by Mortimer Adler in the Paideia Proposal 30 years ago (which led to a large network of schools). Many teachers – me in the day, my wife, and my daughter! – have taught this way. It also has a long history in prep schools, especially Exeter where it started as the so-called Harkness method, named thusly because a Mr Harkness gave beautiful tables to Exeter 100 years ago for students and teacher to sit around and discuss.

We used the Paidiea structure in the Coalition of Essential Schools to help people understand the Coalition mantra Student as Worker, Teacher as Coach. I routinely trained people in such teaching 25 years ago, and at many of our workshops we do a mock seminar with Plato’s Allegory of the Cave. UbD was an outgrowth of much of that CES work. And in UbD we explicitly distinguish between Meaning-making and Acquisition (as well as Transfer) as goals to highlight the difference between ‘teaching’ knowledge and skill and ‘facilitating’ meaning-making by students of texts, data, experiences.

So, this is hardly new stuff. What I think is really underscored by the article is that far too many folks in this field are unaware of ‘best practice’ – whether it be Socratic Seminar, Problem-Based Learning, Reciprocal teaching, UbD, or dozens of other important approaches linked to sophisticated and vital goals.

The scandal here is that many Assistant Supts. subject-area supervisors, and English Dept. Heads are ill-informed; they often don’t know this stuff first-hand or even that there are teachers right now teaching this way in their system – as outliers rather than models. So, leaders too often end up just supervising a safe adoption of textbooks or readings instead of forcing investigation and adoption of a best-practice-based local curriculum. Here, then, is a practical tip. In every curriculum, be it written or online in maps; or in individual units: there should be a column for validating, via footnotes, all instructional choices against best practice, in light of the kind of goal identified, to ensure that pedagogy matches desired outcomes. (In UbD the 3 distinct goals of transfer/meaning/acquisition have to be identified by every teacher-designer, and the essence of the design process is to force alignment of goals/assessments/instruction.)

I want to make clear, however, that my belief in this kind of pedagogy in no way sanctions an unthinking and excessive use of it in schools. There is a kind of naiveté permeating the Common Core support materials so far. (I have found David Coleman’s otherwise interesting videos on this approach very thin gruel in terms of how to actually be such a facilitator, not to mention move the change forward. He doesn’t even really model being such a facilitator; he just kind of talks you through a close read.)

We know from first-hand experience in doing model classes that when you have 7 different grade levels of reading ability in a class and a great deal of pent-up student boredom and intellectual laziness that this vital approach won’t work quickly; you can’t just plunk Socratic Seminar into conventional classrooms without hardship (hence, firm leadership). On the other hand, when my colleague Denise did a mock Seminar with 9th graders in a poor Louisiansa HS, the immediate reaction of kids was – this was way more interesting than typical class. And the Principal blurted out something that was unfortunate but revealing – wow! I had no ideas our kids could think like this!

That’s the point of academic leadership and professional development: we know it is the right thing to do, so let’s plan backward from it as a result, starting now, working to make the most seamless and happy transition possible.

PS: We stand ready, based on decades of direct use and training of teachers, to provide professional development assistance in this kind of pedagogy.

Now, of course, the naysayers are quick to say that you cannot measure creative thinking. This is silly: here is a rubric for doing so: Creative. We can and do measure anything: critical and creative thinking, wine quality, doctors, meals, athletic potential, etc.  (A plug, once again for You Can Measure Anything.) More to the point, we recognize creative thinking immediately when we see it – much more so, then, say “organization” in writing (which is a far more abstract idea that creative thinking) or “effective collaboration.”

In Bloom’s Taxonomy – designed to categorize and guide the design of measures – Synthesis was the level of thinking for such creativity, as Bloom makes clear in defining it:

Synthesis is here defined as the putting together of elements and parts so as to form a whole. This is a process of working with elements, parts, etc. and combining them in such a way as to constitute a pattern or structure not clearly there before. Generally this would involve a recombination of parts of previous experience with new material, reconstructed into a new and more or less well-integrated whole. This is the category in the cognitive domain which most clearly provides for creative behavior on the part of the learner…

One may view the product or performance as essentially a unique communication…Usually too he tries to communicate for one or more of the following purposes – to inform, to describe, to persuade, to impress, to entertain. Ultimately he wishes to achieve a given effect in some audience. Consequently, he uses a particular medium of expression… the product can be considered “unique” [in that it] does not represent a proposed set of operations to be carried out.

Educators sometimes say that they shy from assessing creative thought for fear of inhibiting students, but this is a grave error in my view, even if the fear should be honored as coming from a desire to help. (But ponder: why don’t they shy from assessing “effective communication” and “collaboration,” however?)

I once worked with a group of ELA teachers on student writing rubrics and portfolios, and when it came time to identify key criteria of story-writing the teachers were very reluctant to use an engaging-boring continuum because it seemed so wrong. But, I protested, don’t you easily recognize boring vs engaging and trite vs creative work when you read the stories? Oh, yes, they said. Isn’t that key to what a good story is about? Well, yes, they said. But it seems wrong to say that a piece is “boring” – even if it is. Why, I persisted? Should we deceive the learner into thinking that their writing is better than it is? Is it right to lie to them about such a basic issue of author purpose and desired result? We don’t have to say “boring” but we should certainly say if the readers were not engaged, shouldn’t we? They reluctantly agreed – and found that their students easily understood the difference between “engaging” and “not engaging” and accepted the assessment criterion as common sense. Oh, you mean you don’t want it to be dull and boring, said one kid? Uh, yes. Oh, we didn’t think that mattered in school writing, said a girl. Exactly.

Ditto and underscored for student oral presentations. I once saw a class at Portland HS in Maine where the student oral presentations were unbelievably good, across the board, with “average” kids. How did you do it, I asked the teacher? Simple, he said: there are only two criteria: Was it factually accurate? Did it keep everyone fully engaged the entire time? There were only two grades: A and F!! (He didn’t blindly average grades when calculating term grades, rest assured).

Note how this idea of “impact” flows right from Bloom’s quote and the whole idea of purpose and audience. I have written elsewhere about the importance of using such “impact” criteria in assessment, and I offer a few examples and tips here:

The point in any performance is to cause the appropriate effects in a performance, i.e. achieve the purpose of the performance. Yes, you get some points for content and process, but impact matters. If they didn’t laugh at your jokes or reflect on the cruelties of life suggested by your sad ironic story, then the performance was unsuccessful and you need to know it. (And you need to know when things do work and why because sometimes that is puzzling, too: we are not always the best judge of the positive impact and value of our own work.)

This idea of focusing on impact is actually key to student autonomy, reflected in self-assessment and self-adjustment. The more we focus on impact – did you achieve the goal of such a performance? – instead of such abstract things as “focus” and “organization” or such indicators such as “eye contact” in speaking (which should not be criteria that are mandatory but indicators of the more general and appropriate criterion of “engaging the audience”), the more students can practice, get feedback, and self-assess and self-adjust on their own. Which is surely far more important than being totally dependent upon teacher feedback that is squeamish.

So, it is vital when asking students to perform or produce a product that you are crystal-clear on the purpose of the task, and that you state the purpose (to make clear that the purpose is to cause an intrinsic effect, NOT please the teacher. That’s one value of our GRASPS acronym in UbD: when the student has clarity about the Goal of the task, their Role, the specific Audience, the specific Setting, the Performance particulars, and the Standards and criteria against which they will be judged, they can be far more effective – and creative! – than without such information. Here are some GRASPS worksheets for download. Wiggins MOD M – GRASPS and ROLES. Here is a whole handout from the Milwaukee Schools based on our work with GRASPS: GRASPS K-12 Writing – Milwaukee.

A noteworthy aside: I was looking at student feedback from our surveys and ran across an interesting pattern of dislikes: rubrics that squash creativity. This is a worrisome misunderstanding: students are coming to believe that rubrics hamper their creativity rather than encouraging it. That can only come from a failure on the part of teachers to use the right criteria and multiple & varied exemplars. If rubrics are sending the message that a formulaic response on an uninteresting task is what performance assessment is all about, then we are subverting our mission as teachers.

PS: I contributed a chapter to an Routledge Press Handbook on creative thinking in which I focused on mathematics teaching (a prime offender in encouraging creative thinking in the subject despite the fact that this is what real mathematicians do all the time – create.) Alas, the book is expensive; maybe you can find it online or through your library.

You can get a free copy of the paper Transfer Goals Clarification Feb 2012 by clicking on the link.

I thought I would check out released test items. Here’s a 10th grade item from the PSSA in Pennsylvania:

Oops! Only 40% got it right. And there are many others like this. Here’s one from NAEP:

Results? Grim:

Now, I am sure there are readers who will sigh and say that these items are not sufficiently interesting/relevant/real-world, etc. and I will agree. But that’s not the point.

The point is that even after a year or two of algebra MOST students cannot use the distributive, (and often the associative, and commutative) properties properly. And that’s a problem with the INSTRUCTION, not the kids. Because the misconceptions are predictable; because it takes a lot of iterations to overcome what is counter-intuitive about much of higher-level math, you have to keep probing for this understanding – as the Exeter – and the Harvard Physics  – example so clearly showed us. But because conventional textbook coverage is so fractured, unfocused, superficial, and unprioritized, there is no guarantee that most students will come out knowing the essential concepts of algebra.

Don’t you math teachers get that there is a problem? The ‘yield’ from your ‘coverage’ is terrible. So, clearly, ‘coverage’ is not the key to optimal performance on tests. Some day we’ll know why so many math (and history and science…) teachers think coverage is optimal preparation for tests.

PS: The NAEP Question Database is here.