"Thanks for the feedback!" NYSED to NYS Educators

Edited on November 23 to add the NYSAPE Common Core Survey. 

For years, likely since the first day the website went up, there has been a "Teacher Participation Opportunities" link on the New York State Education Department's Office of State Assessment (OSA) website.

Following the link leads to a series of options available to NY teachers to participate in a variety of test design and assessment writing activities. These activities typically require sub coverage and travel to Albany, a 9-hour round trip and an overnight stay for those in the Southern Tier. Some are "once and done" work in which the teachers go to Albany, engage in a particular task, get a nice thank you letter, and not know what will become of their work until the test is published or the scores released. Some are extended projects in which teachers return multiple times to Albany or continue the work back at home. The biggest challenges of this approach to getting teacher feedback: 
  •  teachers have to volunteer or be nominated,
  • SED can filter who they bring to do the work, and
  • the proceedings aren't public. 
This novel idea of involving NYS teachers in the design of the NYS tests and exams isn't new. Teachers in 1891 were asked their opinion on the exams.
At least as early as 1891, blanks for suggestions and criticisms "relative to the character and scope of the examinations" were shipped with each set of examination papers. These comments are tabulated and studied carefully.
So basically, teachers have been involved in the writing of the NYS tests and exams since pretty much the beginning. Opinions about if it's the *right* kind of feedback, if the *right* teachers are giving feedback, and what that feedback looks like in the modern area vary.

The feedback process around standards isn't nearly as long. The formal presence of standards didn't start until the 90's. Any NYS teacher of a certain age remembers the booklets with the 1996 standards, printed on really thin paper with different colored covers. Inside the front cover of each book was a list of the teachers who participated in their construction and anchoring. This is from the LOTE standards, the only ones that haven't been updated since 1996. 

When the time came to update the standards following the change in NYS law in 2007, Albany came to the field. In April 2008, I was at the Western NY forum and used this new thing called Twitter to share out what was happening. It's interesting to note that many of the things I tweeted, the things the teachers in the room were asking for, are a part of the Common Core design. But I digress. 

Shortly after the forums concluded, the committee wrote up their findings and began working on what are now called the "lost" standards by some advocates. I prefer the moniker the "paused" standards as NY stopped that work in order to be a part of a new initiative to create multi-state standards. "Common" standards, as it were. NYSED provides a timeline of those decisions here. Opinions about why New York made that decision, if the "paused" standards are better or worse than the CCSS, and what it means to have 50 states with 50 sets of standards area vary.

Which brings us to 2015 and NYS is again seeking out teacher feedback.
  1. Want to comment on each specific Common Core Learning Standard? Commissioner Elia wants to know if the standard is acceptable, if it should be moved, changed, or re-worded.
  2. Want to comment on the CCLS, tests, or APPR in general? Governor Cuomo and his task force are all ears. (It remains to be seen, though, how discrepancies between Elia's survey and Cuomo's task force will be resolved.)
  3. Want to comment on the latest draft of the Science standards? The Science department at NYSED will open a survey on December 2. Draft standards are available now.
  4. Want to comment on the proposed changes to the NYS Social Studies Regents? The look, design, and structure of the exams are open for feedback.
  5. Want to be a part of writing NYS tests, assessments, and exams? The offer from them still stands. (Be sure to check dates though, some have closed for now.)
  6. NYS Allies for Public Education (NYSAPE) created their own survey which touches upon testing, APPR, and the CCLS standards. It's unclear how these data will be used. 
In addition, updates from SED frequently appear on the agenda for events like Middle-Level Liaisons, DATAG,  Social Studies conferences, etc. It's a safe assumption that those SED personnel are talking to the teacher- and administrator-leaders of those organizations. So let it not be said NYSED in 2015 doesn't want your opinion. 

But as we know, that's only step 1. 

Lie back and think of England

There is a certain cognitive twitch that occurs only when one is writing a multiple choice question for poetry. It's a brain hiccup caused by the tension of doing something that shouldn't be done but has to be done.

Everyone in the room, myself included, knew the non-negotiables:
  • goal was to write a common assessment
  • it needed to include poetry
  • it had to generate quantitative data (ergo, scanning MC questions versus hand scoring written responses)
It simply wasn't a viable option at that moment to switch gears completely to curriculum-embedded, performance tasks like some districts had done. Things had been negotiated. Compromises had been reached. So, there we were. Trying to find a poem that was equal parts complexity and simplicity.

As most teachers do, this group found a way through and used student choice, several different poems, and a focus on the CCSS Language standards to make it an assessment that would generate useful information without causing too many brain cramps. 

On my travels to my next adventure, I kept re-living the day. Why didn't we push back harder? Why didn't I advocate more vocally for a better assessment when asked to support this group of teachers? Did we capitulate because as a room full of white women, we were socialized to follow the rules? We did what was asked. We met the mandate while doing our very best to ensure quality assessment. No one left feeling like we would be imposing something unethical or unfair on the students but at the same time... a multiple choice question about poetry. Did we do the right thing?

In a twitter exchange on the theme of the thinking behind certain mandates, Peter Greene tweeted to me, "So just lie back and think of England?" Which, first, no. And second... no. (No time to click the link? The phrase is wrongly attributed as advice from Queen Victoria to her daughter-in-law about producing an heir but has come to represent a trope that women need to suffer through sex for the greater good. Here is where every real nerd will repeat, "The Greater Good.")

It would seem there are three ways to deal with policy mandates with which we disagree.

  1. Refuse it. The Opt Out approach appears to be about changing policy by refusing to participate. It's not necessarily about finding a way through, it's about finding a way around. 
  2. Be excused from it. New Hampshire's approach to annual testing is asking permission to come at it from a totally different direction
  3. Find a way through it. Leveraging mandates to make the best of what's been asked.
I compulsively read everything I can on cognitive biases and how our brains are lazy by design. So I spend a lot of time while traveling trying to figure out the holes in my logic model. See - I'm okay with #3. I'm okay with schools looking at policy and saying, "Welp. This is silly. But, it's policy. How can we attend to this in a way that honors what we value and protects our students?" And then moving on. I don't see it as capitulation, but I suspect that's because I'm treating it as a narrow issue of assessment/curriculum design. 

Some authors like to compare the Opt Out movement to activities through history, especially during the Civil Rights era and each time I read one of those blog posts, I struggle against my instinct to reject them as hyperbole. In some cases, I've no problems connecting parts of a system (impact of cultural appropriation on the well-being of Native Americans) but here... (Opting Out of a state test as a gesture towards more equitable schools), I struggle. 

There is, I suspect, a great deal to be said about what it means to leverage mandates. It's a close cousin to "asking for forgiveness instead of permission" and lives in the narrow space between doing what is required and what is right. Is it a matter of changing of what we can? Or do I have a giant blind spot around the Opt Out movement? 

What exactly is "standardization" in assessment design?

I'm going to do my best to keep this really short and concise and write according to The Notorious RBG: 'Get it right, keep it tight.'"

Peter Greene made a claim that the correct number of standardized tests is zero.
I presented a counterclaim that standardization isn't the problem. 
Greene expanded on his claim to clarify his intentions around the tests. 

While reading Peter's updated claim, I realized that at no point was the phrase "standardized" actually defined. We both gave our opinions on what it means:

From Greene: 
"Standardized" when applied to a test can mean any or all (well, most) of the following: mass-produced, mass-administered, simultaneously mass-administered, objective, created by a third party, scored by a third party, reported to a third party, formative, summative, norm-referenced or criterion referenced.
From Me:
Welp, first, minus ten to me because I didn't state a definition, I asked questions that implied one. So to restate the intention of my questions. Standardized means doing the same thing for a group of students. The "thing" can be the nature of the task, the amount of time, the scoring criteria, or the directions to the students.

This is the quote from Peter that made me consult my bookcase and/or Google.
This broad palate of definitions means that conversations about standardized testing often run at cross-purposes. When Binis talks about the new performance assessment task piloting in NH, she thinks she's making a case for standardization, and I'm think that performance based assessment is pretty much the opposite of standardized testing.
I wasn't making a case for standardization, I was identifying an example in which a standardized process is used to develop a performance-based assessment. This may be switch-tracking (from The Hidden Brain Podcast - check it out. It's really cool!) by both of us but it remains that when we use the word or phrase, we've a different meaning in mind. So... to the Googles!

From Standards for Educational and Psychological Testing AKA "The Testing Standards" (This is basically the sourcebook for writing a quality measure of student learning), written by the AERA, APA, NCME, 2014
A test is a device or procedure in which a sample of an examinee's behavior in a specific domain is obtained and subsequently evaluated and scored by a standardized process. Tests differ on a number of dimensions... but in all cases, however, tests standardize the process by which test takers' responses to test materials are evaluated and scored. 
 According to the alpha and omega, a test by its very nature is standardized. Which makes the phrase "standardized test" redundant, it seems.

From the Code of Fair Testing Practices, which is a supplementary document for the Testing Standards.
The Code applies broadly to testing in education regardless of the mode of presentation, so it is relevant to conventional paper-and-pencil tests, computer-based tests, and performance tests.... Although the Code is not intended to cover tests prepared by teachers for use in their own classrooms, teachers are encouraged to use the guidelines to help improve their testing practices.
From Stanford's primer on performance-based assessments:
[Describing performance-based assessments] Teachers can get information and provide feedback to students as needed, something that traditional standardized tests cannot do.
.... in the early years of performance assessment in the United States, Vermont introduced a portfolio system in writing and mathematics that contained unique choices from each teacher’s class as well as some common pieces. Because of this variation, researchers found that teachers could not score the portfolios consistently enough to accurately compare schools. The key problem was the lack of standardization of the portfolios.
Here, the authors use standardized in two ways: first to refer to the multiple choice test we tend to picture when we hear "standardized test" and then to refer to the process of creating a uniform approach to scoring student writing samples.

From Handbook of Test Development, edited by Downing & Haladyna:
The test administration conditions - standard time limits, proctoring to ensure no irregularities, environmental conditions conducive to test taking, and so all - all seek to control extraneous variables in the experiment and make conditions uniform and identical for all examinees. Without adequate control of all relevant variables affecting test performance, it would be difficult to interpret examinee test scores uniformly and meaningfully. This is the essence of the validity issue for test administration.
Now, for the kicker. Why does any of this matter? Because of this - assessment literacy.  If you follow no other link from this post, please follow that one. Peter and I are reading the same book but we're not on the same page, as it were. He's a teacher, I'm out of the classroom, working with teachers around assessment design. This isn't an issue of "He's right and I'm wrong" or "I'm the expert, trust me." It's more compelling, instead, to consider the implications - and there are many of how we talk about testing and assessment. From teacher preparation, to academic writing, to communicating with parents and the public. I suspect, that until the profession agrees on a common glossary, we're going to keep nibbling at the edges.