Home » Posts tagged 'VAM Score'

Tag Archives: VAM Score


Where is our Flexibility?

Recently, John King, the New York State Education Commissioner, testified before a US Senate Committee about the reauthorization of the No Child Left Behind law (you can view his testimony through this link: http://www.scribd.com/doc/124387499/Esea-Testimony). In his testimony, Commissioner King urged the Senate panel to include a federal requirement for teacher and principal evaluation. “It would be helpful in the potential reauthorization to set a few clear, bright-line parameters, and then to give states flexibility to adapt those parameters to their context,” King testified.

What might those “parameters” be? Well…those of us who have followed the recent “education reform” efforts know these parameters to be veritable pillars of faith. Namely: 1) the inclusion of student performance in the evaluations of teachers and principals, and 2) the use of these performance evaluations when making employment or salary decisions. This is exactly what Commissioner King requested in his testimony, adding that states must be afforded flexibility to adapt these parameters to the specific needs of the state.

John King wants to make sure that the federal government provides him with the flexibility necessary to apply its mandates to the unique needs of New York State. As State Education Commissioner, John King’s fixed views of what works best in schools has prevented local districts any real flexibility in demonstrating high levels of student achievement and teacher competencies.

What’s good for the goose is not good for the gander, it seems.

Last month, local news reports were filled with descriptions of the failure of New York City teachers union (UFT) and the Bloomberg administration to reach an agreement regarding New York State’s APPR legislation. Lost in the battle of blame was the fact that no matter what local agreement was reached regarding the implementation of APPR, the State Education Department could ultimately order a “corrective action plan” if it did not like the results of a local district’s APPR plan. As described in the second paragraph of the cover letter of every district’s approved APPR plan (see http://usny.nysed.gov/rttt/teachers-leaders/plans/home.html), NYSED assures the district superintendent that such a corrective action plan will be required if an analysis of data led it to believe that there was:

“unacceptably low correlation results between the student growth subcomponent and any other measures of teacher and principal effectiveness and/or if the teacher or principal scores or ratings show little differentiation across educators and/or the lack of differentiation is not justified by equivalently consistent student achievement results”

Ah yes…those student growth scores! Despite the numerous studies and explanations showing their unreliability (see, for example, http://schoolfinance101.wordpress.com/2012/11/17/air-pollution-in-ny-state-comments-on-the-ny-state-teacherprincipal-rating-modelsreport/) these student scores trump the judgment of the professionals in the school. If a district’s evaluation of teachers do not correlate with the teacher score that NYS gives through the unreliable and erratic value added or student growth model, the problem must be with the district, which in turn must be forced to change its methods for evaluating teachers. No flexibility there!

My colleague, Carol Burris, described the first part of a training series she attended recently in order to prepare her for Calibration Day. Ostensibly, the purpose of the training was to remove all biases from the observation process in order to achieve the holy grail of a fully objectified observation process. Despite her years of experience supervising teachers and leading one of the country’s highest achieving schools, Carol had to spend the day being told how to conduct a proper observation free of any bias. Of course, the biases being removed were the ones identified by the Master Coder, who is an unidentified individual somewhere in Albany. There was no discussion at this training, no dialogue about the flexibility needed to teach and supervise different classes based on the numerous variables related to a class (e.g., student composition, time of day).

Gone are the days of dialogue in New York State; instead, we are repeatedly reminded that SED has the keys to what works in schools. We are told to simply follow along so that our students will find success. Are you wondering how to evaluate teachers and principals? Forget what years of practice and research tell us about what works. Instead, you must follow SED’s APPR guidelines. Forget what research tells us about developing a positive and collaborative culture  (http://www.ascd.org/publications/educational-leadership/oct09/vol67/num02/Creating-Collaborative-Cultures.aspx). Instead, expect to be told each year that a specific percentage of your faculty are ineffective and must be removed. The collective voices and experience of some of this country’s most effective educators are being ignored. Don’t think; follow.

Recently, New York State announced an exciting grant opportunity “to provide funds to support the dissemination of effective practices and programs that have been developed, tested, and proven successful” in schools (http://www.p12.nysed.gov/funding/currentapps.html#nycs_dissemination). With eighteen of the top hundred high schools in the country (as ranked by Newsweek http://www.thedailybeast.com/newsweek/2012/05/20/america-s-best-high-schools.html), it would seem that such a grant opportunity would be a wonderful opportunity for some of our high achieving schools to share their best practices. But wait! These grants are not for public schools to share their practices. Rather, the grant funds “are made available to assist charter schools in disseminating their successful innovations to any district school(s) in New York through designated partnerships.” Once again, in John King’s rigid view of what works in schools, Charter Schools are the answer; public schools have little to share regarding best practices. This is not what some of the most recent research related to Charter Schools tells us: http://blogs.edweek.org/edweek/inside-school-research/2013/02/charters_struggle_with_staffing_like_traditional_schools_study_finds.html. What message is being sent to the very people that our Commissioner is supposed to be leading?

Why does the Commissioner of Education refuse to engage in any real dialogue about successful practices in New York State schools? Why are the deep concerns of nearly 1600 New York State principals (see www.newyorkprincipals.org) dismissed as simply the anxieties of individuals faced with change (http://www.nytimes.com/2011/11/28/education/principals-protest-increased-use-of-test-scores-to-evaluate-educators.html?pagewanted=all)? Why is he asking for the very type of flexibility from the federal government that he refuses to accord districts in New York?


Meet Ashley, a great teacher with a bad ‘value-added’ score

Originally published on-line at the Washington Post’s AnswerSheet Blog

New York State schools are back in session! With the new school year comes a new responsibility for principals across the state: the need to inform teachers of their “growth score” based on the New York State assessments their students took in the spring. This teacher growth score is one of the parts of the New York State APPR system that was implemented last year in a rushed manner against the very public objection of over one-third of the New York State principals along with thousands of other teachers, administrators, parents and concerned citizens (see www.newyorkprincipals.org for more information).

These state-supplied scores were the missing piece in a teacher’s final end-of-year score — potentially determining whether or not a teacher is deemed Ineffective and therefore subject to requiring a Teacher Improvement Plan (TIP) within 10 days of the start of the school year. These scores were not available to schools until the third week of August. So there you have it: high-stakes information that can potentially have a serious impact on a teacher’s career being supplied well past any sort of reasonable timeframe. Welcome to New York’s APPR system!

As a principal, I sat with each of the teachers who received a score from the state and tried to explain how the state arrived at these scores out of 20 points. One of the first teachers with whom I did this was Ashley.

Ashley is the type of teacher that all parents want for their child: smart in her content area and committed to making a difference in her students’ lives. Ashley works incessantly with her students, both inside and outside of the classroom.

During her free time, Ashley can always be found working with small groups of students in the hallways or any free space in the area. She has taken our school’s math teams on weekend trips as our mathematics team has found success in various competitions. Over the past four years, 91% of her 179 Algebra 1, Geometry or Algebra 2/Trigonometry students have passed the corresponding Regents examination on their first attempt.

At the end of every year, students and parents send in countless notes of thanks to Ashley for her tireless efforts. Ashley has worked with our highest achieving students as well as many of those who struggle with mathematical understanding. For those who struggle, Ashley has a well-deserved reputation for making them more confident, successful and comfortable with the material. Last spring, Ashley was recognized as the Parent Teacher Organization teacher of the year.

So what score did the state assign Ashley? Well, she earned a score of 7 out of 20 points. According to the state’s guidelines, this makes Ashley a Developing teacher. Goodness. To those of us who know Ashley and have had the pleasure of working with her over the years, this is a jaw-dropping result. Ashley’s score defies all understanding of who she is as an educator. Her score flies in the face of how she is valued in our school and what she has done for students in our school. Her score contradicts the thoughtful evaluations given to her over the past five years.

How, then, is one to understand this score?

Officials at our State Education Department have certainly spent countless hours putting together guides explaining the scores. These documents describe what they call an objective teacher evaluation process that is based on student test scores, takes into account students’ prior performance, and arrives at a score that is able to measure teacher effectiveness. Along the way, the guides are careful to walk the reader through their explanations of Student Growth Percentiles (SGPs) and a teacher’s Mean Growth Percentile (MGP), impressing the reader with discussions and charts of confidence ranges and the need to be transparent about the data. It all seems so thoughtful and convincing! After all, how could such numbers fail to paint an accurate picture of a teacher’s effectiveness?

(One of the more audacious claims of this document is that the development of this evaluative model is the result of the collaborative efforts of the Regents Task Force on Teacher and Principal Effectiveness. Those of us who know people who served on this committee are well aware that the recommendations of the committee were either rejected or ignored by State Education officials.)

One of the items missing from this presentation, however, is an explanation of how State officials translated SGPs and MGPs into a number from 1 to 20. In order to find out how the State went from MGPs to a teacher effectiveness score out of 20 points, one needs to refer to the 2010-11 Beta Growth Model for Educator Evaluation Technical Report. Why a separate document for explaining these scores? Most likely because there are few State officials who are fluent in the psychometrics necessary to explain how this part of our APPR system works.

It is incredulous that the state feels that it is perfectly fine to use a statistical model still in a beta phase to arrive at these amorphous teacher effectiveness scores. I make it a point not to use beta software on my computer, for I do not want something untested and filled with bugs to contaminate the programs that are working fine on my machine. It is a shame that the State does not have the same opinion regarding its reform initiatives.

As explained in the technical paper, the SGP model championed by New York State claims to account for students who are English Language Learners (ELL), students with disabilities (SWD) and even economically disadvantaged students as it determines a teachers adjusted mean growth percentage. While the statistical explanation underlying the SGP model is carefully developed, nowhere do the statisticians justify the underlying cause for any change in student score measured. In other words, what is the research basis for attributing any change in score from year to year to the singular variable of a teacher? The reason why this is never explained is because there is virtually no research that justifies attributing the teacher as the sole cause of a change in student score from year to year.

So if it is not solely the teacher who caused the change in score, to what should one attribute a change in student score? Well, that is a question that continues to challenge statisticians and educational researchers. Despite the hopes and declarations of so many of our present-day “reformers,” we simply do not have to tools necessary to quantify the impact a single teacher has on an individual student’s test score over the course of time. Derek Briggs presented a critique of the use of SGPs in this paper.

How can one explain Ashley’s shockingly low score, however? As a principal who has always availed himself of data when evaluating teachers, I would sit down and have a conversation about the test results so that I could put them in context. Here is what we know about the context of Ashley’s score:

* This year, Ashley’s score was based on her two eighth grade classes, not the results of her Regents-level classes

* The two eighth grade classes were different curricula: one was an Algebra course and the other was a Math 8 course.

* The Algebra 8 course is geared towards the Regents exam, which is a high-school level assessment that is beyond the mathematical level of the NYS Math 8 examination. Ninety one percent of Ashley’s students in this class passed the Regents Algebra 1 examination. There is different content on the Math 8 exam, which can make it a challenge for some of our weaker Algebra students. In fact, of the students who took the Algebra course, one-quarter of them passed the Regents examination but scored below proficiency on the Math 8 exam.

* In the two weeks prior to the three-day administration of the Math 8 exam in April 2012, students in Ashley’s class had one week of vacation followed by three days of English testing. In the two weeks leading to the beginning of the Math 8 exam, Ashley saw her class only three times.

Rather than place the student results in context, the State issued a blind judgment based on data that was developed through unproven and invalid calculations. These scores are then distributed with an authority and “scientific objectivity” that is simply unwarranted. Along the way, teacher reputations and careers will be destroyed.

Despite the judgment of the New York State Education Department, Ashley remains a model teacher in our school: beloved by students and parents; respected by colleagues and supervisors. She continues to work on perfecting her practice and helping her students gain confidence and skills. My hope, of course, is that she will continue to feel that she is part of a profession that respects teachers and students alike, not one that reduces them to a poorly conceived and incoherent number.