Lies, Damn Lies, and the Myth of “Standardized” Tests
August 7, 2011
This was written about the WASL, HSPE, and other state assessments in Washington, but the information also helps us understand the Smarter Balanced Assessments that are aligned with the Common Core Standards.
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
Lies, Damn Lies, and the Myth of “Standardized” Tests
By Marda Kirkwood, August 2011
The glory which is built upon a lie soon becomes a most unpleasant incumbrance [sic]. How easy it is to make people believe a lie, and how hard it is to undo that work again!
– Mark Twain in Eruption
So here I go, taking on the monumental task of undoing a lie that is repeated so often in the media, by elected officials (who should know better), and educrats (who, I am convinced, do know better) that it has come to be generally accepted as fact.
Here is the lie: The High School Proficiency Exam (HSPE), the Measurements of Student Progress (MSP), and their precursor, the Washington Assessment of Student Learning (WASL) are “standardized” tests.
What, logically, are the characteristics that would allow us to legitimately label a test as “standardized”? It doesn’t take much thought to come up with a few guidelines.
A “standardized” test must:
- Be completely objective. There can be no judgment involved in determining whether an answer is right or wrong.
- Have specific time constraints. We all remember, “Time is up! Put your pencils down now.”
- Receive the same score no matter when, how, or by whom it is scored. It should not matter if it is a Monday or a Friday, or if test scorers are having a good or a bad day.
- Ask all students of the same grade standardized questions, every year.
- Reliably measure what academic knowledge the student knows.
- Serve as an accurate guideline for teachers, principals, school board members, parents, and others to evaluate the quality of curricula and instruction.
- Be “valid and reliable”. These are terms that are roughly analogous to accuracy and precision in target shooting. Valid means they accurately measure what they are intended to measure. Reliable means they are consistently valid (produce the same score). And just because some government official or entity declares a test “valid and reliable” does not make it so.
- Be norm-referenced. That means the scores have been compared, via a bell-curve, with the scores of other students across the nation who also took the same test. The comparison score is shown as a percentile. A student who receives a 70th percentile score performed better than 70% of the other students in the nation who took the same test.
This also gives us a pretty good idea what real standardized tests are not.
Here are some characteristics of an assessment that is not a standardized test:
- It includes essay or short answer, which have to be subjectively hand-scored and require a fallible human to use judgment to determine a score. Repeat after me, “subjective.”
- It has lots of variation in test questions from year to year, changing types and difficulty levels.
- Students are allowed to take however long they feel like to complete the test.
- A committee votes on a “cut score” (i.e. what level is “passing”) after the tests have been scored.
Guess which list describes the HSPE/MSP/WASL? Yup, these assessments include all the features that characterize what standardized tests are not. These are (theoretically, anyway) standards-based assessments, very different from standardized tests. They are supposedly designed to measure the Essential Academic Learning Requirements (EALRs) – our state learning standards. That is debatable, but food for another topic. The EALRs are a moving target, able to be changed by the Superintendent of Public Instruction (SPI), without the consent of the Legislature, as Terry Bergeson did when she was SPI. We have to assume they measure the EALRs because that is what federal law requires under No Child Left Behind. But the results are meaningless when they can be so easily manipulated by the choice of questions, the choice of cut scores, and the directions to the scorers.
The list of standardized tests includes the Iowa Test of Basic Skills (ITBS), the Comprehensive Test of Basic Skills (CTBS), the Metropolitan Achievement Test (MAT), and the Stanford Achievement Test (SAT – not the college entrance one, which is the Scholastic Aptitude Test). Even these are beginning to become WASL-ized, as some of them have started adding writing sections. Washington eliminated the use of norm-referenced, standardized tests in 2005.
During the few years when Washington used both the WASL and a standardized test, WASL scores steadily increased, while the scores on the standardized test remained flat. Are you now surprised that the use of the standardized tests was eliminated?
Another little fact for your file, in the era of recession and government cutbacks: Standardized tests are very cheap, as they can be scored by a computer in seconds. Standards-based assessments (since they must be scored by armies of humans) cost a bundle. In 2003 the ITBS cost $2.88 per student. The same year, the WASL cost $73 per student. Oh, and that $73 only includes the cost of printing and scoring the assessments. No costs are included in that figure for development, administration of the test to the students, or test security. It also doesn’t include what economists call opportunity costs. All the many hours students are forced to spend on test preparation and practice tests are lost opportunities to learn new things.
So, next time you hear someone refer to Washington’s “standardized” tests, remember what you learned here. And explain it to your neighbors. The only way to combat lies is with the truth.
Tags: HSPE, MSP, WASL, assessments
This was written about the WASL, HSPE, and other state assessments in Washington, but the information also helps us understand the Smarter Balanced Assessments that are aligned with the Common Core Standards.
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
Lies, Damn Lies, and the Myth of “Standardized” Tests
By Marda Kirkwood, August 2011
The glory which is built upon a lie soon becomes a most unpleasant incumbrance [sic]. How easy it is to make people believe a lie, and how hard it is to undo that work again!
– Mark Twain in Eruption
So here I go, taking on the monumental task of undoing a lie that is repeated so often in the media, by elected officials (who should know better), and educrats (who, I am convinced, do know better) that it has come to be generally accepted as fact.
Here is the lie: The High School Proficiency Exam (HSPE), the Measurements of Student Progress (MSP), and their precursor, the Washington Assessment of Student Learning (WASL) are “standardized” tests.
What, logically, are the characteristics that would allow us to legitimately label a test as “standardized”? It doesn’t take much thought to come up with a few guidelines.
A “standardized” test must:
- Be completely objective. There can be no judgment involved in determining whether an answer is right or wrong.
- Have specific time constraints. We all remember, “Time is up! Put your pencils down now.”
- Receive the same score no matter when, how, or by whom it is scored. It should not matter if it is a Monday or a Friday, or if test scorers are having a good or a bad day.
- Ask all students of the same grade standardized questions, every year.
- Reliably measure what academic knowledge the student knows.
- Serve as an accurate guideline for teachers, principals, school board members, parents, and others to evaluate the quality of curricula and instruction.
- Be “valid and reliable”. These are terms that are roughly analogous to accuracy and precision in target shooting. Valid means they accurately measure what they are intended to measure. Reliable means they are consistently valid (produce the same score). And just because some government official or entity declares a test “valid and reliable” does not make it so.
- Be norm-referenced. That means the scores have been compared, via a bell-curve, with the scores of other students across the nation who also took the same test. The comparison score is shown as a percentile. A student who receives a 70th percentile score performed better than 70% of the other students in the nation who took the same test.
This also gives us a pretty good idea what real standardized tests are not.
Here are some characteristics of an assessment that is not a standardized test:
- It includes essay or short answer, which have to be subjectively hand-scored and require a fallible human to use judgment to determine a score. Repeat after me, “subjective.”
- It has lots of variation in test questions from year to year, changing types and difficulty levels.
- Students are allowed to take however long they feel like to complete the test.
- A committee votes on a “cut score” (i.e. what level is “passing”) after the tests have been scored.
Guess which list describes the HSPE/MSP/WASL? Yup, these assessments include all the features that characterize what standardized tests are not. These are (theoretically, anyway) standards-based assessments, very different from standardized tests. They are supposedly designed to measure the Essential Academic Learning Requirements (EALRs) – our state learning standards. That is debatable, but food for another topic. The EALRs are a moving target, able to be changed by the Superintendent of Public Instruction (SPI), without the consent of the Legislature, as Terry Bergeson did when she was SPI. We have to assume they measure the EALRs because that is what federal law requires under No Child Left Behind. But the results are meaningless when they can be so easily manipulated by the choice of questions, the choice of cut scores, and the directions to the scorers.
The list of standardized tests includes the Iowa Test of Basic Skills (ITBS), the Comprehensive Test of Basic Skills (CTBS), the Metropolitan Achievement Test (MAT), and the Stanford Achievement Test (SAT – not the college entrance one, which is the Scholastic Aptitude Test). Even these are beginning to become WASL-ized, as some of them have started adding writing sections. Washington eliminated the use of norm-referenced, standardized tests in 2005.
During the few years when Washington used both the WASL and a standardized test, WASL scores steadily increased, while the scores on the standardized test remained flat. Are you now surprised that the use of the standardized tests was eliminated?
Another little fact for your file, in the era of recession and government cutbacks: Standardized tests are very cheap, as they can be scored by a computer in seconds. Standards-based assessments (since they must be scored by armies of humans) cost a bundle. In 2003 the ITBS cost $2.88 per student. The same year, the WASL cost $73 per student. Oh, and that $73 only includes the cost of printing and scoring the assessments. No costs are included in that figure for development, administration of the test to the students, or test security. It also doesn’t include what economists call opportunity costs. All the many hours students are forced to spend on test preparation and practice tests are lost opportunities to learn new things.
So, next time you hear someone refer to Washington’s “standardized” tests, remember what you learned here. And explain it to your neighbors. The only way to combat lies is with the truth.
Tags: HSPE, MSP, WASL, assessments