CURE demands that OSPI follow the law; challenges WASL’s accuracy
April 19, 2010
CURE Responds to SPI Bergeson’s Defense of the Indefensible WASL
By Joanne McCann and Marda Kirkwood
February 2001
SPI said: “WASL scores are reliable and accurate. The scores on the WASL are reliable and accurate. The process for scoring the WASL is sound. … [T]he lowest level of scorer agreement on the test as a whole was 96%. The typical level of agreement was 98%”
Fact: Dr. Thomas Hirsch, a consultant to OSPI and Riverside (the assessment developer and scorer), has examined the percent of times that two different scorers give the exact same score for each item they scored on the WASL. Examples he cites are:
“… at grade 4 in 1998, the percent of times that scorers gave the exact same score on Reading, Listening, and Mathematics questions ranged from 76 to 97 across all questions. … At grade 7 in 1998, exact agreement for Mathematics questions ranged from 69 to 98 percent across all questions. … At grade 10 in 1999 … Exact agreement for Mathematics questions ranged from 70 to 91 percent.” (OSPI – WASL Scoring Q & A, p. 6 of 11)
CURE’s position: In a high stakes assessment, only 100% agreement in scoring is acceptable. Mathematics is an exacting science, requiring only one correct answer. How can two scorers have so much trouble agreeing on the correct answer? OSPI shortchanges children and teachers by accepting as reliable and valid an assessment with low standards of agreement. · SPI claims that scorers follow Washington State scoring rules. Fact: National Computer Systems (NCS) does not follow Washington state scoring rules. Dr. Catherine Taylor, U of W, working for OSPI wrote: “Qualifications of Readers: Highly qualified experienced readers (scorers) were essential to achieving and maintaining consistency and reliability when scoring student-constructed (open-ended) responses. Readers selected for the Washington Assessments were required to have the following qualifications:
- minimum of a bachelor´s degree in an appropriate academic discipline (such as English, English Education, Math, Math Education or related fields):
- Demonstrable ability in performance assessment scoring:
- Teaching experience, especially at the elementary or secondary level, was preferred.”
(Washington Assessment of Student Learning, Grade 4, 1998, Technical Report, p. 4-1)
However, NCS scorers in Iowa and Arizona are only required to have a degree, no teaching experience and receive minimal training. After training, the applicants are required in order to qualify for jobs as scorers, to demonstrate only 65% scoring accuracy. “For example in writing the scores…must agree with the already assigned scorers at least 65% of the time.” (OSPI, WASL Scoring Q & A, pp. 3 – 4 of 11–Q: How do scorers qualify to work as scorers after training?)
CURE’s position: Again, in a high stakes assessment, only 100% agreement in scoring is acceptable and only the most highly qualified, experienced scorers must be hired. What happens if all the students from one school have their math assessments scored by an experienced math teacher and all the students from another school have theirs scored by a 24-year-old law student, who is studying for the bar by day and scoring WASL’s by night? In order to facilitate rapid, inexpensive scoring, OSPI has compromised its own requirements. OSPI has lowered the scoring rules bar. This must explain scoring discrepancies, especially the writing scores.
SPI said: “WASL scores do not impact school funding.”
Fact: WASL development, scoring and implementation drain resources. Since 1993, OSPI has spent over $50 million developing the WASL. 16 ½ new employees currently implement the WASL for OSPI. Private schools pay $23 per child, while public schools are charged $27 per student, per WASL. OSPI demands countless hours and dollars per school for conferences, in-service, substitutes, etc., all in the name of WASL implementation.
Cure Position: The time and money could be better spent for teacher salaries, classroom resources and lowering the taxpayer burden. How much money and classroom time has your district spent just for WASL related activities? OSPI, while constantly demanding more money and teacher time, denies that the WASL has impacted funding.
Tags: WASL, high stakes, reliability, scoring, validity
CURE Responds to SPI Bergeson’s Defense of the Indefensible WASL
By Joanne McCann and Marda Kirkwood
February 2001
SPI said: “WASL scores are reliable and accurate. The scores on the WASL are reliable and accurate. The process for scoring the WASL is sound. … [T]he lowest level of scorer agreement on the test as a whole was 96%. The typical level of agreement was 98%”
Fact: Dr. Thomas Hirsch, a consultant to OSPI and Riverside (the assessment developer and scorer), has examined the percent of times that two different scorers give the exact same score for each item they scored on the WASL. Examples he cites are:
“… at grade 4 in 1998, the percent of times that scorers gave the exact same score on Reading, Listening, and Mathematics questions ranged from 76 to 97 across all questions. … At grade 7 in 1998, exact agreement for Mathematics questions ranged from 69 to 98 percent across all questions. … At grade 10 in 1999 … Exact agreement for Mathematics questions ranged from 70 to 91 percent.” (OSPI – WASL Scoring Q & A, p. 6 of 11)
CURE’s position: In a high stakes assessment, only 100% agreement in scoring is acceptable. Mathematics is an exacting science, requiring only one correct answer. How can two scorers have so much trouble agreeing on the correct answer? OSPI shortchanges children and teachers by accepting as reliable and valid an assessment with low standards of agreement. · SPI claims that scorers follow Washington State scoring rules. Fact: National Computer Systems (NCS) does not follow Washington state scoring rules. Dr. Catherine Taylor, U of W, working for OSPI wrote: “Qualifications of Readers: Highly qualified experienced readers (scorers) were essential to achieving and maintaining consistency and reliability when scoring student-constructed (open-ended) responses. Readers selected for the Washington Assessments were required to have the following qualifications:
- minimum of a bachelor´s degree in an appropriate academic discipline (such as English, English Education, Math, Math Education or related fields):
- Demonstrable ability in performance assessment scoring:
- Teaching experience, especially at the elementary or secondary level, was preferred.”
(Washington Assessment of Student Learning, Grade 4, 1998, Technical Report, p. 4-1)
However, NCS scorers in Iowa and Arizona are only required to have a degree, no teaching experience and receive minimal training. After training, the applicants are required in order to qualify for jobs as scorers, to demonstrate only 65% scoring accuracy. “For example in writing the scores…must agree with the already assigned scorers at least 65% of the time.” (OSPI, WASL Scoring Q & A, pp. 3 – 4 of 11–Q: How do scorers qualify to work as scorers after training?)
CURE’s position: Again, in a high stakes assessment, only 100% agreement in scoring is acceptable and only the most highly qualified, experienced scorers must be hired. What happens if all the students from one school have their math assessments scored by an experienced math teacher and all the students from another school have theirs scored by a 24-year-old law student, who is studying for the bar by day and scoring WASL’s by night? In order to facilitate rapid, inexpensive scoring, OSPI has compromised its own requirements. OSPI has lowered the scoring rules bar. This must explain scoring discrepancies, especially the writing scores.
SPI said: “WASL scores do not impact school funding.”
Fact: WASL development, scoring and implementation drain resources. Since 1993, OSPI has spent over $50 million developing the WASL. 16 ½ new employees currently implement the WASL for OSPI. Private schools pay $23 per child, while public schools are charged $27 per student, per WASL. OSPI demands countless hours and dollars per school for conferences, in-service, substitutes, etc., all in the name of WASL implementation.
Cure Position: The time and money could be better spent for teacher salaries, classroom resources and lowering the taxpayer burden. How much money and classroom time has your district spent just for WASL related activities? OSPI, while constantly demanding more money and teacher time, denies that the WASL has impacted funding.
Tags: WASL, high stakes, reliability, scoring, validity