2013-03-29

Godzilla vs. Consortia

We previously examined in detail the content of a sample question coming out of Smarter Balanced Assessment Consortium, and it would be only fair to have a look at one of PARCC’s high school questions as well.  At the high school level, PARCC to date has played it safe by only releasing three official questions, all of Type I, its most basic level, which in the following example requires students to perform a mathematical procedure and enter numbers into empty boxes:
To work out this math problem, students must still resort to rewriting the entire equation on a piece of paper, with the very real possibility of a transcription error, so exactly what advantage for students has been gained by moving such assessments onto a computer?

PARCC’s release notes accompanying this problem gloat ``the task does not clue the student that the equation is quadratic or that it has two solutions; students must recognize the nature of the equation from its structure,’’ yet the format reveals at least one answer will be a fraction.  If an answer were integral, should students enter digits with a 1 in the denominator?  What if the answer were irrational?  Complex?  What if a fraction were entered not in lowest terms?  What about a negative number?  If negative, can there be a space between the negative sign and the number, i.e., –6 or – 6, or will one of the two be uncredited?

True, the question doesn’t clue students there may be two solutions, but it does suggest students look for a fraction, which in fact each of the two solutions is.

The notes further brag ``[u]nlike traditional multiple-choice tests, the technology in this task prevents guessing and working backwards...A further enhancement is that the item format does not immediately indicate the number of solutions.’’

Indeed it doesn’t.  We are gluttons for punishment, so we kept clicking on the lavender + button (soothing colors are welcome on a high stakes math exam) to see how many solutions the question would allow us to enter.  Fearing the onset of carpal tunnel syndrome, we gave up at 501.  So a computer-based problem doesn’t limit the number of solutions to a quadratic equation, and therefore what advantage does it hold over a paper-and-pencil test?
To continue our analysis we pressed ``done’’, but nothing happened; and so discovered PARCC problems aren’t fully coded and we couldn’t proceed further.  We’ll have to change course mid-stream in today’s analysis.

***

As anyone who has endured the tyranny of Microsoft Windows can attest, computer code is not infallible.  There are bugs.  Lots of them.  And as Microsoft knows full well, notwithstanding extensive beta (or ``pilot’’) testing, once software is put into the hands of the end user, it doesn’t take long for those bugs to surface.

We tried out online one of SBAC’s Grade 3 questions where an on-screen interface allows you click on digits or math symbols to enter the solution:
Mouse click as we did, we could not get an actual numeral to appear, but by clicking on the [x/y] and other buttons, we created the following mess in the answer box (an unaltered screen dump):
Underlying even the simplest interface is a significant amount of JavaScript or other computer code, and with the inevitable bugs, on a computer-based exam being taken by millions of students all over the country, the possibility of even a single software glitch on exam day raises the specter of teachers and administrators running around, crying children, and endless busy signals as everyone tries to call technical support at the same time.

Most logistics concerns over the new assessments seem to focus on upgrades in bandwidth or hardware, but it’s ultimately the software that will induce testing nightmares.

***

Another issue in designing an individual question is its interface: what do you need to do to enter your response and how do you know?  The previous problem requires clicking a series of buttons with recognizable numbers and symbols, an input method which seems fairly obvious, but not every question calls for a numerical answer.

In the following Grade 7 PARCC question, the student must first calculate four separate rates (we’re not analyzing the caliber and clarity of the mathematical task here, although we should):
This question, as in the quadratic equation assessment, necessitates copying information from the computer screen to paper, doing arithmetic on the paper, and then transferring those answers back to the computer screen—but how to submit the response?

The instructions, ``drag and drop the object names in order from greatest speed to least speed in the table provided’’, raise a number of interface issues.  First, it is not immediately and unequivocally clear exactly what you’re supposed to drag and to where.  Mousing over the vertical list of objects in green shaded rectangles changes the cursor to a pointing finger which answers the first question, that Object A is an object name, but then you have to realize that the ``table provided’’ is the four empty boxes to the right, so you have to drag the rectangles to the right side in the correct order.  It may be obvious to some, but not to everyone.

Then there’s an interface headache.  Once you’ve moved a green shaded rectangle from the left to the right, it can’t be moved again.  If you make a mistake, you have to press ``reset’’, undoing everything, including correct answers, and start again.

Interface designers would have a field day.  Where’s Steve Jobs when you need him?

For example, why not create an interface which allows the 4 rectangles to be reordered?  ``Drag the objects into the correct order from greatest speed to least speed.’’  Is PARCC concerned that some rectangles may remain in the same position and they want students to act affirmatively for each object?  That should be the least of PARCC’s concerns in a problem like this.

PARCC, take a hint from Yahoo! in how to create a simple interface:

The instructions can be explicit or not, but everyone knows #1 is the fastest.  The point is, the interface should be easy to use and unambiguous.  The beauty of the original Macintosh computer was its uniformity of interface across applications, but in these assessment questions, each one has a different gimmick and requires the student to figure out what to do.  Students will be under test pressure as it is, and poor interfaces will frustrate (or worse, penalize) some students unnecessarily, increasing inequity.

Our response to PARCC’s final boast from before: does either PARCC’s interface or our suggested alternative deter guessing to a greater extent than a multiple choice response?  We don’t see how.  Guessers will just put the boxes in any order.

***

The following two-part PARCC question is for third graders:
Again, we won’t discuss either the mathematical content of the question or the teaching of equivalent fractions in Grade 3 (follow our ongoing discussion of Common Core’s treatment of fractions), but the interface of this task is awful on many levels.

The grammatical oddness of the instruction ``[d]rag the soybean to the field as many times as needed’’ is only somewhat justified when, after dragging the DayGlo green soybean pictured there to one of the rectangles, another soybean magically appears!  Great coding job!  And if farming were so easy, we’d relocate to Nebraska from the city!
As in the rate question, once an individual soybean is placed, it cannot be moved or removed.  If third graders make a mistake, do they know the meaning of ``reset’’?  Lawyers, take a lesson from PARCC on how not to write plain English.  How about ``start over’’ as an alternative?

In Part B of the task, the student has to type in a fraction equivalent to 3/4.  Presumably PARCC expects most correct answers to be 6/8.

We deliberately typed a wrong answer in the boxes, and when we tried to erase the numbers by pressing reset, the numbers remained, but the soybeans we had dragged over in Part A disappeared, and we were returned to Part A, even though there’s already an arrow we can press to take us there.

Second time around, we went further and typed an explanation, and when we hit reset, as before, the equivalent fraction remained, the soybeans disappeared and we were returned to Part A.  When we clicked the arrow to return to Part B, our explanation was gone.

Finally, let’s not forget that this question is for third graders who haven’t yet taken Mavis Beacon Teaches Typing and don’t even know the historical basis for the QWERTY keyboard, but to imagine the disproportionate amount of time on a math test it will take students to find the letters and type out an explanation (Mr. Sholes, is this enough words?  Does spelling count?  ur serious?  y?) that has to be hand-graded nonetheless on a high cost high tech assessment and we’ve reached the outer boundaries of absurdity.

You think a whole class of third graders is going to get through an entire assessment psychologically unscathed?

***

Due to PARCC’s dearth of sample assessments, we return to SBAC for a Grades 6-8 task:
Because the underlying mathematics is simple, it is possible to decipher this wacky interface and complete the task in less than a minute, but we don’t expect that to be the norm.

This question presents the student with three sales tax calculators, A, B & C, from unknown states.  Which states is the mystery which students must solve by finding clues.  If you begin on the left side and enter $100 for the Purchase Price (our educated choice, but is it obvious to everyone?), and then choose, in succession, Calculator A → Find Sales Tax → Calculator B → Find Sales Tax → Calculator C → Find Sales Tax, the answers 6.25, 5.00 and 6.88 appear in the cyan colored table next to the corresponding calculator.

Then you have to go to Part 2 and realize ``Calculator A’’ is a click-and-draggable object that goes in the empty box to the right side of Illinois (because $6.25 sales tax is 6.250% of $100). Calculator B goes next to Maine, Calculator C next to Minnesota (rounding!) and the remainder of the states get nothing.  Mystery solved!

Wow, what a lot of words, instructions, and underlying code for only one question.

***

How many lengthy-coded and carefully-vetted assessment questions will be in PARCC’s and SBAC’s databases?  Enough that students won’t encounter the same questions repeatedly or be able to study beforehand questions that their peers have already seen and can tell them about?  That requires an awfully big repository, each task with its own interface and underlying code, any single question of which can stymie students or crash the whole system during an assessment.

We’re creating a monster.

13 comments:

  1. I have done some of the SBAC questions and OMG I know the correct answer, but struggle to figure out how I am able to display the answer. Very convoluted and I can just see my 7th graders trying to figure things out.

    ReplyDelete
  2. Excellent work to put this up. You have shown that these are interface tests not math tests. And the test makers failed.

    ReplyDelete
  3. This makes me want to cry. What about visually impaired students. Dslexia will kill you on these math tests. Can teachers help with the interface? What about children who have problems using PCs in general? Dear god, just remembered that their will be time limits on these. It should not take longer to answer the question then to solve the question.

    What ever happened to Solve this equation. Show your work. That way if there is a transcription error or arithmetic issue when solving a quadratic equation you will not get a zero.

    Sobs

    ReplyDelete
  4. The real answer is of course a written assessment with human graders but, while everyone has a desire to know what students know, no one has enough respect for students to personally sit down and give the students' work the attention it deserves.

    What has this to do with students' capacity to create or to make and defend decisions? Where's that going to be on the test and who will actually give students' words and arguments the respect that kind of work demands? I will in my classroom, but will the bureaucrats? will the computers? will the test publishers?

    School is the place where students should learn literacy, numeracy, reasoning, etiquette, and professionalism even as they discover what they do best, and then it should be the place where students learn to be the best they can at what they do best, and once the student has access to a better deal -- a job, an apprenticeship, a college admission -- school should recede into fond and grateful memories.

    This isn't any of that. This is how we "lose" students. The students can't stand or understand it and we teachers can't and shouldn't defend it.

    ReplyDelete
  5. Having spent 25 years in the Mac user interface design community (which also means serving as a tester for much imperfect software along the way), I can promise that just the few catastrophes you have identified here will bring Smarter Balance and PARCC to its knees.

    Looking ambivalently forward to seeing the &**@^% that ensues.

    - Elizabeth (@cheesemonkeysf)

    ReplyDelete
  6. Anonymous06 May, 2013

    For the first problem there is
    (3x-2)(3x-2)=6x-4
    (3x-2)(3x-2)=2(3x-2)
    Dividing both sides of the equation by (3x-2)
    3x-2=2
    3x-2+2=2+2
    3x=4
    3x/3=4/3
    x=4/3
    What is the other value of "X?"

    ReplyDelete
    Replies
    1. Anonymous07 May, 2013

      If 3x-2=0, then your first two equations are clearly satisfied, but then you divided both sides by zero and lost that solution. Divide with care!

      Delete
    2. So to be elear, the two solutions are the solutions of

      3x-2 = 2 and

      3x-2 = 0 (if you check, this converts the original equation to 0=0 which is an equality.)

      So the two solutions are x = 4/3 and x = 2/3 . I am not too worried about the student starting out knowing a fraction is involved -- even the quadratic formula is in the form of a fraction, if the student doesn't see the division by (3x-2), expands the whole mess out, applies the quadratic formula (I guess they haven't learned that in basic HS math but it is one way to solve the problem, the slow way).

      I think it's a fine question, nothing wrong with it. As long as the interface works reliably and the students learn to use it effectively. This testing interface looks like a mess now, but in a year or two it might be pretty slick, and then it's usable forever.

      Frankly, one thing I like is that it's not about justifying your answer, creating making and defending decisions, etc. It's math, which is what there should be in math class. And to heck with any Common Core ideas that it should be diluted with a bunch of other stuff.

      Delete
  7. Anonymous08 May, 2013

    What a mess we have on our hands, this information must and will be passed on to all educators that I know. We are in the process of being trained on Common Core and have not been given a precise answer about the assessments. Now I know why. What a waste of our hard earned middle class dollars!

    ReplyDelete
  8. Anonymous24 July, 2013

    The only way for children to have a chance to pass such exams is for them to spend a minimum of two, but more likely more than four, hours per day sitting at a computer learning to deal with the interface while trying to think of the answers. There will be no time left in the school day for lessons involving labs, divergent thinking, and creative thinking. High school students in my district participated in the SB pilot this spring. No pencil nor paper was allowed. The students could go back and review documents and videos (Some students were attempting to do this with a 3" screen on a phone!) but could make no notes; they had to hold everything in their heads. Content may have been tested, but only after a fatiguing (3-4 hours) trial of working memory.

    ReplyDelete
  9. Are we all just a bunch of idiots to subject ourselves to this? How long will it take to drive a nation to lunacy?

    ReplyDelete
  10. This is truly astonishing, I can not believe that a rolled-out system can be this bad. According to the literature, these initiatives are supposed to make our kids more competitive in the world marketplace. If anything, it appears fiddling around with these widgets will result in a lot of agitated students, frustrated teachers, and administrators not getting their promised bonuses. Perhaps the Washington D.C. proponents of this monstrosity are shareholders in the computer company that will be paid and paid again for upgrades (e.g. fixes). Thank you for posting this, but it made me sad for the children.

    ReplyDelete
  11. This is not designed to measure anything useful. We had perfectly good assessments already, ones that measured student achievement and understanding without being massively frustrating and bug-ridden. This is the Progressive need to fiddle with everything in order to make it impossible for individuals to understand or function without "experts" telling them what is going on. It's about taking control over education away from local administrations and teachers and depositing it in the hands of distant, unaccountable bureaucrats. Students will be handicapped by these new "standards" and be unable to compete against anybody. The whole CCSSI is a Rube Goldberg contraption, set up by people less interested in education than in making money and holding on to power.

    ReplyDelete