Godzilla vs. Consortia

We previously examined in detail the content of a sample question coming out of Smarter Balanced Assessment Consortium, and it would be only fair to have a look at one of PARCC’s high school questions as well.  At the high school level, PARCC to date has played it safe by only releasing three official questions, all of Type I, its most basic level, which in the following example requires students to perform a mathematical procedure and enter numbers into empty boxes:

To work out this math problem, students must still resort to rewriting the entire equation on a piece of paper, with the very real possibility of a transcription error, so exactly what advantage for students has been gained by moving such assessments onto a computer?

PARCC’s release notes accompanying this problem gloat ``the task does not clue the student that the equation is quadratic or that it has two solutions; students must recognize the nature of the equation from its structure,’’ yet the format reveals at least one answer will be a fraction.  If an answer were integral, should students enter digits with a 1 in the denominator?  What if the answer were irrational?  Complex?  What if a fraction were entered not in lowest terms?  What about a negative number?  If negative, can there be a space between the negative sign and the number, i.e., –6 or – 6, or will one of the two be uncredited?

True, the question doesn’t clue students there may be two solutions, but it does suggest students look for a fraction, which in fact each of the two solutions is.

The notes further brag ``[u]nlike traditional multiple-choice tests, the technology in this task prevents guessing and working backwards...A further enhancement is that the item format does not immediately indicate the number of solutions.’’

Indeed it doesn’t.  We are gluttons for punishment, so we kept clicking on the lavender + button (soothing colors are welcome on a high stakes math exam) to see how many solutions the question would allow us to enter.  Fearing the onset of carpal tunnel syndrome, we gave up at 501.  So a computer-based problem doesn’t limit the number of solutions to a quadratic equation, and therefore what advantage does it hold over a paper-and-pencil test?

To continue our analysis we pressed ``done’’, but nothing happened; and so discovered PARCC problems aren’t fully coded and we couldn’t proceed further.  We’ll have to change course mid-stream in today’s analysis.

***

As anyone who has endured the tyranny of Microsoft Windows can attest, computer code is not infallible.  There are bugs.  Lots of them.  And as Microsoft knows full well, notwithstanding extensive beta (or ``pilot’’) testing, once software is put into the hands of the end user, it doesn’t take long for those bugs to surface.

We tried out online one of SBAC’s Grade 3 questions where an on-screen interface allows you click on digits or math symbols to enter the solution:

Mouse click as we did, we could not get an actual numeral to appear, but by clicking on the [x/y] and other buttons, we created the following mess in the answer box (an unaltered screen dump):

Underlying even the simplest interface is a significant amount of JavaScript or other computer code, and with the inevitable bugs, on a computer-based exam being taken by millions of students all over the country, the possibility of even a single software glitch on exam day raises the specter of teachers and administrators running around, crying children, and endless busy signals as everyone tries to call technical support at the same time.

Most logistics concerns over the new assessments seem to focus on upgrades in bandwidth or hardware, but it’s ultimately the software that will induce testing nightmares.

***

Another issue in designing an individual question is its interface: what do you need to do to enter your response and how do you know?  The previous problem requires clicking a series of buttons with recognizable numbers and symbols, an input method which seems fairly obvious, but not every question calls for a numerical answer.

In the following Grade 7 PARCC question, the student must first calculate four separate rates (we’re not analyzing the caliber and clarity of the mathematical task here, although we should):

This question, as in the quadratic equation assessment, necessitates copying information from the computer screen to paper, doing arithmetic on the paper, and then transferring those answers back to the computer screen—but how to submit the response?

The instructions, ``drag and drop the object names in order from greatest speed to least speed in the table provided’’, raise a number of interface issues.  First, it is not immediately and unequivocally clear exactly what you’re supposed to drag and to where.  Mousing over the vertical list of objects in green shaded rectangles changes the cursor to a pointing finger which answers the first question, that Object A is an object name, but then you have to realize that the ``table provided’’ is the four empty boxes to the right, so you have to drag the rectangles to the right side in the correct order.  It may be obvious to some, but not to everyone.

Then there’s an interface headache.  Once you’ve moved a green shaded rectangle from the left to the right, it can’t be moved again.  If you make a mistake, you have to press ``reset’’, undoing everything, including correct answers, and start again.

Interface designers would have a field day.  Where’s Steve Jobs when you need him?

For example, why not create an interface which allows the 4 rectangles to be reordered?  ``Drag the objects into the correct order from greatest speed to least speed.’’  Is PARCC concerned that some rectangles may remain in the same position and they want students to act affirmatively for each object?  That should be the least of PARCC’s concerns in a problem like this.

PARCC, take a hint from Yahoo! in how to create a simple interface:

The instructions can be explicit or not, but everyone knows #1 is the fastest.  The point is, the interface should be easy to use and unambiguous.  The beauty of the original Macintosh computer was its uniformity of interface across applications, but in these assessment questions, each one has a different gimmick and requires the student to figure out what to do.  Students will be under test pressure as it is, and poor interfaces will frustrate (or worse, penalize) some students unnecessarily, increasing inequity.

Our response to PARCC’s final boast from before: does either PARCC’s interface or our suggested alternative deter guessing to a greater extent than a multiple choice response?  We don’t see how.  Guessers will just put the boxes in any order.

***

The following two-part PARCC question is for third graders:

Again, we won’t discuss either the mathematical content of the question or the teaching of equivalent fractions in Grade 3 (follow our ongoing discussion of Common Core’s treatment of fractions), but the interface of this task is awful on many levels.

The grammatical oddness of the instruction ``[d]rag the soybean to the field as many times as needed’’ is only somewhat justified when, after dragging the DayGlo green soybean pictured there to one of the rectangles, another soybean magically appears!  Great coding job!  And if farming were so easy, we’d relocate to Nebraska from the city!

As in the rate question, once an individual soybean is placed, it cannot be moved or removed.  If third graders make a mistake, do they know the meaning of ``reset’’?  Lawyers, take a lesson from PARCC on how not to write plain English.  How about ``start over’’ as an alternative?

In Part B of the task, the student has to type in a fraction equivalent to 3/4.  Presumably PARCC expects most correct answers to be 6/8.

We deliberately typed a wrong answer in the boxes, and when we tried to erase the numbers by pressing reset, the numbers remained, but the soybeans we had dragged over in Part A disappeared, and we were returned to Part A, even though there’s already an arrow we can press to take us there.

Second time around, we went further and typed an explanation, and when we hit reset, as before, the equivalent fraction remained, the soybeans disappeared and we were returned to Part A.  When we clicked the arrow to return to Part B, our explanation was gone.

Finally, let’s not forget that this question is for third graders who haven’t yet taken Mavis Beacon Teaches Typing and don’t even know the historical basis for the QWERTY keyboard, but to imagine the disproportionate amount of time on a math test it will take students to find the letters and type out an explanation (Mr. Sholes, is this enough words?  Does spelling count?  ur serious?  y?) that has to be hand-graded nonetheless on a high cost high tech assessment and we’ve reached the outer boundaries of absurdity.

You think a whole class of third graders is going to get through an entire assessment psychologically unscathed?

***

Due to PARCC’s dearth of sample assessments, we return to SBAC for a Grades 6-8 task:

Because the underlying mathematics is simple, it is possible to decipher this wacky interface and complete the task in less than a minute, but we don’t expect that to be the norm.

This question presents the student with three sales tax calculators, A, B & C, from unknown states.  Which states is the mystery which students must solve by finding clues.  If you begin on the left side and enter $100 for the Purchase Price (our educated choice, but is it obvious to everyone?), and then choose, in succession, Calculator A → Find Sales Tax → Calculator B → Find Sales Tax → Calculator C → Find Sales Tax, the answers 6.25, 5.00 and 6.88 appear in the cyan colored table next to the corresponding calculator.

Then you have to go to Part 2 and realize ``Calculator A’’ is a click-and-draggable object that goes in the empty box to the right side of Illinois (because $6.25 sales tax is 6.250% of $100). Calculator B goes next to Maine, Calculator C next to Minnesota (rounding!) and the remainder of the states get nothing.  Mystery solved!

Wow, what a lot of words, instructions, and underlying code for only one question.

***

How many lengthy-coded and carefully-vetted assessment questions will be in PARCC’s and SBAC’s databases?  Enough that students won’t encounter the same questions repeatedly or be able to study beforehand questions that their peers have already seen and can tell them about?  That requires an awfully big repository, each task with its own interface and underlying code, any single question of which can stymie students or crash the whole system during an assessment.

We’re creating a monster.