We previously examined in detail the content of a sample question
coming out of Smarter Balanced Assessment Consortium, and it would be only fair to have a look at one of PARCC’s high school questions as well.
At the high school level, PARCC to date has played it safe by only
releasing three official questions, all of Type I, its most basic level,
which in the following example requires students to perform a
mathematical procedure and enter numbers into empty boxes:

To work out this math problem, students must still
resort to rewriting the entire equation on a piece of paper, with the very real possibility of a transcription error, so
exactly what advantage for students has been gained by moving such assessments onto a computer?

True,
the question doesn’t clue students there may be two solutions, but it
does suggest students look for a fraction, which in fact each of the two
solutions is.

The
notes further brag ``[u]nlike traditional multiple-choice tests, the
technology in this task prevents guessing and working backwards...A
further enhancement is that the item format does not immediately
indicate the number of solutions.’’

Indeed
it doesn’t. We are gluttons for punishment, so we kept clicking on the
lavender + button (soothing colors are welcome on a high stakes math
exam) to see how many solutions the question would allow us to enter.
Fearing the onset of carpal tunnel syndrome, we gave up at 501. So a
computer-based problem doesn’t limit the number of solutions to a
quadratic equation, and therefore what advantage does it hold over a
paper-and-pencil test?

To
continue our analysis we pressed ``done’’, but nothing happened; and so
discovered PARCC problems aren’t fully coded and we couldn’t proceed
further. We’ll have to change course mid-stream in today’s analysis.

***

As
anyone who has endured the tyranny of Microsoft Windows can attest, computer
code is not infallible. There are bugs. Lots of them. And as
Microsoft knows full well, notwithstanding extensive beta (or ``pilot’’) testing, once software is put into the hands of the end user, it doesn’t take long for those bugs to surface.

We
tried out online one of SBAC’s Grade 3 questions where an on-screen
interface allows you click on digits or math symbols to enter the
solution:

Mouse
click as we did, we could not get an actual numeral to appear, but by
clicking on the [x/y] and other buttons, we created the following mess
in the answer box (an unaltered screen dump):

Underlying
even the simplest interface is a significant amount of JavaScript or
other computer code, and with the inevitable bugs, on a computer-based exam being taken by millions of students
all over the country, the possibility of even a single software glitch
on exam day raises the specter of teachers and administrators running
around, crying children, and endless busy signals as everyone tries to
call technical support at the same time.

Most
logistics concerns over the new assessments seem to focus on upgrades in bandwidth or hardware, but it’s ultimately
the software that will induce testing nightmares.

***

Another
issue in designing an individual question is its interface: what do you
need to do to enter your response and how do you know? The previous
problem requires clicking a series of buttons with recognizable numbers
and symbols, an input method which seems fairly obvious, but not every
question calls for a numerical answer.

In
the following Grade 7 PARCC question, the student must first calculate
four separate rates (we’re not analyzing the caliber and clarity of the
mathematical task here, although we should):

This
question, as in the quadratic equation assessment, necessitates copying
information from the computer screen to paper, doing arithmetic on the
paper, and then transferring those answers back to the computer
screen—but how to submit the response?

The
instructions, ``drag and drop the object names in order from greatest
speed to least speed in the table provided’’, raise a number of
interface issues. First, it is not immediately and unequivocally clear
exactly what you’re supposed to drag and to where. Mousing over the
vertical list of objects in green shaded rectangles changes the cursor
to a pointing finger which answers the first question, that Object A is
an object name, but then you have to realize that the ``table provided’’
is the four empty boxes to the right, so you have to drag the
rectangles to the right side in the correct order. It may be obvious to
some, but not to everyone.

Then
there’s an interface headache. Once you’ve moved a green shaded
rectangle from the left to the right, it can’t be moved again. If you
make a mistake, you have to press ``reset’’, undoing everything,
including correct answers, and start again.

Interface designers would have a field day. Where’s Steve Jobs when you need him?

For
example, why not create an interface which allows the 4 rectangles to
be reordered? ``Drag the objects into the correct order from greatest
speed to least speed.’’ Is PARCC concerned that some rectangles may
remain in the same position and they want students to act affirmatively
for each object? That should be the least of PARCC’s concerns in a
problem like this.

PARCC, take a hint from Yahoo! in how to create a simple interface:

The instructions can be explicit or not, but everyone knows #1 is the fastest. The point is, the interface should be easy to use and unambiguous. The beauty of the original Macintosh computer was its uniformity of interface across applications, but in these assessment questions, each one has a different gimmick and requires the student to figure out what to do. Students will be under test pressure as it is, and poor interfaces will frustrate (or worse, penalize) some students unnecessarily, increasing inequity.

Our
response to PARCC’s final boast from before: does either PARCC’s
interface or our suggested alternative deter guessing to a greater
extent than a multiple choice response? We don’t see how. Guessers
will just put the boxes in any order.

***

The following two-part PARCC question is for third graders:

Again,
we won’t discuss either the mathematical content of the question or the
teaching of equivalent fractions in Grade 3 (follow our ongoing discussion of Common Core’s treatment of fractions), but the interface of this task is awful on many levels.

The
grammatical oddness of the instruction ``[d]rag the soybean to the
field as many times as needed’’ is only somewhat justified when, after
dragging the DayGlo green soybean pictured there to one of the
rectangles, another soybean magically appears! Great coding job! And
if farming were so easy, we’d relocate to Nebraska from the city!

As
in the rate question, once an individual soybean is placed, it cannot
be moved or removed. If third graders make a mistake, do they know the
meaning of ``reset’’? Lawyers, take a lesson from PARCC on how not to
write plain English. How about ``start over’’ as an alternative?

In
Part B of the task, the student has to type in a fraction equivalent to
3/4. Presumably PARCC expects most correct answers to be 6/8.

We deliberately typed a wrong answer in the boxes, and when we tried to erase the numbers by pressing reset, the numbers remained, but the soybeans we had dragged over in Part A disappeared, and we were returned to Part A, even though there’s already an arrow we can press to take us there.

Second
time around, we went further and typed an explanation, and when we hit
reset, as before, the equivalent fraction remained, the soybeans
disappeared and we were returned to Part A. When we clicked the arrow
to return to Part B, our explanation was gone.

Finally,
let’s not forget that this question is for third graders who haven’t
yet taken Mavis Beacon Teaches Typing and don’t even know the historical basis for the QWERTY keyboard,
but to imagine the disproportionate amount of time on a math test it
will take students to find the letters and type out an explanation (Mr.
Sholes, is this enough words? Does spelling count? ur serious? y?)
that has to be hand-graded nonetheless on a high cost high tech
assessment and we’ve reached the outer boundaries of absurdity.

You think a whole class of third graders is going to get through an entire assessment psychologically unscathed?

***

Due to PARCC’s dearth of sample assessments, we return to SBAC for a Grades 6-8 task:

Because
the underlying mathematics is simple, it is possible to decipher this
wacky interface and complete the task in less than a minute, but we
don’t expect that to be the norm.
This
question presents the student with three sales tax calculators, A, B
& C, from unknown states. Which states is the mystery which
students must solve by finding clues. If you begin on the left side and
enter $100 for the Purchase Price (our educated choice, but is it
obvious to everyone?), and then choose, in succession, Calculator A →
Find Sales Tax → Calculator B → Find Sales Tax → Calculator C → Find
Sales Tax, the answers 6.25, 5.00 and 6.88 appear in the cyan colored
table next to the corresponding calculator.

Then
you have to go to Part 2 and realize ``Calculator A’’ is a
click-and-draggable object that goes in the empty box to the right side
of Illinois (because $6.25 sales tax is 6.250% of $100). Calculator B
goes next to Maine, Calculator C next to Minnesota (rounding!) and the
remainder of the states get nothing. Mystery solved!

Wow, what a lot of words, instructions, and underlying code for only one question.

***

How
many lengthy-coded and carefully-vetted assessment questions will be in
PARCC’s and SBAC’s databases? Enough that students won’t encounter the
same questions repeatedly or be able to study beforehand questions that
their peers have already seen and can tell them about? That requires
an awfully big repository, each task with its own interface and
underlying code, any single question of which can stymie students or
crash the whole system during an assessment.

I have done some of the SBAC questions and OMG I know the correct answer, but struggle to figure out how I am able to display the answer. Very convoluted and I can just see my 7th graders trying to figure things out.

ReplyDeleteExcellent work to put this up. You have shown that these are interface tests not math tests. And the test makers failed.

ReplyDeleteThis makes me want to cry. What about visually impaired students. Dslexia will kill you on these math tests. Can teachers help with the interface? What about children who have problems using PCs in general? Dear god, just remembered that their will be time limits on these. It should not take longer to answer the question then to solve the question.

ReplyDeleteWhat ever happened to Solve this equation. Show your work. That way if there is a transcription error or arithmetic issue when solving a quadratic equation you will not get a zero.

Sobs

The real answer is of course a written assessment with human graders but, while everyone has a desire to know what students know, no one has enough respect for students to personally sit down and give the students' work the attention it deserves.

ReplyDeleteWhat has this to do with students' capacity to create or to make and defend decisions? Where's that going to be on the test and who will actually give students' words and arguments the respect that kind of work demands? I will in my classroom, but will the bureaucrats? will the computers? will the test publishers?

School is the place where students should learn literacy, numeracy, reasoning, etiquette, and professionalism even as they discover what they do best, and then it should be the place where students learn to be the best they can at what they do best, and once the student has access to a better deal -- a job, an apprenticeship, a college admission -- school should recede into fond and grateful memories.

This isn't any of that. This is how we "lose" students. The students can't stand or understand it and we teachers can't and shouldn't defend it.

Having spent 25 years in the Mac user interface design community (which also means serving as a tester for much imperfect software along the way), I can promise that just the few catastrophes you have identified here will bring Smarter Balance and PARCC to its knees.

ReplyDeleteLooking ambivalently forward to seeing the &**@^% that ensues.

- Elizabeth (@cheesemonkeysf)

For the first problem there is

ReplyDelete(3x-2)(3x-2)=6x-4

(3x-2)(3x-2)=2(3x-2)

Dividing both sides of the equation by (3x-2)

3x-2=2

3x-2+2=2+2

3x=4

3x/3=4/3

x=4/3

What is the other value of "X?"

If 3x-2=0, then your first two equations are clearly satisfied, but then you divided both sides by zero and lost that solution. Divide with care!

DeleteSo to be elear, the two solutions are the solutions of

Delete3x-2 = 2 and

3x-2 = 0 (if you check, this converts the original equation to 0=0 which is an equality.)

So the two solutions are x = 4/3 and x = 2/3 . I am not too worried about the student starting out knowing a fraction is involved -- even the quadratic formula is in the form of a fraction, if the student doesn't see the division by (3x-2), expands the whole mess out, applies the quadratic formula (I guess they haven't learned that in basic HS math but it is one way to solve the problem, the slow way).

I think it's a fine question, nothing wrong with it. As long as the interface works reliably and the students learn to use it effectively. This testing interface looks like a mess now, but in a year or two it might be pretty slick, and then it's usable forever.

Frankly, one thing I like is that it's not about justifying your answer, creating making and defending decisions, etc. It's math, which is what there should be in math class. And to heck with any Common Core ideas that it should be diluted with a bunch of other stuff.

What a mess we have on our hands, this information must and will be passed on to all educators that I know. We are in the process of being trained on Common Core and have not been given a precise answer about the assessments. Now I know why. What a waste of our hard earned middle class dollars!

ReplyDeleteThe only way for children to have a chance to pass such exams is for them to spend a minimum of two, but more likely more than four, hours per day sitting at a computer learning to deal with the interface while trying to think of the answers. There will be no time left in the school day for lessons involving labs, divergent thinking, and creative thinking. High school students in my district participated in the SB pilot this spring. No pencil nor paper was allowed. The students could go back and review documents and videos (Some students were attempting to do this with a 3" screen on a phone!) but could make no notes; they had to hold everything in their heads. Content may have been tested, but only after a fatiguing (3-4 hours) trial of working memory.

ReplyDeleteAre we all just a bunch of idiots to subject ourselves to this? How long will it take to drive a nation to lunacy?

ReplyDeleteThis is truly astonishing, I can not believe that a rolled-out system can be this bad. According to the literature, these initiatives are supposed to make our kids more competitive in the world marketplace. If anything, it appears fiddling around with these widgets will result in a lot of agitated students, frustrated teachers, and administrators not getting their promised bonuses. Perhaps the Washington D.C. proponents of this monstrosity are shareholders in the computer company that will be paid and paid again for upgrades (e.g. fixes). Thank you for posting this, but it made me sad for the children.

ReplyDeleteThis is not designed to measure anything useful. We had perfectly good assessments already, ones that measured student achievement and understanding without being massively frustrating and bug-ridden. This is the Progressive need to fiddle with everything in order to make it impossible for individuals to understand or function without "experts" telling them what is going on. It's about taking control over education away from local administrations and teachers and depositing it in the hands of distant, unaccountable bureaucrats. Students will be handicapped by these new "standards" and be unable to compete against anybody. The whole CCSSI is a Rube Goldberg contraption, set up by people less interested in education than in making money and holding on to power.

ReplyDelete