Computer Grading?

Robert Yagelowich pointed this article out to me, and like him what I read initially made me pretty skeptical. The article describes a computer program that's being used to grade student essays. Not just grade the spelling and punctuation, but the content itself. Since computers can't even be relied on to spellcheck very well, I couldn't imagine how they would grade content. I had suspicions of another ChatNannies type of hoax. But apparently computer-grading is real. The software, called E-Rater, has been developed by Educational Testing Services, and they provide an online demo of how it works. I used to be a teaching assistant at UC San Diego, and I graded thousands of student essays. And I have to admit that human graders are often pretty fallible. By the time you get to the bottom of a stack of essays, you're just going through those things as fast as possible, barely reading them. So maybe a computer could grade essays better than a human. The computer, at least, wouldn't grade differently depending on how much coffee it had drunk, or how little sleep it had got.

Technology

Posted on Thu May 20, 2004


Comments

I'm wondering if the 'demo' is a hoax... there's nothing in it that actually 'demos' the product.
Posted by Bob S. Yunkle  on  Fri May 21, 2004  at  10:42 AM
Good point Bob

the buttons just use javascript to open html pages with the "results"
Posted by Peter  on  Fri May 21, 2004  at  02:29 PM
A lot of employers use similar programs to evaluate the resumes of prospective employers, so I guess it's not implausible. A lot of profs grade essays based on a "hit list," anyway--checking off mentions of key concepts with little regard to composition or original insight.
Posted by James Joyner  on  Sun May 23, 2004  at  08:28 AM
I agree with Bob that there is a failure to demonstration the product. The fact that the product only seems to require the topic of the essay to judge quality is unsettling since there is no way that a program can derive meaning from all possible contexts and language patterns. What is shown by the "demo" isn't impressive anyway. The topic as well as the work itself was simplistic. The authors of the essays used terms from the topic--"what they want to do" and "what they feel they should do" in almost every instance. That's standardized testing style writing--poor and repetitive. I doubt this program, if it even exists, could grade anything past a 5th grade end of year state exam (and manage to fail the most creative students with the most elegant prose while it's at it).
Posted by Mary  on  Fri Jul 02, 2004  at  08:36 AM
Commenting is not available in this channel entry.