Robert Yagelowich pointed this article out to me, and like him what I read initially made me pretty skeptical. The article describes a computer program that's being used to grade student essays. Not just grade the spelling and punctuation, but the content itself. Since computers can't even be relied on to spellcheck very well, I couldn't imagine how they would grade content. I had suspicions of another ChatNannies type of hoax. But apparently computer-grading is real. The software, called E-Rater, has been developed by Educational Testing Services, and they provide an online demo of how it works. I used to be a teaching assistant at UC San Diego, and I graded thousands of student essays. And I have to admit that human graders are often pretty fallible. By the time you get to the bottom of a stack of essays, you're just going through those things as fast as possible, barely reading them. So maybe a computer could grade essays better than a human. The computer, at least, wouldn't grade differently depending on how much coffee it had drunk, or how little sleep it had got.
I'm wondering if the 'demo' is a hoax... there's nothing in it that actually 'demos' the product.Posted by Bob S. Yunkle on Fri May 21, 2004 at 12:42 PM
Good point BobPosted by Peter in London on Fri May 21, 2004 at 04:29 PM
A lot of employers use similar programs to evaluate the resumes of prospective employers, so I guess it's not implausible. A lot of profs grade essays based on a "hit list," anyway--checking off mentions of key concepts with little regard to composition or original insight.Posted by James Joyner on Sun May 23, 2004 at 10:28 AM
I agree with Bob that there is a failure to demonstration the product. The fact that the product only seems to require the topic of the essay to judge quality is unsettling since there is no way that a program can derive meaning from all possible contexts and language patterns. What is shown by the "demo" isn't impressive anyway. The topic as well as the work itself was simplistic. The authors of the essays used terms from the topic--"what they want to do" and "what they feel they should do" in almost every instance. That's standardized testing style writing--poor and repetitive. I doubt this program, if it even exists, could grade anything past a 5th grade end of year state exam (and manage to fail the most creative students with the most elegant prose while it's at it).Posted by Mary on Fri Jul 02, 2004 at 10:36 AM
Commenting is no longer available in this channel entry.