“It’s a beautiful thing, the destruction of words,” George Orwell, “1984.”
George Orwell’s “1984” is a cautionary tale — the story of a dystopian society where thought was lost forever. Largely due to its literary subtleties and message that expression is critical to individualism, human readers widely regard “1984” as a masterpiece. But if robots had graded Orwell’s classic work, it probably wouldn’t have done very well.
Let’s back up. Robo graders may sound fantastical, like something out of “1984,” but Automated Essay Scoring (AES) software is here today and could be grading your work in the near future.
AES is a type of grading software that uses text analysis to score essays from one to six based on predetermined criteria that include sentence structure, word length and formatting technique. Unfortunately AES, although effective at grading for formating technicalities such as word length, paragraph length and sentence structure, is incapable of grading for content and doesn’t score analytical thinking, accuracy, themes or symbols. Technology that ignores the meaning and purpose of written language has no place in or near our classrooms.
The idea of AES, which dates back to 1966, is nothing new, but support for AES is at an all time high as new curriculum standards are implemented in Massachusetts and across the US during the 2013-2014 school year. Looking to capitalize on the desire for fast and inexpensive grading, about a half-dozen companies are competing for government contracts to sell their automated grading software according to “Salon.”
As we move towards new, federal curriculum standards known as the Common Core, the possibility of more standardized tests has increased the allure surrounding cheap and fast grading. PARCC, the new Common Core standardized test replacing MCAS, will come to Wayland High School in 2015, and AES could come with it.
AES is indeed fast and cheap — some AES software is capable of grading 16,000 essays in as little as 20 seconds according to “The New York Times.” A nationwide transition to AES would save the time of our already overworked educators, lower costs to test takers and taxpayers and speed up test return time. But fast and cheap don’t always mean better, and although AES is good at grading massive quantities, it’s incapable of grading for quality.
AES can’t understand the meaning of words; according to “The New York Times” it grades entirely based on formatting technicalities such as average word, sentence, paragraph and essay length. Because AES grades for structure but not for content, it is easy to fool. As long as an essay has big words and long paragraphs, it’ll do well when graded by AES.
Les Perelman, director of the Writing Across the Curriculum program at MIT, has discovered just how easy it is to trick AES. Perelman wrote an essay critical of high college tuition, passionately asserting that greedy, private jet owning teaching assistants are to blame; his essay received a perfect score. As Perelman noted in a critique of a pro-AES study, “Human graders may be significantly more reliable than machines.” For graders to score effectively, they must understand the tested material. AES understands nothing of the sort.
Unreliability is bad, but AES’s problems run far deeper. Generally AES is pretty good at grading for format but content, which AES is incapable of understanding, is far more important. AES has no problem with an essay that says that aliens were the primary cause of the Great Depression; it can’t be bothered with facts. The purpose of writing is to communicate effectively, not to use words like “circumlocution” and “discursiveness,” but AES doesn’t care what ideas are expressed, only how they are technically written. To accept AES’s grading criteria is to accept that the lengths of words and sentences are more important than their meanings.
When creating a standardized test, it’s important to consider that many teachers, under great pressure to prove their students are learning, will teach to the test. Schools implement entire curricula focused on preparing students to succeed on test day. When teachers teach to the test, they don’t want to “waste” time going over topics not covered on the test. Since AES doesn’t grade for content, why bother teaching it? After all, the test scores wouldn’t reflect strong content. AES-graded tests, instead of rewarding good analysis, favor big words, long paragraphs and complex sentences.
Practicing for AES-graded tests would create homogeneous writers by encouraging curricula that teach students how to spell and format but not how to voice their ideas. If AES were to be universally adopted, the humanity of our writing would be hit at its very core — at schools where kids learn to express themselves.
Perhaps AES can play a useful role in our education system by giving students format-based feedback on rough drafts in place of or in addition to peer editing, but AES clearly does not belong in the test grading business. Students study for tests, and no one should practice writing the contentless shells of essays that AES loves. Because these computers can’t yet grade ideas, we must leave the interpretation of the humanities to humans.
As it turns out, the destruction of words isn’t so beautiful. Maybe, just maybe, Orwell was being satirical when he said otherwise. Too bad AES can’t score based on irony.
Curious • Feb 9, 2014 at 6:55 PM
Was the school planning to implement AES in any aspect of grading?
benporterwspn • Feb 9, 2014 at 7:38 PM
The scariest part of this is that it really isn't up to the school. If the federally mandated PARCC test adopts AES then, whether the school likes it or not, AES will be grading WHS student essays.
ThereBeDragons • Feb 9, 2014 at 8:21 PM
I feel like this could work in the students favor. You could learn what the machine is looking for in an essay, and just write an essay full of gibberish that the machine will only check for grammar, length, key words, etc…If there's a will, there's a way.
guest • Feb 9, 2014 at 8:24 PM
Only downside is then you don't learn how to write but who needs that when you get an A??
guest^2 • Feb 9, 2014 at 8:45 PM
I mean the grade is all that really matters lets be real