The Android English Teacher: Writing Education in the Age of Automation

In an era of widespread automation—from grocery store self-checkout machines to self-driving cars—it is not outrageous to wonder: can teachers be automated? And more specifically, can automated computer teachers instruct students how to write? Automated computer programs have long been used in summative writing evaluation efforts, such as scoring standardized essay exams, ranking placement essays, or facilitating programmatic outcomes assessments. However, new claims about automated writing evaluation’s (AWE) formative educational potential mark a significant shift. My project questions the effectiveness of using AWE technology for formative educational efforts such as improving and teaching writing. Taken seriously, these efforts portend a future embrace of semi, or even fully, automated writing classes, an unprecedented development in writing pedagogy.

Supported by a summer-long grant from the Purdue Research Foundation, I conducted a small-n quasi-experiment to test claims by online college tutoring site Chegg.com that its EasyBib Plus AWE tool can improve both writing and writers. The experiment involved four college English instructors reading pairs of essays comprising one AWE-treated and untreated version per pair. Using a comparative judgment model, a rubric-free method of writing assessment based on Thurstone’s law, raters read and designated one of each pair “better.” Across four raters and 160 essays, I found that AWE-treated essays were designated better only 30% of the time (95% confidence interval: 20-40%), a statistically significant difference from the null hypothesis of 50%. The results suggest that Chegg’s EasyBib Plus tool offers no discernible improvement to student writing, and potentially even worsens it.

Finally, I analyze Chegg’s recent partnership with the Purdue Writing Lab and Online Writing Lab (OWL). The Purdue-Chegg partnership offers a useful test case for anticipating the effects of higher education’s embrace of automated educational technology going forward. Drawing on the history of writing assessment and the results of the experiment, I argue against using AWE for formative writing instruction. In an era of growing automation, I maintain that a human-centered pedagogy remains one of the most durable, important, effective, and transformative ingredients of a quality education.