Analyzing the Grammar Feedback of Two Automated Writing Evaluation Systems

Abstract

英語為第二語言的學生須要加強寫作能力以通過各式語言能力測驗。為了減輕老師的負擔與提供學生立即的回饋，市場上已有幾個商業化的寫作自動評量系統可供選擇。雖然這些創新系統很受歡迎，但針對系統提供的回饋進行檢測的實證研究卻很少。由於越來越多人使用這些系統，瞭解系統優缺點的需求也與日俱增。本文將檢驗兩個著名的自動寫作評量系統Vantage My Access 與 ETS Criterion 所提供的文法回饋。本文主要目的在於發現此二系統文法回饋的優點與缺點。首先，我們先收集此二系統針對269篇學生文章所提供的文法錯誤回饋並加以分析。結果顯示此二系統提供各約30種不同的回饋訊息。然而，My Access 提供了過多的錯誤回饋。Criterion 的表現較好，但它仍然無法偵測一些學生的常見錯誤，像是詞序、語氣、時態、搭配詞、連接詞、選字、與代名詞等。很明顯的此二系統對於英語為第二語言的學生無法提供令人滿意的協助。欲使用此類寫作自動評量系統的老師與學生須留意它們的不足之處與可能遇到的問題。

ESL/EFL students need to enhance writing skills to pass various language tests. To help teachers reduce their teaching load and to give students faster feedback, several commercial AWE (automated writing evaluation) tools are available. Although these innovative tools have become popular, few empirical studies have examined the quality of feedback generated by these systems. There is a pressing need to better understand the strengths and weaknesses of these widely used AWE systems. In this paper, we examined the grammar feedback provided by two well-known systems, Vantage My Access and ETS Criterion. The purpose of this paper is to uncover the strengths and weaknesses of the feedback provided by these two systems. Automated feedback messages on 269 student essays were collected and analyzed. The results showed that these two systems can provide about 30 different types of feedback messages. However, My Access provided many false alarm messages. Criterion performed much better than My Access, but it still failed to deal with several common errors related to word order, modals, tenses, collocations, conjuncts, word choice, and pronouns. It is clear that the two AWE systems do not provide satisfactory help to ESL students. Teachers and students who use these new tools should be aware of these problems and limitations.