From Human Raters to AI: A Critical Review of Research on ChatGPT in Writing Assessment
DOI:
https://doi.org/10.57125/FED.2026.06.12Keywords:
ChatGPT, assessment, writing, internal consist, inter-rater correlationAbstract
ChatGPT holds great promise for assessing student writing, an area greatly in need of research and exploration. The purpose of the current paper is to synthesize studies that used ChatGPT to assess student writing, focusing on the following questions: (1) What is the internal consistency of ChatGPT in writing assessment? (2) How does ChatGPT assessment performance compare with that of human raters? and (3) What factors influence ChatGPT performance in assessing student writing? The findings of the paper provide important insights on accuracy, consistency, and efficacy of ChatGPT as a tool for writing assessment. This research is significant and timely as educational institutions continue to seek a pedagogically valuable and cost-effective method for assessing student writing.
References
Ariyanto, M. S. A., Mukminatien, N., & Tresnadewi, S. (2021). College students’ perceptions of an automated writing evaluation as a supplementary feedback tool in a writing class. Jurnal Ilmu Pendidikan, 27(1), 41.
Arnold, K. M., Umanath, S., Thio, K., Reilly, W. B., McDaniel, M. A., & Marsh, E. J. (2017). Understanding the cognitive processes involved in writing to learn. Journal of Experimental Psychology: Applied, 23(2), 115.
Attali, Y. (2004). Exploring the feedback and revision features of Criterion. Journal of Second Language Writing, 14(3), 1-20.
Awidi, I. T. (2024). Comparing expert tutor evaluation of reflective essays with marking by generative artificial intelligence (AI) tool. Computers and Education: Artificial Intelligence, 6, 100226.
Bouziane, K., & Bouziane, A. (2024). Exploring the role of AI in essay evaluation: A comparative analysis of ChatGPT and human corrections. Research Square. https://doi.org/10.21203/rs.3.rs-4139088/v1
Bucol, J. L., & Sangkawong, N. (2024). Exploring ChatGPT as a writing assessment tool. Innovations in Education and Teaching International, 1-16.
Bui, N. M., & Barrot, J. S. (2025). ChatGPT as an automated essay scoring tool in the writing classrooms: How it compares with human scoring. Education and Information Technologies, 30(2),2041-2058.
Burstein, J. (2003). The E-rater® scoring engine: Automated essay scoring with natural language processing. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 113– 121). Routledge. https://doi.org/10.4324/9781410606860
Chen, Chi-Fen, & Cheng, Wei-Yuan. (2008). Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in EFL writing classes. Language Learning and Technology, 12(2), 94–112.
Cheville, J. (2004). Automated scoring technologies and the rising influence of error. English Journal, 93(4), 47-52.
Correnti, R., Matsumura, L. C., Wang, E. L., Litman, D., & Zhang, H. (2022). Building a validity argument for an automated writing evaluation system (eRevise) as a formative assessment. Computers and Education Open, 3, 100084.
Cotos, E. (2011). Potential of automated writing evaluation feedback. Calico Journal, 28(2), 420-459.
Deane, P. (2022). The importance of assessing student writing and improving writing instruction. Research notes. Educational Testing Service.
Dockrell, J. E., & Connelly, V. (2021). Capturing the challenges in assessing writing. Executive Functions and Writing, 103.
Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023). Automated feedback and writing: A multi-level meta-analysis of effects on students' performance. Frontiers in Artificial Intelligence, 6, 1162454.
Fokides, E., & Peristeraki, E. (2025). Comparing ChatGPT's correction and feedback comments with that of educators in the context of primary students' short essays written in English and Greek. Education and Information Technologies, 30(2), 2577-2621.
Gao, J. (2021). Exploring the feedback quality of an automated writing evaluation system pigai. International Journal of Emerging Technologies in Learning (iJET), 16(11), 322-330.
Geçkin, V., Kızıltaş, E., & Çınar, Ç. (2023). Assessing second-language academic writing: AI vs. Human raters. Journal of Educational Technology and Online Learning, 6(4), 1096-1108.
Graham, S. (2019). Changing how writing is taught. Review of Research in Education, 43(1), 277-303.
Grimes, D., &Warschauer, M. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. Journal of Technology, Learning, and Assessment, 8(6), 4–44
Hahn, M. G., Navarro, S. M. B., Valentín, L. D. L. F., & Burgos, D. (2021). A systematic review of the effects of automatic scoring and automatic feedback in educational settings. IEEE Access, 9, 108190-108198.
Hayes, J. R. (2012). Modeling and remodeling writing. Written Communication, 29(3), 369-388.
Hayes, J. R., & Flower, L. S. (1983). Uncovering cognitive processes in writing: An introduction to protocol analysis. In P. Mosenthal, L. Tamor, & S. A. Walmsley (Eds.), Research on writing: Principles and methods (pp. 207-220). New York: Longman.
Hessler, T., Konrad, M., & Alber-Morgan, S. (2009). Assess student writing. Intervention in School and Clinic, 45(1), 68-71.
Horvath, B. K. (1984). The components of written response: A practical synthesis of current views. Rhetoric Review, 2(2), 136-156.
Hussein, M. A., Hassan, H., & Nassef, M. (2019). Automated language essay scoring systems: A literature review. PeerJ Computer Science, 5, e208.
Jackaria, P. M., Hajan, B. H., & Mastul, A. R. H. (2024). A comparative analysis of the rating of college students’ essays by ChatGPT versus human raters. International Journal of Learning, Teaching and Educational Research, 23(2), 478-492.
Judy, S. N. (1973). Writing for the here and now: An approach to assessing student writing. English Journal, 62(1), 69-79.
Kim, H., Baghestani, Sh., Yin, Sh., Karatay, Y., Kurt, S., Beck, J., & Karatay, L. (2024). ChatGPT for writing evaluation: Examining the accuracy and reliability of AI-generated scores compared to human raters. In C. A. Chapelle, G. H. Beckett, & J. Ranalli (Eds.), Exploring artificial intelligence in applied linguistics (pp. 73–95). Iowa State University Digital Press. https://doi. org/10.31274/isudp.2024.154.06
Kukich, K. (2000). Beyond automated essay scoring, the debate on automated essay grading. IEEE intelligent systems, 15(5), 22-27.
Landauer, T. K., Laham, D., & Foltz, P. D. (2003). Automated scoring and annotation of essays with the Intelligent Essay Assessor. In Shermis, M.D. and Burstein, J.C. (eds), Automated essay scoring: A cross-disciplinary perspective, (pp. 87-112). Lawrence Erlbaum Associates.
Lee, Y. J. (2020). The long-term effect of automated writing evaluation feedback on writing development. English Teaching, 75(1), 67-92.
Li, J., Huang, J., Wu, W., & Whipple, P. B. (2024). Evaluating the role of ChatGPT in enhancing EFL writing assessments in classroom settings: A preliminary investigation. Humanities and Social Sciences Communications, 11(1), 1-9.
Li, J., Jangamreddy, N. K., Hisamoto, R., Bhansali, R., Dyda, A., Zaphir, L., & Glencross, M. (2024). AI-assisted marking: Functionality and limitations of ChatGPT in written assessment evaluation. Australasian Journal of Educational Technology, 40(4), 56-72.
Lim, C. T., Bong, C. H., Wong, W. S., & Lee, N. K. (2021). A comprehensive review of automated essay scoring (AES) research and development. Pertanika Journal of Science & Technology, 29(3), 1875-1899.
Link, S., Mehrzad, M., & Rahimi, M. (2022). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning, 35(4), 605-634.
Lu, Q., Yao, Y., Xiao, L., Yuan, M., Wang, J., & Zhu, X. (2024). Can ChatGPT effectively complement teacher assessment of undergraduate students’ academic writing? Assessment & Evaluation in Higher Education, 49(5), 616–633. doi.org/10.1080/02602938.2024.2301722
McNamara, D. S., Crossley, S. A., Roscoe, R. D., Allen, L. K., & Dai, J. (2015). A hierarchical classification approach to automated essay scoring. Assessing Writing, 23, 35-59.
Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 100050.
Mizumoto, A., Shintani, N., Sasaki, M., & Teng, M. F. (2024). Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment. Research Methods in Applied Linguistics, 3(2), 100116.
Neff Lippman, J. (2003). Assessing writing. In I. L. Clark (Ed.), Concepts in composition: Theory and practice in the teaching of writing (pp. 199-240). Mahwah, New Jersey: Lawrence Erlbaum Associates.
Page, E.B. (2003). Project essay grade: PEG. In Shermis, M.D. and Burstein, J.C. (eds), Automated essay scoring: A cross-disciplinary perspective. Lawrence Erlbaum, 43–54.
Parra G, L., & Calero S, X. (2019). Automated writing evaluation tools in the improvement of the writing skill. International Journal of Instruction, 12(2), 209-226.
Pati, D., & Lorusso, L. N. (2018). How to write a systematic review of the literature. HERD: Health Environments Research & Design Journal, 11(1), 15-30.
Pfau, A., Polio, C., & Xu, Y. (2023). Exploring the potential of ChatGPT in assessing L2 writing accuracy for research purposes. Research Methods in Applied Linguistics, 2(3), 100083.
Quah, B., Zheng, L., Sng, T. J. H., Yong, C. W., & Islam, I. (2024). Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations. BMC Medical Education, 24(1), 962. doi.org/10.1186/s12909-024-05881-6
Palermo, C., & Wilson, J. (2020). Implementing automated writing evaluation in different instructional contexts: A mixed-methods study. Journal of Writing Research, 12(1), 63-108.
Ramesh, D., & Sanampudi, S. K. (2022). An automated essay scoring system: A systematic literature review. Artificial Intelligence Review, 55(3), 2495-2527.
Rudner, L. M., & Gagne, P. (2001). An overview of three approaches to scoring written essays by computer. Practical Assessment, Research, and Evaluation, 7(26),1-6.
Scardamalia, M., & Bereiter, C. (1991). Literate expertise. In K. A. Ericsson & J. Smith (Eds.), Toward a general theory of expertise (pp. 172–194). Cambridge, UK: Cambridge University Press.
Shabara, R., ElEbyary, K., & Boraie, D. (2024). Teachers or ChatGPT: The issue of accuracy and consistency in L2 assessment. Teaching English with Technology, 24(2), 71-92.
Shermis, M. (2024). Using ChatGPT to score essays and short-form constructed responses. doi.org/10.48550/arXiv.2408.09540.
Shermis, M. D., & Burstein, J. C (Eds.). (2003). Automatic Essay Scoring: A Cross Disciplinary Perspective. Mahwah, NJ: Erlbaum.
Shermis, M. D., Rasmussen, J. L., Rajecki, D. W., Olsen, J., & Marsiglio, C. (2001). All prompts are created equal, but some prompts are more equal than other prompts. Journal of Applied Measurement, 2(2), 154-170.
Shin, D., & Lee, J. H. (2024). Exploratory study on the potential of ChatGPT as a rater of second language writing. Education and Information Technologies, 1-23.
Smith, D. A. (2017). Collaborative peer feedback. International Association for Development of the Information Society.
Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., ... & Olson, C. B. (2024). Comparing the quality of human and ChatGPT feedback of students’ writing. Learning and Instruction, 91, 101894.
Stevenson, M. (2016). A critical interpretative synthesis: The integration of automated writing evaluation into classroom writing instruction. Computers and Composition, 42, 1-16.
Tang, J., & Rich, C. S. (2017). Automated writing evaluation in an EFL setting: Lessons from China. The JALT CALL Journal, 13(2), 117-146.
Uymaz, E. (2019). The effects of peer feedback on the essay writing performances of EFL students. International Journal of Curriculum and Instruction, 11(2), 20-37.
Valenti, S., Neri, F., & Cucchiarelli, A. (2003). An overview of current research on automated essay grading. Journal of Information Technology Education: Research, 2(1), 319-330.
Wang, P. L. (2015). Effects of an automated writing evaluation program: Student experiences and perceptions. Electronic Journal of Foreign Language Teaching, 12(1).
Wang, Y. J., Shang, H. F., & Briody, P. (2013). Exploring the impact of using automated writing evaluation in English as a foreign language university students' writing. Computer Assisted Language Learning, 26(3), 234-257.
Warschauer, M., & Grimes, D. (2008). Automated writing assessment in the classroom. Pedagogies, 3, 22–36.
Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research 10(2), 1–24.
Weigle, S. C. (2002) Assessing writing. Cambridge: Cambridge University Press.
Wilson, J., Ahrendt, C., Fudge, E. A., Raiche, A., Beard, G., & MacArthur, C. (2021). Elementary teachers’ perceptions of automated feedback and automated scoring: Transforming the teaching and learning of writing using automated writing evaluation. Computers & Education, 168, 104208.
Wilson, J., & Roscoe, R. D. (2020). Automated writing evaluation and feedback: Multiple metrics of efficacy. Journal of Educational Computing Research, 58(1), 87-125.
Yang, Y. (2024). The reliability of using ChatGPT in rating EFL writings. Shanlax International Journal of Education, 12(4), 49-59.
Yoon, S. Y., Miszoglad, E., & Pierce, L. R. (2023). Evaluation of ChatGPT Feedback on ELL Writers' Coherence and Cohesion. arXiv preprint arXiv:2310.06505.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 authors
This work is licensed under a Creative Commons Attribution 4.0 International License.
