Dr. Román Mendoza and Prof. Rubio Sánchez (Universidad de Burgos) presented findings from a broad empirical study evaluating feedback provided by generative artificial intelligence, its acceptance by L2 teachers, and its usefulness for autonomous learning at the conference Accessing Written Information: Aiming for Readability organized in Salamanca (Spain). In this first phase, the reliability and readability of recommendations generated by ChatGPT-4 for improving the written output of Spanish as a Second Language (L2) students were analyzed.
To conduct this initial study, three GPT models were developed to identify and annotate spelling, lexical, and grammatical errors and provide feedback in Spanish and English. These models were applied to 50 beginner-level and 17 high-intermediate-level Spanish texts (around 7,500 words) written by English-speaking learners from the Written Corpus of Spanish as a Second Language (CEDEL2). The suggestions were analyzed to assess their validity, readability, and systematicity using adapted methodologies from previous studies on automated writing assessment and automated writing feedback (cf. Hassanzadeh & Fotoohnejad, 2021; Shi & Aryadoust, 2024).
Initial findings indicate that the latest versions of ChatGPT detect spelling and lexical errors with higher precision compared to previous iterations and support error labeling. An improvement in grammatical error detection was also observed, though longer texts remain challenging. Regarding the systematicity and readability of the feedback, it was found that, unlike feedback from software packages for automated writing evaluation, the variety of feedback provided by ChatGPT can enhance student engagement but may also reduce accessibility, particularly for beginner-level learners.
For more information about this study or to access the developed GPT models, please contact Prof. Román Mendoza.
November 09, 2024