When ChatGPT burst onto the scene in November 2022, many educators and parents worried that new writing tools powered by artificial intelligence (AI) would help their students bypass important learning opportunities. Instead, as University of Delaware associate professor Joshua Wilson has shown, AI-powered writing and evaluation tools have actually helped students develop their writing skills and have supported teachers in providing meaningful feedback.

Now, in a recent study published in Learning and Instruction, Wilson and his co-authors turn their attention to elementary English learners (EL), investigating how this growing population of students interacts with and benefits from automated writing evaluation (AWE) software. They found that AWE technologies are equally beneficial for ELs as they are for non-ELs. Study participants accessed writing feedback to a similar extent, achieved equal gains in writing quality, focused on consistent dimensions of writing when revising and endorsed the AWE system to similar degrees, regardless of their language status.

"As AI-based feedback applications become increasingly prevalent, it's critical that researchers examine the consequences of implementing those tools in authentic educational settings, with a particular focus on equity," said Wilson, who specializes in literacy in UD's College of Education and Human Development (CEHD). "This study represents a novel step forward in the field of AWE by focusing on students' multifaceted engagement and by ensuring that there were not systematic differences in engagement that might disadvantage vulnerable subgroups. This approach sets a precedent for other investigations into AI-based feedback applications, ensuring that these technologies support equitable learning outcomes for all students."

Automated essay evaluation software

AWE refers to a class of educational technology tools that use natural language processing and AI to provide students with automated formative feedback that supports improvements in writing quality. Wilson's study focuses on MI Write, an AWE system designed to improve the teaching and learning of writing by providing students with automated feedback and writing scores.

When a student drafts an essay and submits it within MI Write, their essay is instantly analyzed by its scoring and feedback algorithms, which then deliver immediate automated feedback directly to the student. In addition to providing the student with a holistic writing score, it also provides a specific writing score and feedback on idea development, organization, style, sentence fluency, word choice and conventions. MI Write also includes peer review capabilities, offers multimedia lessons for skill-building and allows teachers to communicate with students through commenting features.

"[The students] seem more determined and [MI Write is] so catered to them," said a fourth grade teacher in Wilson's study. "It's almost like there's a person, like somebody [teaching] them, conferencing with them, telling them how they can improve, and it's all of them at once. Whereas before, I wouldn't be physically able to accomplish that quickly."

Benefits for English learners

To assess how ELs interacted with and benefited from AWE technology, Wilson and his co-authors collected data from nearly 3,500 students in grades 3-5 within a Mid-Atlantic district that implemented MI Write in all 14 of its elementary schools during the 2017-18 school year. They collected data from ELs - students whose home language is not English and who qualify for English language services - and from non-ELs.

To investigate interaction with the AWE software, Wilson and his co-authors looked at three dimensions of engagement: behavioral, or the actions students take in response to feedback; cognitive, or the thinking and revision strategies that students use in response to feedback; and affective, or how students feel about and perceive feedback.

Across all three dimensions, Wilson and his co-authors found similar levels of engagement across all students. For example, some students chose not to access the feedback that AWE provided, while others accessed it several times. But these differences in behavioral engagement were not associated with language status. Similarly, ELs made productive revisions to their texts to the same extent as non-ELs and they often focused on the same set of textual features when revising.