Measuring the Expressive Language and Vocabulary of Latino English Learners Using Hand Transcribed Speech Data and Automated Scoring

Description

The purpose of this study was to explore how different automated scoring (AS) models score the expressive language and vocabulary knowledge of second-grade Latino English learners. The authors analyzed 13,471 English utterances from 217 Latino English learners with random forest, end-to-end memory networks, long short-term memory, and other AS models. Random forest outperformed the other AS models, as measured by the mean of quadratic weighted kappa (QWK = 0.70), followed by the end-to-end memory networks, and long short-term memory (QWK = 0.69) across all tasks and data points. The QWK between humans was 0.90, and the human-machine agreement of three AS models and humans ranged from 0.66 to 0.70. Practical implications include examining misclassifications between human and machine scoring to better understand the specific words and structures the systems did not capture. The authors discuss findings in the context of developing efficient and reliable ways to analyze the natural speech of young English learners. This information could guide vocabulary and language proficiency instruction in the early grades.

Citation