Microsoft Reading Progress as Capt Tool

Marek Molenda; Izabela Grabarczyk

doi:10.18778/1731-7533.20.2.05

Authors

Marek Molenda University of Łódź, Poland
Izabela Grabarczyk University of Łódź, Poland

DOI:

https://doi.org/10.18778/1731-7533.20.2.05

Keywords:

CAPT, EFL, ASR

Abstract

The paper explores the accuracy of feedback provided to non-native learners of English by a pronunciation module included in Microsoft Reading Progress. We compared pronunciation assessment offered by Reading Progress against two university pronunciation teachers. Recordings from students of English who aim for native-like pronunciation were assessed independently by Reading Progress and the human raters. The output was standardized as negative binary feedback assigned to orthographic words, which matches the Microsoft format.
Our results indicate that Reading Progress is not yet ready to be used as a CAPT tool. Inter-rater reliability analysis showed a moderate level of agreement for all raters and a good level of agreement upon eliminating feedback from Reading Progress. Meanwhile, the qualitative analysis revealed certain problems, notably false positives, i.e., words pronounced within the boundaries of academic pronunciation standards, but still marked as incorrect by the digital rater.
We recommend that EFL teachers and researchers approach the current version of Reading Progress with caution, especially as regards automated feedback. However, its design may still be useful for manual feedback. Given Microsoft declarations that Reading Progress would be developed to include more accents, it has the potential to evolve into a fully-functional CAPT tool for EFL pedagogy and research.

References

Bajorek, Joan. 2017. L2 pronunciation tools: The unrealized potential of prominent computer-assisted language learning software, Issues and Trends in Educational Technology, 5(2). DOI: https://doi.org/10.2458/azu_itet_v5i1_bajorek

Baker, Ann. 2007. Ship or sheep? An intermediate pronunciation course (3rd ed.). Cambridge University Press.

Bernstein, Jared, Cohen, Michael, Murveit, Hy, Rtischev, Dimitry and Weintraub, Mitchel. 1990. Automatic evaluation and training in English pronunciation. ICSLP, 1185-1188. DOI: https://doi.org/10.21437/ICSLP.1990-313

Bialik, Kristen, Scheller, Alissa and Walker, Kristi. 2018. 6 facts about English language learners in U.S. public schools. Pew Research Center. Retrieved April 13, 2022, from https://www.pewresearch.org/fact-tank/2018/10/25/6-facts-about-english-language-learners-in-u-s-public-schools/

Chen, Liang-Yu and Jang, Jyh -Shing Roger. 2015. Automatic pronunciation scoring with score combination by learning to rank and class-normalized DP-based quantization. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(11), 1737–1749. DOI: https://doi.org/10.1109/TASLP.2015.2449089

Coniam, David. 1999. Voice recognition software accuracy with second language speakers of English. System, 27(1), 49–64. https://doi.org/10.1016/S0346-251X(98)00049-9 DOI: https://doi.org/10.1016/S0346-251X(98)00049-9

Cucchiarini, Catia, Strik, Helmer and Boves, Lou. 2000. Different aspects of expert pronunciation quality ratings and their relation to scores produced by speech recognition algorithms. Speech Communication, 30(2-3), 109–119. https://doi.org/10.1016/S0167-6393(99)00040-0 DOI: https://doi.org/10.1016/S0167-6393(99)00040-0

Davey, Melissa. 2017, August 9. Outsmarting the computer: The secret to passing Australia’s English-proficiency test. The Guardian.

Derwing, Tracy M., Munro, Murray J., and Carbonaro, M. 2000. Does popular speech recognition software work with ESL speech? TESOL Quarterly, 34(3), 592–603. https://doi.org/10.2307/3587748 DOI: https://doi.org/10.2307/3587748

Dixon, Robert M. W. and Aikhenvald, Alexandra Y. (eds.) 2002. Word: A cross-linguistic typology. Phonology, 20, 425–429. https://doi.org/10.1017/S0952675704210119 DOI: https://doi.org/10.1017/S0952675704210119

Getting started with Reading Progress in Teams. (n.d.). Microsoft. Retrieved December 19, 2021, from https://support.microsoft.com/en-us/topic/getting-started-with-reading-progress-in-teams-7617c11c-d685-4cb7-8b75-3917b297c407?storagetype=live#ID0EDD=Educators

Grant, Malcolm J., Button, Cathryn M. and Snook, Brent. 2017. An evaluation of interrater reliability measures on binary tasks using d-prime. Applied Psychological Measurement, 41(4), 264-276. https://doi.org/10.1177/0146621616684584 DOI: https://doi.org/10.1177/0146621616684584

Han, Jeonghye. 2012. Emerging technologies: Robot-assisted language learning. Language Learning & Technology, 16(3), 1–9.

Han, ZhaoHong. 2004. Fossilization: Five central issues. International Journal of Applied Linguistics, 14(2), 212–242. https://doi.org/10.1111/j.1473-4192.2004.00060.x DOI: https://doi.org/10.1111/j.1473-4192.2004.00060.x

Henrichsen, Lynn E. 2021. An illustrated taxonomy of online CAPT resources, RELC Journal, 52(1), 179–188. https://doi.org/10.1177/0033688220954560 DOI: https://doi.org/10.1177/0033688220954560

Hodges, Charles B., Moore, Stephanie, Lockee, Barb B., Trust, Torrey and Bond, M. Aron. 2020. The difference between emergency remote teaching and online learning. Educase. https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remote-teaching-and-online-learning

In-Seok Kim. 2006. Automatic speech recognition: Reliability and pedagogical implications for teaching pronunciation. Journal of Educational Technology & Society, 9(1), 322–334. http://www.jstor.org/stable/jeductechsoci.9.1.322

Koo, Terry K. and Li, Mae Y. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155-163. http://dx.doi.org/10.1016/j.jcm.2016.02.012 DOI: https://doi.org/10.1016/j.jcm.2016.02.012

Landis, J. Richard and Koch, Gary G. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. PMID: 843571. DOI: https://doi.org/10.2307/2529310

Levis, John M. 2005. Changing contexts and shifting paradigms in pronunciation teaching. TESOL Quarterly, 39(3), 369-377. https://doi.org/10.2307/3588485 DOI: https://doi.org/10.2307/3588485

Levis, John M. 2007. Computer technology in teaching and researching pronunciation, Annual Review of Applied Linguistics, 27, 184–202. https://doi.org/10.1017/S0267190508070098 DOI: https://doi.org/10.1017/S0267190508070098

Li, Kun, Qioan, Xiaojun and Meng, Helen. 2017. Mispronunciation detection and diagnosis in L2 English speech using multidistribution deep neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(1), 193-207. https://doi.org/10.1109/TASLP.2016.2621675 DOI: https://doi.org/10.1109/TASLP.2016.2621675

Research related to Reading Progress. (n.d.). Microsoft. Retrieved December 19, 2021, from https://education.microsoft.com/en-us/resource/9d24e8eb

Marks, Johathan, Hancock, Mark, Hewings, Martin and Donna, Silvie. 2012. English pronunciation in use (Vol. 1-3). Cambridge University Press.

Mahdi, Hassan Saleh and Al Khateeb, Ahmed Abdulateef. 2019, The effectiveness of computer-assisted pronunciation training: A meta-analysis. Review of Education, 7, 733-753. https://doi.org/10.1002/rev3.3165 DOI: https://doi.org/10.1002/rev3.3165

Molenda, Marek, Adamczyk, Michał and Rybińska, Paulina. 2018. Beyond the CAPT – Automatic Speech Recognition in pronunciation training. In J. Pitura & S. Sauro (Eds.), CALL for mobility, 32–153. Peter Lang. https://doi.org/10.3726/b13451 DOI: https://doi.org/10.3726/b13451

Morril, Tuuli, Baese-Berk, Melissa., Heffner, Christopher and Dilley, Laura. 2015. Interactions between distal speech rate, linguistic knowledge, and speech environment. Psychonomic Bulletin & Review, 22(5), 1451–1457. https://doi.org/10.3758/s13423-015-0820-9 DOI: https://doi.org/10.3758/s13423-015-0820-9

Nushi, Musa and Sadeghi, Mahsa. 2021. A critical review of ELSA: A pronunciation app. Computer Assisted Language Learning Electronic Journal, 22(3), 287-302.

Paige, David D., Rasinski, Timothy V. and Magpuri-Lavell, Theresa. 2012. Is fluent, expressive reading important for high school readers? Journal of Adolescent & Adult Literacy, 56(1), 67–76. https://doi.org/10.1002/JAAL.00103 DOI: https://doi.org/10.1002/JAAL.00103

Pennington, Martha C. and Rogerson-Revell, Pamela. 2019. English pronunciation teaching and research. Palgrave Macmillan. https://doi.org/10.1057/978-1-137-47677-7 DOI: https://doi.org/10.1057/978-1-137-47677-7

Qian, Xiaojun and Meng, Helen. 2017. A two-pass framework for mispronunciation detection and diagnosis for computer-aided pronunciation training. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(6), 1020–1028. DOI: https://doi.org/10.1109/TASLP.2016.2526782

Rasinski, Timothy, Rikli, Andrew, & Johnston, Susan. 2009. Reading fluency: More than automaticity? More than a concern for the primary grades? Literacy Research and Instruction, 48(4), 350–361. https://doi.org/10.1080/19388070802468715 DOI: https://doi.org/10.1080/19388070802468715

Ray, Susanna. 2021. Students have a new, less stressful way to improve their reading – and it’s easier for teachers, too. Microsoft. https://news.microsoft.com/features/reading-progress/

Retrospective study of ReadWorks use and effectiveness Web analytics and survey findings. (2014). Rockman et al.

Roach, Peter. 2009. English phonetics and phonology: A practical course (4th ed.). Cambridge University Press.

Rogerson-Revell, Pamela M. 2021. Computer-Assisted Pronunciation Training (CAPT): Current issues and future directions, RELC Journal, 52(1), 189–205. https://doi.org/10.1177/0033688220977406 DOI: https://doi.org/10.1177/0033688220977406

Sobkowiak, Włodzimierz. 2004. English phonetics for Poles (2nd ed.). Wydawnictwo Poznańskie.

Souza, Hanna Kivistö and Gottardi, William. 2022. How well can ASR technology understand foreign-accented speech?. SciELO Preprints. https://doi.org/10.1590/010318138668782v61n32022 DOI: https://doi.org/10.1590/010318138668782v61n32022

Yaman, İsmail and Ekmekçi, Emrah. 2016. Shift from CALL to MALL? Participatory Educational Research, special issue 2016-IV, 25-32.

Wexler, Jade, Vaughn, Sharon, Edmonds, Meaghan and Reutebuch, Colleen Klein. 2008. A synthesis of fluency interventions for secondary struggling readers. Reading and Writing, 21(4), 317–347. https://doi.org/10.1007/s11145-007-9085-7 DOI: https://doi.org/10.1007/s11145-007-9085-7