Microsoft Reading Progress as Capt Tool

Marek Molenda; Izabela Grabarczyk

doi:10.18778/1731-7533.20.2.05

Authors

Marek Molenda University of Łódź, Poland
Izabela Grabarczyk University of Łódź, Poland

DOI:

https://doi.org/10.18778/1731-7533.20.2.05

Keywords:

CAPT, EFL, ASR

Abstract

The paper explores the accuracy of feedback provided to non-native learners of English by a pronunciation module included in Microsoft Reading Progress. We compared pronunciation assessment offered by Reading Progress against two university pronunciation teachers. Recordings from students of English who aim for native-like pronunciation were assessed independently by Reading Progress and the human raters. The output was standardized as negative binary feedback assigned to orthographic words, which matches the Microsoft format.
Our results indicate that Reading Progress is not yet ready to be used as a CAPT tool. Inter-rater reliability analysis showed a moderate level of agreement for all raters and a good level of agreement upon eliminating feedback from Reading Progress. Meanwhile, the qualitative analysis revealed certain problems, notably false positives, i.e., words pronounced within the boundaries of academic pronunciation standards, but still marked as incorrect by the digital rater.
We recommend that EFL teachers and researchers approach the current version of Reading Progress with caution, especially as regards automated feedback. However, its design may still be useful for manual feedback. Given Microsoft declarations that Reading Progress would be developed to include more accents, it has the potential to evolve into a fully-functional CAPT tool for EFL pedagogy and research.

References

Bajorek, Joan. 2017. L2 pronunciation tools: The unrealized potential of prominent computer-assisted language learning software, Issues and Trends in Educational Technology, 5(2).
Google Scholar DOI: https://doi.org/10.2458/azu_itet_v5i1_bajorek

Baker, Ann. 2007. Ship or sheep? An intermediate pronunciation course (3rd ed.). Cambridge University Press.
Google Scholar

Bernstein, Jared, Cohen, Michael, Murveit, Hy, Rtischev, Dimitry and Weintraub, Mitchel. 1990. Automatic evaluation and training in English pronunciation. ICSLP, 1185-1188.
Google Scholar DOI: https://doi.org/10.21437/ICSLP.1990-313

Bialik, Kristen, Scheller, Alissa and Walker, Kristi. 2018. 6 facts about English language learners in U.S. public schools. Pew Research Center. Retrieved April 13, 2022, from https://www.pewresearch.org/fact-tank/2018/10/25/6-facts-about-english-language-learners-in-u-s-public-schools/
Google Scholar

Chen, Liang-Yu and Jang, Jyh -Shing Roger. 2015. Automatic pronunciation scoring with score combination by learning to rank and class-normalized DP-based quantization. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(11), 1737–1749.
Google Scholar DOI: https://doi.org/10.1109/TASLP.2015.2449089

Coniam, David. 1999. Voice recognition software accuracy with second language speakers of English. System, 27(1), 49–64. https://doi.org/10.1016/S0346-251X(98)00049-9
Google Scholar DOI: https://doi.org/10.1016/S0346-251X(98)00049-9

Cucchiarini, Catia, Strik, Helmer and Boves, Lou. 2000. Different aspects of expert pronunciation quality ratings and their relation to scores produced by speech recognition algorithms. Speech Communication, 30(2-3), 109–119. https://doi.org/10.1016/S0167-6393(99)00040-0
Google Scholar DOI: https://doi.org/10.1016/S0167-6393(99)00040-0

Davey, Melissa. 2017, August 9. Outsmarting the computer: The secret to passing Australia’s English-proficiency test. The Guardian.
Google Scholar

Derwing, Tracy M., Munro, Murray J., and Carbonaro, M. 2000. Does popular speech recognition software work with ESL speech? TESOL Quarterly, 34(3), 592–603. https://doi.org/10.2307/3587748
Google Scholar DOI: https://doi.org/10.2307/3587748

Dixon, Robert M. W. and Aikhenvald, Alexandra Y. (eds.) 2002. Word: A cross-linguistic typology. Phonology, 20, 425–429. https://doi.org/10.1017/S0952675704210119
Google Scholar DOI: https://doi.org/10.1017/S0952675704210119

Getting started with Reading Progress in Teams. (n.d.). Microsoft. Retrieved December 19, 2021, from https://support.microsoft.com/en-us/topic/getting-started-with-reading-progress-in-teams-7617c11c-d685-4cb7-8b75-3917b297c407?storagetype=live#ID0EDD=Educators
Google Scholar

Grant, Malcolm J., Button, Cathryn M. and Snook, Brent. 2017. An evaluation of interrater reliability measures on binary tasks using d-prime. Applied Psychological Measurement, 41(4), 264-276. https://doi.org/10.1177/0146621616684584
Google Scholar DOI: https://doi.org/10.1177/0146621616684584

Han, Jeonghye. 2012. Emerging technologies: Robot-assisted language learning. Language Learning & Technology, 16(3), 1–9.
Google Scholar

Han, ZhaoHong. 2004. Fossilization: Five central issues. International Journal of Applied Linguistics, 14(2), 212–242. https://doi.org/10.1111/j.1473-4192.2004.00060.x
Google Scholar DOI: https://doi.org/10.1111/j.1473-4192.2004.00060.x

Henrichsen, Lynn E. 2021. An illustrated taxonomy of online CAPT resources, RELC Journal, 52(1), 179–188. https://doi.org/10.1177/0033688220954560
Google Scholar DOI: https://doi.org/10.1177/0033688220954560

Hodges, Charles B., Moore, Stephanie, Lockee, Barb B., Trust, Torrey and Bond, M. Aron. 2020. The difference between emergency remote teaching and online learning. Educase. https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remote-teaching-and-online-learning
Google Scholar

In-Seok Kim. 2006. Automatic speech recognition: Reliability and pedagogical implications for teaching pronunciation. Journal of Educational Technology & Society, 9(1), 322–334. http://www.jstor.org/stable/jeductechsoci.9.1.322
Google Scholar

Koo, Terry K. and Li, Mae Y. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155-163. http://dx.doi.org/10.1016/j.jcm.2016.02.012
Google Scholar DOI: https://doi.org/10.1016/j.jcm.2016.02.012

Landis, J. Richard and Koch, Gary G. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. PMID: 843571.
Google Scholar DOI: https://doi.org/10.2307/2529310

Levis, John M. 2005. Changing contexts and shifting paradigms in pronunciation teaching. TESOL Quarterly, 39(3), 369-377. https://doi.org/10.2307/3588485
Google Scholar DOI: https://doi.org/10.2307/3588485

Levis, John M. 2007. Computer technology in teaching and researching pronunciation, Annual Review of Applied Linguistics, 27, 184–202. https://doi.org/10.1017/S0267190508070098
Google Scholar DOI: https://doi.org/10.1017/S0267190508070098

Li, Kun, Qioan, Xiaojun and Meng, Helen. 2017. Mispronunciation detection and diagnosis in L2 English speech using multidistribution deep neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(1), 193-207. https://doi.org/10.1109/TASLP.2016.2621675
Google Scholar DOI: https://doi.org/10.1109/TASLP.2016.2621675

Research related to Reading Progress. (n.d.). Microsoft. Retrieved December 19, 2021, from https://education.microsoft.com/en-us/resource/9d24e8eb
Google Scholar

Marks, Johathan, Hancock, Mark, Hewings, Martin and Donna, Silvie. 2012. English pronunciation in use (Vol. 1-3). Cambridge University Press.
Google Scholar

Mahdi, Hassan Saleh and Al Khateeb, Ahmed Abdulateef. 2019, The effectiveness of computer-assisted pronunciation training: A meta-analysis. Review of Education, 7, 733-753. https://doi.org/10.1002/rev3.3165
Google Scholar DOI: https://doi.org/10.1002/rev3.3165

Molenda, Marek, Adamczyk, Michał and Rybińska, Paulina. 2018. Beyond the CAPT – Automatic Speech Recognition in pronunciation training. In J. Pitura & S. Sauro (Eds.), CALL for mobility, 32–153. Peter Lang. https://doi.org/10.3726/b13451
Google Scholar DOI: https://doi.org/10.3726/b13451

Morril, Tuuli, Baese-Berk, Melissa., Heffner, Christopher and Dilley, Laura. 2015. Interactions between distal speech rate, linguistic knowledge, and speech environment. Psychonomic Bulletin & Review, 22(5), 1451–1457. https://doi.org/10.3758/s13423-015-0820-9
Google Scholar DOI: https://doi.org/10.3758/s13423-015-0820-9

Nushi, Musa and Sadeghi, Mahsa. 2021. A critical review of ELSA: A pronunciation app. Computer Assisted Language Learning Electronic Journal, 22(3), 287-302.
Google Scholar

Paige, David D., Rasinski, Timothy V. and Magpuri-Lavell, Theresa. 2012. Is fluent, expressive reading important for high school readers? Journal of Adolescent & Adult Literacy, 56(1), 67–76. https://doi.org/10.1002/JAAL.00103
Google Scholar DOI: https://doi.org/10.1002/JAAL.00103

Pennington, Martha C. and Rogerson-Revell, Pamela. 2019. English pronunciation teaching and research. Palgrave Macmillan. https://doi.org/10.1057/978-1-137-47677-7
Google Scholar DOI: https://doi.org/10.1057/978-1-137-47677-7

Qian, Xiaojun and Meng, Helen. 2017. A two-pass framework for mispronunciation detection and diagnosis for computer-aided pronunciation training. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(6), 1020–1028.
Google Scholar DOI: https://doi.org/10.1109/TASLP.2016.2526782

Rasinski, Timothy, Rikli, Andrew, & Johnston, Susan. 2009. Reading fluency: More than automaticity? More than a concern for the primary grades? Literacy Research and Instruction, 48(4), 350–361. https://doi.org/10.1080/19388070802468715
Google Scholar DOI: https://doi.org/10.1080/19388070802468715

Ray, Susanna. 2021. Students have a new, less stressful way to improve their reading – and it’s easier for teachers, too. Microsoft. https://news.microsoft.com/features/reading-progress/
Google Scholar

Retrospective study of ReadWorks use and effectiveness Web analytics and survey findings. (2014). Rockman et al.
Google Scholar

Roach, Peter. 2009. English phonetics and phonology: A practical course (4th ed.). Cambridge University Press.
Google Scholar

Rogerson-Revell, Pamela M. 2021. Computer-Assisted Pronunciation Training (CAPT): Current issues and future directions, RELC Journal, 52(1), 189–205. https://doi.org/10.1177/0033688220977406
Google Scholar DOI: https://doi.org/10.1177/0033688220977406

Sobkowiak, Włodzimierz. 2004. English phonetics for Poles (2nd ed.). Wydawnictwo Poznańskie.
Google Scholar

Souza, Hanna Kivistö and Gottardi, William. 2022. How well can ASR technology understand foreign-accented speech?. SciELO Preprints. https://doi.org/10.1590/010318138668782v61n32022
Google Scholar DOI: https://doi.org/10.1590/010318138668782v61n32022

Yaman, İsmail and Ekmekçi, Emrah. 2016. Shift from CALL to MALL? Participatory Educational Research, special issue 2016-IV, 25-32.
Google Scholar

Wexler, Jade, Vaughn, Sharon, Edmonds, Meaghan and Reutebuch, Colleen Klein. 2008. A synthesis of fluency interventions for secondary struggling readers. Reading and Writing, 21(4), 317–347. https://doi.org/10.1007/s11145-007-9085-7
Google Scholar DOI: https://doi.org/10.1007/s11145-007-9085-7