February 2017 | Technical Memorandum NTIA TM-17-523
A Crowdsourced Speech Intelligibility Test that Agrees With, Has Higher Repeatability Than, Lab Tests
Cite This Publication
Stephen D. Voran and Andrew A. Catellier, “A Crowdsourced Speech Intelligibility Test that Agrees With, Has Higher Repeatability Than, Lab Tests,” Technical Memorandum NTIA TM-17-523, U.S. Department of Commerce, National Telecommunications and Information Administration, Institute for Telecommunication Sciences, February 2017.
Stephen D. Voran 
and Andrew A. Catellier
Abstract:
Crowdsourcing of subjective speech, audio, and video quality of experience (QoE) tests has received much interest and study, but crowdsourcing of speech intelligibility testing has not. We hypothesize that speech intelligibility tests offer a unique crowdsourcing opportunity because, unlike QoE testing, each trial has a correct answer. That allows us to motivate and evaluate listeners. We describe the design, implementation, and analysis of a Crowdsourced Modified Rhyme Test (CMRT) that replicates our recent Laboratory MRT (LMRT) work. Our results show that CMRT results are more repeatable than LMRT results, CMRT repeats LMRT better than LMRT repeats itself, and application of a simple listener selection rule produces per-condition CMRT results that almost exactly agree with reference LMRT results.
Keywords: modified rhyme test (MRT); speech intelligibility; subjective test; crowdsource
For technical information concerning this report, contact:
Stephen D. Voran
Institute for Telecommunication Sciences
(720) 446-6425
svoran@ntia.gov
For funding information concerning this report, click this link.
Funding Information
Performing Agency
Disclaimer:
Certain commercial equipment, components, and software may be identified in this report to specify adequately the technical aspects of the reported results. In no case does such identification imply recommendation or endorsement by the National Telecommunications and Information Administration, nor does it imply that the equipment or software identified is necessarily the best available for the particular application or uses.
For questions or information on this or any other NTIA scientific publication, contact the ITS Publications Office at ITSinfo@ntia.gov or 303-497-3572.