November 2008 | Technical Report NTIA TR-09-457
Techniques for Evaluating Objective Video Quality Models Using Overlapping Subjective Data Sets
Cite This Publication
Margaret H. Pinson and Stephen Wolf, “Techniques for Evaluating Objective Video Quality Models Using Overlapping Subjective Data Sets,” Technical Report NTIA TR-09-457, U.S. Department of Commerce, National Telecommunications and Information Administration, Institute for Telecommunication Sciences, November 2008.
Margaret H. Pinson and Stephen Wolf
Abstract:
This report presents techniques for evaluating objective video quality models using overlapping subjective data sets. The techniques are demonstrated using data from the Video Quality Experts Group (VQEG) Multi-Media (MM) Phase I experiments. These results also provide a supplemental analysis of the performance achieved by the objective models that were submitted to the MM Phase I experiments. The analysis presented herein uses the subjective scores from the common set of video clips to map all the subjective scores from the 13 or 14 experiments (at a given image resolution) onto a single subjective scale. This mapping greatly increases the available data and thus allows for more powerful analysis techniques to be performed. Resolving power values are presented for each model and image resolution. On a per-clip level, models' responses to stimuli are analyzed with respect to all stimuli, each coding algorithm, coding-only impairments, and transmission error impairments. The models' responses to stimuli are also analyzed on per-system and per-scene levels. Results indicate the amount of improvement possible when averaging over multiple scenes or systems.
Keywords: performance; video; quality; objective; subjective; correlation; combining; mapping; multimedia; Video Quality Experts Group (VQEG)
For technical information concerning this report, contact:
Margaret H. Pinson
Institute for Telecommunication Sciences
(303) 497-3579
mpinson@ntia.doc.gov
Disclaimer: Certain commercial equipment, components, and software may be identified in this report to specify adequately the technical aspects of the reported results. In no case does such identification imply recommendation or endorsement by the National Telecommunications and Information Administration, nor does it imply that the equipment or software identified is necessarily the best available for the particular application or uses.
For questions or information on this or any other NTIA scientific publication, contact the ITS Publications Office at ITSinfo@ntia.gov or 303-497-3572.