Search

In this paper we reproduce experimental results presented in our earlier work titled "Describing Subjective Experiment Consistency by $p$-Value P-P Plot" that was presented in the course of the 28th ACM International Conference on Multimedia. The paper aims at verifying the soundness of our prior results and helping others understand our software framework. We present artifacts that help reproduce tables, figures and all the data derived from raw subjective responses that were included in our earlier work. Using the artifacts we show that our results are reproducible. We invite everyone to use our software framework for subjective responses analyses going beyond reproducibility efforts., Comment: Please refer to the original publication: https://dl.acm.org/doi/10.1145/3474085.3477935 Related paper: https://dl.acm.org/doi/abs/10.1145/3394171.3413749 or arXiv:2009.13372

It was once thought that high QoS (Quality of Service) performance solves recurrent problems of low-quality multimedia services. Since then, solutions have been proposed to ensure a high level of QoE (Quality of Experience). In this document, the authors attempt to outline their understanding of an accurate meaning of quality of multimedia services. Starting from QoS and passing through generalised QoE, the authors focus on aspects of subjective and objective quality modelling and optimisation of visual performance for TRV (Target Recognition Video) applications (such as video surveillance), outlining the path of ITU-T standardisation in this area. The authors revised the ITU-T Recommendation P.912 to reflect improved subjective test techniques developed since this Recommendation was approved. The authors also attempt to predict at least some existing errors of reasoning, which are likely to become evident for the industry in the next decade.

There are phenomena that cannot be measured without subjective testing. However, subjective testing is a complex issue with many influencing factors. These interplay to yield either precise or incorrect results. Researchers require a tool to classify results of subjective experiment as either consistent or inconsistent. This is necessary in order to decide whether to treat the gathered scores as quality ground truth data. Knowing if subjective scores can be trusted is key to drawing valid conclusions and building functional tools based on those scores (e.g., algorithms assessing the perceived quality of multimedia materials). We provide a tool to classify subjective experiment (and all its results) as either consistent or inconsistent. Additionally, the tool identifies stimuli having irregular score distribution. The approach is based on treating subjective scores as a random variable coming from the discrete Generalized Score Distribution (GSD). The GSD, in combination with a bootstrapped G-test of goodness-of-fit, allows to construct $p$-value P-P plot that visualizes experiment's consistency. The tool safeguards researchers from using inconsistent subjective data. In this way, it makes sure that conclusions they draw and tools they build are more precise and trustworthy. The proposed approach works in line with expectations drawn solely on experiment design descriptions of 21 real-life multimedia quality subjective experiments., Comment: 11 pages, 3 figures. Accepted to 28th ACM International Conference on Multimedia (MM '20). For associated data sets, source codes and documentation, see https://github.com/Qub3k/subjective-exp-consistency-check

The conventional video subjective test design, in which subjects view and rate multiple versions of each source video sequence, was used for decades. New technology, like adaptive streaming, makes it almost impossible to use this design since much longer sequences are needed. In this paper we examine three experiment designs: the conventional design and two alternatives that use each source sequence only once. Based on data collected by three laboratories, we compare the accuracy and scoring behavior of these three designs. We check whether there is a significant difference in scoring behavior between the experiment designs. One of the proposed experiment designs is proposed for immediate use.

In a subjective experiment to evaluate the perceptual audiovisual quality of multimedia and television services, raw opinion scores offered by subjects are often noisy and unreliable. Recommendations such as ITU-R BT.500, ITU-T P.910 and ITU-T P.913 standardize post-processing procedures to clean up the raw opinion scores, using techniques such as subject outlier rejection and bias removal. In this paper, we analyze the prior standardized techniques to demonstrate their weaknesses. As an alternative, we propose a simple model to account for two of the most dominant behaviors of subject inaccuracy: bias (aka systematic error) and inconsistency (aka random error). We further show that this model can also effectively deal with inattentive subjects that give random scores. We propose to use maximum likelihood estimation (MLE) to jointly estimate the model parameters, and present two numeric solvers: the first based on the Newton-Raphson method, and the second based on alternating projection. We show that the second solver can be considered as a generalization of the subject bias removal procedure in ITU-T P.913. We compare the proposed methods with the standardized techniques using real datasets and synthetic simulations, and demonstrate that the proposed methods have advantages in better model-data fit, tighter confidence intervals, better robustness against subject outliers, shorter runtime, the absence of hard coded parameters and thresholds, and auxiliary information on test subjects. The source code for this work is open-sourced at this https URL.

The use of no-reference image quality evaluation tools that produce MOS scores, like the VIQET tool which was released by the Video Quality Expert Group, raises the question of whether the produced MOS differences between images correspond to noticeable differences in quality by the consumers. In this work, we attempted to approximate the minimum MOS difference that is required in order for people to be able to distinguish between a higher and a lower quality image under realistic conditions that are commonly encountered in the current consumer space. 91 people participated in a subjective just-noticeable-differences study across three countries that used non-simulated image stimuli, produced and evaluated through crowd sourcing for the validation of the VIQET no-reference image quality tool. The image dataset consisted of 15 different scenes belonging to three different scene types, with a total of 210 different image pairs being used. After evaluating the quality of the collected data, a logistic regression analysis approach was employed in order to estimate the minimum MOS difference required between two images in order for a given percentage of people to be able to detect the higher quality image.

The paper proposes a No-Reference (NR) metric to objectively assess H.264/AVC video quality and its lightweight implementation. The proposed model takes into account typical artefacts introduced by hybrid block-based motion-compensated predictive video codecs such as the one related to the H.264/AVC standard. More specifically, these artefacts are the blockiness introduced at the boundaries of each coded block, and the temporal flickering due to different coding modes used for the same macro-block along the video sequence. Furthermore, a flickering metric for intra-coded frames is also derived. The quality prediction accuracy of the proposed NR quality metric is validated over subjective data collected during a subjective video evaluation experiment. Moreover, quality prediction accuracy is also compared with that provided by the well-known state-of-the-art Structural Similarity Index Metric (SSIM) which works in a Full-Reference (FR) mode. The proposed metric achieves a higher Pearson's correlation coefficient with subjective scores than the one achieved by the SSIM metric. The proposed implementation of algorithms is sufficiently lightweight to perform real-time calculations of video streaming quality on PCs in order to assure a Quality of Experience (QoE)-aware service. Graphical abstractDisplay Omitted HighlightsWe propose NR metric implementation to objectively assess H.264/AVC quality.Proposed model takes into account typical artefacts introduced by video codecs.Accuracy of proposed quality metric is validated over subjective data and SSIM.Our metric has higher Pearson's correlation with subjective scores than SSIM.Our implementation is lightweight to perform real-time quality calculations.

This article describes how to perform a video quality subjective test. For companies, these tests can greatly facilitate video product development; for universities, removing perceived barriers to conducting such tests allows expanded research opportunities. This tutorial assumes no prior knowledge and focuses on proven techniques.

Video bit rate reduction is very important for all video streaming applications. One possibility involves quantization domain and the majority of the work devoted to bit rate reduction focuses on this aspect only. The other possibility is to modify a video in time or space domain i.e. change the frames per second FPS rate or frame resolution FR. In this paper we present two no reference metrics mapping FPS rate and FR into MOS (Mean Opinion Scale). The performance of both models is significantly improved by incorporating content characteristics such as spatial information SI and temporal information TI. The impact on Quality of Experience (QoE) of both content characteristics is discussed with relation to the FPS rate and FR changes and general conclusions are drawn. The models were estimated and verified upon results of subjective experiments performed using video sequences of diverse spatial and temporal variability. The considered FPS rate was changed from 5 to 30 and the considered FR was changed from SQCIF to SD.

Video bit rate reduction can be very important for all video streaming application. One of the possible ways to reduce bit rate is decreasing change in time or space domain i.e. changing frame rate or resolution. In this paper we present two no reference metrics mapping frame rate or resolution into MOS. Both models use simple to calculate parameters expressed by sequence spatial and temporal information. The models were estimated and verified upon distinctive video sequence sets. The considered frame rate change varies from 5 to 30 frames per second. The considered resolutions changes from SQCIF to SD.

Aggregated traffic traces are commonly used in network engineering for QoS or performance parameters evaluation. Many performance models come from such aggregated traces. However, real traffic is a marked point process combining two processes: one for the arrival times of packets and the other for their size in bytes. This paper deals with assessing whether aggregated traces are a good representation of real traffic. Based on the analysis of many traffic traces, and focusing only on loss probability, it is shown that the packet drop probability obtained for the aggregated traffic traces can significantly differ from the real packet drop probability obtained for the real traffic traces. Then, a solution which enables one to obtain correct loss probability based on aggregated traffic traces is proposed by determining the correct aggregation scale and traffic parameters to be applied.

Downlink throughput is the most widely used and accepted Quality of Service (QoS) related feature within the networking community, specially in the operational field. Current quality monitoring and reporting systems as well as quality benchmarking campaigns use the Average Downlink Throughput (ADT) as metric to assess the performance of the network. For example, flow-based monitoring systems normally report the per flow ADT as a throughput-based Key Performance Indicator (KPI), which is then aggregated at different temporal (e.g., 15 minutes), geographical (e.g., per radio cell) and/or logical (e.g., per service) scales to reflect the health of the network. A similar direction is currently followed within the Quality of Experience (QoE) research domain, where ADT is translated into a measure of user satisfaction for bandwidth-sensitive services (e.g., video streaming, file sharing, etc.). We claim that the ADT is not always an accurate KPI in terms of QoE for bandwidth-sensitive services, and present results showing that the variation of downlink throughput can actually have a major impact on the perceived quality of the end user. In this paper we present a complete study of the QoE undergone by 52 mobile users in controlled subjective lab tests, using different mobile applications such as YouTube, Facebook and Gmaps. By shaping the traffic of the users through multiple bandwidth fluctuation patterns, we conclude that novel downlink-throughput related KPIs must be defined for QoE-based traffic analysis in mobile networks. Based on this observation, we propose some very first simple throughput fluctuation models to define such KPIs.

The systematic investigation of network bandwidth fluctuations and their impact on Quality of Experience (QoE) has recently gained center stage, not only in academia but also among network providers. This development is driven by the fact that in a real world usage scenario, volatile network performance (and thus varying QoE) actually is the norm, particularly in mobile contexts. However, we are still witnessing a lack of validated fluctuation models and applicable performance indicators to represent the QoE impact of network quality variations in provider-relevant network planning & management processes. In addition, fundamental phenomena like the impact of single bandwidth outage events on Web QoE (which occur e.g. during a handover between different base stations) have not been studied yet. In this paper we present the results of an empirical Web-QoE user study which a) lay the path towards the analysis of different bandwidth fluctuation models and b) provide insights into the impact of a single outage (i.e. zero network bandwidth) events on subjective quality perception. Furthermore, our experimental setup that exerts short & repetitive testing-conditions provides methodological guidance on how to perform subjective quality testing in the context of fluctuations and Web QoE.

In this paper, a practical solution for optimal coding parameters using different bandwidth requirements is presented. The obtained specification is based on the analysis of a large database of more than 10,000 sequences compressed with different parameters. The obtained parameters can be used both for adaptive streaming or storage optimization.

Recently, a new 802.11aa amendment has been released. Its main goal is to increase quality of service support for audio-video traffic streams. Among others, it defines the intra-access category prioritization mechanism coupled with appropriate traffic selection procedures, in order to increase the granularity of traffic prioritization in comparison to the currently used EDCA. This paper presents the preliminary study of the possible impact of the new feature on quality of experience (QoE) in case of H.264 video streams. The obtained results show that IEEE 802.11aa and 802.1Q parameters should be tuned to efficiently use the available bandwidth.

Wi-Fi networks are based on the cooperation of users in sharing a common resource — the radio channel. This is a security risk because users may behave selfishly to increase their own throughput but at the same time decrease the overall network performance. Many scientific analyses have focused on this problem, but none have taken into account real user behavior. We present the initial results of a work-in-progress in which we studied a group of users in terms of their online behavior as well as their psychological characteristics. We have found that users behave selfishly in a wireless setting, regardless of their cooperative nature. We provide lessons learned as well as pose open questions for further research in this field.

Recent advances in video distribution over the Internet have inevitably led to the need for accompanying the video related value-added services with the means for maintaining integrity and quality of distributed media to optimize Quality of Experience (QoE). Current systems cover the aspects of maintaining the quality of 2D (only) media by conducting user requirements surveys and 2D QoE assessments. This paper presents the work performed in the EU FP7 SARACEN research project on personalized 3D media streaming over P2P architectures, together with evaluation tests and results as regards scalable and adaptive coding for 3D video streams.

Video communications require the use of networks capable of optimally allocating resources and adapting dynamically themselves to the changing operating conditions. When a video stream has to be delivered to various receivers, the appropriate service to accomplish the delivery process is the multicast communication service. In the particular case of the widely used IEEE 802.11 Wireless LANs (WLANs), the multicast service is simply specified as an unreliable broadcasting service, i.e., it does not include any error recovery mechanism. The absence of feedback from the receivers does not only prevent the sender from taking proper action to recover the corrupted packets, but it also makes unfeasible the implementation of a channel rate adaptation mechanism as the one used by the IEEE 802.11 unicast service. The absence of such mechanism may render the multicast service useless if the channel conditions are so that it makes unreliable the delivery process of the multicast traffic. Even though the IEEE has been working on an amendment to the standard specifying the operation of various error recovery mechanisms for the multicast service, these services have not been designed to support any rate adaptation mechanism. In this paper, we present a novel auto rate selection multicast mechanism, whose design is based on the collision prevention to increase the reliability. We have developed QoS and QoE evaluations via simulation, demonstrating that our proposal outperforms the video quality perceived by the end-user.

Video distribution over error-prone Internet Protocol (IP) networks results in visual impairments on the received video streams. Objective impairment detection algorithms are crucial for maintaining a high Quality of Experience (QoE) as provided with IPTV distribution. There is a lot of research invested in H.264/AVC impairment detection models and questions rise if these turn obsolete with a transition to the successor of H.264/AVC, called High Efficiency Video Coding (HEVC). In this paper, first we show that impairments on HEVC compressed sequences are more visible compaired to H.264/AVC encoded sequences. We also show that an impairment detection model designed for H.264/AVC could be reused on HEVC, but that caution is advised. A more accurate model taking into account content classification needed slight modification to remain applicable for HEVC compression video content.

International audience; Crosstalk is one of the main display-related perceptual factors degrading image quality and causing visual discomfort on 3D-displays. It causes visual artifacts such as ghosting effects, blurring, and lack of color fidelity which are considerably annoying and can lead to difficulties to fuse stereoscopic images. On stereoscopic LCD with shutter-glasses, crosstalk is mainly due to dynamic temporal aspects: imprecise target luminance (highly dependent on the combination of left-view and right-view pixel color values in disparity regions) and synchronization issues between shutter-glasses and LCD. These different factors influence largely the reproducibility of crosstalk measurements across laboratories and need to be evaluated in several different locations involving similar and differing conditions. In this paper we propose a fast and reproducible measurement procedure for crosstalk based on high-frequency temporal measurements of both display and shutter responses. It permits to fully characterize crosstalk for any right/left color combination and at any spatial position on the screen. Such a reliable objective crosstalk measurement method at several spatial positions is considered a mandatory prerequisite for evaluating the perceptual influence of crosstalk in further subjective studies.

The paper proposes a No-Reference (NR) metric to objectively assess the H.264/AVC video quality. The proposed model takes into account the typical artefacts introduced by hybrid block-based motion compensated predictive video codecs as the one related to the H.264/AVC standard. More specifically, these artefacts are the blockiness introduced at the boundaries of each coded block and the temporal flickering due to different coding modes used for the same macroblock along the video sequence. Furthermore, a flickering metric for intra coded frames is also derived. The quality prediction accuracy of the proposed NR quality metric is validated over subjective data collected during a video subjective evaluation experiments. Moreover, the quality prediction accuracy is also compared with the one provided by the well known state-of-the-art Structural SIMilarity (SSIM) metric which works in a full-reference mode. The proposed metric achieves a higher Pearson's correlation coefficient with subjective scores than the one achieved by the SSIM metric.

Cross talk is one of the most important parameters of the 3D displays' quality. Different cross talk definitions exist, which makes cross talk measurement and comparison difficult. We take a step back and focus on a detailed 3D display luminance analysis. The conclusions we draw from the temporal luminance analysis can be used to propose an effective approach to cross talk measurements. In scope of the presented work we have measured four different 3D displays.

Since 1997, the Video Quality Experts Group (VQEG) has been active in the field of subjective and objective video quality assessment. The group has validated competitive quality metrics throughout several projects. Each of these projects requires mandatory actions such as creating a testplan and obtaining databases consisting of degraded video sequences with corresponding subjective quality ratings. Recently, VQEG started a new open initiative, the Joint Effort Group (JEG), for encouraging joint collaboration on all mandatory actions needed to validate video quality metrics. Within the JEG, effort is made to advance the field of both subjective and objective video quality measurement by providing proper software tools and subjective databases to the community. One of the subprojects of the JEG is the joint development of a hybrid H.264/AVC objective quality metric. In this paper, we introduce the JEG and provide an overview of the different ongoing activities within this newly started group.

In the field of still and moving pictures digital imaging one of the most important factors affecting the quality of the acquisition is a correct exposure. This factor is especially important for video surveillance. It is particularly applicable in the case of CCTV cameras often operating in low-light conditions or blinded by an external light source. It is quite common that the lighting conditions are well below the adaptive capacity of a camera. Therefore, it is necessary to continuously monitor the level of exposure to determine image quality. It should be noted that the level of exposure is not expressed using quantitative parameters of acquisition (being sometimes meaningless from the perceived quality assessment point of view) but is measured directly on the image. The purpose of the presented research was to develop a No Reference (NR) metric assessing video Quality of Experience (QoE) affected with the exposure distortion. It was presumed that both over- and under-exposure degrade QoE, so the research task was to derive a proper mapping function between QoE and exposure. The presented exposure metric is unique and no similar research was found in the literature.

In this study we address the problem of an image quality trade-off that can be observed when dealing with content reconstruction schemes based on self-embedding. We derive two models for the estimation of optimal system parameters and the optimization of the overall image quality. This goal is achieved by balancing the distortions of a different nature that affect the resulting images. The performance of the derived models is verified with an accurate reference model and compared to traditional parameter selection strategies. The models are based on basic image features only and allow for rapid prediction of the best values for system parameters in a fully automatic manner.

In this paper, we propose an accurate sampling scheme for defeating SYN flooding attacks as well as TCP portscan activity. The scheme examines TCP segments to find at least one of multiple ACK segments coming from the server. The method is simple and scalable, because it achieves good detection performance with false positive rate close to zero even for very low sampling rates. Our trace-based simulations show that the effectiveness of the proposed scheme only relies on the sampling rate regardless on the sampling method.

This paper introduces a novel approach to a qualitative assessment of images affected by multi-modal distortions. The idea is to assess the image quality perceived by an end user in an automatic way in order to avoid the usual time-consuming, costly and non-repeatable method of collecting subjective scores during a psycho-physical experiment. This is achieved by computing quantitative image distortions and mapping results on qualitative scores. Useful mapping models have been proposed and constructed using the generalised linear model (GLZ), which is a generalisation of the least squares regression in statistics for ordinal data. Overall qualitative image distortion is computed based on partial quantitative distortions from component algorithms operating on specified image features. Seven such algorithms are applied to successfully analyse the seven image distortions in relation to the original image. A survey of over 12,000 subjective quality scores has been carried out in order to determine the influence of these features on the perceived image quality. The results of quantitative assessments are mapped on the surveyed scores to obtain an overall quality score of the image. The proposed models have been validated in order to prove that the above technique can be applied to automatic image quality assessment.

Live mobile streaming applications will be among the most important applications in future wireless multimedia systems. Hence, a Quality of Experience (QoE) assessment control mechanism is an essential requirement to assure the video quality level, whiling maximizing profits to service providers and keeping and attracting new customers. This paper studies the requirements to develop a video artifacts assessment mechanism for live mobile streaming applications, introduce a new assessment solution and no-reference QoE metrics. The proposed schemes are evaluated based on psycho-physical experiments faced with artifacts measurements. The results present the benefits of the assessment mechanism in estimating the quality level of live mobile streaming applications.

The aim of this paper is to determine how to model router interface in order to accurately predict packet drops. There is an enormous amount of research on traffic models reported, however, a model of router interface has not gained proper consideration yet. Our experiments reveal that an incorrect model of the router interface can result in a significant disparity between drop probabilities measured on a physical interface and derived from a trace-driven simulation study. In this paper an accurate model of the router interface (Cisco IOS-based routers with a non-distributed architecture) was presented.