Search

International audience; The ranking of sound event detection (SED) systems may be biased by assumptions inherent to evaluation criteria and to the choice of an operating point. This paper compares conventional event-based and segment-based criteria against the Polyphonic Sound Detection Score (PSDS)'s intersection-based criterion, over a selection of systems from DCASE 2020 Challenge Task 4. It shows that, by relying on collars , the conventional event-based criterion introduces different strictness levels depending on the length of the sound events, and that the segment-based criterion may lack precision and be application dependent. Alternatively, PSDS's intersection-based criterion overcomes the dependency of the evaluation on sound event duration and provides robustness to labelling subjectivity, by allowing valid detections of interrupted events. Furthermore, PSDS enhances the comparison of SED systems by measuring sound event modelling performance independently from the systems' operating points.

This work defines a new framework for performance evaluation of polyphonic sound event detection (SED) systems, which overcomes the limitations of the conventional collar-based event decisions, event F-scores and event error rates. The proposed framework introduces a definition of event detection that is more robust against labelling subjectivity. It also resorts to polyphonic receiver operating characteristic (ROC) curves to deliver more global insight into system performance than F1-scores, and proposes a reduction of these curves into a single polyphonic sound detection score (PSDS), which allows system comparison independently from operating points (OPs). The presented method also delivers better insight into data biases and classification stability across sound classes. Furthermore, it can be tuned to varying applications in order to match a variety of user experience requirements. The benefits of the proposed approach are demonstrated by re-evaluating the baseline and two of the top-performing systems from DCASE 2019 Task 4., Accepted to ICASSP 2020

Nonnegative matrix factorization (NMF) and nonnegative tensor factorization (NTF) are important tools for modeling nonnegative data, which gained increasing popularity in various fields, a significant one of which is audio processing. However, there are still many problems in audio processing, for which the NMF (or NTF) model has not been successfully utilized. In this paper, we propose a new algorithm based on the NMF (and NTF) in the short-time Fourier domain for solving a large class of audio inverse problems with missing or corrupted time-domain samples. The proposed approach overcomes the difficulty of employing a model in the frequency domain to recover time-domain samples with the help of probabilistic modeling. Its performance is demonstrated for the following applications: audio declipping and declicking (never solved with NMF/NTF modeling prior to this paper); joint audio declipping/declicking and source separation (never solved with NMF/NTF modeling or any other method prior to this paper); and compressive sampling recovery and compressive sampling-based informed source separation (an extremely low complexity encoding scheme that is possible with the proposed approach and has never been proposed prior to this paper).

In this paper we propose a new method to automatically select the rank of linear transforms during supervised learning. Our approach relies on a sparsity-enforcing element-wise soft-thresholding operation applied after the linear transform. This novel approach to supervised rank learning has the important advantage that it is very simple to implement and incurs no extra complexity relative to linear transform learning. Furthermore, we propose a simple Stochastic Gradient Descent (SGD) implementation suitable for large scale learning, where SGD solvers have established themselves as the default workhorse. We compare our method to various other metric learning techniques in the application of image retrieval. This is one of the remaining few areas where supervised learning of low-rank linear transforms has not been fully exploited. The main reason for this is the lack of adequate datasets that are large enough, and hence we further introduce a new dataset consisting of groups of matching images derived from Cable News Network (CNN) videos using geometric verification and manual selection to find matching frames with adequate variability.

We address efficient compression and real-time streaming of stereoscopic video over the current Internet. We first propose content-adaptive stereo video coding (CA-SC), where additional coding gain, over that can be achieved by exploiting only inter-view correlations, is targeted by down-sampling one of the views spatially or temporally depending on the content, based on the well-known theory that the human visual system can perceive high frequencies in three-dimensional (3D) from the higher quality view. We also developed stereoscopic 3D video streaming server and clients by modifying available open source platforms, where each client can view the video in mono or stereo mode depending on its display capabilities. The performance of the end-to-end stereoscopic streaming system is demonstrated using subjective quality tests.

International audience; Audio inpainting and audio declipping are important problems in audio signal processing, which are encountered in various practical applications. A number of approaches has been proposed in the literature to address these problems, most successful of which are based on sparsity of the audio signals in certain dictionary representations. Non-negative matrix factorization (NMF) is another powerful tool that has been successfully used in applications such as audio source separation. In this paper we propose a new algorithm that makes use of a low rank NMF model to perform audio inpainting and declipping. In addition to utilizing for the first time the NMF model to perform audio inpainting in presence of arbitrary losses in time domain, the proposed approach also introduces a novel way to enforce additional constraints on the signal magnitude in order to improve the performance in declipping applications. The proposed approach is shown to have a comparable performance with the state of the art dictionary based methods while providing a number of advantages.

International audience; Despite being two important problems in audio signal processing that are interconnected in practice, audio inpainting and audio source separation have not been considered jointly. It is not uncommon in practice to have the mixtures to be separated which also suffer from artifacts due to clipping or other losses. In present work, we consider this problem of source separation using partially observed mixtures. We introduce a flexible framework based on non-negative tensor factorisation (NTF) to attack this new task, and we apply it to source separation with clipped mixtures. It allows us to perform declipping and source separation either in turn or jointly. We investigate experimentally these two regimes and report large performance gains compared to source separation with clipping artefacts being ignored, which is the common approach in practice.

International audience; The paradigm of using a very simple encoder and a sophisticated decoder for compression of signals became popular with the theory of distributed coding and it has been exercised for the compression of various types of signals such as images and video. The theory of compressive sampling later introduced a similar concept but with the focus on guarantees of signal recovery using sparse and low rank priors lying in an incoherent domain to the domain of sampling. In this paper, we bring together the concepts introduced in distributed coding and compressive sampling with the informed source separation , in which the goal is to efficiently compress the audio sources so that they can be decoded with the knowledge of the mixture of the sources. The proposed framework uses a very simple time domain sampling scheme to encode the sources, and a sophisticated decoding algorithm that makes use of the low rank non-negative tensor factorization model of the distribution of short-time Fourier transform coefficients to recover the sources, which is a direct application of the principles of both compressive sampling and distributed coding.

International audience; We investigate a compressive sensing framework in which the sensors introduce a distortion to the measurements in the form of unknown gains. We focus on blind calibration, using measures performed on multiple unknown (but sparse) signals and formulate the joint recovery of the gains and the sparse signals as a convex optimization problem. We divide this problem in 3 subproblems with different conditions on the gains, specifially (i) gains with different amplitude and the same phase, (ii) gains with the same amplitude and different phase and (iii) gains with different amplitude and phase. In order to solve the first case, we propose an extension to the basis pursuit optimization which can estimate the unknown gains along with the unknown sparse signals. For the second case, we formulate a quadratic approach that eliminates the unknown phase shifts and retrieves the unknown sparse signals. An alternative form of this approach is also formulated to reduce complexity and memory requirements and provide scalability with respect to the number of input signals. Finally for the third case, we propose a formulation that combines the earlier two approaches to solve the problem. The performance of the proposed algorithms is investigated extensively through numerical simulations, which demonstrates that simultaneous signal recovery and calibration is possible with convex methods when sufficiently many (unknown, but sparse) calibrating signals are provided.

Magnetic Resonance Imaging (MRI) is one of the fields that the compressed sensing theory is well utilized to reduce the scan time significantly leading to faster imaging or higher resolution images. It has been shown that a small fraction of the overall measurements are sufficient to reconstruct images with the combination of compressed sensing and parallel imaging. Various reconstruction algorithms has been proposed for compressed sensing, among which Augmented Lagrangian based methods have been shown to often perform better than others for many different applications. In this paper, we propose new Augmented Lagrangian based solutions to the compressed sensing reconstruction problem with analysis and synthesis prior formulations. We also propose a computational method which makes use of properties of the sampling pattern to significantly improve the speed of the reconstruction for the proposed algorithms in Cartesian sampled MRI. The proposed algorithms are shown to outperform earlier methods especially for the case of dynamic MRI for which the transfer function tends to be a very large matrix and significantly ill conditioned. It is also demonstrated that the proposed algorithm can be accelerated much further than other methods in case of a parallel implementation with graphics processing units (GPUs)., Submitted to IEEE JETCAS, Special Issue on Circuits, Systems and Algorithms for Compressed Sensing

Many areas in signal processing have benefited from the emergence of compressed sensing and sparse reconstruction methods, one of which is magnetic resonance imaging (MRI). Recent studies showed that MRI acquisition can be highly accelerated with the joint use of compressed sensing and parallel imaging methods. It is also suggested that dynamic MRI can be further improved by making use of temporal correlations. Although methods using motion compensation has been proposed to exploit temporal dependence, most of these require reference frames and/or a sub-portion of k-space to be fully sampled. In this paper we propose a new approach to exploit the motion information during compressed sensing reconstruction without any requirement for reference frames, modeled motion or a specific sampling pattern on the k-space measurements.

In traditional wireless multicast systems, the system is optimized to provide high reliability to users with the worst channel conditions. However this results in a waste of available bandwidth for users with a good channel conditions. In particular for transmission of video, this wasted bandwidth can be used to deliver higher quality video to some of the users. This paper addresses this problem by using diversity embedded space time codes (DESTC) to send multiple layers of video simultaneously. DESTC provides unequal error protection to different video layers, thereby delivering high quality video to users with good channel conditions while still providing acceptable quality to users with poor channel conditions. The performance of DESTC is compared to a single-layer orthogonal space-time code used for single-layer video delivery as well as layered video delivery with time division multiplexing. The results show that the use of DESTC is advantageous over both strategies in multicast video delivery.

In this paper we propose a novel framework for the streaming of 3-D representations in the form of Multi- View Videos (MVV). The proposed streaming system is completely standards based, flexible and backwards compatible in order to support monoscopic streaming to legacy clients. We demonstrate compatibility of the proposed system with various possible encoding schemes and operating scenarios. In the current implementation, the MVVs in the server are compressed using a simplified form of MVC with negligible loss of compression efficiency and streamed using Real Time Streaming Protocol (RTSP), Session Description Protocol (SDP) and Real Time Protocol (RTP) to the clients. We describe our extensions to SDP and discuss a preliminary RTP payload format for MVC. The clients in this implementation perform basic error concealment to reduce the effects of packet losses and decode MVC in near-real-time. The modular clients can display decoded 3-D content on a multitude of 3-D display systems.

Stereoscopic video is one of the emerging research areas especially among the video coding community. Along with the studies for efficiently compressing the stereoscopic and multiview video, new error concealment and error protection methods are also necessary to overcome the problems due to erroneous channel conditions in practical applications. In this paper we propose a full frame loss concealment algorithm for stereoscopic sequences. The proposed method uses redundancy and disparity between the two views and motion information between the previously decoded frames to estimate the lost frame. The results show that, the proposed algorithm outperforms the monoscopic methods when they are applied to the same view as they are simulcast coded.

The utilization of forward error correction (FEC) schemes for stereo video streaming is investigated. Stereo video is categorized in 3 layers and each layer is protected with different protection ratios for efficient streaming. Systematic Reed-Solomon (RS) and Luby Transform (LT) codes are utilized as the error protection schemes. Detailed simulations are performed in order to observe the optimum unequal error protection (UEP) strategies for the defined video layers. Moreover, as a result of these simulations the performance comparison of RS and LT codes for video streaming is provided.

Date of Conference: 7-9 May 2007 Conference Name: 3DTV Conference, IEEE 2007 In this paper, error resilient stereoscopic video streaming problem is addressed. Two different Forward Error Correction (FEC) codes namely Systematic LT and RS codes are utilized to protect the stereoscopic video data against transmission errors. Initially, the stereoscopic video is categorized in 3 layers with different priorities. Then, a packetization scheme is used to increase the efficiency of error protection. A comparative analysis of RS and LT codes are provided via simulations to observe the optimum packetization and UEP strategies.

Date of Conference: 12-13 November 2007 Conference Name: Packet Video 2007 - 16th International Packet Video Workshop A near optimal streaming system for stereoscopic video is proposed. Initially, the stereoscopic video is separated into three layers and the approximate analytical model of the Rate-Distortion (RD) curve of each layer is calculated from sufficient number of rate and distortion samples. The analytical modeling includes the interdependency of the defined layers. Then, the analytical models are used to derive the optimal source encoding rates for a given channel bandwidth. The distortion in the quality of the stereoscopic video that is caused by losing a NAL unit from the defined layers is estimated to minimize the average distortion of a single NAL unit loss. The minimization is performed over protection rates allocated to each layer. Raptor codes are utilized as the error protection scheme due to their novelty and suitability in video transmission. The layers are protected unequally using Raptor codes according to the parity ratios allocated to the layers. Comparison of the defined scheme with two other protection allocation schemes is provided via simulations to observe the quality of stereoscopic video.

H.264 is the current state-of-the-art monoscopic video codec providing almost twice the coding efficiency with the same quality comparing the previous codecs. With the increasing interest in 3D TV, multi-view video sequences that are provided by multiple cameras capturing the three dimensional objects and/or scene are more widely used. Compressing multi-view sequences independently with H.264 (simulcast) is not efficient since the redundancy between the closer cameras is not exploited. In order to reduce these redundancies, we propose a multi-view video codec based on H.264 using disparity estimation/compensation as well as motion estimation/compensation. In order to effectively search for disparity/motion without increasing computational complexity, we modified the buffering structure of H.264 and implemented several referencing modes. Our results show that for closely located cameras, our codec outperforms simulcast H.264 coding. For sparsely located cameras, our method can still improve coding gain depending on the video characteristics.

Today, stereoscopic and multi-view video are among the popular research areas in the multimedia world. In this study, we have designed and built a platform consisting of stereo-view capturing, real-time transmission and display. At the display stage, end users view video in 3D by using polarized glasses. Multi-view video is compressed in an efficient way by using multi-view video coding techniques and streamed using standard real-time transport protocols. The entire system is built by modifying available open source systems whenever possible. Receiver can view the content of the video built from multiple channels as mono or stereo depending on its display and bandwidth capabilities.