Search

Hyperspectral data provides rich information about a scene in terms of spectral details since it encapsulates measurements/observations from a wide large range of spectrum. To this end, it has been used in different problems mostly related to identification and detection processes. However, the main limitation arises for the accessibility of data. More precisely, there is no sufficient amount of hyperspectral data available compared to visible range data for trainable models. In this paper, we tackle an inverse problem to estimate the relative lidar depth from hyperspectral data. To solve its limitation, we integrate semantic information existed in data with supervised labels to decrease the possibility of parameter overfitting. Moreover, details of the output responses are enhanced with Laplacian pyramids and attention layers in which the model makes predictions from each subsequent scale instead of a single shot prediction from the top of the model. In our experiments, we use the 2018 IEEE GRSS Data Fusion Challenge dataset. From the experimental results, we prove that use of hyperspectral data instead of visible range data improves the performance. Moreover, we show that results are significantly improved if a sparse set of depth measurements is used along with hyperspectral data. Lastly, the integration of semantic information to the solution yields more stable and better results compared to the baselines.

In this study, the effect of visual context information to the performance of learning-based techniques for the super resolution problem is analyzed. Beside the interpretation of the experimental results in detail, its theoretical reasoning is also achieved in the paper. For the experiments, two different visual datasets composed of natural and remote sensing scenes are utilized. From the experimental results, we observe that keeping visual context information in the course of parameter learning for convolutional neural networks yields better performance compared to the baselines. Moreover, we summarize that finetuning pre-trained parameters with the related context yet fewer samples improves the results.

Hyperspectral sensors have high spectral resolution by capturing images in hundreds of bands. Despite the high spectral resolution, low spatial resolution of these sensors restricts the performance of the hyperspectral imaging applications such as target tracking and image classification. Fusing the hyper-spectral image (HSI) with higher spatial resolution RGB or multispectral image (MSI) data is a commonly used method in the resolution enhancement of the HSIs. In this paper, we propose a new fusion technique for the HSI super-resolution. The main contribution of this study is formulating the fusion problem in a quadratic manner and also regularizing the solution quadratically using smoothness prior. Moreover, another contribution of the proposed method is converting the fusion problem from spectral domain to the abundance map domain which gives more robust and spectrally consistent results. In the proposed method, first, abundance maps are obtained using linear spectral unmixing and then a quadratic energy function is obtained using these maps and high resolution (HR) RGB image. In addition, quadratic function is regularized using additional constraints. Solving the regularized quadratic function gives the HR abundance maps and these maps are used to reconstruct HR HSI. Experiments show that proposed method yields better performance as compared to state of the art methods in different performance metrics.

Inverse problems in imaging such as denoising, deblurring, superresolution (SR) have been addressed for many decades. In recent years, convolutional neural networks (CNNs) have been widely used for many inverse problem areas. Although their indisputable success, CNNs are not mathematically validated as to how and what they learn. In this paper, we prove that during training, CNN elements solve for inverse problems which are optimum solutions stored as CNN neuron filters. We discuss the necessity of mutual coherence between CNN layer elements in order for a network to converge to the optimum solution. We prove that required mutual coherence can be provided by the usage of residual learning and skip connections. We have set rules over training sets and depth of networks for better convergence, i.e. performance., PostPrint IET Signal Processing Journal

In this study, we propose a novel single image Bayesian super-resolution (SR) algorithm where the hyperspectral image (HSI) is the only source of information. The main contribution of the proposed approach is to convert the ill-posed SR reconstruction (SRR) problem in the spectral domain to a quadratic optimization problem in the abundance map domain. In order to do so, Markov Random Field (MRF) based energy minimization approach is proposed and proved that the solution is quadratic. The proposed approach consists of five main steps. First, the number of endmembers in the scene is determined using virtual dimensionality. Second, the endmembers and their low resolution abundance maps are computed using simplex identification via the splitted augmented Lagrangian (SISAL) and fully constrained least squares (FCLS) algorithms. Third, high resolution (HR) abundance maps are obtained using our proposed maximum a posteriori (MAP) based energy function. This energy function is minimized subject to smoothness, unity and boundary constraints. Fourth, the HR abundance maps are further enhanced with texture preserving methods. Finally, HR HSI is reconstructed using the extracted endmembers and the enhanced abundance maps. The proposed method is tested on three real HSI datasets; namely the Cave, Harvard and Hyperspectral Remote Sensing Scenes (HRSS) and compared to state-of-the-art alternative methods using peak signal to noise ratio, structural similarity, spectral angle mapper and relative dimensionless global error in synthesis metrics. It is shown that the proposed method outperforms the state of the art methods in terms of quality while preserving the spectral consistency.

Inpainting applications include object removal on images and videos, crack filling, error concealment, texture synthesis, where in this paper, its usage for image coherence and perspective emphasis on video frames in 2D image-to-video conversion system is analysed. Besides, the performance of different techniques in object removal and image reconstruction is compared using visual experiments and quality metrics.

In this paper, we propose a novel hyperspectral unmixing technique based on deep spectral convolution networks (DSCN). Particularly, three important contributions are presented throughout this paper. First, fully-connected linear operation is replaced with spectral convolutions to extract local spectral characteristics from hyperspectral signatures with a deeper network architecture. Second, instead of batch normalization, we propose a spectral normalization layer which improves the selectivity of filters by normalizing their spectral responses. Third, we introduce two fusion configurations that produce ideal abundance maps by using the abstract representations computed from previous layers. In experiments, we use two real datasets to evaluate the performance of our method with other baseline techniques. The experimental results validate that the proposed method outperforms baselines based on Root Mean Square Error (RMSE)., Comment: Accepted to ICIP 2018

The cost-effective visual representation and fast query-by-example search are two challenging goals that should be maintained for web-scale visual retrieval tasks on moderate hardware. This paper introduces a fast and robust method that ensures both of these goals by obtaining state-of-the-art performance for an image-to-video search scenario. Hence, we present critical enhancements to well-known indexing and visual representation techniques by promoting faster, better and moderate retrieval performance. We also boost the superiority of our method for some visual challenges by exploiting individual decisions of local and global descriptors at query time. For instance, local content descriptors represent copied/duplicated scenes with large geometric deformations such as scale, orientation and affine transformation. In contrast, the use of global content descriptors is more practical for near-duplicate and semantic searches. Experiments are conducted on a large-scale Stanford I2V dataset. The experimental results show that our method is useful in terms of complexity and query processing time for large-scale visual retrieval scenarios, even if local and global representations are used together. The proposed method is superior and achieves state-of-the-art performance based on the mean average precision (MAP) score of this dataset. Lastly, we report additional MAP scores after updating the ground annotations unveiled by retrieval results of the proposed method, and it shows that the actual performance., Comment: ICPR 2020

Frame-level visual features are generally aggregated in time with the techniques such as LSTM, Fisher Vectors, NetVLAD etc. to produce a robust video-level representation. We here introduce a learnable aggregation technique whose primary objective is to retain short-time temporal structure between frame-level features and their spatial interdependencies in the representation. Also, it can be easily adapted to the cases where there have very scarce training samples. We evaluate the method on a real-fake expression prediction dataset to demonstrate its superiority. Our method obtains 65% score on the test dataset in the official MAP evaluation and there is only one misclassified decision with the best reported result in the Chalearn Challenge (i.e. 66:7%) . Lastly, we believe that this method can be extended to different problems such as action/event recognition in future., Comment: Submitted to International Conference on Computer Vision Workshops

A vehicle classification system, which uses features based on radiometry, is developed for single band infrared (IR) image sequences. In this context, the process is divided into three components. These are moving vehicle detection, radiance estimation, and classification. The major contribution of this paper lies in the usage of the radiance values as features, other than the raw output of IR camera output, to improve the classification performance of the detected objects. The motivation behind this is that each vehicle class has a discriminating radiance value that originates from the source temperature of the object modified by the intrinsic characteristics of the radiating surface and the environment. As a consequence, significant performance gains are achieved due to the use of radiance values in classification for the utilized measurement system.

In this paper, an application of short video synthesis from single frame images is realized and a comparative analysis of different methods on image inpainting, which is a computationally costly part of the whole procedure, is provided. Our work is fortified with experiments in order to measure the computational performances and efficiencies of the proposed method and the ones existing in literature.

Infrared imagery is widely used in many applications in both civilian and military areas. In landmine detection, the goal is to detect the anomalies between mine surface and soil from variation of reflected/emitted thermal radiation. In this study, various types of anomaly detection techniques of IR are investigated and the feasibility of these techniques for use in landmine detection is analyzed. Additionally, effects of parameters for algorithms are compared and the parameters are optimized for increasing detection accuracy. Furthermore, fusion of the algorithms is performed to reduce False Alarm Rate (FAR). We also prepare an experimental setup to reflect the effects of environmental changes on FLIR imagery recording. Soil and various types of landmine mock-ups are also examined in this setup. Finally, all anomalies are mapped into local coordinate space for indicating possible landmines locations.

Super-resolution Reconstruction (SRR) is technique to increase the spatial resolution of images. It is especially useful for hyperspectral images (HSI), which have good spectral resolution but low spatial resolution. In this study, we propose an improvement to our previous work and present a novel MAP-MRF (maximum a posteriori-Markov random Fields) based approach for the SRR of HSI. The key point of our approach is to find the abundance maps of an HSI and perform SRR on the abundance maps using MRF based energy minimization, without needing any other additional source of information. In order to do so, first, PCA is used to determine the endmembers. Second, SISAL and fully constraint least squares (FCLS) are used to estimate the abundance maps. Third, in order to find the high resolution abundance maps, the ill-posed inverse SRR problem for abundances is regularized with a MAP-MRF based approach. The MAP-MRF formulation is restricted with the constraints which are specific to the abundances. Using the non-linear programming (NLP) techniques, the convex MAP formulation is minimized and High Resolution (HR) abundance maps are obtained. Then, these maps are used to construct the HR HSI. This improved SRR method is verified on real data sets, and quantitative performance comparison is achieved using PSNR, SSIM and PSNR metrics. Our results indicate that this improved method gives very close results to the original high resolution images, keeps the spectral consistency, and performs better than the compared algorithms.

In this paper, a decision level fusion using multiple pre-screener algorithms is proposed for the detection of buried landmines from Ground Penetrating Radar (GPR) data. The Kernel Least Mean Square (KLMS) and the Blob Filter pre-screeners are fused together to work in real time with less false alarms and higher true detection rates. The effect of the kernel variance is investigated for the KLMS algorithm. Also, the results of the KLMS and KLMS+Blob filter algorithms are compared to the LMS method in terms of processing time and false alarm rates. Proposed algorithm is tested on both simulated data and real data collected at the field of IPA Defence at METU, Ankara, Turkey.

This paper describes a novel pre-screener algorithm for landmine detection with a ground penetrating radar (GPR). The pre-screener algorithms are used for finding anomalies that are potential locations of interest. Thus, their processing time is as important as their true detection rate and false alarm rate. The proposed approach is based on Kernel Least Mean Square algorithm. Although Least Mean Square (LMS) based approach has already been used in the literature, KLMS based approach is a novel application for landmine detection with GPR. In this study, KLMS approach is compared with LMS approach in terms of processing time, false alarm rate, and true detection rate.

In this study, we propose a method for panoramic background subtraction by using Pan-Tilt cameras in real-time. The proposed method is based on parallelization of image registration, panorama generation and background subtraction operations to run on Graphics Processing Unit (GPU). Experiments results showed that GPU usage increases speed of the algorithm 33 times without considerable performance loss and makes working real-time possible.

The Sign Language Recognition (SLR) Problem is a highly important research topic, because of its ability to increase the interaction between the people who are hearing-impaired or impediment in speech. We propose a simple but robust system. The proposed system consists of three main steps. First we apply segmentation to the face and hand region by using Fuzzy C-Means Clustering (FCM) and Thresholding. FCM is a clustering technique which employs fuzzy partitioning, in an iterative algorithm. After the face and hands are segmented, the feature vectors are extracted. The feature vectors are chosen among the low level features such as the bounding ellipse, bounding box, and center of mass coordinates, since they are known to be more robust to segmentation errors due to low resolution images. In total there are 23 features for each hand. After the feature vectors are extracted, they are used for recognition with discrete Hidden Markov Model (HMM). Recognition stage is composed of two stages, namely training and classification. The Baum Welch algorithm is used for HMM training. In classification part the likelihood of each HMM is calculated and the HMM with the highest likelihood is chosen. In order to measure the success rate of the system, the eNTERFACE dataset is used. In this dataset 8 different American Sign Language example classified and in user independent case, is shown to be working with 94.19% accuracy.

Despite their high spectral resolution, hyperspectral images have low spatial resolution which adversely affects the applications that use hyperspectral images. In this study, instead of the traditional way of using spectral images, abundances of the endmembers are used in resolution enhancement. In the proposed method, first, endmembers are extracted with the SISAL algorithm. Then, the abundance maps are estimated using FCLS. From the low resolution abundance maps, high resolution abundance maps are obtained with a total variation based minimization. Finally, high resolution hyperspectral images are constructed from high resolution abundance maps. The proposed method is tested on real hyperspectral images. The experimental results and comparative analysis show the effectiveness of the proposed method.

Aggressive driving behavior is among the important causes of traffic accidents. Hence, detection of driver aggressiveness has an importance in terms of decreasing the number of traffic accidents. Collected driving data while the vehicle is in traffic can be used to make inferences about the aggressiveness of the driver. In this study, a multimodal method is proposed in order to detect driver aggressiveness. The proposed method is based on utilizing the visual data taken from the on vehicle camera and sensor data taken from the controller area network bus (CAN-bus) in order to decide whether the driving session involves aggressive driving behavior. Lane following pattern and vehicle following distance information is obtained from the data collected by camera while vehicle speed and engine speed information is obtained from CAN-bus. These information is fused to conceive feature vectors that represent the driving session and aggressiveness decision is made according to the classification of these feature vectors.

In this paper, we present an approach that can detect scene nudity level with high precision using different deep net configurations. For this purpose, a recent approach [1] which has intense and very deep convolution layers is used. During net modelling, we strive to obtain most successful net configuration by comparing different Dropout models and image sizes -64 × 64, 128 × 128-. Additionally, leveraging the generalization capability of Support Vector Machine (SVM), improvement on success rate is demonstrated by retraining the features obtained at different output levels of the nets with SVM. At test and training stages, scene is investigated under three nudity levels: regular, semi-nudity and full-nudity. In order to evaluate false alarm rates of the net models, tests are conducted on different datasets which are SUN2012, LEAR Human and a dataset contains only semi-nudity samples besides validation set determined for each class. The results indicate that high precision rates can be achieved with low false alarm rate exploiting deep net models.

8th International Conference on Quality of Multimedia Experience (QoMEX) -- JUN 06-08, 2016 -- Lisbon, PORTUGAL WOS: 000391251500046 In this study a novel, spatio-temporal characteristics based video content analysis method is presented. The proposed method has been evaluated on different video quality assessment databases, which include videos with different characteristics and distortion types. Test results obtained on different databases demonstrate the robustness and accuracy of the proposed content analysis method. Moreover, this analysis method is employed in order to examine the performance improvement in audiovisual quality assessment when the video content is taken into consideration. IEEE, IEEE Signal Proc Soc, NOS, YouTube, EURASIP, Qualinet, QoENet, Inst telecomunicacoes, Tecnico Lisboa

In this paper, we tackle content-based image-in-video search on large video archive. Particularly in this problem, the computation cost of a proposed method is another important parameter that should be considered along side of the success rate achieved due to the high volume of data. With our proposed method in this paper, content representation of multimedia data is achieved quite fast compared to other local approaches. Additionally, approximately % 16 improvement is introduced at the success rate with respect to the method that uses global approach for content representation on Stanford I2V dataset.

Animated meshes represented as sequences of static meshes sharing the same connectivity require efficient compression. Among the compression techniques, layered predictive coding methods efficiently encode the animated meshes in a structured way such that the successive reconstruction with an adaptable quality can be performed. The decoding quality heavily depends on how well the prediction is performed in the encoder. Due to this fact, in this paper, three novel prediction structures are proposed and integrated into a state of the art layered predictive coder. The proposed structures are based on weighted spatial prediction with its weighted refinement and angular relations of triangles between current and previous frames. The experimental results show that compared to the state of the art scalable predictive coder, up to 30% bitrate reductions can be achieved with the combination of proposed prediction schemes depending on the content and quantization level.

In this paper, we propose three novel multiple description coding (MDC) methods for reliable transmission of compressed animated meshes represented by series of 3D static meshes with same connectivity. The proposed methods trade off reconstruction quality for error resilience to provide the best expected reconstruction of 3D mesh sequence at the decoder side. The methods are based on layer duplication and partitioning of the set of vertices of a scalable coded animated mesh by either spatial or temporal subsampling. Each partitioned set is encoded separately to generate independently decodable bitstreams or so-called descriptions. In addition, the proposed MDC methods can achieve varying redundancy allocations by including a number of encoded spatial or temporal layers from the other description. The algorithms are evaluated with redundancy-rate-distortion (RRD) curves and per-frame reconstruction analysis. RRD performances show that vertex partitioning-based MDC performs better at low redundancies for especially spatially dense models. Temporal subsampling-based MDC performs better at moderate redundancies as well as low redundancies for spatially coarse models. Layer duplication-based MDC can achieve the lowest redundancies with flexible redundancy allocation capability and can be designed to achieve the smallest variance of reconstruction quality between consecutive frames.

This paper aims at providing an overview of the core technologies enabling the delivery of 3-D Media to next-generation mobile devices. To succeed in the design of the corresponding system, a profound knowledge about the human visual system and the visual cues that form the perception of depth, combined with understanding of the user requirements for designing user experience for mobile 3-D media, are required. These aspects are addressed first and related with the critical parts of the generic system within a novel user-centered research framework. Next-generation mobile devices are characterized through their portable 3-D displays, as those are considered critical for enabling a genuine 3-D experience on mobiles. Quality of 3-D content is emphasized as the most important factor for the adoption of the new technology. Quality is characterized through the most typical, 3-D-specific visual artifacts on portable 3-D displays and through subjective tests addressing the acceptance and satisfaction of different 3-D video representation, coding, and transmission methods. An emphasis is put on 3-D video broadcast over digital video broadcasting—handheld (DVB-H) in order to illustrate the importance of the joint source-channel optimization of 3-D video for its efficient compression and robust transmission over error-prone channels. The comparative results obtained identify the best coding and transmission approaches and enlighten the interaction between video quality and depth perception along with the influence of the context of media use. Finally, the paper speculates on the role and place of 3-D multimedia mobile devices in the future internet continuum involving the users in cocreation and refining of rich 3-D media content. © 2010 IEEE.

3D video based on stereo/multi-view representations is becoming widely popular. Real-time encoding/decoding of such video is an important concern as the number and spatial/temporal resolution of views increase. We present a systematic method for design and optimization of multi-threaded multi-view video encoding/decoding algorithms using multi-core processors and provide benchmark results for real-time decoding. The proposed multi-core decoding architectures are compliant with the current MVC extension of H.264/AVC international standard, and enable multi-threaded processing with negligible loss of encoding efficiency and minimum processing overhead. Benchmark results show that multi-core processors and multi-threading decoding are necessary for real-time high-definition multi-view video decoding and display.

Mobile devices such as mobile phones, personal digital assistants and personal video/game players are somehow converging and getting more powerful, thus enabling 3D mobile devices a reality. In order to store or transmit stereo video in these devices, coding techniques from both monoscopic video coding and multi-view video coding can be used. In this work, we analyze the possible stereoscopic encoding schemes for mobile devices. We have used rate-distortion curves for coding efficiency and decoding speed tests for decoder complexity. Depending on the processing power and memory of the mobile device, we concluded to use two of the settings used in our experiments.

This paper presents modelling and analysis of a mobile 3DTV broadcast system for handheld devices. The underlying technology behind our system is the DVB-H specification which brings ordinary TV broadcast services to battery-powered handheld receivers. In our system, we simulated the broadcast of stereo video content over DVB-H to mobile handheld devices with autostereoscopic displays. In this way, mobile users can watch 3D content without need for eyeglasses or any special equipment.

A European consortium of six partners has been developing core technological components of a mobile 3D television system over DVB-H channel. In this overview paper, we present our current results on developing optimal methods for stereo-video content creation, coding and transmission and emphasize their significance for the power-constrained mobile platform, equipped with auto-stereoscopic display. We address the user requirements by applying modern usercentered approaches taking into account different user groups and usage contexts in contrast to the laboratory assessment methods which, though standardized, offer limited applicability to real applications. To this end, we have been aiming at developing a methodological framework for the whole system development process. One of our goals has been to further develop the user-centered approach towards experienced quality of critical system components. In this paper, we classify different research methods and technological solutions analyzing their pros and constraints. Based on this analysis we present the user-centered methodological framework being used throughout the whole development process of the system and aimed at achieving the best performance and quality appealing to the end user.

We consider an error-resilient stereoscopic streaming system that uses an H.264-based multiview video codec and a rateless Raptor code for recovery from packet losses. One aim of the present work is to suggest a heuristic methodology for modeling the end-to-end rate-distortion (RD) characteristic of such a system. Another aim is to show how to make use of such a model to optimally select the parameters of the video codec and the Raptor code to minimize the overall distortion. Specifically, the proposed system models the RD curve of video encoder and performance of channel codec to jointly derive the optimal encoder bit rates and unequal error protection (UEP) rates specific to the layered stereoscopic video streaming. We define analytical RD curve modeling for each layer that includes the interdependency of these layers. A heuristic analytical model of the performance of Raptor codes is also defined. Furthermore, the distortion on the stereoscopic video quality caused by packet losses is estimated. Finally, analytical models and estimated single-packet loss distortions are used to minimize the end-to-end distortion and to obtain optimal encoder bit rates and UEP rates. The simulation results clearly demonstrate the significant quality gain against the nonoptimized schemes.

In this paper, we propose a distributed 3D dynamic mesh coding system. The system is based on Slepian and Wolf's and Wyner and Ziv's information-theoretic results. Our system extends the ideas in distributed video coding to 3D dynamic meshes with constant connectivity. The connectivity of the sequence and key frames are encoded and decoded by a conventional static mesh coder. The Wyner-Ziv frames are encoded independent of key frames but decoded jointly with decoded key frames. The joint decoding is performed by the low density parity check codes and the side information generated by linear interpolation of decoded key frames. Experimental results show that better rate-distortion performance is obtained compared to encoding each frame by a static mesh coder.