Search

Image deblurring based only on the blurry image is challenging as motion information is lost while imaging. Event cameras capture the texture of moving objects in high temporal resolution with asynchronous events. In this paper, we extract motion features from events and fuse them with background features from the image for event-based image deblurring. Spiking neural network (SNN), a widely recognized event feature extractor, is well suited for motion feature extraction due to its high temporal resolution. However, extracting motion information from events exclusively with SNN is challenging. We propose a novel Temporal-local-Spatio Spiking Transformer (TSST) to extract motion intensity and motion attention regions in the spatio-temporal domain. Motion intensity extracted from spiking features is represented as a high temporal resolution motion attention map to guide the fusion of the two networks. In the temporal domain, motion intensity maps spiking features to CNN features as motion features to avoid blurring. In the spatial domain, the motion intensity shows the motion regions and gives the weight of the motion feature during fusion. Moreover, a hybrid feature extraction encoder (HFEE) is introduced, which fully fuses the motion and background features for deblurring. The gradient is back-propagated from CNN to SNN, and the hybrid deblurring network is jointly optimized. We evaluated the performance of our model on the public dataset GoPro and a real event dataset we captured. Codes and pretrained models are available at https://github.com/XDULzx/MotionSNN.

Despite the considerable effort devoted to high-generalizable blind image quality assessment (BIQA), the generalization performance of the state-of-the-art metrics remains limited when facing new visual scenes. A straightforward way to address the dilemma is labeling a great number of images from the new scene and subsequently training a new model, which is quite labor-intensive and cost-expensive. Hence, there is an urgent need to mitigate the dependency on labeled samples by designing a data-efficient BIQA algorithm. Motivated by the above facts, this paper presents an Active Learning-based IQA (AL-IQA) framework, which reduces the requirement for training samples by selecting representative images from two perspectives, including distortion and content. Specifically, in terms of distortion, we design distortion prompts and adopt Contrastive Language-Image Pre-Training (CLIP) to predict image distortion in a zero-shot manner. Then, we employ curriculum learning-inspired strategy to select samples with gradually increasing difficulty (measured by prediction uncertainty of CLIP), in order to facilitate model training. Meantime, in terms of content, we adopt distribution matching-based dataset distillation to distill unlabeled images into several high-density informative synthetic images. Then, feature distances between unlabeled images and distilled images are compared to identify images with the most representative content. Finally, Borda count is adopted to capture a consensus of both distortion and content through weighted counting, and prompt tuning is utilized for adapting the model to the IQA task. Extensive experiments are conducted on five IQA datasets, and the results demonstrate that the proposed AL-IQA not only effectively reduces the number of training samples but also achieves state-of-the-art prediction accuracy and generalization performance. The source code is available at https://github.com/esnthere/AL-IQA.

The task of hyperspectral image (HSI) classification is fundamental and crucial in HSI processing. Currently, domain adaptive methods have become a research hotspot in HSI classification. However, most domain adaptive methods ignore the class alignment in different domains. Additionally, HSIs have the characteristics of category imbalance and complex spatial–spectral distribution, which restricts the adaptation performance in HSIs. To address these problems, a class-aligned and class-balancing generative domain adaptation (CCGDA) method is proposed for HSI classification. The architecture of CCGDA is designed by using the classifier, domain discriminator, sampler, and two weight-sharing generators. In the classifier, split-level capsule network (CapsNet) is constructed by extracting rich spatial information of shallow layer and spectral features of deep layer with equivariant characteristic. Then, the classifier provides the pseudo-label of samples in the target domain. To prevent the generators from mode collapse caused by category imbalance, the sampler is designed. It samples and resamples the samples of the target domain in an adaptive proportion according to the statistical calculation through confidence and distribution of pseudo-labels. Finally, a novel class-aligned domain adversarial loss is defined to jointly optimize the generators and discriminator. It incorporates the class shift adjusting and adaptive sampling for the samples of the target domain to better adapt the discriminant boundary of the classifier to the target domain. Experiments on benchmark HSI datasets verify the superiority of the proposed method for domain adaptive classification.

Omnidirectional image quality assessment (OIQA) has become a research hotspot in the field of image processing. The existing OIQA metrics have the common problem of incomplete perceptual information representation, which results in performance limitation. To deal with this problem, a perceptual information completion-based Siamese (PICS) OIQA network is proposed. Specifically, the input OI is first reprojected to generate another OI with complementary perceptual information. Both the original input OI and the generated OI are simultaneously fed into the following QA network, forming the Siamese network structure. For each branch of the QA network, the multiresolution local features and the complementary global features are severally extracted by convolution operations and self-attention modules and then are hierarchically aggregated to more fully simulate the visual perception mechanism. Finally, the aggregated features of the output of two branches are further fused to measure the quality of the OI to be tested. Through extensive experiments on two public OIQA datasets and deep analysis of the experimental results, the performance advancement of the PICS method is demonstrated in the OIQA task.

People usually assess image aesthetics according to visual attributes, e.g., interesting content, good lighting and vivid color, etc. Further, the perception of visual attributes depends on the image theme. Therefore, the inherent relationship between visual attributes and image theme is crucial for image aesthetics assessment (IAA), which has not been comprehensively investigated. With this motivation, this paper presents a new IAA model based on Theme-Aware Visual Attribute Reasoning (TAVAR). The underlying idea is to simulate the process of human perception in image aesthetics by performing bilevel reasoning. Specifically, a visual attribute analysis network and a theme understanding network are first pre-trained to extract aesthetic attribute features and theme features, respectively. Then, the first level Attribute-Theme Graph (ATG) is built to investigate the coupling relationship between visual attributes and image theme. Further, a flexible aesthetics network is introduced to extract general aesthetic features, based on which we built the second level Attribute-Aesthetics Graph (AAG) to mine the relationship between theme-aware visual attributes and aesthetic features, producing the final aesthetic prediction. Extensive experiments on four public IAA databases demonstrate the superiority of the proposed TAVAR model over the state-of-the-arts. Furthermore, TAVAR features better explainability due to the use of visual attributes.

Depth quality of stereoscopic three-dimensional (S3D) videos is a significant factor which directly affects the quality of experience (QoE) associated with 3D video applications and services. Nevertheless, there remain limited reports on the investigation of depth perception and depth quality evaluation of S3D videos, which impedes further advancement and deployment of 3D video technology. This paper reports a series of subjective experiments which have been conducted to investigate the depth perception and its related properties of the human visual system (HVS) using S3D video compressed by the H.264/AVC standard. The experimental results reveal that the HVS response in depth perception varies at different frequencies and in varying orientations, and the distortions introduced by video coding can cause the loss of and/or variation in depth perception. By integration of binocular and monocular features (BM) extracted from left and right views of S3D video with respect to depth perception, a depth quality assessment model, herein referred to as BM-DQAM, is devised by training these stereoscopic and spatial orientation structural features with a support vector regression model. It is shown that the BM-DQAM provides a novel no-reference metric for evaluation of the depth quality of S3D videos. Based on two publicly available 3D video databases and the proposed depth perception assessment database, the experimental results show that the BM-DQAM has demonstrated better performance in assessing the depth quality in S3D video viewing than that of other metrics reported in the published literatures, correlating well with the HVS response in the depth perception assessment experiment.

Image reconstruction from partial observations has attracted increasing attention. Conventional image reconstruction methods with hand-crafted priors often fail to recover fine image details due to the poor representation capability of the hand-crafted priors. Deep learning methods attack this problem by directly learning mapping functions between the observations and the targeted images can achieve much better results. However, most powerful deep networks lack transparency and are nontrivial to design heuristically. This paper proposes a novel image reconstruction method based on the Maximum a Posterior (MAP) estimation framework using learned Gaussian Scale Mixture (GSM) prior. Unlike existing unfolding methods that only estimate the image means (i.e., the denoising prior) but neglected the variances, we propose characterizing images by the GSM models with learned means and variances through a deep network. Furthermore, to learn the long-range dependencies of images, we develop an enhanced variant based on the Swin Transformer for learning GSM models. All parameters of the MAP estimator and the deep network are jointly optimized through end-to-end training. Extensive simulation and real data experimental results on spectral compressive imaging and image super-resolution demonstrate that the proposed method outperforms existing state-of-the-art methods.

Both network pruning and neural architecture search (NAS) can be interpreted as techniques to automate the design and optimization of artificial neural networks. In this paper, we challenge the conventional wisdom of training before pruning by proposing a joint search-and-training approach to learn a compact network directly from scratch. Using pruning as a search strategy, we advocate three new insights for network engineering: 1) to formulate adaptive search as a cold start strategy to find a compact subnetwork on the coarse scale; and 2) to automatically learn the threshold for network pruning; 3) to offer flexibility to choose between efficiency and robustness. More specifically, we propose an adaptive search algorithm in the cold start by exploiting the randomness and flexibility of filter pruning. The weights associated with the network filters will be updated by ThreshNet, a flexible coarse-to-fine pruning method inspired by reinforcement learning. In addition, we introduce a robust pruning strategy leveraging the technique of knowledge distillation through a teacher-student network. Extensive experiments on ResNet and VGGNet have shown that our proposed method can achieve a better balance in terms of efficiency and accuracy and notable advantages over current state-of-the-art pruning methods in several popular datasets, including CIFAR10, CIFAR100, and ImageNet. The code associate with this paper is available at: https://see.xidian.edu.cn/faculty/wsdong/Projects/AST-NP.htm.

Lignocellulosic biomass is still considered a feasible source of bioethanol production. Saccharomyces cerevisiaecan adapt to detoxify lignocellulose-derived inhibitors, including furfural. Tolerance of strain performance has been measured by the extent of the lag phase for cell proliferation following the furfural inhibitor challenge. The purpose of this work was to obtain a tolerant yeast strain against furfural through overexpression of YPR015Cusing the in vivohomologous recombination method. The physiological observation of the overexpressing yeast strain showed that it was more resistant to furfural than its parental strain. Fluorescence microscopy revealed improved enzyme reductase activity and accumulation of oxygen reactive species due to the harmful effects of furfural inhibitor in contrast to its parental strain. Comparative transcriptomic analysis revealed 79 genes potentially involved in amino acid biosynthesis, oxidative stress, cell wall response, heat shock protein, and mitochondrial-associated protein for the YPR015Coverexpressing strain associated with stress responses to furfural at the late stage of lag phase growth. Both up- and down-regulated genes involved in diversified functional categories were accountable for tolerance in yeast to survive and adapt to the furfural stress in a time course study during the lag phase growth. This study enlarges our perceptions comprehensively about the physiological and molecular mechanisms implicated in the YPR015Coverexpressing strain’s tolerance under furfural stress.

The neuromorphic event cameras can efficiently sense the latent geometric structures and motion clues of a scene by generating asynchronous and sparse event signals. Due to the irregular layout of the event signals, how to leverage their plentiful spatio-temporal information for recognition tasks remains a significant challenge. Existing methods tend to treat events as dense image-like or point-serie representations. However, they either suffer from severe destruction on the sparsity of event data or fail to encode robust spatial cues. To fully exploit their inherent sparsity with reconciling the spatio-temporal information, we introduce a compact event representation, namely 2D-1T event cloud sequence (2D-1T ECS). We couple this representation with a novel light-weight spatio-temporal learning framework (ECSNet) that accommodates both object classification and action recognition tasks. The core of our framework is a hierarchical spatial relation module. Equipped with specially designed surface-event-based sampling unit and local event normalization unit to enhance the inter-event relation encoding, this module learns robust geometric features from the 2D event clouds. And we propose a motion attention module for efficiently capturing long-term temporal context evolving with the 1T cloud sequence. Empirically, the experiments show that our framework achieves par or even better state-of-the-art performance. Importantly, our approach cooperates well with the sparsity of event data without any sophisticated operations, hence leading to low computational costs and prominent inference speeds.

The prevalence of short-video applications imposes more requirements for video quality assessment (VQA). User-generated content (UGC) videos are captured under an unprofessional environment, thus suffering from various dynamic degradations, such as camera shaking. To cover the dynamic degradations, existing recurrent neural network-based UGC-VQA methods can only provide implicit modeling, which is unclear and difficult to analyze. In this work, we consider explicit motion representation for dynamic degradations, and propose a motion-enhanced UGC-VQA method based on decomposition and recomposition. In the decomposition stage, a dual-stream decomposition module is built, and VQA task is decomposed into single frame-based quality assessment problem and cross frames-based motion understanding. The dual streams are well grounded on the two-pathway visual system during perception, and require no extra UGC data due to knowledge transfer. Hierarchical features from shallow to deep layers are gathered to narrow the gaps from tasks and domains. In the recomposition stage, a progressively residual aggregation module is built to recompose features from the dual streams. Representations with different layers and pathways are interacted and aggregated in a progressive and residual manner, which keeps a good trade-off between representation deficiency and redundancy. Extensive experiments on UGC-VQA databases verify that our method achieves the state-of-the-art performance and keeps a good capability of generalization. The source code will be available in https://github.com/Sissuire/DSD-PRO.

Target detection becomes increasingly important in hyperspectral image (HSI) analysis but is limited by difficulties in acquiring accurate pixel-level training labels. This article proposes a multiple-instance metric learning neural network (MIML-Net) for hyperspectral target detection tasks, which only requires region-level labels and greatly alleviates the laborious pixel-level annotation problems. Our method learns the embeddings of regions with weak labels under attention-based multiple-instance learning (MIL) framework. Based on which, we impose a novel metric-based regularizer to constrain target and background embeddings to two learnable compact clusters with distinct centroids, which further boosts the spectral feature representation ability. The proposed metric-based regularizer enforces a discriminative detector due to its capability to reduce the intraclass variations and encourage the interclass separations simultaneously. Extensive experimental results from both simulated and real-field datasets demonstrate the effectiveness of the proposed MIML-Net in comparison with the state-of-the-art weakly supervised techniques.

Conventional wisdom in model-based computational imaging incorporates physics-based imaging models, noise characteristics, and image priors into a unified Bayesian framework. Rapid advances in deep learning have inspired a new generation of data-driven computational imaging systems with performances even better than those of their model-based counterparts. However, the design of learning-based algorithms for computational imaging often lacks transparency, making it difficult to optimize the entire imaging system in a complete manner.

Despite the impressive progress of supervised methods in quality assessment for in- the-wild videos, models trained on one domain often fail to generalize well to others due to the domain shifts caused by distortion diversity and content variation. Domain generalizable video quality assessment (VQA) methods that can work across domains remain an open research challenge. Although combining more data following the mixed-domain training strategy can improve the generalization performance to a certain extent, the specific knowledge from each source domain, which could potentially be useful for improving unseen domain generalization, is ignored in this principle. Motivated by this, we propose a domain generalizable VQA method named Dynamic Ensemble of Expert-Knowledge (DEEK), a novel framework that dynamically exploits the expert-knowledge from each source domain to achieve a generalizable ensemble prediction. Specifically, based on the multiple experts each trained to specialize in a particular source domain, we aim to exploit complementary information provided by the expert-knowledge. We effectively train an ensemble model by proposing a quality-sensitive InfoNCE loss to regularize the collaborative training of all experts in the contrastive learning formulation, aiming to exploit complementary information provided by the expert-knowledge when forming the ensemble. By dynamically integrating the experts according to their relevances to the target data, these expert-knowledge could be leveraged for better generalization. Experiments on five VQA datasets verify that our approach outperforms the state-of-the-arts by large margins.

In multidimensional signal processing, such as image and video processing, superresolution (SR) imaging is a classical problem. Over the past 25 years, academia and industry have been interested in reconstructing high-resolution (HR) images from their low-resolution (LR) counterparts. We review the development of SR technology in this tutorial article based on the evolution of key insights associated with the prior knowledge or regularization method from analytical representations to data-driven deep models. The coevolution of SR with other technical fields, such as autoregressive modeling, sparse coding, and deep learning, will be highlighted in both model-based and learning-based approaches. Model-based SR includes geometry-driven, sparsity-based, and gradient-profile priors; learning-based SR covers three types of neural network (NN) architectures, namely residual networks (ResNet), generative adversarial networks (GANs), and pretrained models (PTMs). Both model-based and learning-based SR are united by highlighting their limitations from the perspective of model-data mismatch. Our new perspective allows us to maintain a healthy skepticism about current practice and advocate for a hybrid approach that combines the strengths of model-based and learning-based SR. We will also discuss several open challenges, including arbitrary-ratio, reference-based, and domain-specific SR.

Blind image quality assessment (BIQA) for authentic distortions is still a great challenge, even in today’s deep learning era. It has been widely acknowledged that local and global features are both indispensable for IQA, which play complementary roles. While combining local and global features is straightforward in traditional handcrafted feature-based IQA metrics, it is not an easy task in the deep learning framework. This is mainly due to the fact that deep neural networks typically require input images with a fixed size. Current metrics either resize the image or use local patches as input, which are problematic in that they cannot integrate local and global aspects as well as their interactions to achieve comprehensive quality evaluation. Motivated by the above facts, this paper presents a new BIQA metric for authentic distortions by aggregating local and global deep features in a Vision-Transformer framework. In the proposed metric, selective local regions and global content are simultaneously input for complementary feature extraction, and the Vision-Transformer is employed to build the relationship between different local patches and image quality. Self-attention mechanism is further adopted to explore the interaction between local and global deep features, producing the final image quality score. Extensive experiments on five authentically distorted IQA databases demonstrate that the proposed metric outperforms the state-of-the-arts in terms of both prediction performance and generalization ability.

Herein, we report a preparation method based on a modified zeolite imidazolate skeleton loose nanofiltration membrane used to achieve the efficient separation of salt and dye from dye wastewater. In this study, negatively charged hydrophilic TA@ZIF-8 was synthesized using a tannin surface-functionalization-assisted etching process by tannic acid (TA). A novel, highly negatively charged, and loose nanofiltration membrane was then prepared by adding TA@ZIF-8 to a polyethyleneimine (PEI) and tricarboxylic acid (TMA) crosslinking system. The addition of TA@ZIF-8 nanoparticles not only changed the degree of crosslinking of PEI/TMA but also provided a preferential channel for water molecules, thereby improving the flux of the membrane. Then, the success of TA@ZIF-8 NPs and membranes preparation were verified by SEM, XRD, FT-IR, TGA, BET, XPS and other methods. The results showed that the optimal preparation conditions were 0.020 wt% TA@ZIF-8 loading, and the contact angle of the membranes decreased from 79.8° to 53.7°. The water flux of the membrane is 81 L·m−2·h−1·bar−1, which is 1.5 times that of PEI/TMA membrane. In addition, TA@ZIF-8–2 membrane not only has high rejection rate for Methyl blue (98.7 %), Toluidine blue (89.6 %), Cr6+(91.2 %), but also maintains high transmittance for NaCl and Na2SO4. Besides, the separation performance of the membrane was stable for a long time in a mixed solution of dye and salt and excellent antifouling performance. Therefore, the loose TA@ZIF-8/PEI/TMA nanofiltration membrane has broad application prospects in printing and dyeing wastewater treatments.

The concept of just noticeable difference (JND), which accounts for the visibility threshold (visual redundancy) of the human visual system, is useful in perception-oriented signal processing systems. In this work, we present a comprehensive review of JND estimation technology. First, the visual mechanism and its corresponding computational modules are illustrated. These include luminance adaptation, contrast masking, pattern masking, and the contrast sensitivity function. Next, the existing pixel domain and subband domain JND models are presented and analyzed. Finally, the challenges associated with JND estimation are discussed.

In this study, aggregation-induced emission luminogens (AIEgen) are used for the detection of per- and poly-fluoroalkyl substances (PFAS) including perfluorooctanoic acid (PFOA), perfluorooctanesulfonic acid (PFOS) and 1H,1H,2H,2H-perfluorooctanesulfonic acid (6:2FTS). A solution of acetone–water (solvent) containing AIE-PFAS is first formulated. From this solution, one droplet (1–2 μL) is dropped and trapped into a hole in a chip made of glass slide. When the droplet is exposed to air, both water and acetone can vaporise, but acetone vaporises much more quickly than water does due to its higher vapour pressure. Consequently, with the solvent volume shrinking, the concentrations of AIE and PFAS increase in the shrinking droplet while the water percentage increases. At a certain stage of this process, the micelle of PFAS surfactant forms, accompanied by AIE aggregation. After completely vaporising and drying on the bottom of the chip-hole, the aggregated AIE features fluorescence, the density of which can be effectively linked to the concentration of PFAS surfactant. Thus, a simple sensor is developed for PFAS detection in the range of 0.1–100 μM (41 ppb to 41 ppm for PFOA) within around 1 min using a 1–2 μL PFAS sample.

Image enhancement is a popular technique, which is widely used to improve the visual quality of images. While image enhancement has been extensively investigated, the relevant quality assessment of enhanced images remains an open problem, which may hinder further development of enhancement techniques. In this paper, a no-reference quality metric for digitally enhanced images is proposed. Three kinds of features are extracted for characterizing the quality of enhanced images, including non-structural information, sharpness and naturalness. Specifically, a total of 42 perceptual features are extracted and used to train a support vector regression (SVR) model. Finally, the trained SVR model is used for predicting the quality of enhanced images. The performance of the proposed method is evaluated on several enhancement-related databases, including a new enhanced image database built by the authors. The experimental results demonstrate the efficiency and advantage of the proposed metric.

In this work, the NH2-MIL-125/F127/PES membranes with superior ultrafiltration, anti-fouling and self-cleaning were constructed via the RTIPS method. Characterizations showed the successful fabrication of NH2-MIL-125 with good photocatalysis activities, together with its stable anchoring in PES membrane. The loss of NH2-MIL-125 was suppressed by hydrogen bond between F127 and NH2-MIL-125, where hydrophilic F127 significantly boosted permeability and anti-fouling property. The permeability-selectivity trade-off effect was alleviated by NH2-MIL-125 with a regular size window. It was observed that the membranes prepared by the RTIPS method occurred a spongy-like cross-section and a porous surface, which imparted the membrane with improved ultrafiltration and mechanical properties over the finger-like cross-section and dense skin caused by the NIPS method. The optimal membranes exhibited decent permeability (1511 L/m2h), BSA rejection rate (85.3%) and high COD removal rate over the commercial membranes, whose flux recovery rate was up to 76.91% after two cycles under light irradiation by the synergistic effect of F127 and NH2-MIL-125. The membrane fouled with RhB exhibited excellent regeneration properties benefiting from the holes (h+) and superoxide free radicals (•O2−), as well as reactive •OH. Overall, the membrane with F127 and NH2-MIL-125 presented a highly promising in improving ultrafiltration and self-cleaning via the RTIPS method.

Without Abstract: