Search

• A feasible strategy was suggested to produce "finer grains + high-density TBs + precipitates" architectures in LPBF-fabricated Inconel 718. • A favorable strength and excellent plasticity (1088 MPa, 1369 MPa, 30 %) was obtained. • The intensive interactions between GBs/TBs/precipitates and dislocations under finer grain scale offers a strong strain hardening effect. • Deformation modes of SFs, L-C locks, primary DTs, and secondary twins sustaining an additional and higher strain hardening rate. Tailoring high-density annealing twins in laser powder bed fusion (LPBF)-fabricated alloys based on their intrinsic residual stress requires high annealing temperatures and/or long-term annealing, resulting in the abnormal growth of large recrystallized grains, which is detrimental to mechanical properties. This work proposes a new strategy for achieving a favorable strength–plasticity synergy of the LPBF-fabricated Inconel 718 superalloy by performing a deep cryogenic treatment (DCT) with the subsequent heat treatment (including annealing and double aging) to tailor fine grains with "high-density annealing twins + precipitates" architectures and compares the obtained material with an alloy subjected to a direct heat treatment without a prior DCT. The obtained results reveal that the additional internal stress generated during DCT increases the stored energy and dislocation density, which provide a sufficient driving force for activating high-density annealing twin boundaries (63.2 %) with fine grains (31.6 μm) within a short annealing time. The more homogeneous tailored microstructure with the "finer grains + high-density twins + precipitates" architectures decreases the mean free path of slipping dislocations, promoting intensive interactions with dislocations and inducing a strong strain hardening effect. The multiple deformation modes of stacking faults coupled with Lomer–Cottrell locks, thin primary deformation twins, and secondary twins activated during tensile loading, sustaining a strong work hardening ability and delaying the plastic instability, which exhibits a high strength (yield strength of 1088 MPa and tensile strength of 1369 MPa) and excellent plasticity (elongation of 30 %). This work not only describes a feasible method for simultaneously enhancing the strength and plasticity in additively manufactured (AM) alloys but also provides new insights into increasing the fraction of twins at a small grain size to improve the grain boundary-related properties without destroying the AM alloy shape. [Display omitted] [ABSTRACT FROM AUTHOR]

The present study investigates the synergistic enhancement of hot corrosion resistance (900 °C, 75%Na2SO4+25%NaCl) and mechanical properties of IN738LC alloy prepared by selective laser melting after heat treatment. The results indicate that the M23C6precipitation at the grain boundaries coupled with smaller-sized γ′ phase distribution in the inner grain is conducive to the formation of thicker Cr2O3outer layer and Al2O3inner layer for the samples, which effectively hinders the diffusion of S and Cl elements penetrate to the matrix, enhancing the hot corrosion resistance. Additionally, the presence of small-sized γ′ phase and re-crystallization-induced grain refinement contribute to an increase in yield strength (σYS = 1261 MPa) and ultimate tensile strength (σUTS = 1360 MPa) for the alloy.

High-purity tantalum is used as the sputtering target material for integrated circuits, requiring fine grains and random texture. Understanding different orientation deformation mechanisms, especially the metastable orientation (here {112} crystallographic orientations), is beneficial for controlling the structure of tantalum target materials. This paper employs EBSD and crystallographic calculations to regulate the grain deformation behavior by introducing a reverse strain at 135° clock rolling. The deformation behavior and dislocation density distribution of the metastable oriented grains under different strain paths are quantitatively analyzed. Results show that unidirectional rolling (UR) generates stress concentration, causing an uneven distribution of geometrically necessary dislocation density. The splitting behavior of {112} and {111} grains is promoted, and they have similar deformation behavior in UR process, while the deformation of {100} oriented grains is significantly suppressed. Furthermore, changing the strain path promotes the splitting of {100} grains, and makes the {112} change from the soft orientation to the hard orientation. This change makes it easier to start multi-slipping, resulting in a more uniform deformation. Additionally, crystallographic calculations show that Schmid factor is ineffective in explaining the strain localization problem in BCC metals. However, Schmid factor difference ratio makes it possible to analyze the distribution of dislocation density in local regions through the quantitative calculation of multi-slipping.

Diabetic retinopathy (DR) is the leading cause of preventable blindness worldwide. The risk of DR progression is highly variable among different individuals, making it difficult to predict risk and personalize screening intervals. We developed and validated a deep learning system (DeepDR Plus) to predict time to DR progression within 5 years solely from fundus images. First, we used 717,308 fundus images from 179,327 participants with diabetes to pretrain the system. Subsequently, we trained and validated the system with a multiethnic dataset comprising 118,868 images from 29,868 participants with diabetes. For predicting time to DR progression, the system achieved concordance indexes of 0.754–0.846 and integrated Brier scores of 0.153–0.241 for all times up to 5 years. Furthermore, we validated the system in real-world cohorts of participants with diabetes. The integration with clinical workflow could potentially extend the mean screening interval from 12 months to 31.97 months, and the percentage of participants recommended to be screened at 1–5 years was 30.62%, 20.00%, 19.63%, 11.85% and 17.89%, respectively, while delayed detection of progression to vision-threatening DR was 0.18%. Altogether, the DeepDR Plus system could predict individualized risk and time to DR progression over 5 years, potentially allowing personalized screening intervals.

The point cloud keypoint detection algorithm like USIP that uses downsampling first and then fine-tuning the sampling points cannot effectively detect the defect part of the single view defect point cloud, resulting in the inability to output the keypoints of the defect part. Therefore, this paper proposes the twin structure key point detection algorithm named TSKPD based on the idea of contrastive learning, which uses two single-view defect point clouds to synthesize relatively more complete key points for learning, so as to promote the network model to learn the features of the complete point cloud. The robustness of key point detection of point cloud is effectively improved, and the detection of single view defect point cloud is realized. The test results on ModelNet40 and ShapeNet datasets show that the coverage rate of TSKPD on the missing part of the single view defect point cloud is 12.62 higher than the existing optimal algorithm.

Primary diabetes care and diabetic retinopathy (DR) screening persist as major public health challenges due to a shortage of trained primary care physicians (PCPs), particularly in low-resource settings. Here, to bridge the gaps, we developed an integrated image–language system (DeepDR-LLM), combining a large language model (LLM module) and image-based deep learning (DeepDR-Transformer), to provide individualized diabetes management recommendations to PCPs. In a retrospective evaluation, the LLM module demonstrated comparable performance to PCPs and endocrinology residents when tested in English and outperformed PCPs and had comparable performance to endocrinology residents in Chinese. For identifying referable DR, the average PCP’s accuracy was 81.0% unassisted and 92.3% assisted by DeepDR-Transformer. Furthermore, we performed a single-center real-world prospective study, deploying DeepDR-LLM. We compared diabetes management adherence of patients under the unassisted PCP arm (n= 397) with those under the PCP+DeepDR-LLM arm (n= 372). Patients with newly diagnosed diabetes in the PCP+DeepDR-LLM arm showed better self-management behaviors throughout follow-up (P< 0.05). For patients with referral DR, those in the PCP+DeepDR-LLM arm were more likely to adhere to DR referrals (P< 0.01). Additionally, DeepDR-LLM deployment improved the quality and empathy level of management recommendations. Given its multifaceted performance, DeepDR-LLM holds promise as a digital solution for enhancing primary diabetes care and DR screening.

Few-shot semantic segmentation (FSS) aims to segment objects of unseen classes in query images with only a few annotated support images. Existing FSS algorithms typically focus on mining category representations from the single-view support to match semantic objects of the single-view query. However, the limited annotated samples render the single-view matching struggle to perceive the varying characteristics of novel objects, which results in a restricted learning space for novel categories and further induces a biased segmentation with demoted parsing performance. To address this challenge, inspired by the semantic transform invariance, this paper proposes a fresh few-shot segmentation framework to break the bias and perform invariant segmentation in a multi-view matching manner. Specifically, original and transform support features from different perspectives with the same semantics are learnable fused to obtain the transform invariance prototype with a stronger category representation ability. Simultaneously, aiming at providing better parsing guidance, the Transform Invariance Guidance Mask Generation (TIGM) module is proposed to integrate prior knowledge from different perspectives. Finally, segmentation predictions from varying views are complementarily merged in the Transform Invariance Semantic Prediction (TISP) module to decide the uncertain area and yield precise segmentation predictions. Extensive experiments on both PASCAL- $5^{i}$ and COCO- $20^{i}$ datasets demonstrate the effectiveness of our approach and show that our method could achieve state-of-the-art performance. Code is available at https://github.com/caoql98/BBD.

Lane detection is a critical component of autonomous driving systems and has been the subject of extensive research. Unlike object detection, identifying car lanes requires extracting features from multi-scale information since they are slender, sparse, and distributed in the entire image. Unfortunately, many previous works have overlooked this requirement, resulting in less robust lane feature extraction. To address this issue, we propose an attention mechanism that is exceptional at extracting global and long-dependence lane features compared to CNNs. Our attention-based structure, called attention in attention, explores the relationship between various correlations and mitigates mismatched correlation problems in attention computation. Furthermore, we introduce a novel feature fusion structure in the backbone called the double feature pyramid network, which effectively gathers feature information with various dimensions and enlarges the receptive field. Our network is based on the BEV-LaneDet and achieves impressive performance on the OpenLane dataset. Notably, experiments demonstrate that our method surpasses BEV-LaneDet by 4.4% in terms of F-Score on OpenLane.

World models learn the consequences of actions in vision-based interactive systems. However, in practical scenarios like autonomous driving, noncontrollable dynamics that are independent or sparsely dependent on action signals often exist, making it challenging to learn effective world models. To address this issue, we propose Iso-Dream++, a model-based reinforcement learning approach that has two main contributions. First, we optimize the inverse dynamics to encourage the world model to isolate controllable state transitions from the mixed spatiotemporal variations of the environment. Second, we perform policy optimization based on the decoupled latent imaginations, where we roll out noncontrollable states into the future and adaptively associate them with the current controllable state. This enables long-horizon visuomotor control tasks to benefit from isolating mixed dynamics sources in the wild, such as self-driving cars that can anticipate the movement of other vehicles, thereby avoiding potential risks. On top of our previous work (Pan et al. 2022), we further consider the sparse dependencies between controllable and noncontrollable states, address the training collapse problem of state decoupling, and validate our approach in transfer learning setups. Our empirical study demonstrates that Iso-Dream++ outperforms existing reinforcement learning models significantly on CARLA and DeepMind Control.

Few-shot aerial image semantic segmentation is a challenging task that requires precisely parsing unseen-category objects in query aerial images with limited annotated support aerial images. Formally, category prototypes would be extracted from support samples to segment query images in a pixel-to-pixel matching manner. However, aerial objects in aerial images are often distributed with arbitrary orientations, and varying orientations could cause a dramatic feature change. This unique property of aerial images renders conventional matching manner without consideration of orientations fails to activate same-category objects with different orientations. Furthermore, the oscillation of the confidence scores in existing rotation-insensitive algorithms, engendered by the striking changes of object orientations, often leads to false recognition of lower scored rotated semantic objects. To tackle these challenges, inspired by the intrinsic rotation invariance in aerial images, we propose a novel few-shot rotation-invariant aerial semantic segmentation network (FRINet) to efficiently segment aerial semantic objects with diverse orientations. Specifically, through extracting orientation-varying yet category-consistent support information, FRINet provides rotation-adaptive matching for each query feature in a feature-aggregation manner. Meanwhile, to encourage consistent predictions for aerial objects with arbitrary orientations, segmentation predictions from different orientations are supervised by the same label and further fused to obtain the final rotation-invariant prediction in a complementary manner. Moreover, aiming at providing a better solution searching space, the backbones are newly pretrained in the base category to basically boost the segmentation performance. Extensive experiments on the few-shot aerial image semantic segmentation benchmark demonstrate that the proposed FRINet achieves a new state-of-the-art performance. The code is available at https://github.com/caoql98/FRINet.

The high-quality pathological microscopic images are essential for physicians or pathologists to make a correct diagnosis. Image quality assessment (IQA) can quantify the visual distortion degree of images and guide the imaging system to improve image quality, thus raising the quality of pathological microscopic images. Current IQA methods are not ideal for pathological microscopy images due to their specificity. In this paper, we present deep learning-based blind image quality assessment model with saliency block and patch block for pathological microscopic images. The saliency block and patch block can handle the local and global distortions, respectively. To better capture the area of interest of pathologists when viewing pathological images, the saliency block is fine-tuned by eye movement data of pathologists. The patch block can capture lots of global information strongly related to image quality via the interaction between different image patches from different positions. The performance of the developed model is validated by the home-made Pathological Microscopic Image Quality Database under Screen and Immersion Scenarios (PMIQD-SIS) and cross-validated by the five public datasets. The results of ablation experiments demonstrate the contribution of the added blocks. The dataset and the corresponding code are publicly available at: https://github.com/mikugyf/PMIQD-SIS.

Mesh is a type of data structure commonly used for 3-D shapes. Representation learning for 3-D meshes is essential in many computer vision and graphics applications. The recent success of convolutional neural networks (CNNs) for structured data (e.g., images) suggests the value of adapting insights from CNN for 3-D shapes. However, 3-D shape data are irregular since each node’s neighbors are unordered. Various graph neural networks for 3-D shapes have been developed with isotropic filters or predefined local coordinate systems to overcome the node inconsistency on graphs. However, isotropic filters or predefined local coordinate systems limit the representation power. In this article, we propose a local structure-aware anisotropic convolutional operation (LSA-Conv) that learns adaptive weighting matrices for each template’s node according to its neighboring structure and performs shared anisotropic filters. In fact, the learnable weighting matrix is similar to the attention matrix in the random synthesizer—a new Transformer model for natural language processing (NLP). Since the learnable weighting matrices require large amounts of parameters for high-resolution 3-D shapes, we introduce a matrix factorization technique to notably reduce the parameter size, denoted as LSA-small. Furthermore, a residual connection with a linear transformation is introduced to improve the performance of our LSA-Conv. Comprehensive experiments demonstrate that our model produces significant improvement in 3-D shape reconstruction compared to state-of-the-art methods.

Recently neural architecture (NAS) search has attracted great interest in academia and industry. It remains a challenging problem due to the huge search space and computational costs. Recent studies in NAS mainly focused on the usage of weight sharing to train a SuperNet once. However, the corresponding branch of each subnetwork is not guaranteed to be fully trained. It may not only incur huge computation costs but also affect the architecture ranking in the retraining procedure. We propose a multi-teacher-guided NAS, which proposes to use the adaptive ensemble and perturbation-aware knowledge distillation algorithm in the one-shot-based NAS algorithm. The optimization method aiming to find the optimal descent directions is used to obtain adaptive coefficients for the feature maps of the combined teacher model. Besides, we propose a specific knowledge distillation process for optimal architectures and perturbed ones in each searching process to learn better feature maps for later distillation procedures. Comprehensive experiments verify our approach is flexible and effective. We show improvement in precision and search efficiency in the standard recognition dataset. We also show improvement in correlation between the accuracy of the search algorithm and true accuracy by NAS benchmark datasets.

Graph matching (GM) has been a long-standing combinatorial problem due to its NP-hard nature. Recently (deep) learning-based approaches have shown their superiority over the traditional solvers while the methods are almost based on supervised learning which can be expensive or even impractical. We develop a unified unsupervised framework from matching two graphs to multiple graphs, without correspondence ground truth for training. Specifically, a Siamese-style unsupervised learning framework is devised and trained by minimizing the discrepancy of a second-order classic solver and a first-order (differentiable) Sinkhorn net as two branches for matching prediction. The two branches share the same CNN backbone for visual graph matching. Our framework further allows unsupervised learning with graphs from a mixture of modes which is ubiquitous in reality. Specifically, we develop and unify the graduated assignment (GA) strategy for matching two-graph, multi-graph, and graphs from a mixture of modes, whereby two-way constraint and clustering confidence (for mixture case) are modulated by two separate annealing parameters, respectively. Moreover, for partial and outlier matching, an adaptive reweighting technique is developed to suppress the overmatching issue. Experimental results on real-world benchmarks including natural image matching show our unsupervised method performs comparatively and even better against two-graph based supervised approaches.

The rapid development of deep learning has made a great progress in image segmentation, one of the fundamental tasks of computer vision. However, the current segmentation algorithms mostly rely on the availability of pixel-level annotations, which are often expensive, tedious, and laborious. To alleviate this burden, the past years have witnessed an increasing attention in building label-efficient, deep-learning-based image segmentation algorithms. This paper offers a comprehensive review on label-efficient image segmentation methods. To this end, we first develop a taxonomy to organize these methods according to the supervision provided by different types of weak labels (including no supervision, inexact supervision, incomplete supervision and inaccurate supervision) and supplemented by the types of segmentation problems (including semantic segmentation, instance segmentation and panoptic segmentation). Next, we summarize the existing label-efficient image segmentation methods from a unified perspective that discusses an important question: how to bridge the gap between weak supervision and dense prediction – the current methods are mostly based on heuristic priors, such as cross-pixel similarity, cross-label constraint, cross-view consistency, and cross-image relation. Finally, we share our opinions about the future research directions for label-efficient deep image segmentation.

This paper reports a quasi in-situ study on the plastic deformation behavior of high purity tantalum, particularly the slip activation and slip transfer, using the electron backscatter diffraction method. We found that the rotation degree of grains is in the range of 0–15°; the rotation axes of slip-activated grains are mainly concentrated in the three corners, while the axes of slip-transfer ones are concentrated near {111} corner in the stereographic triangle. Microstructural parameters including grain size and grain aspect ratio were found to have no apparent relationships with the slip activation or slip transfer process. Most m'factors for the grains underwent slip-transfer are larger than 0.27, and most slip systems involved in the slip-transfer observations are associated with relatively high Schmid factors (SFs), indicating that slip would more easily transfer through the boundaries with large m'factors. In addition, according to the distribution of geometrical necessary dislocation density and the SF of slipping activated grains, we found the slip transfer process is affected by a combination of factors, including the SF, m'factor, and the strain concentration in local regions.

Small and cluttered objects are common in real-world which are challenging for detection. The difficulty is further pronounced when the objects are rotated, as traditional detectors often routinely locate the objects in horizontal bounding box such that the region of interest is contaminated with background or nearby interleaved objects. In this paper, we first innovatively introduce the idea of denoising to object detection. Instance-level denoising on the feature map is performed to enhance the detection to small and cluttered objects. To handle the rotation variation, we also add a novel IoU constant factor to the smooth L1 loss to address the long standing boundary problem, which to our analysis, is mainly caused by the periodicity of angular (PoA) and exchangeability of edges (EoE). By combing these two features, our proposed detector is termed as SCRDet++. Extensive experiments are performed on large aerial images public datasets DOTA, DIOR, UCAS-AOD as well as natural image dataset COCO, scene text dataset ICDAR2015, small traffic light dataset BSTLD and our released S $^{2}$2 TLD by this paper. The results show the effectiveness of our approach. The released dataset S $^{2}$2 TLD is made public available, which contains 5,786 images with 14,130 traffic light instances across five categories.

Capturing the interactions of human articulations lies in the center of skeleton-based action recognition. Recent graph-based methods are inherently limited in the weak spatial context modeling capability due to fixed interaction pattern and inflexible shared weights of GCN. To address above problems, we propose the multi-view interactional graph network (MV-IGNet) which can construct, learn and infer multi-level spatial skeleton context, including view-level (global), group-level, joint-level (local) context, in a unified way. MV-IGNet leverages different skeleton topologies as multi-views to cooperatively generate complementary action features. For each view, separable parametric graph convolution (SPG-Conv) enables multiple parameterized graphs to enrich local interaction patterns, which provides strong graph-adaption ability to handle irregular skeleton topologies. We also partition the skeleton into several groups and then the higher-level group contexts including inter-group and intra-group, are hierarchically captured by above SPG-Conv layers. A simple yet effective global context adaption (GCA) module facilitates representative feature extraction by learning the input-dependent skeleton topologies. Compared to the mainstream works, MV-IGNet can be readily implemented while with smaller model size and faster inference. Experimental results show the proposed MV-IGNet achieves impressive performance on large-scale benchmarks: NTU-RGB+D and NTU-RGB+D 120.

Learning to re-identify or retrieve a group of people across non-overlapped camera systems has important applications in video surveillance. However, most existing methods focus on (single) person re-identification (re-id), ignoring the fact that people often walk in groups in real scenarios. In this work, we take a step further and consider employing context information for identifying groups of people, i.e., group re-id. On the one hand, group re-id is more challenging than single person re-id, since it requires both a robust modeling of local individual person appearance (with different illumination conditions, pose/viewpoint variations, and occlusions), as well as full awareness of global group structures (with group layout and group member variations). On the other hand, we believe that person re-id can be greatly enhanced by incorporating additional visual context from neighboring group members, a task which we formulate as group-aware (single) person re-id. In this paper, we propose a novel unified framework based on graph neural networks to simultaneously address the above two group-based re-id tasks, i.e., group re-id and group-aware person re-id. Specifically, we construct a context graph with group members as its nodes to exploit dependencies among different people. A multi-level attention mechanism is developed to formulate both intra-group and inter-group context, with an additional self-attention module for robust graph-level representations by attentively aggregating node-level features. The proposed model can be directly generalized to tackle group-aware person re-id using node-level representations. Meanwhile, to facilitate the deployment of deep learning models on these tasks, we build a new group re-id dataset which contains more than $3.8K$3.8K images with $1.5K$1.5K annotated groups, an order of magnitude larger than existing group re-id datasets. Extensive experiments on the novel dataset as well as three existing datasets clearly demonstrate the effectiveness of the proposed framework for both group-based re-id tasks.

Anomalies are ubiquitous in real-world time-series data which call for effective and timely detection, especially in an unsupervised setting for labeling cost saving. In this paper, we develop an unsupervised density reconstruction model for multi-dimensional time-series anomaly detection. In particular, it directly handles an important realistic setting that the detection is achieved towards raw time-series contaminated with noise for training, in contrast to most existing anomaly detection works that assume the training data is in general clean i.e., not contaminated with anomaly. It extends recent advancements in deep generative models and state space models to achieve robust anomaly detection. Our approach comprises of a novel state space based generative model, a filtering based inference model, together with a carefully-designated emission model based on robust statistics theory. Extensive experimental results are conducted to show that our approach can adapt to complex patterns even given severely contaminated training data. We also develop visualization techniques to help better understand the behavior of the anomaly detection models. Empirical results show that our method outperforms state-of-the-arts on both synthetic and real-world datasets.

Siamese trackers have received a lot of attentions due to their promising performance and real-time high speed. However, the robustness is generally limited especially in challenging conditions, such as occlusion. Motivated by that tracking failure does not always occur and the target movement tends to follow certain patterns, in the paper, we propose a generic, fast and flexible approach to improve the robustness of Siamese trackers with two light-load novel modules: Trajectory Guidance Module (TGM) and Selective Refinement Module (SRM). Specifically, TGM encourages to pay a soft attention on possible target location based on short-term historical trajectory. SRM selectively remedies the tracking results at the risk of failure with little impact on the speed. The proposed algorithm can be easily establish upon state-of-the-art Siamese trackers and obtains better performance on seven benchmarks with high real-time tracking speed. The code is available at https://github.com/TJUMMG/TGSR.

Time-series and event sequences are widely collected data types in real-world applications. Modeling and forecasting of such temporal data play an important role in an informed decision-making process. A major limitation of previous methods is that they either focus on time-series or events, rather than the combination of the two worlds. In fact, the two types of data often provide complementary information, emphasizing the necessity of jointly modeling the both. In this paper, we propose the RNN-ODE collaborative model for joint modeling and forecasting of heterogeneous time-series and event sequence data, which combines several useful techniques from both Bayesian and deep learning for its interpretability. Specifically, we devise a tailored encoder to combine the advances in deep temporal point processes models and variational recurrent neural networks. To predict the probability of event occurrence over an arbitrary continuous-time horizon, we base our model on the mathematical foundation of Neural Ordinary Differential Equations (NODE). Extensive experimental results on simulations and real data sets show that compared with existing methods, our integrated approach can achieve more competitive forecasting performance of both time-series and event sequences.

The explosive growth of image data facilitates the fast development of image processing and computer vision methods for emerging visual applications, meanwhile introducing novel distortions to processed images. This poses a grand challenge to existing blind image quality assessment (BIQA) models, which are weak at adapting to subpopulation shift. Recent work suggests training BIQA methods on the combination of all available human-rated IQA datasets. However, this type of approach is not scalable to a large number of datasets and is cumbersome to incorporate a newly created dataset as well. In this paper, we formulate continual learning for BIQA, where a model learns continually from a stream of IQA datasets, building on what was learned from previously seen data. We first identify five desiderata in the continual setting with three criteria to quantify the prediction accuracy, plasticity, and stability, respectively. We then propose a simple yet effective continual learning method for BIQA. Specifically, based on a shared backbone network, we add a prediction head for a new dataset and enforce a regularizer to allow all prediction heads to evolve with new data while being resistant to catastrophic forgetting of old data. We compute the overall quality score by a weighted summation of predictions from all heads. Extensive experiments demonstrate the promise of the proposed continual learning method in comparison to standard training techniques for BIQA, with and without experience replay. We made the code publicly available at https://github.com/zwx8981/BIQA_CL.

Graphical abstract:

We consider a new problem of adapting a human mesh reconstruction model to out-of-domain streaming videos, where the performance of existing SMPL-based models is significantly affected by the distribution shift represented by different camera parameters, bone lengths, backgrounds, and occlusions. We tackle this problem through online adaptation, gradually correcting the model bias during testing. There are two main challenges: First, the lack of 3D annotations increases the training difficulty and results in 3D ambiguities. Second, non-stationary data distribution makes it difficult to strike a balance between fitting regular frames and hard samples with severe occlusions or dramatic changes. To this end, we propose the Dynamic Bilevel Online Adaptation algorithm (DynaBOA). It first introduces the temporal constraints to compensate for the unavailable 3D annotations and leverages a bilevel optimization procedure to address the conflicts between multi-objectives. DynaBOA provides additional 3D guidance by co-training with similar source examples retrieved efficiently despite the distribution shift. Furthermore, it can adaptively adjust the number of optimization steps on individual frames to fully fit hard samples and avoid overfitting regular frames. DynaBOA achieves state-of-the-art results on three out-of-domain human mesh reconstruction benchmarks.

Graph matching aims to establish node correspondence between two graphs, which has been a fundamental problem for its NP-hard nature. One practical consideration is the effective modeling of the affinity function in the presence of noise, such that the mathematically optimal matching result is also physically meaningful. This paper resorts to deep neural networks to learn the node and edge feature, as well as the affinity model for graph matching in an end-to-end fashion. The learning is supervised by combinatorial permutation loss over nodes. Specifically, the parameters belong to convolutional neural networks for image feature extraction, graph neural networks for node embedding that convert the structural (beyond second-order) information into node-wise features that leads to a linear assignment problem, as well as the affinity kernel between two graphs. Our approach enjoys flexibility in that the permutation loss is agnostic to the number of nodes, and the embedding model is shared among nodes such that the network can deal with varying numbers of nodes for both training and inference. Moreover, our network is class-agnostic. Experimental results on extensive benchmarks show its state-of-the-art performance. It bears some generalization capability across categories and datasets, and is capable for robust matching against outliers.

To reduce noncommutative error (NCE) and improve the attitude-update accuracy in the attitude algorithm of the strapdown inertial navigation system (SINS), we assume that angular rate is represented as a polynomial to deduce the relationship between the angular increment sampling sequence and the derivatives of the quaternion. Subsequently, a high-order numerical solution of the quaternion differential equation is obtained by substituting the derivatives of the quaternion into its Taylor expansion. Generally, the quaternion will be constructed with a series of angular increment samples in a constant updating interval. Furthermore, the piecewise quaternion high-order numerical (PWQHN) algorithm is deduced by reconstructing quaternion in each sampling interval to ulteriorly improve attitude accuracy. The simulation results indicate that the PWQHN algorithm is stable in two kinds of simulation conditions. In the coning motion condition, PWQHN algorithm has the highest accuracy. In the high-dynamic condition, the extreme accuracy of PWQHN is not significantly higher than other algorithms. However, PWQHN can still output accurate attitude in the situation without many angular increment subsamples.

Temporal data are ubiquitous in real-world applications, and they can be generally divided into two categories: 1) synchronous temporal data which are basically equivalent to time series data; and 2) the asynchronous data which are often in the form of event data with a time stamp in continuous time-space. In fact, the event data are often converted to the time series by aggregating the event count in equal time intervals in many previous approaches. While it is often of one’s greater interest to directly establish models based on the raw event data whose time stamps carry useful information, especially for those time-sensitive tasks, ranging from earthquake prediction, crime analysis, to infectious disease diffusion forecasting, etc. Developing the spatio-temporal point process and the related applications is the theme of this Special Issue, which treats an event as a point in the spatio-temporal space, with possibly extra attributes. The model captures the instantaneous happening rate of the events and their potential dependency. The derived use cases often refer to future events prediction, and causality estimation.

Rare diseases, which are severely underrepresented in basic and clinical research, can particularly benefit from machine learning techniques. However, current learning-based approaches usually focus on either mono-modal image data or matched multi-modal data, whereas the diagnosis of rare diseases necessitates the aggregation of unstructured and unmatched multi-modal image data due to their rare and diverse nature. In this study, we therefore propose diagnosis-guided multi-to-mono modal generation networks (TMM-Nets) along with training and testing procedures. TMM-Nets can transfer data from multiple sources to a single modality for diagnostic data structurization. To demonstrate their potential in the context of rare diseases, TMM-Nets were deployed to diagnose the lupus retinopathy (LR-SLE), leveraging unmatched regular and ultra-wide-field fundus images for transfer learning. The TMM-Nets encoded the transfer learning from diabetic retinopathy to LR-SLE based on the similarity of the fundus lesions. In addition, a lesion-aware multi-scale attention mechanism was developed for clinical alerts, enabling TMM-Nets not only to inform patient care, but also to provide insights consistent with those of clinicians. An adversarial strategy was also developed to refine multi- to mono-modal image generation based on diagnostic results and the data distribution to enhance the data augmentation performance. Compared to the baseline model, the TMM-Nets showed 35.19% and 33.56% F1 score improvements on the test and external validation sets, respectively. In addition, the TMM-Nets can be used to develop diagnostic models for other rare diseases.

The microstructure evolution is very complex during the processing of tantalum sputtering target for integrated circuits, and the deformation mechanisms for this evolution is still unclear. This work aims to investigate the deformation mechanism in a quasi-in-situ way, particularly the activation of slip systems, lattice rotation, and strain concentration, utilizing the electron backscatter diffraction (EBSD) method. We found that the maximal orientation density increased after the deformation, especially the content of both the {111} and {100} components increased while the {110} orientations decreased after the deformation. The lattice rotation leads to misorientation between grain orientations before and after the deformation, and the misorientation axes seem to be randomly distributed during the deformation. The rotation behavior can be adequately described by the activation of both {110}<111> and {112}<111> slip systems, which were found to have no linear relationship with grain size or grain aspect ratio, but are related to the Schmid factors and strain concentration situations. In particular, the inhomogeneous geometrical necessary dislocations distribution indicates the complicated situation of flow stress, and the slip systems activated with smaller Schmid factors are associated with the local heterogeneous strain concentrations.

Extraction of moving objects & an important and fundamental research topic for many digital video applications. This paper addresses an efficient spatio-temporal segmentation scheme to extract moving objects from video sequences. The temporal segmentation yields a temporal mask that indicates moving regions and static regions for each frame. For localization of moving objects, a block-based motion detection method considering a novel feature measure is proposed to detect changed regions. These changed regions are coarse and need accurate spatial compensation. An edge-based morphological dilation method is presented to achieve the anisotropic expansion of the changed regions. Furthermore, to solve the temporarily stopping problem of moving objects, the inertia information of moving objects is considered in the temporal segmentation. The spatial segmentation based on the watershed algorithm is performed to provide homogeneous regions with closed and precise boundaries. It considers the global information to improve the accuracy of the boundaries. To reduce over-segmentation in the watershed segmentation, a novel mean filter is proposed to suppress some minima. A fusion of the spatial and temporal segmentation results produces complete moving objects faithfully. Compared with the reference algorithms, the fusion threshold in our scheme is fixed for different sequences. Experiments on typical sequences have successfully demonstrated the validity of the proposed scheme. Index Terms--watershed, dilation, moving object, spatio-temporal segmentation.

Traffic prediction generated by multimedia sources is needed for effective dynamic bandwidth allocation and for multimedia quality of services (QoS) control strategies implemented at the network edges. In this paper, An [H.sub.[infinity]]--optimal mode is proposed for video traffic prediction, which not only minimizes the worst possible effect in the estimation error but also does not require a priori knowledge of the noise source statistics. The proposed method has been applied to modeling and prediction MPEG-4 variable bit rate (VBR) video traffic. Simulation results show that the proposed model is more robust compared with the Kalman prediction model and is able to predict the original traffic more accurately, thus more suitable for real time video traffic prediction and resource reservation. Index Terms--[H.sub.[infinity]] model, video traffic, VBR, MPEG-4 video.

We described a challenge named “DRAC - Diabetic Retinopathy Analysis Challenge” in conjunction with the 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022). Within this challenge, we provided the DRAC datset, an ultra-wide optical coherence tomography angiography (UW-OCTA) dataset (1,103 images), addressing three primary clinical tasks: diabetic retinopathy (DR) lesion segmentation, image quality assessment, and DR grading. The scientific community responded positively to the challenge, with 11, 12, and 13 teams submitting different solutions for these three tasks, respectively. This paper presents a concise summary and analysis of the top-performing solutions and results across all challenge tasks. These solutions could provide practical guidance for developing accurate classification and segmentation models for image quality assessment and DR diagnosis using UW-OCTA images, potentially improving the diagnostic capabilities of healthcare professionals. The dataset has been released to support the development of computer-aided diagnostic systems for DR evaluation.

Current top-performing saliency prediction methods of omnidirectional images (ODIs) depend on deep feedforward convolutional neural networks (CNNs), benefiting from their powerful multiscale representation ability. Although these methods adopt deep feedforward CNNs to achieve superb performance in saliency prediction task, they have the following limitations: 1) these deep feedforward CNNs are difficult to map to ventral stream structure of the brain visual system due to their vast number of layers and missing biologically important connections, such as recurrence and 2) most deep feedforward CNNs represent the multiscale features in a layerwise manner. To tackle these issues, models that could learn multiscale features yet share the similarities with human brain are needed. In this article, we propose a novel multiscale brain-like network (MBN) model to predict saliency of head fixations on ODIs. Specifically, our proposed model consists of two major modules: 1) a brain-like CORnet-S module and 2) a multiscale feature extraction module. The CORnet-S module is a lightweight backbone network with four anatomically mapped areas (V1, V2, V4, and IT) and it can simulate the visual processing mechanism of ventral visual stream in the human brain. The multiscale feature extraction module is inspired by the multiscale brain structure, which represents multiscale features at a granular level and increases the range of receptive fields for each network layer. Extensive experiments and ablation studies conducted on two major benchmarks demonstrate the superiority of the proposed MBN model over the state-of-the-art methods.

Real-world sequential data are often generated based on complicated and latent mechanisms, which can be formulated as event sequences occurring in the continuous time domain. In addition, continuous signals may often be associated with event sequences and be formulated as time series with fixed time lags. Traditionally, event sequences are often modeled by parametric temporal point processes, which use explicitly defined conditional intensity functions to quantify the occurrence rates of events. However, these parametric models often merely take one-side information from event sequences into account while ignoring the information from concurrent time series, and their intensity functions are usually designed for specific tasks dependent on prior knowledge. To tackle the above-mentioned problems, we propose a model called recurrent point process networks which instantiates temporal point process models with temporal recurrent neural networks (RNNs). In particular, the intensity functions of the proposed model are modeled by two RNNs: one temporal RNN capturing the relationships among events and the other RNN updating intensity functions based on time series. Furthermore, an attention mechanism is introduced, which uncovers influence strengths among events with good interpretability. Focusing on challenging tasks such as temporal event prediction and underlying relational network mining, we demonstrate the superiority of our model on both synthetic and real-world data.

The increasing prevalence of diabetes, high avoidable morbidity and mortality due to diabetes and diabetic complications, and related substantial economic burden make diabetes a significant health challenge worldwide. A shortage of diabetes specialists, uneven distribution of medical resources, low adherence to medications, and improper self-management contribute to poor glycemic control in patients with diabetes. Recent advancements in digital health technologies, especially artificial intelligence (AI), provide a significant opportunity to achieve better efficiency in diabetes care, which may diminish the increase in diabetes-related health-care expenditures. Here, we review the recent progress in the application of AI in the management of diabetes and then discuss the opportunities and challenges of AI application in clinical practice. Furthermore, we explore the possibility of combining and expanding upon existing digital health technologies to develop an AI-assisted digital health-care ecosystem that includes the prevention and management of diabetes.

Phrase-based visual grounding aims to localise the object in the image referred by a textual query phrase. Most existing approaches adopt a two-stage mechanism to address this problem: first, an off-the-shelf proposal generation model is adopted to extract region-based visual features, and then a deep model is designed to score the proposals based on the query phrase and extracted visual features. In contrast to that, the authors design an end-to-end approach to tackle the visual grounding problem in this study. They use a region proposal network to generate object proposals and the corresponding visual features simultaneously, and multi-modal factorised bilinear pooling model to fuse the multi-modal features effectively. After that, two novel losses are posed on top of the multi-modal features to rank and refine the proposals, respectively. To verify the effectiveness of the proposed approach, the authors conduct experiments on three real-world visual grounding datasets, namely Flickr-30k Entities, ReferItGame and RefCOCO. The experimental results demonstrate the significant superiority of the proposed method over the existing state-of-the-arts.

The true strain data and true stress data are obtained from the isothermal compression tests under a wide range of strain rates (0.1–20 s−1) and temperatures (933–1,133 K) over the Gleeble-3500 thermomechanical simulator. The data are employed to generate the constitutive equations according to four constitutive models, respectively, the strain-compensated Arrhenius-type model, the modified Zerilli–Armstrong (ZA) model, the modified Johnson–Cook (JC) model and the JC model. In the meanwhile, a comparative research was made over the capacities of these four models and hence to represent the elevated temperature flow behavior of TA2. Besides, a comparison of the accuracy of the predictions of average absolute relative error, correlation coefficient (R) and the deformation behavior was made to test the sustainability level of these four models. It is shown from these results that the JC model is not suitable for the description of flow behavior of TA2 alloy in α+βphase domain, while the predicted values of modified JC model, modified ZA model and the strain-compensated Arrhenius-type model could be consistent well with the experimental values except under some deformation conditions. Moreover, the strain-compensated Arrhenius-type model can be also used to track the deformation behavior more precisely in comparison with other models.

A fundamental bottleneck for achieving highly discriminative action representation is that local motion/appearance features are usually not semantic aligned. Namely, a local feature, such as a motion vector or motion trajectory, does not possess any attribute that indicates which moving body part or operated object it is associated with. This mostly leads to global feature pooling/representation learning methods that are often too coarse. Inspired by the recent success of end-to-end (pixel-to-pixel) deep convolutional neural networks (DCNNs), in this paper, we first propose a DCNN architecture, which maps a human centric image region onto human body part response maps. Based on these response maps, we propose a second DCNN, which achieves semantic-aligned feature representation learning. Prior knowledge that only a few parts are responsible for a certain action is also utilized by introducing a group (part) sparseness prior during feature learning. The learned semantic-aligned feature not only boosts the discriminative capability of action representation, but also possesses the good nature of robustness to pose variations and occlusions. Finally, an iterative mining method is employed for learning discriminative action primitive detectors. Extensive experiments on action recognition benchmarks demonstrate a superior recognition performance of the proposed framework.

To promote element diffusion in the solid-liquid two-phase region, optimize solidification microstructure, and improve the matching degree of strength and toughness, Ti–46Al–8Nb-2.6C-xTa (at. %) alloys were prepared by arc melting under the action of ultrasound. The lamellar colony, precipitate phase, elements distribution, and related mechanisms were analyzed. Results show that the relative content of the γ phase decreases, while the relative content of the α2phase and Ti2AlC phase increases as the addition of Ta increases. The coarse long rod-shaped Ti2AlC transforms into short rod-shaped particles. The lamellar colony size reduces from 32.1 to 18.6 μm, the lamellar spacing reduces from 1.4 to 0.7 μm, and the aspect ratio of Ti2AlC reduces from 5.8 to 2.5 as the addition of Ta increases from 0 to 1.0 at. %. The acoustic streaming effect of ultrasound promotes the diffusion of tantalum in the solid-liquid two-phase region for a certain ultrasonic processing time. Tantalum is a heterogeneous nucleation particle of Ti2AlC particles and causes composition supercooling at the forefront of the solidification interface, which is the main reason for refining the microstructure. Compressive strength increases from 1996 to 2596 MPa and the strain increases from 25.4 to 29.3% as the addition of Ta increases from 0 to 0.6 at. %. The main reasons for improving the properties of alloy are the refinement of the microstructure and the formation of a large number of dislocations.

We described a challenge named “Diabetic Retinopathy (DR)—Grading and Image Quality Estimation Challenge” in conjunction with ISBI 2020 to hold three sub-challenges and develop deep learning models for DR image assessment and grading. The scientific community responded positively to the challenge, with 34 submissions from 574 registrations. In the challenge, we provided the DeepDRiD dataset containing 2,000 regular DR images (500 patients) and 256 ultra-widefield images (128 patients), both having DR quality and grading annotations. We discussed details of the top 3 algorithms in each sub-challenges. The weighted kappa for DR grading ranged from 0.93 to 0.82, and the accuracy for image quality evaluation ranged from 0.70 to 0.65. The results showed that image quality assessment can be used as a further target for exploration. We also have released the DeepDRiD dataset on GitHub to help develop automatic systems and improve human judgment in DR screening and diagnosis.

By employing a quasi-in-situ method, we investigated the crystallographic texture evolution process of high purity tantalum during compression deformation. The influence of grain size, aspect ratio, and initial orientations on the crystallographic lattice rotation process was analyzed in-depth using the electron backscatter diffraction technique. It is found that the randomly distributed grain orientations mainly evolve toward {111} and {100} orientations during compression, forming the two typical deformation textures in body-centered cubic metals. Furthermore, the crystallographic rotation direction and degree have no apparent linear relationship with grain size or grain aspect ratio but are mainly related to the grain orientation. Especially, the slight crystallographic rotation angle for {100} grain is primarily associated with the activation of multiple slip systems. Multiple slip systems mean different slip planes and different slip directions, and it is difficult for {100} grains to be dominated by a certain slip system and rotate to a large extent like {111} grains. In comparison, the large crystallographic rotation angle for {110} grain is mainly related to the strain coordination in local regions. {110} grains are easier to active slip systems than grains with other crystal plane orientations inferred from the critical shear stress, thereby the rotation of grains and inhomogeneous deformation occurred during the compression process.

Twitter displays the tweets a user received in a reversed chronological order, which is not always the best choice. As Twitter is full of messages of very different qualities, many informative or relevant tweets might be flooded or displayed at the bottom while some nonsense buzzes might be ranked higher. In this work, we present a supervised learning method for personalized tweets reordering based on user interests. User activities on Twitter, in terms of tweeting, retweeting, and replying, are leveraged to obtain the training data for reordering models. Through exploring a rich set of social and personalized features, we model the relevance of tweets by minimizing the pairwise loss of relevant and irrelevant tweets. The tweets are then reordered according to the predicted relevance scores. Experimental results with real twitter user activities demonstrated the effectiveness of our method. The new method achieved above 30% accuracy gain compared with the default ordering in twitter based on time.

Many evolving video services and applications for intelligent security systems require reliable transmission of high quality video to diverse clients over heterogeneous networks using available system resources. Scalable video coding (SVC) is one of the emerging video compression technologies with such potential capabilities. Advances in lifting-based motion-compensated temporal filtering (MCTF) have enabled highly efficient and flexible spatial, temporal, signal-to-noise ratio (SNR), and complexity scalability to be realized over a wide range of bit rates. In this paper, we present an algorithm to improve the update step of MCTF, which serves as an important informative step for the coding performance of SVC. A novel update-step algorithm, which takes advantage of the chrominance information of the video sequence and the correlation of the motion vectors (MVs) of the neighboring blocks as well as the correlation of the derived update MVs in the low-pass frames, is proposed to improve update step of MCTF by (1) computing correct update motion information, (2) generating correct amount of energy contained in the high-pass frames. Experimental results show that the proposed algorithm can significantly improve the quality of the reconstructed video sequence in visual quality.

We present an efficient rate control strategy for H.264 in order to maximize the video quality by appropriately determining the quantization parameter (QP) for each macroblock. To break the chicken-and-egg dilemma resulting from QP-dependent rate-distortion optimization (RDO) in H.264, a preanalysis phase is conducted to gain the necessary source information, and then the coarse QP is decided for rate-distortion (RD) estimation. After motion estimation, we further refine the QP of each mode using the obtained actual standard deviation of motion-compensated residues. In the encoding process, RDO only performs once for each macroblock, thus one-pass, while QP determination is conducted twice. Therefore, the increase of computational complexity is small compared to that of the JM 9.3 software. Experimental results indicate that our rate control scheme with two-step QP determination but single-pass encoding not only effectively improves the average PSNR but also controls the target bit rates well.

Conventional statistical multiplexing methods for MPEG-1/2 programs involve high computationally complex transcoding (decoding and re-encoding) process to convert the original bit-rate into the target bit-rate. To avoid a time-consuming transcoding process, a new statistical multiplexer is proposed to enable content provider to easily convert the bit-rates by exploiting the MPEG-4 fine granularity scalability (FGS) coding scheme. The proposed statistical multiplexer is particularly useful for multiple-program broadcasting applications, including Internet television and video on demand, as well as value-added MPEG-4 video streaming services for DVB and ATSC digital TV systems. The proposed multiplexer mainly includes two parts: the FGS-based frame lag scheme and the optimal bit-plane truncation scheme. The FGS-based frame lag scheme exploits intra- and inter-layer correlations exist in MPEG-4 FGS bit-streams. The optimal bit-plane truncation scheme dynamically truncates enhancement layers of FGS bit-streams under the available bandwidth constraint and the quality/smoothness constraint. Experimental results show that high statistical multiplexing efficiency, inter-program fairness, and intra-program smoothness are achieved by the proposed multiplexer.

Abstract Fully automatic image colorization is extensively developed using a large number of examples. However, this kind of methods need further detailed revision if users require fine colorization for delicate objects. This paper proposes a novel scribble-based colorization scheme which automatically generates scribbles and propagates color information from representative pixels within homogeneous superpixels. The scribbles are placed in the regions with minimal entropy to ensure color propagation from representative pixels with high confidence. To avoid color bleeding and under colorization, colors are propagated under a robust similarity metric using a set of hierarchical features, where the low level features consist of quaternion Gabor phases and the high level features are extracted using VGG-19 pre-trained deep model. Experimental results on open dataset demonstrate that the proposed colorization scheme can achieve more consistent results with the users' purposes. [ABSTRACT FROM AUTHOR]

We present a novel postprocessing algorithm for blocking artifact removal in the discrete Hadamard transform (DHT) domain, which does not require prior knowledge of quantization parameters and features low computational complexity. All block-based video coding methods suffer annoying blocking artifacts at low bit rates. We first acquire edge information for the frame, then calculate block activities from DHT coefficients so as to classify smooth and coarse regions adaptively. Blocking artifacts are adaptively filtered in the DHT domain according to block activities. Experimental results show that the proposed method is able to remove blocking artifacts effectively while preserving image details and object edges well, and that it achieves better visual quality compared with other methods.