Search

Multimodal Large Language Models (MLLMs) have shown significant progress in offline video understanding. However, applying these models to real-world scenarios, such as autonomous driving and human-computer interaction, presents unique challenges due to the need for real-time processing of continuous online video streams. To this end, this paper presents systematic efforts from three perspectives: evaluation benchmark, model architecture, and training strategy. First, we introduce OVBench, a comprehensive question-answering benchmark specifically designed to evaluate models' ability to perceive, memorize, and reason within online video contexts. It features six core task types across three temporal contexts-past, present, and future-forming 16 subtasks from diverse datasets. Second, we propose a new Pyramid Memory Bank (PMB) that effectively retains key spatiotemporal information in video streams. Third, we proposed an offline-to-online learning paradigm, designing an interleaved dialogue format for online video data and constructing an instruction-tuning dataset tailored for online video training. This framework led to the development of VideoChat-Online, a robust and efficient model for online video understanding. Despite the lower computational cost and higher efficiency, VideoChat-Online outperforms existing state-of-the-art offline and online models across popular offline video benchmarks and OVBench, demonstrating the effectiveness of our model architecture and training strategy.

We discuss the landscape of flavor physics at the Circular Electron-Positron Collider (CEPC), based on the nominal luminosity outlined in its Technical Design Report. The CEPC is designed to operate in multiple modes to address a variety of tasks. At the $Z$ pole, the expected production of 4 Tera $Z$ bosons will provide unique and highly precise measurements of $Z$ boson couplings, while the substantial number of boosted heavy-flavored quarks and leptons produced in clean $Z$ decays will facilitate investigations into their flavor physics with unprecedented precision. We investigate the prospects of measuring various physics benchmarks and discuss their implications for particle theories and phenomenological models. Our studies indicate that, with its highlighted advantages and anticipated excellent detector performance, the CEPC can explore beauty and $\tau$ physics in ways that are superior to or complementary with the Belle II and Large-Hadron-Collider-beauty experiments, potentially enabling the detection of new physics at energy scales of 10 TeV and above. This potential also extends to the observation of yet-to-be-discovered rare and exotic processes, as well as testing fundamental principles such as lepton flavor universality, lepton and baryon number conservation, etc., making the CEPC a vibrant platform for flavor physics research. The $WW$ threshold scan, Higgs-factory operation and top-pair productions of the CEPC further enhance its merits in this regard, especially for measuring the Cabibbo-Kobayashi-Maskawa matrix elements, and Flavor-Changing-Neutral-Current physics of Higgs boson and top quarks. We outline the requirements for detector performance and considerations for future development to achieve the anticipated scientific goals.

Recently, Text-to-Image (T2I) generation models have achieved significant advancements. Correspondingly, many automated metrics have emerged to evaluate the image-text alignment capabilities of generative models. However, the performance comparison among these automated metrics is limited by existing small datasets. Additionally, these datasets lack the capacity to assess the performance of automated metrics at a fine-grained level. In this study, we contribute an EvalMuse-40K benchmark, gathering 40K image-text pairs with fine-grained human annotations for image-text alignment-related tasks. In the construction process, we employ various strategies such as balanced prompt sampling and data re-annotation to ensure the diversity and reliability of our benchmark. This allows us to comprehensively evaluate the effectiveness of image-text alignment metrics for T2I models. Meanwhile, we introduce two new methods to evaluate the image-text alignment capabilities of T2I models: FGA-BLIP2 which involves end-to-end fine-tuning of a vision-language model to produce fine-grained image-text alignment scores and PN-VQA which adopts a novel positive-negative VQA manner in VQA models for zero-shot fine-grained evaluation. Both methods achieve impressive performance in image-text alignment evaluations. We also use our methods to rank current AIGC models, in which the results can serve as a reference source for future study and promote the development of T2I generation. The data and code will be made publicly available.

Diffusion models (DMs) have demonstrated exceptional performance in text-to-image (T2I) tasks, leading to their widespread use. With the introduction of classifier-free guidance (CFG), the quality of images generated by DMs is improved. However, DMs can generate more harmful images by maliciously guiding the image generation process through CFG. Some safe guidance methods aim to mitigate the risk of generating harmful images but often reduce the quality of clean image generation. To address this issue, we introduce the Harmful Guidance Redirector (HGR), which redirects harmful CFG direction while preserving clean CFG direction during image generation, transforming CFG into SafeCFG and achieving high safety and quality generation. We train HGR to redirect multiple harmful CFG directions simultaneously, demonstrating its ability to eliminate various harmful elements while preserving high-quality generation. Additionally, we find that HGR can detect image harmfulness, allowing for unsupervised fine-tuning of safe diffusion models without pre-defined clean or harmful labels. Experimental results show that by incorporating HGR, images generated by diffusion models achieve both high quality and strong safety, and safe DMs trained through unsupervised methods according to the harmfulness detected by HGR also exhibit good safety performance. The codes will be publicly available.

Video has emerged as a favored multimedia format on the internet. To better gain video contents, a new topic HIREST is presented, including video retrieval, moment retrieval, moment segmentation, and step-captioning. The pioneering work chooses the pre-trained CLIP-based model for video retrieval, and leverages it as a feature extractor for other three challenging tasks solved in a multi-task learning paradigm. Nevertheless, this work struggles to learn the comprehensive cognition of user-preferred content, due to disregarding the hierarchies and association relations across modalities. In this paper, guided by the shallow-to-deep principle, we propose a query-centric audio-visual cognition (QUAG) network to construct a reliable multi-modal representation for moment retrieval, segmentation and step-captioning. Specifically, we first design the modality-synergistic perception to obtain rich audio-visual content, by modeling global contrastive alignment and local fine-grained interaction between visual and audio modalities. Then, we devise the query-centric cognition that uses the deep-level query to perform the temporal-channel filtration on the shallow-level audio-visual representation. This can cognize user-preferred content and thus attain a query-centric audio-visual representation for three tasks. Extensive experiments show QUAG achieves the SOTA results on HIREST. Further, we test QUAG on the query-based video summarization task and verify its good generalization., Comment: Accepted by AAAI 2025

Deep learning has fundamentally reshaped the landscape of artificial intelligence over the past decade, enabling remarkable achievements across diverse domains. At the heart of these developments lie multi-layered neural network architectures that excel at automatic feature extraction, leading to significant improvements in machine learning tasks. To demystify these advances and offer accessible guidance, we present a comprehensive overview of the most influential deep learning algorithms selected through a broad-based survey of the field. Our discussion centers on pivotal architectures, including Residual Networks, Transformers, Generative Adversarial Networks, Variational Autoencoders, Graph Neural Networks, Contrastive Language-Image Pre-training, and Diffusion models. We detail their historical context, highlight their mathematical foundations and algorithmic principles, and examine subsequent variants, extensions, and practical considerations such as training methodologies, normalization techniques, and learning rate schedules. Beyond historical and technical insights, we also address their applications, challenges, and potential research directions. This survey aims to serve as a practical manual for both newcomers seeking an entry point into cutting-edge deep learning methods and experienced researchers transitioning into this rapidly evolving domain.

Given a piece of text, a video clip, and a reference audio, the movie dubbing task aims to generate speech that aligns with the video while cloning the desired voice. The existing methods have two primary deficiencies: (1) They struggle to simultaneously hold audio-visual sync and achieve clear pronunciation; (2) They lack the capacity to express user-defined emotions. To address these problems, we propose EmoDubber, an emotion-controllable dubbing architecture that allows users to specify emotion type and emotional intensity while satisfying high-quality lip sync and pronunciation. Specifically, we first design Lip-related Prosody Aligning (LPA), which focuses on learning the inherent consistency between lip motion and prosody variation by duration level contrastive learning to incorporate reasonable alignment. Then, we design Pronunciation Enhancing (PE) strategy to fuse the video-level phoneme sequences by efficient conformer to improve speech intelligibility. Next, the speaker identity adapting module aims to decode acoustics prior and inject the speaker style embedding. After that, the proposed Flow-based User Emotion Controlling (FUEC) is used to synthesize waveform by flow matching prediction network conditioned on acoustics prior. In this process, the FUEC determines the gradient direction and guidance scale based on the user's emotion instructions by the positive and negative guidance mechanism, which focuses on amplifying the desired emotion while suppressing others. Extensive experimental results on three benchmark datasets demonstrate favorable performance compared to several state-of-the-art methods., Comment: Under review

The \emph{Swift} Burst Alert Telescope (BAT), operating in the 15--150 keV energy band, struggles to detect the peak energy ($E_{\rm p}$) of gamma-ray bursts (GRBs), as most GRBs have $E_{\rm p}$ values typically distributed between 200-300 keV, exceeding BAT's upper limit. To address this, we develop an innovative method to robustly estimate the lower limit of $E_{\rm p}$ for GRBs with $E_{\rm p}>150$ keV. This approach relies on the intrinsic curvature of GRB spectra, which is already evident within the BAT energy range for such GRBs. By fitting BAT spectra with a cutoff power-law model and extrapolating the spectral curvature beyond BAT's range, we, therefore, can estimate the cutoff energy ($E^{'}_{\rm c}$) beyond 150 keV and the corresponding peak energy ($E^{'}_{\rm p}$). We applied this method to 17 GRBs, categorizing them into two main groups. Group I (10 bursts) maintains $\alpha$ within a typical range (from $\sim$ -0.8 to $\sim$ -1.20) with increasing $E_{\rm c}$; Group II (2 bursts) maintains $E_{\rm c}$ within a typical range (300-500 keV) but with varying $\alpha$. Our results show that for $E_{\rm c}\lesssim $1000 keV, the estimated $E^{'}_{\rm c}$ aligns well with observed values. Moreover, the reliability of $E^{'}_{\rm c}$ also depends on $\alpha$: bursts with harder $\alpha$ (e.g., $\alpha \gtrsim -2/3$) show reduced accuracy, while bursts with softer $\alpha$ (e.g., $\alpha \lesssim -2/3$) yield more precise estimates. In conclusion, this method is well-suited for GRB spectra with moderately observed $E_{\rm c}$ ($E_{\rm p}$) values and $\alpha$ indices that are not too hard., Comment: 54 pages, 21 figures including 61 panels, 20 tables,Submitted to The Astrophysical Journal Supplement Series

Implicit neural representations and 3D Gaussian splatting (3DGS) have shown great potential for scene reconstruction. Recent studies have expanded their applications in autonomous reconstruction through task assignment methods. However, these methods are mainly limited to single robot, and rapid reconstruction of large-scale scenes remains challenging. Additionally, task-driven planning based on surface uncertainty is prone to being trapped in local optima. To this end, we propose the first 3DGS-based centralized multi-robot autonomous 3D reconstruction framework. To further reduce time cost of task generation and improve reconstruction quality, we integrate online open-vocabulary semantic segmentation with surface uncertainty of 3DGS, focusing view sampling on regions with high instance uncertainty. Finally, we develop a multi-robot collaboration strategy with mode and task assignments improving reconstruction quality while ensuring planning efficiency. Our method demonstrates the highest reconstruction quality among all planning methods and superior planning efficiency compared to existing multi-robot methods. We deploy our method on multiple robots, and results show that it can effectively plan view paths and reconstruct scenes with high quality.

Wave/Schr\"{o}dinger equations with potentials naturally originates from both the quantum physics and the study of nonlinear equations. The distractive Coulomb potential is a quantum mechanical description of distractive Coulomb force between two particles with the same charge. The spectrum of the operator $-\Delta +1/|x|$ is well known and there are also a few results on the Strichartz estimates, local and global well-posedness and scattering result about the nonlinear Schr\"{o}dinger equation with a distractive Coulomb potential. In the contrast, much less is known for the global and asymptotic behaviour of solutions to the corresponding wave equations with a Coulomb potential. In this work we consider the wave equation with a distractive Coulomb potential in dimensions $d\geq 3$. We first describe the asymptotic behaviour of the solutions to the linear homogeneous Coulomb wave equation, especially their energy distribution property and scattering profiles, then show that the radial finite-energy solutions to suitable defocusing Coulomb wave equation are defined for all time and scatter in both two time directions, by establishing a family of radial Strichartz estimates and combining them with the decay of the potential energy., Comment: 120 pages, 6 figures

In this letter, we propose an energy-efficient split learning (SL) framework for fine-tuning large language models (LLMs) using geo-distributed personal data at the network edge, where LLMs are split and alternately across massive mobile devices and an edge server. Considering the device heterogeneity and channel dynamics in edge networks, a \underline{C}ut l\underline{A}yer and computing \underline{R}esource \underline{D}ecision (CARD) algorithm is developed to minimize training delay and energy consumption. Simulation results demonstrate that the proposed approach reduces the average training delay and server's energy consumption by 70.8% and 53.1%, compared to the benchmarks, respectively., Comment: 5 pages, 6 figures

We employ the microscopic time-dependent Hartree-Fock (TDHF) theory to study the 48Ca+249Bk and 48Ti+238U systems, taking into account the dependence on orientation for deformed nuclei and full range of impact parameters. By analyzing fragment distributions of neutron and proton numbers, we assess the influence of different isoscalar and isovector tensor coupling constants of the effective nucleon-nucleon interaction. The quasifission yield distributions of 48Ca + 249Bk collision system utilizing SLy5t and T31 parametrizations exhibit more pronounced spherical shell effects compared to those using SLy5, T44 and T62 sets. Furthermore, within each parametrization group, the distributions for SLy5t and T31 are closely aligned, as are those for SLy5, T44, and T62. Similarly, the yield distributions for the 48Ti + 238U system using SLy5t and T31 also reflect the more pronounced spherical shell effects relative to SLy5 and T62, while the charge distribution shows much better agreement with experimental results for the SLy5t and T62 parametrizations compared to SLy5 and T31. The yield distributions for the 48Ca+249Bk and 48Ti+238U systems, when compared across the SLy5, SLy5t, T31, T44, and T62 parametrizations, indicate that the influence of tensor forces on quasifission fragments is reflected in the prominence of shell effects. This influence appears to be sensitive only in specific regions within the isoscalar and isovector coupling constant parameter space. In the 48Ti + 238U system, the prominence of shell effects is manifested not only through shifts in peak positions but also through narrower yield distributions., Comment: Phys. Rev. C (in press), 15 pages, 11 figures

Lead halide perovskite nanocrystals (LHP-NCs) embedded in a plastic matrix are highly promising for a variety of photonic technologies and are quickly gaining attention as ultrafast, radiation-resistant nanoscintillators for radiation detection. However, advancements in LHP-NC-based photonics are hindered by their well-known thermal instability, which makes them unsuitable for industrial thermally activated mass polymerization processes - crucial for creating polystyrene-based scintillating nanocomposites. In this study, we address this challenge by presenting the first thermal nanocomposite scintillators made from CsPbBr3 NCs passivated with fluorinated ligands that remain attached to the particles surfaces even at high temperatures, enabling their integration into mass-cured polyvinyl toluene without compromising optical properties. Consequently, these nanocomposites demonstrate scintillation light yields reaching 10,400 photons/MeV, sub-nanosecond scintillation kinetics, and remarkable radiation resilience, able to withstand gamma radiation doses of up to 1 MGy. This performance not only meets but also exceeds the scintillation of plastic scintillators, despite the radiation-induced damage to the host matrix.

This paper explores the Class-Incremental Source-Free Unsupervised Domain Adaptation (CI-SFUDA) problem, where the unlabeled target data come incrementally without access to labeled source instances. This problem poses two challenges, the disturbances of similar source-class knowledge to target-class representation learning and the new target knowledge to old ones. To address them, we propose the Multi-Granularity Class Prototype Topology Distillation (GROTO) algorithm, which effectively transfers the source knowledge to the unlabeled class-incremental target domain. Concretely, we design the multi-granularity class prototype self-organization module and prototype topology distillation module. Firstly, the positive classes are mined by modeling two accumulation distributions. Then, we generate reliable pseudo-labels by introducing multi-granularity class prototypes, and use them to promote the positive-class target feature self-organization. Secondly, the positive-class prototypes are leveraged to construct the topological structures of source and target feature spaces. Then, we perform the topology distillation to continually mitigate the interferences of new target knowledge to old ones. Extensive experiments demonstrate that our proposed method achieves state-of-the-art performances on three public datasets., Comment: 10 pages, 6 figures

This paper shows a proof-of-concept that, given a typical 3-channel images but in a randomly permuted channel order, a model (termed as Chanel-Orderer) with ad-hoc inductive biases in terms of both architecture and loss functions can accurately predict the channel ordering and knows how to make it right. Specifically, Chanel-Orderer learns to score each of the three channels with the priors of object semantics and uses the resulting scores to predict the channel ordering. This brings up benefits into a typical scenario where an \texttt{RGB} image is often mis-displayed in the \texttt{BGR} format and needs to be corrected into the right order. Furthermore, as a byproduct, the resulting model Chanel-Orderer is able to tell whether a given image is a near-gray-scale image (near-monochromatic) or not (polychromatic). Our research suggests that Chanel-Orderer mimics human visual coloring of our physical natural world.

DarkSHINE is a newly proposed fixed-target experiment initiative to search for the invisible decay of Dark Photon via missing energy/momentum signatures, based on the high repetition rate electron beam to be deployed/delivered by the Shanghai High repetition rate XFEL and Extreme light facility (SHINE). This report elaborates the baseline design of DarkSHINE experiment by introducing the physics goals, experimental setups, details of each sub-detector system technical designs, signal and backgground modelings, expected search sensitivities and future prospects, which mark an important step towards the further prototyping and technical demonstrations.

In this paper, we present TexPro, a novel method for high-fidelity material generation for input 3D meshes given text prompts. Unlike existing text-conditioned texture generation methods that typically generate RGB textures with baked lighting, TexPro is able to produce diverse texture maps via procedural material modeling, which enables physical-based rendering, relighting, and additional benefits inherent to procedural materials. Specifically, we first generate multi-view reference images given the input textual prompt by employing the latest text-to-image model. We then derive texture maps through a rendering-based optimization with recent differentiable procedural materials. To this end, we design several techniques to handle the misalignment between the generated multi-view images and 3D meshes, and introduce a novel material agent that enhances material classification and matching by exploring both part-level understanding and object-aware material reasoning. Experiments demonstrate the superiority of the proposed method over existing SOTAs and its capability of relighting., Comment: In submission. Supplementary material is included at the end of the main paper (5 pages, 2 figures)

Deep learning-based feature matching has shown great superiority for point cloud registration in the absence of pose priors. Although coarse-to-fine matching approaches are prevalent, the coarse matching of existing methods is typically sparse and loose without consideration of geometric consistency, which makes the subsequent fine matching rely on ineffective optimal transport and hypothesis-and-selection methods for consistency. Therefore, these methods are neither efficient nor scalable for real-time applications such as odometry in robotics. To address these issues, we design a consistency-aware spot-guided Transformer (CAST), which incorporates a spot-guided cross-attention module to avoid interfering with irrelevant areas, and a consistency-aware self-attention module to enhance matching capabilities with geometrically consistent correspondences. Furthermore, a lightweight fine matching module for both sparse keypoints and dense features can estimate the transformation accurately. Extensive experiments on both outdoor LiDAR point cloud datasets and indoor RGBD point cloud datasets demonstrate that our method achieves state-of-the-art accuracy, efficiency, and robustness., Comment: Accepted by NeurIPS 2024 as poster

Conductor materials with good mechanical performance as well as high electrical- and thermal-conductivities are particularly important to break through the current bottle-neck limit ($\sim 100$ T) of pulsed magnets. Here we perform systematic studies on the elastic properties of the Cu-6wt\%Ag alloy wires, a promising candidate material for the new-generation pulsed magnets, by employing two independent ultrasonic techniques - resonant ultrasound spectroscopy (RUS) and ultrasound pulse-echo experiments. Our RUS measurements manifest that the elastic properties of the Cu-6wt\%Ag alloy wires can be improved by an electroplastic drawing procedure as compared with the conventional cold drawing. We also take this chance to test the availability of our newly-built ultrasound pulse-echo facility at Wuhan National High Magnetic Field Center (WHMFC, China), and the results suggest that the elastic performance of the electroplastically-drawn Cu-6wt\%Ag alloy wire remains excellent without anomalous softening under extreme conditions, e.g., ultra-high magnetic field up to 50 T, nitrogen / helium cryogenic liquids., Comment: 7 pages, 4 figures, 1 table

We study the evolution of a newly formed magnetized neutron-star (NS) as a power source of gamma-ray bursts (GRBs) in the light of both gravitational wave (GW) and electromagnetic (EM) radiations. The compressible and incompressible fluids are employed in order to model the secular evolution of Maclaurian spheroids. It is shown that the GW and EM light curves evolve as a function of eccentricity and rotational frequency with time. We find that the light curve characteristics crucially depend on NS parameters such as magnitude and structure of magnetic field, ellipticity and the equation of state (EOS) of the fluid. The presence of X-ray flares, whose origins are not yet well understood, can be captured in our model regarding some specific nuclear EOSs. Our model allowing us to explain flares that occur within the wide range of $ 10$ to $10^4$ seconds and the peak luminosity in the order of $10^{46}$ - $10^{51}$ $\rm \text{erg}/s$ by using a reasonable set of parameters such as magnetic field strength around $10^{14}-10^{16}$ Gauss, the quadrupole to dipole ratio of magnetic field up to 500. By applying our model to a sample of GRB X-ray flares observed by Swift/XRT, we try to constraint the crucial parameters of a deformed magnetar via MCMC fitting method. Our analysis shows that ongoing and upcoming joint multi-messenger detections can be used to understand the nature of GRB's central engine and its evolution at the early times of the burst formation.

In this paper, we introduce CalliffusionV2, a novel system designed to produce natural Chinese calligraphy with flexible multi-modal control. Unlike previous approaches that rely solely on image or text inputs and lack fine-grained control, our system leverages both images to guide generations at fine-grained levels and natural language texts to describe the features of generations. CalliffusionV2 excels at creating a broad range of characters and can quickly learn new styles through a few-shot learning approach. It is also capable of generating non-Chinese characters without prior training. Comprehensive tests confirm that our system produces calligraphy that is both stylistically accurate and recognizable by neural network classifiers and human evaluators., Comment: 11 pages, 7 figures

This paper pioneers the use of quantum machine learning (QML) for modeling the Ohmic contact process in GaN high-electron-mobility transistors (HEMTs) for the first time. Utilizing data from 159 devices and variational auto-encoder-based augmentation, we developed a quantum kernel-based regressor (QKR) with a 2-level ZZ-feature map. Benchmarking against six classical machine learning (CML) models, our QKR consistently demonstrated the lowest mean absolute error (MAE), mean squared error (MSE), and root mean squared error (RMSE). Repeated statistical analysis confirmed its robustness. Additionally, experiments verified an MAE of 0.314 ohm-mm, underscoring the QKR's superior performance and potential for semiconductor applications, and demonstrating significant advancements over traditional CML methods., Comment: This is the manuscript in the conference version. An expanded version for the journal will be released later and more information will be added. The author list, content, conclusion, and figures may change due to further research

Producing currents predominantly from a single valley, namely valley-polarized currents, at optical-cycle timescales is an important aspect of the petahertz valleytronics, yet it remains less developed. This work exhibits the feasibility of achieving this goal using bichromatic optical fields, which allow for the precise control of sub-cycle electron dynamics. The combined effect of the helical and asymmetric waveforms of the optical fields leads to highly asymmetric excitation at different valleys and displacement of the excited electrons concurrently, thereby inducing valley-polarized currents with high valley purity, on the sub-optical-cycle timescale. Inherently, the purity of the currents built up from the optical approach remains high even for materials with short decoherence time. Moreover, the direction of the currents can be precisely controlled by adjusting the relative phase of the bichromatic components. Our work offers a promising avenue for generating and modulating high-purity valley-polarized currents at the femtosecond timescale, facilitating the development of petahertz valleytronics.

Existing symbolic music generation methods usually utilize discriminator to improve the quality of generated music via global perception of music. However, considering the complexity of information in music, such as rhythm and melody, a single discriminator cannot fully reflect the differences in these two primary dimensions of music. In this work, we propose to decouple the melody and rhythm from music, and design corresponding fine-grained discriminators to tackle the aforementioned issues. Specifically, equipped with a pitch augmentation strategy, the melody discriminator discerns the melody variations presented by the generated samples. By contrast, the rhythm discriminator, enhanced with bar-level relative positional encoding, focuses on the velocity of generated notes. Such a design allows the generator to be more explicitly aware of which aspects should be adjusted in the generated music, making it easier to mimic human-composed music. Experimental results on the POP909 benchmark demonstrate the favorable performance of the proposed method compared to several state-of-the-art methods in terms of both objective and subjective metrics., Comment: Accepted by ICPR2024

This paper presents the design and optimization of a LYSO crystal electromagnetic calorimeter (ECAL) for the DarkSHINE experiment, which aims to search for dark photons as potential mediators of dark forces. The ECAL design was evaluated through comprehensive simulations, focusing on optimizing dimensions, material selection, energy distribution, and energy resolution. The ECAL configuration consists of 21$\times$21$\times$11 LYSO crystals, each measuring 2.5$\times$2.5$\times$4 cm$^3$, arranged in a staggered layout to improve signal detection efficiency. A 4 GeV energy dynamic range was established to ensure accurate energy measurements without saturation, which is essential for background rejection and signal identification. A detailed digitization model was developed to simulate the scintillation, SiPM, and ADC behaviors, providing a more realistic representation of detector performance. Additionally, the study assessed radiation damage in the ECAL region, highlighting the necessity of radiation-resistant scintillators and silicon sensors.

Active learning (AL) is designed to construct a high-quality labeled dataset by iteratively selecting the most informative samples. Such sampling heavily relies on data representation, while recently pre-training is popular for robust feature learning. However, as pre-training utilizes low-level pretext tasks that lack annotation, directly using pre-trained representation in AL is inadequate for determining the sampling score. To address this problem, we propose a downstream-pretext domain knowledge traceback (DOKT) method that traces the data interactions of downstream knowledge and pre-training guidance for selecting diverse and instructive samples near the decision boundary. DOKT consists of a traceback diversity indicator and a domain-based uncertainty estimator. The diversity indicator constructs two feature spaces based on the pre-training pretext model and the downstream knowledge from annotation, by which it locates the neighbors of unlabeled data from the downstream space in the pretext space to explore the interaction of samples. With this mechanism, DOKT unifies the data relations of low-level and high-level representations to estimate traceback diversity. Next, in the uncertainty estimator, domain mixing is designed to enforce perceptual perturbing to unlabeled samples with similar visual patches in the pretext space. Then the divergence of perturbed samples is measured to estimate the domain uncertainty. As a result, DOKT selects the most diverse and important samples based on these two modules. The experiments conducted on ten datasets show that our model outperforms other state-of-the-art methods and generalizes well to various application scenarios such as semantic segmentation and image captioning.

Joint Detection and Embedding (JDE) trackers have demonstrated excellent performance in Multi-Object Tracking (MOT) tasks by incorporating the extraction of appearance features as auxiliary tasks through embedding Re-Identification task (ReID) into the detector, achieving a balance between inference speed and tracking performance. However, solving the competition between the detector and the feature extractor has always been a challenge. Meanwhile, the issue of directly embedding the ReID task into MOT has remained unresolved. The lack of high discriminability in appearance features results in their limited utility. In this paper, a new learning approach using cross-correlation to capture temporal information of objects is proposed. The feature extraction network is no longer trained solely on appearance features from each frame but learns richer motion features by utilizing feature heatmaps from consecutive frames, which addresses the challenge of inter-class feature similarity. Furthermore, our learning approach is applied to a more lightweight feature extraction network, and treat the feature matching scores as strong cues rather than auxiliary cues, with an appropriate weight calculation to reflect the compatibility between our obtained features and the MOT task. Our tracker, named TCBTrack, achieves state-of-the-art performance on multiple public benchmarks, i.e., MOT17, MOT20, and DanceTrack datasets. Specifically, on the DanceTrack test set, we achieve 56.8 HOTA, 58.1 IDF1 and 92.5 MOTA, making it the best online tracker capable of achieving real-time performance. Comparative evaluations with other trackers prove that our tracker achieves the best balance between speed, robustness and accuracy. Code is available at https://github.com/yfzhang1214/TCBTrack., Comment: A submission to IJCV

Change captioning aims to succinctly describe the semantic change between a pair of similar images, while being immune to distractors (illumination and viewpoint changes). Under these distractors, unchanged objects often appear pseudo changes about location and scale, and certain objects might overlap others, resulting in perturbational and discrimination-degraded features between two images. However, most existing methods directly capture the difference between them, which risk obtaining error-prone difference features. In this paper, we propose a distractors-immune representation learning network that correlates the corresponding channels of two image representations and decorrelates different ones in a self-supervised manner, thus attaining a pair of stable image representations under distractors. Then, the model can better interact them to capture the reliable difference features for caption generation. To yield words based on the most related difference features, we further design a cross-modal contrastive regularization, which regularizes the cross-modal alignment by maximizing the contrastive alignment between the attended difference features and generated words. Extensive experiments show that our method outperforms the state-of-the-art methods on four public datasets. The code is available at https://github.com/tuyunbin/DIRL., Comment: Accepted by ECCV 2024

Training latency is critical for the success of numerous intrigued applications ignited by federated learning (FL) over heterogeneous mobile devices. By revolutionarily overlapping local gradient transmission with continuous local computing, FL can remarkably reduce its training latency over homogeneous clients, yet encounter severe model staleness, model drifts, memory cost and straggler issues in heterogeneous environments. To unleash the full potential of overlapping, we propose, FedEx, a novel \underline{fed}erated learning approach to \underline{ex}pedite FL training over mobile devices under data, computing and wireless heterogeneity. FedEx redefines the overlapping procedure with staleness ceilings to constrain memory consumption and make overlapping compatible with participation selection (PS) designs. Then, FedEx characterizes the PS utility function by considering the latency reduced by overlapping, and provides a holistic PS solution to address the straggler issue. FedEx also introduces a simple but effective metric to trigger overlapping, in order to avoid model drifts. Experimental results show that compared with its peer designs, FedEx demonstrates substantial reductions in FL training latency over heterogeneous mobile devices with limited memory cost., Comment: 21 pages, 10 figures, Submitted to Sensys2024

The distributed inference paradigm enables the computation workload to be distributed across multiple devices, facilitating the implementations of deep learning based intelligent services on extremely resource-constrained Internet of Things (IoT) scenarios. Yet it raises great challenges to perform complicated inference tasks relying on a cluster of IoT devices that are heterogeneous in their computing/communication capacity and prone to crash or timeout failures. In this paper, we present RoCoIn, a robust cooperative inference mechanism for locally distributed execution of deep neural network-based inference tasks over heterogeneous edge devices. It creates a set of independent and compact student models that are learned from a large model using knowledge distillation for distributed deployment. In particular, the devices are strategically grouped to redundantly deploy and execute the same student model such that the inference process is resilient to any local failures, while a joint knowledge partition and student model assignment scheme are designed to minimize the response latency of the distributed inference system in the presence of devices with diverse capacities. Extensive simulations are conducted to corroborate the superior performance of our RoCoIn for distributed inference compared to several baselines, and the results demonstrate its efficacy in timely inference and failure resiliency.

The intersection of physics-based vision and deep learning presents an exciting frontier for advancing computer vision technologies. By leveraging the principles of physics to inform and enhance deep learning models, we can develop more robust and accurate vision systems. Physics-based vision aims to invert the processes to recover scene properties such as shape, reflectance, light distribution, and medium properties from images. In recent years, deep learning has shown promising improvements for various vision tasks, and when combined with physics-based vision, these approaches can enhance the robustness and accuracy of vision systems. This technical report summarizes the outcomes of the Physics-Based Vision Meets Deep Learning (PBDL) 2024 challenge, held in CVPR 2024 workshop. The challenge consisted of eight tracks, focusing on Low-Light Enhancement and Detection as well as High Dynamic Range (HDR) Imaging. This report details the objectives, methodologies, and results of each track, highlighting the top-performing solutions and their innovative approaches., Comment: CVPR 2024 PBDL Challenges: https://pbdl-ws.github.io/pbdl2024/challenge/index.html

Multi-change captioning aims to describe complex and coupled changes within an image pair in natural language. Compared with single-change captioning, this task requires the model to have higher-level cognition ability to reason an arbitrary number of changes. In this paper, we propose a novel context-aware difference distilling (CARD) network to capture all genuine changes for yielding sentences. Given an image pair, CARD first decouples context features that aggregate all similar/dissimilar semantics, termed common/difference context features. Then, the consistency and independence constraints are designed to guarantee the alignment/discrepancy of common/difference context features. Further, the common context features guide the model to mine locally unchanged features, which are subtracted from the pair to distill locally difference features. Next, the difference context features augment the locally difference features to ensure that all changes are distilled. In this way, we obtain an omni-representation of all changes, which is translated into linguistic sentences by a transformer decoder. Extensive experiments on three public datasets show CARD performs favourably against state-of-the-art methods.The code is available at https://github.com/tuyunbin/CARD., Comment: Accepted by ACL 2024 main conference (long paper)

Considered herein is the global existence of weak, strong solutions and Rayleigh-Taylor (RT) instability for 2D semi-dissipative Boussinesq equations in an infinite strip domain $\Omega_{\infty}$ subject to Navier boundary conditions with non-positive slip coefficients. We first prove the global existence of weak and strong solutions on bounded domain $\Omega_{R}$ via the Galerkin method, characteristic analyzing technique and Stokes estimates etc. Based on above results, we further derive the uniform estimates, independent of the length of horizontal direction of $\Omega_{R}$, ensuring the global existence of weak and strong solutions in unbounded case $\Omega_{\infty}$ by utilizing the domain expansion method. Moreover, when the steady temperature is higher with decreasing height (i.e., RT steady-state) on certain region, we demonstrate that the steady-state is linear unstable through the construction of energy functional and the settlement of a family of modified variational problems. Furthermore, with the help of unstable solutions constructed in linear instability and global existence theorems, we confirm the instability of nonlinear problem in a Lipschitz structural sense. Finally, we give a series of rigorous verification (see Appendix) including the spectra of Stokes equations with Navier boundary conditions, Sobolev embedding inequalities, trace inequalities, and Stokes estimates under Navier boundary conditions etc, used in the proof of main conclusions.

Image-guided depth completion aims at generating a dense depth map from sparse LiDAR data and RGB image. Recent methods have shown promising performance by reformulating it as a classification problem with two sub-tasks: depth discretization and probability prediction. They divide the depth range into several discrete depth values as depth categories, serving as priors for scene depth distributions. However, previous depth discretization methods are easy to be impacted by depth distribution variations across different scenes, resulting in suboptimal scene depth distribution priors. To address the above problem, we propose a progressive depth decoupling and modulating network, which incrementally decouples the depth range into bins and adaptively generates multi-scale dense depth maps in multiple stages. Specifically, we first design a Bins Initializing Module (BIM) to construct the seed bins by exploring the depth distribution information within a sparse depth map, adapting variations of depth distribution. Then, we devise an incremental depth decoupling branch to progressively refine the depth distribution information from global to local. Meanwhile, an adaptive depth modulating branch is developed to progressively improve the probability representation from coarse-grained to fine-grained. And the bi-directional information interactions are proposed to strengthen the information interaction between those two branches (sub-tasks) for promoting information complementation in each branch. Further, we introduce a multi-scale supervision mechanism to learn the depth distribution information in latent features and enhance the adaptation capability across different scenes. Experimental results on public datasets demonstrate that our method outperforms the state-of-the-art methods. The code will be open-sourced at [this https URL](https://github.com/Cisse-away/PDDM)., Comment: The article is accepted by IEEE Transactions on Instrumentation & Measurement

The observation of a gamma-ray burst (GRB) associated with a supernova (SN) coincides remarkably with the energy output from a binary system comprising a very massive carbon-oxygen (CO) core and an associated binary neutron star (NS) by the Binary-Driven Hypernova (BdHN) model. The dragging effect in the late evolution of such systems leads to co-rotation, with binary periods on the order of minutes, resulting in a very fast rotating core and a binary NS companion at a distance of $\sim 10^5$ km. Such a fast-rotating CO core, stripped of its hydrogen and helium, undergoes gravitational collapse and, within a fraction of seconds, leads to a supernova (SN) and a newly born, fast-spinning neutron star ($\nu$NS), we name the emergence of the SN and the $\nu$NS as the SN-rise and $\nu$NS-rise. Typically, the SN energies range from $10^{51}$ to $10^{53}$ erg. We address this issue by examining 10 cases of Type-I BdHNe, the most energetic ones, in which SN accretion onto the companion NS leads to the formation of a black hole (BH). In all ten cases, the energetics of the SN events are estimated, ranging between $0.18$ and $12 \times 10^{52}$ erg. Additionally, in all 8 sources at redshift $z$ closer than $4.61$, a clear thermal blackbody component has been identified, with temperatures between $6.2$ and $39.99$ keV, as a possible signature of pair-driven SN. The triggering of the X-ray afterglow induced by the $\nu$NS-rise are identified in three cases at high redshift where early X-ray observations are achievable, benefits from the interplay of cosmological effects., Comment: The previous version was uploaded on arXiv before being submitted to any Journal, this new version matches the one actually submitted for publication

As a popular distributed learning paradigm, federated learning (FL) over mobile devices fosters numerous applications, while their practical deployment is hindered by participating devices' computing and communication heterogeneity. Some pioneering research efforts proposed to extract subnetworks from the global model, and assign as large a subnetwork as possible to the device for local training based on its full computing and communications capacity. Although such fixed size subnetwork assignment enables FL training over heterogeneous mobile devices, it is unaware of (i) the dynamic changes of devices' communication and computing conditions and (ii) FL training progress and its dynamic requirements of local training contributions, both of which may cause very long FL training delay. Motivated by those dynamics, in this paper, we develop a wireless and heterogeneity aware latency efficient FL (WHALE-FL) approach to accelerate FL training through adaptive subnetwork scheduling. Instead of sticking to the fixed size subnetwork, WHALE-FL introduces a novel subnetwork selection utility function to capture device and FL training dynamics, and guides the mobile device to adaptively select the subnetwork size for local training based on (a) its computing and communication capacity, (b) its dynamic computing and/or communication conditions, and (c) FL training status and its corresponding requirements for local training contributions. Our evaluation shows that, compared with peer designs, WHALE-FL effectively accelerates FL training without sacrificing learning accuracy.