Search

In this paper, we introduce Hunyuan-Large, which is currently the largest open-source Transformer-based mixture of experts model, with a total of 389 billion parameters and 52 billion activation parameters, capable of handling up to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior performance across various benchmarks including language understanding and generation, logical reasoning, mathematical problem-solving, coding, long-context, and aggregated tasks, where it outperforms LLama3.1-70B and exhibits comparable performance when compared to the significantly larger LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale synthetic data that is orders larger than in previous literature, a mixed expert routing strategy, a key-value cache compression technique, and an expert-specific learning rate strategy. Additionally, we also investigate the scaling laws and learning rate schedule of mixture of experts models, providing valuable insights and guidances for future model development and optimization. The code and checkpoints of Hunyuan-Large are released to facilitate future innovations and applications. Codes: https://github.com/Tencent/Hunyuan-Large Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large, Comment: 17 pages, 4 Figures

Diffusion models have achieved remarkable progress in the field of image generation due to their outstanding capabilities. However, these models require substantial computing resources because of the multi-step denoising process during inference. While traditional pruning methods have been employed to optimize these models, the retraining process necessitates large-scale training datasets and extensive computational costs to maintain generalization ability, making it neither convenient nor efficient. Recent studies attempt to utilize the similarity of features across adjacent denoising stages to reduce computational costs through simple and static strategies. However, these strategies cannot fully harness the potential of the similar feature patterns across adjacent timesteps. In this work, we propose a novel pruning method that derives an efficient diffusion model via a more intelligent and differentiable pruner. At the core of our approach is casting the model pruning process into a SubNet search process. Specifically, we first introduce a SuperNet based on standard diffusion via adding some backup connections built upon the similar features. We then construct a plugin pruner network and design optimization losses to identify redundant computation. Finally, our method can identify an optimal SubNet through few-step gradient optimization and a simple post-processing procedure. We conduct extensive experiments on various diffusion models including Stable Diffusion series and DiTs. Our DiP-GO approach achieves 4.4 x speedup for SD-1.5 without any loss of accuracy, significantly outperforming the previous state-of-the-art methods.

Mother-child interaction is a highly dynamic process neurally characterized by inter-brain synchrony (IBS) at {\theta} and/or {\alpha} rhythms. However, their establishment, dynamic changes, and roles in mother-child interactions remain unknown. Through dynamic analysis of dual-EEG from 40 mother-child dyads during turn-taking cooperation, we uncover that {\theta}-IBS and {\alpha}-IBS alternated with interactive behaviors, with EEG frequency-shift as a prerequisite for IBS transitions. When mothers attempt to track their children's attention and/or predict their intentions, they will adjust their EEG frequencies to align with their children's {\theta} oscillations, leading to a higher occurrence of the {\theta}-IBS state. Conversely, the {\alpha}-IBS state, accompanied by the EEG frequency-shift to the {\alpha} range, is more prominent during mother-led interactions. Further exploratory analysis reveals greater presence and stability of the {\theta}-IBS state during cooperative than non-cooperative conditions, particularly in dyads with stronger emotional attachments and more frequent interactions in their daily lives. Our findings shed light on the neural oscillational substrates underlying the IBS dynamics during mother-child interactions., Comment: 27 Pages,6 figures

With the rapid development of wearable device technologies, accelerometers can record minute-by-minute physical activity for consecutive days, which provides important insight into a dynamic association between the intensity of physical activity and mental health outcomes for large-scale population studies. Using Shanghai school adolescent cohort we estimate the effect of health assessment results on physical activity profiles recorded by accelerometers throughout a week, which is recognized as repeatedly measured functional data. To achieve this goal, we propose an innovative two-dimensional functional mixed-effect model (2dFMM) for the specialized data, which smoothly varies over longitudinal day observations with covariate-dependent mean and covariance functions. The modeling framework characterizes the longitudinal and functional structures while incorporating two-dimensional fixed effects for covariates of interest. We also develop a fast three-stage estimation procedure to provide accurate fixed-effect inference for model interpretability and improve computational efficiency when encountering large datasets. We find strong evidence of intraday and interday varying significant associations between physical activity and mental health assessments among our cohort population, which shed light on possible intervention strategies targeting daily physical activity patterns to improve school adolescent mental health. Our method is also used in environmental data to illustrate the wide applicability. Supplementary materials for this article are available online., Comment: 50 pages, 8 figures in main, 6 figures in supp

The OTFS (Orthogonal Time Frequency Space) is widely acknowledged for its ability to combat Doppler spread in time-varying channels. In this paper, another advantage of OTFS over OFDM (Orthogonal Frequency Division Multiplexing) will be demonstrated: much reduced channel training overhead. Specifically, the sparsity of the channel in delay-Doppler (D-D) domain implies strong correlation of channel gains in time-frequency (T-F) domain, which can be harnessed to reduce channel training overhead through interpolation. An immediate question is how much training overhead is needed in doubly-dispersive channels? A conventional belief is that the overhead is only dependent on the product of delay and Doppler spreads, but we will show that it's also dependent on the T-F window size. The finite T-F window leads to infinite spreading in D-D domain, and aliasing will be inevitable after sampling in T-F domain. Two direct consequences of the aliasing are increased channel training overhead and interference. Another factor contributing to channel estimation error is the inter-symbol-carrier-interference (ISCI), resulting from the uncertainty principle. Both aliasing and ISCI are considered in channel modelling, a low-complexity algorithm is proposed for channel estimation and interpolation through FFT. A large T-F window is necessary for reduced channel training overhead and aliasing, but increases processing delay. Fortunately, we show that the proposed algorithm can be implemented in a pipeline fashion. Further more, we showed that data-aided channel tracking is possible in D-D domain to further reduce the channel estimation frequency, i.e., channel extrapolation. The impacts of aliasing and ISCI on channel interpolation error are analyzed.

Generating realistic animated videos from static images is an important area of research in computer vision. Methods based on physical simulation and motion prediction have achieved notable advances, but they are often limited to specific object textures and motion trajectories, failing to exhibit highly complex environments and physical dynamics. In this paper, we introduce an open-domain controllable image animation method using motion priors with video diffusion models. Our method achieves precise control over the direction and speed of motion in the movable region by extracting the motion field information from videos and learning moving trajectories and strengths. Current pretrained video generation models are typically limited to producing very short videos, typically less than 30 frames. In contrast, we propose an efficient long-duration video generation method based on noise reschedule specifically tailored for image animation tasks, facilitating the creation of videos over 100 frames in length while maintaining consistency in content scenery and motion coordination. Specifically, we decompose the denoise process into two distinct phases: the shaping of scene contours and the refining of motion details. Then we reschedule the noise to control the generated frame sequences maintaining long-distance noise correlation. We conducted extensive experiments with 10 baselines, encompassing both commercial tools and academic methodologies, which demonstrate the superiority of our method. Our project page: https://wangqiang9.github.io/Controllable.github.io/, Comment: https://wangqiang9.github.io/Controllable.github.io/

Integrated communications and localization (ICAL) will play an important part in future sixth generation (6G) networks for the realization of Internet of Everything (IoE) to support both global communications and seamless localization. Massive multiple-input multiple-output (MIMO) low earth orbit (LEO) satellite systems have great potential in providing wide coverage with enhanced gains, and thus are strong candidates for realizing ubiquitous ICAL. In this paper, we develop a wideband massive MIMO LEO satellite system to simultaneously support wireless communications and localization operations in the downlink. In particular, we first characterize the signal propagation properties and derive a localization performance bound. Based on these analyses, we focus on the hybrid analog/digital precoding design to achieve high communication capability and localization precision. Numerical results demonstrate that the proposed ICAL scheme supports both the wireless communication and localization operations for typical system setups., Comment: 14 pages, 7 figures, to appear in IEEE Transactions on Wireless Communications

Cross-lingual open domain question answering (CLQA) is a complex problem, comprising cross-lingual retrieval from a multilingual knowledge base, followed by answer generation in the query language. Both steps are usually tackled by separate models, requiring substantial annotated datasets, and typically auxiliary resources, like machine translation systems to bridge between languages. In this paper, we show that CLQA can be addressed using a single encoder-decoder model. To effectively train this model, we propose a self-supervised method based on exploiting the cross-lingual link structure within Wikipedia. We demonstrate how linked Wikipedia pages can be used to synthesise supervisory signals for cross-lingual retrieval, through a form of cloze query, and generate more natural questions to supervise answer generation. Together, we show our approach, \texttt{CLASS}, outperforms comparable methods on both supervised and zero-shot language adaptation settings, including those using machine translation., Comment: EMNLP 2024 Main

Traditional channel-wise pruning methods by reducing network channels struggle to effectively prune efficient CNN models with depth-wise convolutional layers and certain efficient modules, such as popular inverted residual blocks. Prior depth pruning methods by reducing network depths are not suitable for pruning some efficient models due to the existence of some normalization layers. Moreover, finetuning subnet by directly removing activation layers would corrupt the original model weights, hindering the pruned model from achieving high performance. To address these issues, we propose a novel depth pruning method for efficient models. Our approach proposes a novel block pruning strategy and progressive training method for the subnet. Additionally, we extend our pruning method to vision transformer models. Experimental results demonstrate that our method consistently outperforms existing depth pruning methods across various pruning configurations. We obtained three pruned ConvNeXtV1 models with our method applying on ConvNeXtV1, which surpass most SOTA efficient models with comparable inference performance. Our method also achieves state-of-the-art pruning performance on the vision transformer model.

Indoor wireless ranging localization is a promising approach for low-power and high-accuracy localization of wearable devices. A primary challenge in this domain stems from non-line of sight propagation of radio waves. This study tackles a fundamental issue in wireless ranging: the unpredictability of real-time multipath determination, especially in challenging conditions such as when there is no direct line of sight. We achieve this by fusing range measurements with inertial measurements obtained from a low cost Inertial Measurement Unit (IMU). For this purpose, we introduce a novel asymmetric noise model crafted specifically for non-Gaussian multipath disturbances. Additionally, we present a novel Levenberg-Marquardt (LM)-family trust-region adaptation of the iSAM2 fusion algorithm, which is optimized for robust performance for our ranging-IMU fusion problem. We evaluate our solution in a densely occupied real office environment. Our proposed solution can achieve temporally consistent localization with an average absolute accuracy of $\sim$0.3m in real-world settings. Furthermore, our results indicate that we can achieve comparable accuracy even with infrequent (1Hz) range measurements.

A convolutional encoder is said to be catastrophic if it maps an information sequence of infinite weight into a code sequence of finite weight. As a consequence of this mapping, a finite number of channel errors may cause an infinite number of information bit errors when decoding. This situation should be avoided. A catastrophic condition to determine if a time-invariant convolutional encoder is catastrophic or not is stated in \cite{Massey:LSC}. Palazzo developed this condition for periodically time-varying convolutional encoders in \cite{Palazzo:Analysis}. Since Palazzo's condition is based on the state transition table of the constituent encoders, its complexity increases exponentially with the number of memory elements in the encoders. A novel catastrophic condition making use of time-varying equivalent convolutional encoders is presented in this letter. A technique to convert a catastrophic periodically time-varying convolutional encoder into a non-catastrophic one can also be developed based on these encoders. Since they do not involve the state transitions of the convolutional encoder, the time complexity of these methods grows linearly with the encoder memory.

This paper looks into the gain or loss from rolling a fair die multiple times and choosing the highest or lowest number as the outcome over rolling the die just once. Specifically, this paper gives a general formula for the expected value of choosing the highest or lowest value of any number of die rolls and sides. It also shows how, for a fixed number of rolls, the ratio between this expected value and the number of sides converges as the number of sides increases asymptotically. The converging behavior of this ratio helps formulate the aforementioned gain or loss.

Designing protein mutants of both high stability and activity is a critical yet challenging task in protein engineering. Here, we introduce PRIME, a deep learning model, which can suggest protein mutants of improved stability and activity without any prior experimental mutagenesis data of the specified protein. Leveraging temperature-aware language modeling, PRIME demonstrated superior predictive power compared to current state-of-the-art models on the public mutagenesis dataset over 283 protein assays. Furthermore, we validated PRIME's predictions on five proteins, examining the top 30-45 single-site mutations' impact on various protein properties, including thermal stability, antigen-antibody binding affinity, and the ability to polymerize non-natural nucleic acid or resilience to extreme alkaline conditions. Remarkably, over 30% of the AI-recommended mutants exhibited superior performance compared to their pre-mutation counterparts across all proteins and desired properties. Moreover, we have developed an efficient, and successful method based on PRIME to rapidly obtain multi-site mutants with enhanced activity and stability. Hence, PRIME demonstrates the general applicability in protein engineering., Comment: arXiv admin note: text overlap with arXiv:2304.03780

Focused plenoptic cameras can record spatial and angular information of the light field (LF) simultaneously with higher spatial resolution relative to traditional plenoptic cameras, which facilitate various applications in computer vision. However, the existing plenoptic image compression methods present ineffectiveness to the captured images due to the complex micro-textures generated by the microlens relay imaging and long-distance correlations among the microimages. In this paper, a lossy end-to-end learning architecture is proposed to compress the focused plenoptic images efficiently. First, a data preprocessing scheme is designed according to the imaging principle to remove the sub-aperture image ineffective pixels in the recorded light field and align the microimages to the rectangular grid. Then, the global attention module with large receptive field is proposed to capture the global correlation among the feature maps using pixel-wise vector attention computed in the resampling process. Also, a new image dataset consisting of 1910 focused plenoptic images with content and depth diversity is built to benefit training and testing. Extensive experimental evaluations demonstrate the effectiveness of the proposed approach. It outperforms intra coding of HEVC and VVC by an average of 62.57% and 51.67% bitrate reduction on the 20 preprocessed focused plenoptic images, respectively. Also, it achieves 18.73% bitrate saving and generates perceptually pleasant reconstructions compared to the state-of-the-art end-to-end image compression methods, which benefits the applications of focused plenoptic cameras greatly. The dataset and code are publicly available at https://github.com/VincentChandelier/GACN., Comment: 14 pages, 15 figures, accepted by IEEE Transactions on Multimedia

Positioning with 5G signals generally requires connection to several base stations (BSs), which makes positioning more demanding in terms of infrastructure than communications. To address this issue, there have been several theoretical studies on single BS positioning, leveraging high-resolution angle and delay estimation and multipath exploitation possibilities at mmWave frequencies. This paper presents the first realistic experimental validation of such studies, involving a commercial 5G mmWave BS and a user equipment (UE) development kit mounted on a test vehicle. We present the relevant signal models, signal processing methods (including channel parameter estimation and position estimation), and validate these based on real data collected in an outdoor science park environment. Our results indicate that positioning is possible, but the performance with a single BS is limited by the knowledge of the position and orientation of the infrastructure and the multipath visibility and diversity.

Mother-child interaction is highly dynamic and reciprocal. Switching roles in these back-and-forth interactions serves as a crucial feature of reciprocal behaviors while the underlying neural entrainment is still not well-studied. Here, we designed a role-controlled cooperative task with dual EEG recording to study how differently two brains interact when mothers and children hold different roles. When children were actors and mothers were observers, mother-child inter-brain synchrony emerged within the theta oscillations and the frontal lobe, which highly correlated with children's attachment to their mothers. When their roles were reversed, this synchrony was shifted to the alpha oscillations and the central area and associated with mothers' perception of their relationship with their children. The results suggested an observer-actor neural alignment within the actor's oscillations, which was modulated by the actor-toward-observer emotional bonding. Our findings contribute to the understanding of how inter-brain synchrony is established and dynamically changed during mother-child reciprocal interaction.

We consider the problem of ESPRIT-oriented precoder design for beamspace angle-of-departure (AoD) estimation in downlink mmWave multiple-input single-output communications. Standard precoders (i.e., directional/sum beams) yield poor performance in AoD estimation, while Cramer-Rao bound-optimized precoders undermine the so-called shift invariance property (SIP) of ESPRIT. To tackle this issue, the problem of designing ESPRIT-oriented precoders is formulated to jointly optimize over the precoding matrix and the SIP-restoring matrix of ESPRIT. We develop an alternating optimization approach that updates these two matrices under unit-modulus constraints for analog beamforming architectures. Simulation results demonstrate the validity of the proposed approach while providing valuable insights on the beampatterns of the ESPRIT-oriented precoders.

Spatial wideband effects are known to affect channel estimation and localization performance in millimeter wave (mmWave) massive multiple-input multiple-output (MIMO) systems. Based on perturbation analysis, we show that the spatial wideband effect is in fact more pronounced than previously thought and significantly degrades performance, even at moderate bandwidths, if it is not properly considered in the algorithm design. We propose a novel channel estimation method based on multidimensional ESPRIT per subcarrier, combined with unsupervised learning for pairing across subcarriers, which shows significant performance gain over existing schemes under wideband conditions.

Reconfigurable intelligent surfaces (RISs) have tremendous potential to boost communication performance, especially when the line-of-sight (LOS) path between the user equipment (UE) and base station (BS) is blocked. To control the RIS, channel state information (CSI) is needed, which entails significant pilot overhead. To reduce this overhead and the need for frequent RIS reconfiguration, we propose a novel framework for integrated localization and communications, where RIS configurations are fixed during location coherence intervals, while BS precoders are optimized every channel coherence interval. This framework leverages accurate location information obtained with the aid of several RISs as well as novel RIS optimization and channel estimation methods. Performance in terms of localization accuracy, channel estimation error, and achievable rate demonstrates the effectiveness of the proposed approach.

5G positioning is a very promising area that presents many opportunities and challenges. Many existing techniques rely on multiple anchor nodes and line-of-sight (LOS) paths, or single reference node and single-bounce non-LOS (NLOS) paths. However, in dense multipath environments, identifying the LOS or single-bounce assumptions is challenging. The multi-bounce paths will make the positioning accuracy deteriorate significantly. We propose a robust 5G positioning algorithm in NLOS multipath environments. The corresponding positioning problem is formulated as an iterative and weighted least squares problem, and different weights are utilized to mitigate the effects of multi-bounce paths. Numerical simulations are carried out to evaluate the performance of the proposed algorithm. Compared with the benchmark positioning algorithms only using the single-bounce paths, similar positioning accuracy is achieved for the proposed algorithm.