Search

The tremendous success of Large Language Models (LLMs) across various complex tasks relies heavily on their substantial scale, which raises challenges during model deployment due to their large memory consumption. Recently, numerous studies have attempted to compress LLMs using one-shot pruning methods. However, these methods often experience considerable performance degradation on complex language understanding tasks, calling into question the feasibility of pruning in LLMs. To address this issue, we propose a pruning pipeline for semi-structured sparse models via retraining, termed Adaptive Sparse Trainer (AST). Unlike previous one-shot pruning methods, AST incrementally transforms dense models into sparse ones by applying decay to masked weights while allowing the model to adaptively select masks throughout the training process. Furthermore, we observe that using distillation with a dense model as the teacher can prevent the sparse model from falling into local optima and accelerate convergence. In addition, we incorporate extra well-initialized parameters to further enhance model performance with minimal increase in memory footprint. AST can significantly enhance model performance, approaching the level of dense models. When applied to the LLaMA2-7B model, AST reduces the zero-shot accuracy gap between dense and semi-structured sparse models to 1.12% across multiple zero-shot tasks, utilizing less than 0.4% of the pretraining tokens. Our work demonstrates the feasibility of deploying semi-structured sparse large language models and introduces a novel method for achieving highly compressed models when combined with existing quantization techniques.

Training large transformers is slow, but recent innovations on GPU architecture give us an advantage. NVIDIA Ampere GPUs can execute a fine-grained 2:4 sparse matrix multiplication twice as fast as its dense equivalent. In the light of this property, we comprehensively investigate the feasibility of accelerating feed-forward networks (FFNs) of transformers in pre-training. First, we define a ``flip rate'' to monitor the stability of a 2:4 training process. Utilizing this metric, we propose three techniques to preserve accuracy: to modify the sparse-refined straight-through estimator by applying the masked decay term on gradients, to determine a feasible decay factor in warm-up stage, and to enhance the model's quality by a dense fine-tuning procedure near the end of pre-training. Besides, we devise two techniques to practically accelerate training: to calculate transposable 2:4 masks by convolution, and to accelerate gated activation functions by reducing GPU L2 cache miss. Experiments show that our 2:4 sparse training algorithm achieves similar convergence to dense training algorithms on several transformer pre-training tasks, while actual acceleration can be observed on different shapes of transformer block apparently. Our toolkit is available at https://github.com/huyz2023/2by4-pretrain.

Pharmaceutical targeting is one of key inputs for making sales and marketing strategy planning. Targeting list is built on predicting physician's sales potential of certain type of patient. In this paper, we present a time-sensitive targeting framework leveraging time series model to predict patient's disease and treatment progression. We create time features by extracting service history within a certain period, and record whether the event happens in a look-forward period. Such feature-label pairs are examined across all time periods and all patients to train a model. It keeps the inherent order of services and evaluates features associated to the imminent future, which contribute to improved accuracy., Comment: Machine Learning for Health (ML4H) Workshop at NeurIPS 2018 arXiv:1811.07216

Modern neuroimaging techniques provide us with unique views on brain structure and function; i.e., how the brain is wired, and where and when activity takes place. Data acquired using these techniques can be analyzed in terms of its network structure to reveal organizing principles at the systems level. Graph representations are versatile models where nodes are associated to brain regions and edges to structural or functional connections. Structural graphs model neural pathways in white matter that are the anatomical backbone between regions. Functional graphs are built based on functional connectivity, which is a pairwise measure of statistical interdependency between activity traces of regions. Therefore, most research to date has focused on analyzing these graphs reflecting structure or function. Graph signal processing (GSP) is an emerging area of research where signals recorded at the nodes of the graph are studied atop the underlying graph structure. An increasing number of fundamental operations have been generalized to the graph setting, allowing to analyze the signals from a new viewpoint. Here, we review GSP for brain imaging data and discuss their potential to integrate brain structure, contained in the graph itself, with brain function, residing in the graph signals. We review how brain activity can be meaningfully filtered based on concepts of spectral modes derived from brain structure. We also derive other operations such as surrogate data generation or decompositions informed by cognitive systems. In sum, GSP offers a novel framework for the analysis of brain imaging data., Comment: Weiyu Huang and Thomas A. W. Bolton contributed equally

This paper presents methods to compare networks where relationships between pairs of nodes in a given network are defined. We define such network distance by searching for the optimal method to embed one network into another network, prove that such distance is a valid metric in the space of networks modulo permutation isomorphisms, and examine its relationship with other network metrics. The network distance defined can be approximated via multi-dimensional scaling, however, the lack of structure in networks results in poor approximations. To alleviate such problem, we consider methods to define the interiors of networks. We show that comparing interiors induced from a pair of networks yields the same result as the actual network distance between the original networks. Practical implications are explored by showing the ability to discriminate networks generated by different models.

Cognitive flexibility describes the human ability to switch between modes of mental function to achieve goals. Mental switching is accompanied by transient changes in brain activity, which must occur atop an anatomical architecture that bridges disparate cortical and subcortical regions by underlying white matter tracts. However, an integrated perspective regarding how white matter networks might constrain brain dynamics during cognitive processes requiring flexibility has remained elusive. To address this challenge, we applied emerging tools from graph signal processing to decompose BOLD signals based on diffusion imaging tractography in 28 individuals performing a perceptual task that probed cognitive flexibility. We found that the alignment between functional signals and the architecture of the underlying white matter network was associated with greater cognitive flexibility across subjects. Signals with behaviorally-relevant alignment were concentrated in the basal ganglia and anterior cingulate cortex, consistent with cortico-striatal mechanisms of cognitive flexibility. Importantly, these findings are not accessible to unimodal analyses of functional or anatomical neuroimaging alone. Instead, by taking a generalizable and concise reduction of multimodal neuroimaging data, we uncover an integrated structure-function driver of human behavior.

This paper considers metric spaces where distances between a pair of nodes are represented by distance intervals. The goal is to study methods for the determination of hierarchical clusters, i.e., a family of nested partitions indexed by a resolution parameter, induced from the given distance intervals of the metric spaces. Our construction of hierarchical clustering methods is based on defining admissible methods to be those methods that abide to the axioms of value - nodes in a metric space with two nodes are clustered together at the convex combination of the distance bounds between them - and transformation - when both distance bounds are reduced, the output may become more clustered but not less. Two admissible methods are constructed and are shown to provide universal upper and lower bounds in the space of admissible methods. Practical implications are explored by clustering moving points via snapshots and by clustering networks representing brain structural connectivity using the lower and upper bounds of the network distance. The proposed clustering methods succeed in identifying underlying clustering structures via the maximum and minimum distances in all snapshots, as well as in differentiating brain connectivity networks of patients from those of healthy controls.

Multimodal neuroimaging studies of corticobasal syndrome using volumetric MRI and DTI successfully discriminate between Alzheimer's disease and frontotemporal lobar degeneration but this evidence has typically included clinically heterogeneous patient cohorts and has rarely assessed the network structure of these distinct sources of pathology. Using structural MRI data, we identify areas in fronto-temporo-parietal cortex with reduced gray matter density in corticobasal syndrome relative to age matched controls. A support vector machine procedure demonstrates that gray matter density poorly discriminates between frontotemporal lobar degeneration and Alzheimer's disease pathology subgroups with low sensitivity and specificity. In contrast, a statistic of local network efficiency demonstrates excellent discriminatory power, with high sensitivity and specificity. Our results indicate that the underlying pathological sources of corticobasal syndrome can be classified more accurately using graph theoretical statistics of white matter microstructure in association cortex than by regional gray matter density alone. These results highlight the importance of a multimodal neuroimaging approach to diagnostic analyses of corticobasal syndrome and suggest that distinct sources of pathology mediate the circuitry of brain regions affected by corticobasal syndrome.

This paper presents methods to analyze functional brain networks and signals from graph spectral perspectives. The notion of frequency and filters traditionally defined for signals supported on regular domains such as discrete time and image grids has been recently generalized to irregular graph domains, and defines brain graph frequencies associated with different levels of spatial smoothness across the brain regions. Brain network frequency also enables the decomposition of brain signals into pieces corresponding to smooth or rapid variations. We relate graph frequency with principal component analysis when the networks of interest denote functional connectivity. The methods are utilized to analyze brain networks and signals as subjects master a simple motor skill. We observe that brain signals corresponding to different graph frequencies exhibit different levels of adaptability throughout learning. Further, we notice a strong association between graph spectral properties of brain networks and the level of exposure to tasks performed, and recognize the most contributing and important frequency signatures at different task familiarity.

High order networks are weighted hypergraphs col- lecting relationships between elements of tuples, not necessarily pairs. Valid metric distances between high order networks have been defined but they are difficult to compute when the number of nodes is large. The goal here is to find tractable approximations of these network distances. The paper does so by mapping high order networks to filtrations of simplicial complexes and showing that the distance between networks can be lower bounded by the difference between the homological features of their respective filtrations. Practical implications are explored by classifying weighted pairwise networks constructed from different gener- ative processes and by comparing the coauthorship networks of engineering and mathematics academic journals. The persistent homology methods succeed in identifying different generative models, in discriminating engineering and mathematics commu- nities, as well as in differentiating engineering communities with different research interests.

This paper presents methods to compare high order networks, defined as weighted complete hypergraphs collecting relationship functions between elements of tuples. They can be considered as generalizations of conventional networks where only relationship functions between pairs are defined. Important properties between relationships of tuples of different lengths are established, particularly when relationships encode dissimilarities or proximities between nodes. Two families of distances are then introduced in the space of high order networks. The distances measure differences between networks. We prove that they are valid metrics in the spaces of high order dissimilarity and proximity networks modulo permutation isomorphisms. Practical implications are explored by comparing the coauthorship networks of two popular signal processing researchers. The metrics succeed in identifying their respective collaboration patterns.

We introduce the diffusion and superposition distances as two metrics to compare signals supported in the nodes of a network. Both metrics consider the given vectors as initial temperature distributions and diffuse heat trough the edges of the graph. The similarity between the given vectors is determined by the similarity of the respective diffusion profiles. The superposition distance computes the instantaneous difference between the diffused signals and integrates the difference over time. The diffusion distance determines a distance between the integrals of the diffused signals. We prove that both distances define valid metrics and that they are stable to perturbations in the underlying network. We utilize numerical experiments to illustrate their utility in classifying signals in a synthetic network as well as in classifying ovarian cancer histologies using gene mutation profiles of different patients. We also reinterpret diffusion as a transformation of interrelated feature spaces and use it as preprocessing tool for learning. We use diffusion to increase the accuracy of handwritten digit classification.