Search

Traditional pruning methods are known to be challenging to work in Large Language Models (LLMs) for Generative AI because of their unaffordable training process and large computational demands. For the first time, we introduce the information entropy of hidden state features into a pruning metric design, namely E-Sparse, to improve the accuracy of N:M sparsity on LLM. E-Sparse employs the information richness to leverage the channel importance, and further incorporates several novel techniques to put it into effect: (1) it introduces information entropy to enhance the significance of parameter weights and input feature norms as a novel pruning metric, and performs N:M sparsity without modifying the remaining weights. (2) it designs global naive shuffle and local block shuffle to quickly optimize the information distribution and adequately cope with the impact of N:M sparsity on LLMs' accuracy. E-Sparse is implemented as a Sparse-GEMM on FasterTransformer and runs on NVIDIA Ampere GPUs. Extensive experiments on the LLaMA family and OPT models show that E-Sparse can significantly speed up the model inference over the dense model (up to 1.53X) and obtain significant memory saving (up to 43.52%), with acceptable accuracy loss.

Understanding the fundamental principle of catalytic performance and the mechanism of multimetal-based electrocatalysts is essential for the rational design of advanced renewable energy systems. Here, highly crystalline MMMoB4 (M = Fe, Co) compounds with controllable compositions of multiple active metal atoms and polyacene-type boron networks were synthesized delicately by a one-step high-pressure technique to explore electrocatalytic selectivity and activity. CoFeMoB4 and Co2MoB4 are revealed to be highly active and durable oxygen evolution reaction (OER) electrocatalysts under alkaline conditions. The mutually promotive activation of metals with amorphous clusters and ultra-small grains on the surface are responsible for the enhanced activity of CoFeMoB4. More specifically, Co and Fe coupling in CoFeMoB4 facilitates surface reconstruction into active Co hydroxide and Fe oxyhydroxide, in contrast to Co oxyhydroxide in Co2MoB4 and Fe oxides in Fe2MoB4. Dissolving Mo may provide potential space for adsorbing hydroxyl, and the optimized electronic structure with boron is mainly responsible for the long-term durability. In contrast, Mo atoms are responsible for hydrogen evolution reaction (HER) properties, and the optimized d-band center and density of states at the Fermi level make Co2MoB4 a superior HER catalyst. Our findings provide insight into distinguishing the catalytic pathway of multi-metal borides with improved OER activity and different roles of Mo and Co/Fe in the HER and OER.

Due to the change of microstructure and increased heterogeneity of ultra-high-performance concrete (UHPC) in comparison with normal concrete, the thermal properties of UHPC are much less uniform within the material. Consequently, the fire performance of UHPC filled steel tubes (UHPCFST) columns are affected by the random distribution of temperature within the concrete. In order to study the distribution of the random temperature field within UFPCFST columns under elevated temperature, high temperature tests on 16 circular UHPCFST columns subjected to the ISO-834 standard fire and a linear heating mode are carried out. The average temperature, temperature difference, temperature probability density distribution and the temporal and spatial correlation of the random temperature field are investigated. It is found that the columns with more steel fiber and coarse aggregate contents have higher temperature, small temperature gradient and higher temperature difference. It is also found that the correlation of the zero-centered temperature in the circumferential direction between different time is strong when the temperature continues to rise or fall. The aims of this paper is to provide the reference for the analysis of temperature field of UHPCFST considering the heterogeneity of UHPC core.

Due to the change of microstructure and increased heterogeneity of ultra-high-performance concrete (UHPC) in comparison with normal concrete, the thermal properties of UHPC are much less uniform within the material. Consequently, the fire performance of UHPC filled steel tubes (UHPCFST) columns are affected by the random distribution of temperature within the concrete. In order to study the distribution of the random temperature field within UFPCFST columns under elevated temperature, high temperature tests on 16 circular UHPCFST columns subjected to the ISO-834 standard fire and a linear heating mode are carried out. The average temperature, temperature difference, temperature probability density distribution and the temporal and spatial correlation of the random temperature field are investigated. It is found that the columns with more steel fiber and coarse aggregate contents have higher temperature, small temperature gradient and higher temperature difference. It is also found that the correlation of the zero-centered temperature in the circumferential direction between different time is strong when the temperature continues to rise or fall. The aims of this paper is to provide the reference for the analysis of temperature field of UHPCFST considering the heterogeneity of UHPC core.

Recently, pre-trained Transformer based language models, such as BERT, have shown great superiority over the traditional methods in many Natural Language Processing (NLP) tasks. However, the computational cost for deploying these models is prohibitive on resource-restricted devices. One method to alleviate this computation overhead is to quantize the original model into fewer bits representation, and previous work has proved that we can at most quantize both weights and activations of BERT into 8-bits, without degrading its performance. In this work, we propose MKQ-BERT, which further improves the compression level and uses 4-bits for quantization. In MKQ-BERT, we propose a novel way for computing the gradient of the quantization scale, combined with an advanced distillation strategy. On the one hand, we prove that MKQ-BERT outperforms the existing BERT quantization methods for achieving a higher accuracy under the same compression level. On the other hand, we are the first work that successfully deploys the 4-bits BERT and achieves an end-to-end speedup for inference. Our results suggest that we could achieve 5.3x of bits reduction without degrading the model accuracy, and the inference speed of one int4 layer is 15x faster than a float32 layer in Transformer based model.

Projecte realitzat en el marc d’un programa de mobilitat amb la Université de Bourgogne, [ANGLÈS] In this thesis, a tunable analog filter with digital control is presented. The digital control is implemented by means of a Spartan 3A FPGA and programmable resistors. The system consists of two identical tunable filters. The master filter filters an input tone and the digital control compares the input and the output and modifies the programmable resistors to tune the central frequency of the filter to the input tone. Then, the same parameters are applied to the slave filter obtaining the tunable filter. Simulations and experimental results show the feasibility of this approach., [CASTELLÀ] En este proyecto se presenta un filtro analógico sintonizable con control digital. El control digital se implementa por medio de una FPGA Spartan 3A y resistencias programables. El sistema consta de dos filtros sintonizables idénticos. El filtro 'master' filtra un tono de entrada y el de control digital compara la entrada y la salida y modifica las resistencias programables para sintonizar la frecuencia central del filtro para el tono de entrada. Entonces, los mismos parámetros se aplican al filtro 'slave' para obtener el filtro sintonizable. Las simulaciones y los resultados experimentales muestran la viabilidad de este enfoque., [CATALÀ] En aquest projecte es presenta un filtre analògic sintonitzable amb control digital. El control digital s'implementa per mitjà d'una FPGA Spartan 3A i resistències programables. El sistema consta de dos filtres sintonitzables idèntics. El filtre 'master' filtra un to d'entrada i el de control digital compara l'entrada i la sortida i modifica les resistències programables per sintonitzar la freqüència central del filtre al to d'entrada. Llavors, els mateixos paràmetres s'apliquen al filtre 'slave' per obtenir el filtre sintonitzable. Les simulacions i els resultats experimentals mostren la viabilitat d'aquest enfocament.