Search

Fine-tuning large language models is a popular choice among users trying to adapt them for specific applications. However, fine-tuning these models is a demanding task because the user has to examine several factors, such as resource budget, runtime, model size and context length among others. A specific challenge is that fine-tuning is memory intensive, imposing constraints on the required hardware memory and context length of training data that can be handled. In this work, we share a detailed study on a variety of fine-tuning optimizations across different fine-tuning scenarios. In particular, we assess Gradient Checkpointing, Low-Rank Adaptation, DeepSpeed's Zero Redundancy Optimizer and FlashAttention. With a focus on memory and runtime, we examine the impact of different optimization combinations on GPU memory usage and execution runtime during fine-tuning phase. We provide our recommendation on the best default optimization for balancing memory and runtime across diverse model sizes. We share effective strategies for fine-tuning very large models with tens or hundreds of billions of parameters and enabling large context lengths during fine-tuning. Furthermore, we propose the appropriate optimization mixtures for fine-tuning under GPU resource limitations., Comment: 10 pages, 4 figures. Revised text for clarity, updated references

Terahertz-band (100 GHz-10 THz) communication is a promising radio technology envisioned to enable ultra-high data rate, reliable and low-latency wireless connectivity in next-generation wireless systems. However, the low transmission power of THz transmitters, the need for high gain directional antennas, and the complex interaction of THz radiation with common objects along the propagation path make crucial the understanding of the THz channel. In this paper, we conduct an extensive channel measurement campaign in an indoor setting (i.e., a conference room) through a channel sounder with 0.1 ns time resolution and 20 GHz bandwidth at 140 GHz. Particularly, the impact of different antenna directivities (and, thus, beam widths) on the channel characteristics is extensively studied. The experimentally obtained dataset is processed to develop the path loss model and, subsequently, derive key channel metrics such as the path loss exponent, delay spread, and K-factor. The results highlight the multi-faceted impact of the antenna gain on the channel and, by extension, the wireless system and, thus, show that an antenna-agnostic channel model cannot capture the propagation characteristics of the THz channel., Comment: Accepted and to be published in IEEE ICC 2024. Copyright \copyright\ 2024 by the Institute of Electrical and Electronics Engineers (IEEE). Permission to make digital or hard copies of portions of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage

In this paper, we introduce a novel fine-tuning technique for language models, which involves incorporating symmetric noise into the embedding process. This method aims to enhance the model's function by more stringently regulating its local curvature, demonstrating superior performance over the current method, NEFTune. When fine-tuning the LLaMA-2-7B model using Alpaca, standard techniques yield a 29.79% score on AlpacaEval. However, our approach, SymNoise, increases this score significantly to 69.04%, using symmetric noisy embeddings. This is a 6.7% improvement over the state-of-the-art method, NEFTune~(64.69%). Furthermore, when tested on various models and stronger baseline instruction datasets, such as Evol-Instruct, ShareGPT, OpenPlatypus, SymNoise consistently outperforms NEFTune. The current literature, including NEFTune, has underscored the importance of more in-depth research into the application of noise-based strategies in the fine-tuning of language models. Our approach, SymNoise, is another significant step towards this direction, showing notable improvement over the existing state-of-the-art method.

Wireless communications in the terahertz band (0.1-10 THz) is a promising and key wireless technology enabling ultra-high data rate communication over multi-gigahertz-wide bandwidths, thus fulfilling the demand for denser networks. The complex propagation environment at such high frequencies introduces several challenges, such as high spreading and molecular absorption losses. As such, intelligent reflecting surfaces have been proposed as a promising solution to enable communication in the presence of blockage or to aid a resource-limited quasi-omnidirectional transmitter direct its radiated power. In this paper, we present a metallic reflectarray design achieving controlled non-specular reflection at true terahertz frequencies (i.e., 1-1.05 THz). We conduct extensive experiments to further characterize and validate its working principle using terahertz time-domain spectroscopy and demonstrate its effectiveness with information-carrying signals using a continuous-wave terahertz testbed. Our results show that the reflectarray can help facilitate robust communication links over non-specular paths and improve the reliability of terahertz communications, thereby unleashing the true potential of the terahertz band., Comment: Accepted and to be published in ACM mmNets 2023. Copyright {\copyright} 2023 by the Association for Computing Machinery, Inc. (ACM). Permission to make digital or hard copies of portions of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage

Near-field mobile terahertz (THz) communications is one of the candidate enablers for high-rate wireless data exchange in sixth-generation (6G) networks. However, operating in the THz near field brings both attractive opportunities and severe challenges. Hence, it becomes of interest to explore if it is possible to design a realistic mobile THz communication system without working in the THz near field. To answer this question, a mathematical framework is presented modeling a mobile THz link that works exclusively in the far field. The study leads to an interesting theoretical conclusion: while the actual frequency is of (almost) no interest, such a system must operate over a limited bandwidth not exceeding a certain threshold. It is then numerically shown that operating only in the far field imposes stringent limitations on mobile THz communications, thus making them less attractive to prospective high-rate services. In contrast, it is shown that a stationary THz link can still be broadband even when staying exclusively in the THz far field. Hence, broadband mobile THz communications MUST be near-field, while broadband stationary THz links do not have to., Comment: Accepted to IEEE Globecom, 2023. Copyright 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material, creating new works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works