Search

Since the invention of GPT2--1.5B in 2019, large language models (LLMs) have transitioned from specialized models to versatile foundation models. The LLMs exhibit impressive zero-shot ability, however, require fine-tuning on local datasets and significant resources for deployment. Traditional fine-tuning techniques with the first-order optimizers require substantial GPU memory that exceeds mainstream hardware capability. Therefore, memory-efficient methods are motivated to be investigated. Model compression techniques can reduce energy consumption, operational costs, and environmental impact so that to support sustainable artificial intelligence advancements. Additionally, large-scale foundation models have expanded to create images, audio, videos, and multi-modal contents, further emphasizing the need for efficient deployment. Therefore, we are motivated to present a comprehensive overview of the prevalent memory-efficient fine-tuning methods over the network edge. We also review the state-of-the-art literatures on model compression to provide a vision on deploying LLMs over the network edge.

By using an parametric value function to replace the Monte-Carlo rollouts for value estimation, the actor-critic (AC) algorithms can reduce the variance of stochastic policy gradient so that to improve the convergence rate. While existing works mainly focus on analyzing convergence rate of AC algorithms under Markovian noise, the impacts of momentum on AC algorithms remain largely unexplored. In this work, we first propose a heavy-ball momentum based advantage actor-critic (\mbox{HB-A2C}) algorithm by integrating the heavy-ball momentum into the critic recursion that is parameterized by a linear function. When the sample trajectory follows a Markov decision process, we quantitatively certify the acceleration capability of the proposed HB-A2C algorithm. Our theoretical results demonstrate that the proposed HB-A2C finds an $\epsilon$-approximate stationary point with $\oo{\epsilon^{-2}}$ iterations for reinforcement learning tasks with Markovian noise. Moreover, we also reveal the dependence of learning rates on the length of the sample trajectory. By carefully selecting the momentum factor of the critic recursion, the proposed HB-A2C can balance the errors introduced by the initialization and the stoschastic approximation.

The next generation of communication is envisioned to be intelligent communication, that can replace traditional symbolic communication, where highly condensed semantic information considering both source and channel will be extracted and transmitted with high efficiency. The recent popular large models such as GPT4 and the boosting learning techniques lay a solid foundation for the intelligent communication, and prompt the practical deployment of it in the near future. Given the characteristics of "training once and widely use" of those multimodal large language models, we argue that a pay-as-you-go service mode will be suitable in this context, referred to as Large Model as a Service (LMaaS). However, the trading and pricing problem is quite complex with heterogeneous and dynamic customer environments, making the pricing optimization problem challenging in seeking on-hand solutions. In this paper, we aim to fill this gap and formulate the LMaaS market trading as a Stackelberg game with two steps. In the first step, we optimize the seller's pricing decision and propose an Iterative Model Pricing (IMP) algorithm that optimizes the prices of large models iteratively by reasoning customers' future rental decisions, which is able to achieve a near-optimal pricing solution. In the second step, we optimize customers' selection decisions by designing a robust selecting and renting (RSR) algorithm, which is guaranteed to be optimal with rigorous theoretical proof. Extensive experiments confirm the effectiveness and robustness of our algorithms.

A wireless federated learning system is investigated by allowing a server and workers to exchange uncoded information via orthogonal wireless channels. Since the workers frequently upload local gradients to the server via bandwidth-limited channels, the uplink transmission from the workers to the server becomes a communication bottleneck. Therefore, a one-shot distributed principle component analysis (PCA) is leveraged to reduce the dimension of uploaded gradients such that the communication bottleneck is relieved. A PCA-based wireless federated learning (PCA-WFL) algorithm and its accelerated version (i.e., PCA-AWFL) are proposed based on the low-dimensional gradients and the Nesterov's momentum. For the non-convex loss functions, a finite-time analysis is performed to quantify the impacts of system hyper-parameters on the convergence of the PCA-WFL and PCA-AWFL algorithms. The PCA-AWFL algorithm is theoretically certified to converge faster than the PCA-WFL algorithm. Besides, the convergence rates of PCA-WFL and PCA-AWFL algorithms quantitatively reveal the linear speedup with respect to the number of workers over the vanilla gradient descent algorithm. Numerical results are used to demonstrate the improved convergence rates of the proposed PCA-WFL and PCA-AWFL algorithms over the benchmarks.

Building fair machine learning models becomes more and more important. As many powerful models are built by collaboration among multiple parties, each holding some sensitive data, it is natural to explore the feasibility of training fair models in cross-silo federated learning so that fairness, privacy and collaboration can be fully respected simultaneously. However, it is a very challenging task, since it is far from trivial to accurately estimate the fairness of a model without knowing the private data of the participating parties. In this paper, we first propose a federated estimation method to accurately estimate the fairness of a model without infringing the data privacy of any party. Then, we use the fairness estimation to formulate a novel problem of training fair models in cross-silo federated learning. We develop FedFair, a well-designed federated learning framework, which can successfully train a fair model with high performance without any data privacy infringement. Our extensive experiments on three real-world data sets demonstrate the excellent fair model training performance of our method.

This work investigates fault-resilient federated learning when the data samples are non-uniformly distributed across workers, and the number of faulty workers is unknown to the central server. In the presence of adversarially faulty workers who may strategically corrupt datasets, the local messages exchanged (e.g., local gradients and/or local model parameters) can be unreliable, and thus the vanilla stochastic gradient descent (SGD) algorithm is not guaranteed to converge. Recently developed algorithms improve upon vanilla SGD by providing robustness to faulty workers at the price of slowing down convergence. To remedy this limitation, the present work introduces a fault-resilient proximal gradient (FRPG) algorithm that relies on Nesterov's acceleration technique. To reduce the communication overhead of FRPG, a local (L) FRPG algorithm is also developed to allow for intermittent server-workers parameter exchanges. For strongly convex loss functions, FRPG and LFRPG have provably faster convergence rates than a benchmark robust stochastic aggregation algorithm. Moreover, LFRPG converges faster than FRPG while using the same communication rounds. Numerical tests performed on various real datasets confirm the accelerated convergence of FRPG and LFRPG over the robust stochastic aggregation benchmark and competing alternatives., Comment: submitted for possible publication, and based on our independent work

User scheduling, beamforming and energy coordination are investigated in smart-grid powered cellular networks (SGPCNs), where the base stations are powered by a smart grid and natural renewable energy sources. Heterogeneous energy coordination is considered in SGPCNs, namely energy merchandizing with the smart grid and energy exchanging among the base stations. A long-term grid-energy expenditure minimization problem with proportional-rate constraints is formulated for SGPCNs. Since user scheduling is coupled with the beamforming vectors, the formulated problem is challenging to handle via standard convex optimization methods. In practice, the beamforming vectors need to be updated over each slot according to the channel variations. User scheduling needs to be updated over several slots (frame) since the frequent scheduling of user equipment can cause reliability issues. Therefore, the Lyapunov optimization method is used to decouple the problem. A practical two-scale algorithm is proposed to schedule users at each frame, and obtain the beamforming vectors and amount of exchanged natural renewable energy at each slot. We prove that the proposed two-scale algorithm can asymptotically achieve the optimal solutions via tuning a control parameter. Numerical results verify the performance of the proposed two-scale algorithm., Comment: accepted journal article

In the small-cell networks with multiple-input-single-output broadcasting (MISO-BC) backhauls, the joint dirty-paper coding and power control are investigated for the \mbox{MISO-BC} backhauls and access links in order to minimize the system transmit power. Considering the proportional rates of MISO-BC backhauls and flow-conservation constraints, the formulated optimization problem is \mbox{non-convex}. Moreover, the formulated problem couples the precoding vectors with the power-control variables. In order to handle the \mbox{non-convex} optimization problem and decouple the backhaul and access links, the structure of the formulated problem is investigated such that the optimal precoding vectors and optimal power-control variables are independently obtained. Moreover, the optimal precoding vectors are obtained in closed-form expressions. Simulation results are used to show the performance improvement over the benchmark scheme., Comment: This work was accepted by IEEE Globecom 2019

In the small-cell networks (SCNs) with multiple small-cell base stations (ScBSs), the joint design of beamforming vectors, user scheduling and ScBS sleeping is investigated with the constraints on proportional rate. A long-term grid-energy expenditure minimization problem is formulated for the considered SCNs, which are powered by the smart grid and natural renewable energy. Since the scheduled user indicators are coupled with the beamforming vectors, the formulated problem is challenging to handle. In order to decouple the beamforming vectors from the scheduled user indicators, the Lyapunov optimization technique is used. As a result, a practical two-scale algorithm is proposed to allocate the user scheduling indicators and ScBS sleeping variables at the coarse-grained granularity (frame) as well as obtain the beamforming vectors at the fine-grained granularity (slot). Numerical results are used to verify the performance of the proposed two-scale algorithm., Comment: This work was accepted by IEEE ICC 2019

The privacy concern exists when the central server has the copies of datasets. Hence, there is a paradigm shift for the learning networks to change from centralized in-cloud learning to distributed \mbox{on-device} learning. Benefit from the parallel computing, the on-device learning networks have a lower bandwidth requirement than the in-cloud learning networks. Moreover, the on-device learning networks also have several desirable characteristics such as privacy preserving and flexibility. However, the \mbox{on-device} learning networks are vulnerable to the malfunctioning terminals across the networks. The worst-case malfunctioning terminals are the Byzantine adversaries, that can perform arbitrary harmful operations to compromise the learned model based on the full knowledge of the networks. Hence, the design of secure learning algorithms becomes an emerging topic in the on-device learning networks with Byzantine adversaries. In this article, we present a comprehensive overview of the prevalent secure learning algorithms for the two promising on-device learning networks: Federated-Learning networks and decentralized-learning networks. We also review several future research directions in the \mbox{Federated-Learning} and decentralized-learning networks., Comment: This work was in part accepted by IEEE Network

The joint design of beamforming vector and artificial noise covariance matrix is investigated for the multiple-input-single-output-multiple-eavesdropper simultaneous wireless information and power transferring \mbox{(MISOME-SWIPT)} systems. In the MISOME-SWIPT system, the base station delivers information signals to the legitimate user equipments and broadcasts jamming signals to the eavesdroppers. A secrecy energy efficiency (SEE) maximization problem is formulated for the considered \mbox{MISOME-SWIPT} system with imperfect channel state information, where the SEE is defined as the ratio of sum secrecy rate over total power consumption. Since the formulated SEE maximization problem is non-convex, it is first recast into a series of convex problems in order to obtain the optimal solution with a reasonable computational complexity. Two suboptimal solutions are also proposed based on the heuristic beamforming techniques that trade performance for computational complexity. In addition, the analysis of computational complexity is performed for the optimal and suboptimal solutions. Numerical results are used to verify the performance of proposed algorithms and to reveal practical insights., Comment: This work was accepted by IEEE JSAC. arXiv admin note: text overlap with arXiv:1808.02004

The joint design of beamforming vector and artificial noise covariance matrix is investigated for multiple-input-single-output-multiple-eavesdropper simultaneous wireless information and power transferring (MISOME-SWIPT) systems. A secrecy energy efficiency (SEE) maximization problem is formulated in the MISOME-SWIPT system with imperfect channel state information and proportional secrecy rate constraints. Since the formulated SEE maximization problem is non-convex, it is first recast into a series of convex problems in order to obtain the optimal solution with a reasonable computational complexity. Numerical results are used to verify the performance of the proposed algorithm and to reveal practical insights., Comment: This work was accepted in IEEE Globecom 2018

With the maturity of unmanned aerial vehicle (UAV) technology, this work investigates the integration of UAV into wireless communication systems. Since the UAV is powered by a capacity-limited battery, this work proposes to use the radio energy harvesting technology at the UAV in order to extend the lifetime of UAV empowered base station. To extract the most weighted throughput of UAV empowered wireless systems, the dirty paper coding scheme and information-theoretic uplink-downlink channel duality are exploited to propose an extracting the most weighted throughput algorithm. Numerical results are used to verify the proposed algorithm., Comment: This work was accepted by BSC 2018, Toronto, Canada

One promising approach to address the supply-demand mismatch between the terrestrial infrastructure and the temporary and/or unexpected traffic demands is to leverage the unmanned aerial vehicle (UAV) technologies. Motivated by the recent advancement of UAV technologies and retromodulator based free space optical communication, we propose a novel edge-computing empowered radio access network architecture where the fronthaul and backhaul links are mounted on the UAVs for rapid event response and flexible deployment. The implementation of hardware and networking technologies for the proposed architecture are investigated. Due to the limited payload and endurance as well as the high mobility of UAVs, research challenges related to the communication resource management and recent research progress are reported., Comment: This work is accepted by IEEE Wireless Communications Magazine

Security is a critical issue in full duplex (FD) communication systems due to the broadcast nature of wireless channels. In this paper, joint design of information and artificial noise beamforming vectors is proposed for the FD simultaneous wireless information and power transferring (FD-SWIPT) systems with loopback self-interference cancellation. To guarantee high security and energy harvesting performance of the FD-SWIPT system, the proposed design is formulated as a secrecy rate maximization problem under energy transfer rate constraints. Although the secrecy rate maximization problem is non-convex, we solve it via semidefinite relaxation and a two-dimensional search. We prove the optimality of our proposed algorithm and demonstrate its performance via simulations., Comment: Accepted in ICC 2018

We investigate the precoding, remote radio head (RRH) selection and signal splitting in the simultaneous wireless information and power transferring (SWIPT) cloud radio access networks \mbox{(C-RANs)}. The objective is to minimize the power consumption of the SWIPT C-RAN. Different from the existing literature, we consider the nonlinear fronthaul power consumption and the multiple antenna RRHs. By switching off the unnecessary RRHs, the group sparsity of the precoding coefficients is introduced, which indicates that the precoding process and the RRH selection are coupled. In order to overcome these issues, a group sparse precoding and signal splitting algorithm is proposed based on the majorization-minimization framework, and the convergence behavior is established. Numerical results are used to verify our proposed studies., Comment: Accepted by IEEE Globecom 2017

The application of renewable energy is a promising solution to realize the Green Communications. However, if the cellular systems are solely powered by the renewable energy, the weather dependence of the renewable energy arrival makes the systems unstable. On the other hand, the proliferation of the smart grid facilitates the loads with two-way energy trading capability. Hence, a hybrid powered cellular system, which combines the smart grid with the base stations, can reduce the grid energy expenditure and improve the utilization efficiency of the renewable energy. In this paper, the long-term grid energy expenditure minimization problem is formulated as a stochastic optimization model. By leveraging the stochastic optimization theory, we reformulate the stochastic optimization problem as a \mbox{per-frame} grid energy plus weighted penalized packet rate minimization problem, which is NP-hard. As a result, two suboptimal algorithms, which jointly consider the effects of the channel quality and the packet reception failure, are proposed based on the successive approximation beamforming (SABF) technique and the \mbox{zero-forcing} beamforming (ZFBF) technique. The convergence properties of the proposed suboptimal algorithms are established, and the corresponding computational complexities are analyzed. Simulation results show that the proposed SABF algorithm outperforms the ZFBF algorithm in both grid energy expenditure and packet delay. By tuning a control parameter, the grid energy expenditure can be traded for the packet delay under the proposed stochastic optimization model., Comment: accepted by IEEE Trans. Wireless Commun

Physical layer security is a key issue in the full duplex (FD) communication systems due to the broadcast nature of wireless channels. In this paper, the joint design of information and artificial noise beamforming vectors is proposed for the FD simultaneous wireless information and power transferring (FD-SWIPT) systems. To guarantee high security and energy harvesting performance of the FD-SWIPT system, the proposed design is formulated as a sum information transmission rate (SITR) maximization problem under information-leakage and energy constraints. In addition, we consider the fairness issue between the uplink and downlink information transmission rates by formulating a \mbox{fairness-aware} SITR-maximization problem. Although the formulated \mbox{SITR-maximization} and \mbox{fairness-aware} \mbox{SITR-maximization} problems are non-convex, we solve them via semidefinite relaxation and one-dimensional search. The optimality of our proposed algorithms is theoretically proved, and the computation complexities are established. Moreover, we propose two suboptimal solutions to the formulated optimization problems. In terms of the SITR-maximization problem, numerical results show that the performance achieved by one of the two suboptimal algorithms is close to the performance of the optimal algorithm with increasing maximum transmission power of the FD-BST., Comment: submitted for journal publication

Ultra dense cloud small cell network (UDCSNet), which combines cloud computing and massive deployment of small cells, is a promising technology for the fifth-generation (5G) LTE-U mobile communications because it can accommodate the anticipated explosive growth of mobile users' data traffic. As a result, fronthauling becomes a challenging problem in 5G LTE-U UDCSNet. In this article, we present an overview of the challenges and requirements of the fronthaul technology in 5G \mbox{LTE-U} UDCSNets. We survey the advantages and challenges for various candidate fronthaul technologies such as optical fiber, millimeter-wave based unlicensed spectrum, Wi-Fi based unlicensed spectrum, sub 6GHz based licensed spectrum, and free-space optical based unlicensed spectrum., Comment: 17 pages