Search

In this paper, we establish novel data-dependent upper bounds on the generalization error through the lens of a "variable-size compressibility" framework that we introduce newly here. In this framework, the generalization error of an algorithm is linked to a variable-size 'compression rate' of its input data. This is shown to yield bounds that depend on the empirical measure of the given input data at hand, rather than its unknown distribution. Our new generalization bounds that we establish are tail bounds, tail bounds on the expectation, and in-expectations bounds. Moreover, it is shown that our framework also allows to derive general bounds on any function of the input data and output hypothesis random variables. In particular, these general bounds are shown to subsume and possibly improve over several existing PAC-Bayes and data-dependent intrinsic dimension-based bounds that are recovered as special cases, thus unveiling a unifying character of our approach. For instance, a new data-dependent intrinsic dimension based bounds is established, which connects the generalization error to the optimization trajectories and reveals various interesting connections with rate-distortion dimension of process, R\'enyi information dimension of process, and metric mean dimension., Comment: 35 pages

Neural network compression has been an increasingly important subject, due to its practical implications in terms of reducing the computational requirements and its theoretical implications, as there is an explicit connection between compressibility and the generalization error. Recent studies have shown that the choice of the hyperparameters of stochastic gradient descent (SGD) can have an effect on the compressibility of the learned parameter vector. Even though these results have shed some light on the role of the training dynamics over compressibility, they relied on unverifiable assumptions and the resulting theory does not provide a practical guideline due to its implicitness. In this study, we propose a simple modification for SGD, such that the outputs of the algorithm will be provably compressible without making any nontrivial assumptions. We consider a one-hidden-layer neural network trained with SGD and we inject additive heavy-tailed noise to the iterates at each iteration. We then show that, for any compression rate, there exists a level of overparametrization (i.e., the number of hidden units), such that the output of the algorithm will be compressible with high probability. To achieve this result, we make two main technical contributions: (i) we build on a recent study on stochastic analysis and prove a 'propagation of chaos' result with improved rates for a class of heavy-tailed stochastic differential equations, and (ii) we derive strong-error estimates for their Euler discretization. We finally illustrate our approach on experiments, where the results suggest that the proposed approach achieves compressibility with a slight compromise from the training and test error., Comment: 28 pages, 1 figure

In this paper, we use tools from rate-distortion theory to establish new upper bounds on the generalization error of statistical distributed learning algorithms. Specifically, there are $K$ clients whose individually chosen models are aggregated by a central server. The bounds depend on the compressibility of each client's algorithm while keeping other clients' algorithms un-compressed, and leverage the fact that small changes in each local model change the aggregated model by a factor of only $1/K$. Adopting a recently proposed approach by Sefidgaran et al., and extending it suitably to the distributed setting, this enables smaller rate-distortion terms which are shown to translate into tighter generalization bounds. The bounds are then applied to the distributed support vector machines (SVM), suggesting that the generalization error of the distributed setting decays faster than that of the centralized one with a factor of $\mathcal{O}(\log(K)/\sqrt{K})$. This finding is validated also experimentally. A similar conclusion is obtained for a multiple-round federated learning setup where each client uses stochastic gradient Langevin dynamics (SGLD)., Comment: Accepted at NeurIPS 2022

It is widely perceived that leveraging the success of modern machine learning techniques to mobile devices and wireless networks has the potential of enabling important new services. This, however, poses significant challenges, essentially due to that both data and processing power are highly distributed in a wireless network. In this paper, we develop a learning algorithm and an architecture that make use of multiple data streams and processing units, not only during the training phase but also during the inference phase. In particular, the analysis reveals how inference propagates and fuses across a network. We study the design criterion of our proposed method and its bandwidth requirements. Also, we discuss implementation aspects using neural networks in typical wireless radio access; and provide experiments that illustrate benefits over state-of-the-art techniques., Comment: Extended version of Globecom'2021 paper; 11 double-column pages; 11 figures; 1 Table. arXiv admin note: substantial text overlap with arXiv:2104.14929

International audience

Publication in the conference proceedings of EUSIPCO, Antalya, Turkey, 2005

International audience; We consider a two-user state-dependent multiaccess channel in which the states of the channel are known, causally or only strictly causally, at both encoders, but not at the decoder. Both encoders transmit a common message and, one of the encoders also transmits an individual message. We study the capacity region of this communication model for both causal and strictly causal settings. For the model with causal states, we find an explicit characterization of the capacity region in the discrete memoryless case. For the model with strictly causal states, we establish inner and outer bounds on the capacity region. The outer bound is non-trivial, has a relatively simple form and has the advantage of incorporating only one auxiliary random variable. In particular, it suggests that there is none, or at best only little, to gain from having the encoder that transmits both messages also sending an individual description of the state to the receiver, in addition to the compressed version that is sent cooperatively with the other encoder. The results shed more light on the utility of delayed channel state information for increasing the capacity region of multiaccess channels; and tie with some recent progress in this framework.

International audience; We study a system in which two sources communicate with a destination with the help of a half-duplex relay. We consider a decoding strategy, based on the compute-and-forward strategy, in which the destination decodes two integer-valued linear combinations that relate the transmitted codewords. In this strategy, the relay compresses its observation using Wyner- Ziv compression and then forwards it to the destination. The destination appropriately combines what it gets from the direct transmission and the relay. Then, using this combination, it computes two integer-valued linear combinations. We discuss the encoding/decoding strategy, and evaluate the achievable sumrate. Next, we consider the problem of allocating the powers and selecting the integer-valued coefficients of the recovered linear combinations in order to maximize the sum-rate. For the model under consideration, the optimization problem is NP hard. We propose an iterative algorithm to solve this problem using coordinate descent method. The results are illustrated through some numerical examples.

International audience; We consider a multiple access relay channel (MARC), in which a relay, based on the recently proposed compute-and-forward protocol, helps two transmitters to communicate with a common destination. The relay decodes a linear combination of the received symbols instead of the individual symbols then forwards the new symbol to the destination. The destination recovers two linear equations from the decoded signals. The two equations relate the transmitted symbols with integer coefficients at different computational rates. We propose an iterative algorithm to optimize the integer coefficients and the power allocation at the transmitters alternatively, so that the sum-rate is maximized. In each iteration, the integer coefficients are updated by solving a mixed-integer quadratic programming (MIQP) problem with quadratic constraints, while the power allocation is updated by solving a series of geometric programs using a successive convex approximation method. The simulation results show that the compute-and-forward strategy and the proposed optimization method can offer substantial gain over the standard amplify-and-forward and decode-and-forward protocols for this model.

We consider the problem of transmission over a relay version of the carbon-copying onto dirty paper. In this setup, additive Gaussian outside interferences corrupt both transmissions to the relay and to the destination; and only the source knows the interferences (in a noncausal manner). We first focus on the case of one interference corrupting both links; and then we focus on the case of two independent interferences. For each of these two models, we establish a lower bound on the channel capacity. The coding schemes for the lower bounds use techniques of dirty paper coding or carbon copying onto dirty paper, interference reduction at the source and decodeand-forward relaying. The results reveal that, by opposition to carbon copying onto dirty paper and its root Costa's initial dirty paper coding (DPC), it may be beneficial in our setup that the informed source uses a part of its power to partially cancel the effect of the interference so that the uninformed relay benefits from this cancellation, and so the source benefits in turn. The established results may be of importance for the emerging field of cooperation in presence of some cognitive radios that might be aware of some of other users messages intended to a common receiver.

In this work, we consider a partially cooperative relay broadcast channel (PC-RBC) controlled by random parameters. We provide rate regions for two different situations: 1) when side information (SI) S^n on the random parameters is non-causally known at both the source and the relay and, 2) when side information S^n is non-causally known at the source only. These achievable regions are derived for the general discrete memoryless case first and then extended to the case when the channel is degraded Gaussian and the SI is additive i.i.d. Gaussian. In this case, the source uses generalized dirty paper coding (GDPC), i.e., DPC combined with partial state cancellation, when only the source is informed, and DPC alone when both the source and the relay are informed. It appears that, even though it can not completely eliminate the effect of the SI (in contrast to the case of source and relay being informed), GDPC is particularly useful when only the source is informed., Comment: 7 pages, Proc. of IEEE International Symposium on Information theory, ISIT 2007, Nice, France