Search

A client device which has access to $n$ training data samples needs to obtain a statistical hypothesis or model $W$ and then to send it to a remote server. The client and the server devices share some common randomness sequence as well as a prior on the hypothesis space. In this problem a suitable hypothesis or model $W$ should meet two distinct design criteria simultaneously: (i) small (population) risk during the inference phase and (ii) small 'complexity' for it to be conveyed to the server with minimum communication cost. In this paper, we propose a joint training and source coding scheme with provable in-expectation guarantees, where the expectation is over the encoder's output message. Specifically, we show that by imposing a constraint on a suitable Kullback-Leibler divergence between the conditional distribution induced by a compressed learning model $\widehat{W}$ given $W$ and the prior, one guarantees simultaneously small average empirical risk (aka training loss), small average generalization error and small average communication cost. We also consider a one-shot scenario in which the guarantees on the empirical risk and generalization error are obtained for every encoder's output message., Comment: Accepted at ISIT 2024

A major challenge in designing efficient statistical supervised learning algorithms is finding representations that perform well not only on available training samples but also on unseen data. While the study of representation learning has spurred much interest, most existing such approaches are heuristic; and very little is known about theoretical generalization guarantees. In this paper, we establish a compressibility framework that allows us to derive upper bounds on the generalization error of a representation learning algorithm in terms of the "Minimum Description Length" (MDL) of the labels or the latent variables (representations). Rather than the mutual information between the encoder's input and the representation, which is often believed to reflect the algorithm's generalization capability in the related literature but in fact, falls short of doing so, our new bounds involve the "multi-letter" relative entropy between the distribution of the representations (or labels) of the training and test sets and a fixed prior. In particular, these new bounds reflect the structure of the encoder and are not vacuous for deterministic algorithms. Our compressibility approach, which is information-theoretic in nature, builds upon that of Blum-Langford for PAC-MDL bounds and introduces two essential ingredients: block-coding and lossy-compression. The latter allows our approach to subsume the so-called geometrical compressibility as a special case. To the best knowledge of the authors, the established generalization bounds are the first of their kind for Information Bottleneck (IB) type encoders and representation learning. Finally, we partly exploit the theoretical results by introducing a new data-dependent prior. Numerical simulations illustrate the advantages of well-chosen such priors over classical priors used in IB., Comment: Accepted and presented at NeurIPS 2023

Neural network compression has been an increasingly important subject, not only due to its practical relevance, but also due to its theoretical implications, as there is an explicit connection between compressibility and generalization error. Recent studies have shown that the choice of the hyperparameters of stochastic gradient descent (SGD) can have an effect on the compressibility of the learned parameter vector. These results, however, rely on unverifiable assumptions and the resulting theory does not provide a practical guideline due to its implicitness. In this study, we propose a simple modification for SGD, such that the outputs of the algorithm will be provably compressible without making any nontrivial assumptions. We consider a one-hidden-layer neural network trained with SGD, and show that if we inject additive heavy-tailed noise to the iterates at each iteration, for any compression rate, there exists a level of overparametrization such that the output of the algorithm will be compressible with high probability. To achieve this result, we make two main technical contributions: (i) we prove a 'propagation of chaos' result for a class of heavy-tailed stochastic differential equations, and (ii) we derive error estimates for their Euler discretization. Our experiments suggest that the proposed approach not only achieves increased compressibility with various models and datasets, but also leads to robust test performance under pruning, even in more realistic architectures that lie beyond our theoretical setting., Comment: 31 pages, 2 figures

We investigate the generalization error of statistical learning models in a Federated Learning (FL) setting. Specifically, we study the evolution of the generalization error with the number of communication rounds $R$ between $K$ clients and a parameter server (PS), i.e., the effect on the generalization error of how often the clients' local models are aggregated at PS. In our setup, the more the clients communicate with PS the less data they use for local training in each round, such that the amount of training data per client is identical for distinct values of $R$. We establish PAC-Bayes and rate-distortion theoretic bounds on the generalization error that account explicitly for the effect of the number of rounds $R$, in addition to the number of participating devices $K$ and individual datasets size $n$. The bounds, which apply to a large class of loss functions and learning algorithms, appear to be the first of their kind for the FL setting. Furthermore, we apply our bounds to FL-type Support Vector Machines (FSVM); and derive (more) explicit bounds in this case. In particular, we show that the generalization bound of FSVM increases with $R$, suggesting that more frequent communication with PS diminishes the generalization power. This implies that the population risk decreases less fast with $R$ than does the empirical risk. Moreover, our bound suggests that the generalization error of FSVM decreases faster than that of centralized learning by a factor of $\mathcal{O}(\sqrt{\log(K)/K})$. Finally, we provide experimental results obtained using neural networks (ResNet-56) which show evidence that not only may our observations for FSVM hold more generally but also that the population risk may even start to increase beyond some value of $R$., Comment: Accepted at ICML 2024

We study the generalization error of statistical learning models in a Federated Learning (FL) setting. Specifically, there are $K$ devices or clients, each holding an independent own dataset of size $n$. Individual models, learned locally via Stochastic Gradient Descent, are aggregated (averaged) by a central server into a global model and then sent back to the devices. We consider multiple (say $R \in \mathbb N^*$) rounds of model aggregation and study the effect of $R$ on the generalization error of the final aggregated model. We establish an upper bound on the generalization error that accounts explicitly for the effect of $R$ (in addition to the number of participating devices $K$ and dataset size $n$). It is observed that, for fixed $(n, K)$, the bound increases with $R$, suggesting that the generalization of such learning algorithms is negatively affected by more frequent communication with the parameter server. Combined with the fact that the empirical risk, however, generally decreases for larger values of $R$, this indicates that $R$ might be a parameter to optimize to reduce the population risk of FL algorithms. The results of this paper, which extend straightforwardly to the heterogeneous data setting, are also illustrated through numerical examples., Comment: Extended version of paper accepted at ISIT 2023

In this paper, we establish novel data-dependent upper bounds on the generalization error through the lens of a "variable-size compressibility" framework that we introduce newly here. In this framework, the generalization error of an algorithm is linked to a variable-size 'compression rate' of its input data. This is shown to yield bounds that depend on the empirical measure of the given input data at hand, rather than its unknown distribution. Our new generalization bounds that we establish are tail bounds, tail bounds on the expectation, and in-expectations bounds. Moreover, it is shown that our framework also allows to derive general bounds on any function of the input data and output hypothesis random variables. In particular, these general bounds are shown to subsume and possibly improve over several existing PAC-Bayes and data-dependent intrinsic dimension-based bounds that are recovered as special cases, thus unveiling a unifying character of our approach. For instance, a new data-dependent intrinsic dimension-based bound is established, which connects the generalization error to the optimization trajectories and reveals various interesting connections with the rate-distortion dimension of a process, the R\'enyi information dimension of a process, and the metric mean dimension., Comment: Accepted for publication in IEEE Transactions on Information Theory

In this paper, we use tools from rate-distortion theory to establish new upper bounds on the generalization error of statistical distributed learning algorithms. Specifically, there are $K$ clients whose individually chosen models are aggregated by a central server. The bounds depend on the compressibility of each client's algorithm while keeping other clients' algorithms un-compressed, and leverage the fact that small changes in each local model change the aggregated model by a factor of only $1/K$. Adopting a recently proposed approach by Sefidgaran et al., and extending it suitably to the distributed setting, this enables smaller rate-distortion terms which are shown to translate into tighter generalization bounds. The bounds are then applied to the distributed support vector machines (SVM), suggesting that the generalization error of the distributed setting decays faster than that of the centralized one with a factor of $\mathcal{O}(\log(K)/\sqrt{K})$. This finding is validated also experimentally. A similar conclusion is obtained for a multiple-round federated learning setup where each client uses stochastic gradient Langevin dynamics (SGLD)., Comment: Accepted at NeurIPS 2022

We study a class of $K$-encoder hypothesis testing against conditional independence problems. Under the criterion that stipulates minimization of the Type II error subject to a (constant) upper bound $\epsilon$ on the Type I error, we characterize the set of encoding rates and exponent for both discrete memoryless and memoryless vector Gaussian settings. For the DM setting, we provide a converse proof and show that it is achieved using the Quantize-Bin-Test scheme of Rahman and Wagner. For the memoryless vector Gaussian setting, we develop a tight outer bound by means of a technique that relies on the de Bruijn identity and the properties of Fisher information. In particular, the result shows that for memoryless vector Gaussian sources the rate-exponent region is exhausted using the Quantize-Bin-Test scheme with \textit{Gaussian} test channels; and there is \textit{no} loss in performance caused by restricting the sensors' encoders not to employ time sharing. Furthermore, we also study a variant of the problem in which the source, not necessarily Gaussian, has finite differential entropy and the sensors' observations noises under the null hypothesis are Gaussian. For this model, our main result is an upper bound on the exponent-rate function. The bound is shown to mirror a corresponding explicit lower bound, except that the lower bound involves the source power (variance) whereas the upper bound has the source entropy power. Part of the utility of the established bound is for investigating asymptotic exponent/rates and losses incurred by distributed detection as function of the number of sensors., Comment: Submitted for publication to the IEEE Transactions of Information Theory. arXiv admin note: substantial text overlap with arXiv:1904.03028, arXiv:1811.03933

It is widely perceived that leveraging the success of modern machine learning techniques to mobile devices and wireless networks has the potential of enabling important new services. This, however, poses significant challenges, essentially due to that both data and processing power are highly distributed in a wireless network. In this paper, we develop a learning algorithm and an architecture that make use of multiple data streams and processing units, not only during the training phase but also during the inference phase. In particular, the analysis reveals how inference propagates and fuses across a network. We study the design criterion of our proposed method and its bandwidth requirements. Also, we discuss implementation aspects using neural networks in typical wireless radio access; and provide experiments that illustrate benefits over state-of-the-art techniques., Comment: Extended version of Globecom'2021 paper; 11 double-column pages; 11 figures; 1 Table. arXiv admin note: substantial text overlap with arXiv:2104.14929

We consider the problem of learning parametric distributions from their quantized samples in a network. Specifically, $n$ agents or sensors observe independent samples of an unknown parametric distribution; and each of them uses $k$ bits to describe its observed sample to a central processor whose goal is to estimate the unknown distribution. First, we establish a generalization of the well-known van Trees inequality to general $L_p$-norms, with $p > 1$, in terms of Generalized Fisher information. Then, we develop minimax lower bounds on the estimation error for two losses: general $L_p$-norms and the related Wasserstein loss from optimal transport., Comment: Short version accepted for publication at the IEEE Information Theory Symposium (ISIT) 2021; this version contains the detailed proofs with some minor corrections

In this paper, we consider a problem in which distributively extracted features are used for performing inference in wireless networks. We elaborate on our proposed architecture, which we herein refer to as "in-network learning", provide a suitable loss function and discuss its optimization using neural networks. We compare its performance with both Federated- and Split learning; and show that this architecture offers both better accuracy and bandwidth savings., Comment: Accepted for publication at the 2021 IEEE 22nd International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), special session on Machine learning at the Edge. arXiv admin note: substantial text overlap with arXiv:2107.03433

In the context of statistical learning, the Information Bottleneck method seeks a right balance between accuracy and generalization capability through a suitable tradeoff between compression complexity, measured by minimum description length, and distortion evaluated under logarithmic loss measure. In this paper, we study a variation of the problem, called scalable information bottleneck, in which the encoder outputs multiple descriptions of the observation with increasingly richer features. The model, which is of successive-refinement type with degraded side information streams at the decoders, is motivated by some application scenarios that require varying levels of accuracy depending on the allowed (or targeted) level of complexity. We establish an analytic characterization of the optimal relevance-complexity region for vector Gaussian sources. Then, we derive a variational inference type algorithm for general sources with unknown distribution; and show means of parametrizing it using neural networks. Finally, we provide experimental results on the MNIST dataset which illustrate that the proposed method generalizes better to unseen data during the training phase., Comment: The short version was submitted to ISIT 2021

The Information Bottleneck method is a learning technique that seeks a right balance between accuracy and generalization capability through a suitable tradeoff between compression complexity, measured by minimum description length, and distortion evaluated under logarithmic loss measure. In this paper, we study a variation of the problem, called scalable information bottleneck, where the encoder outputs multiple descriptions of the observation with increasingly richer features. The problem at hand is motivated by some application scenarios that require varying levels of accuracy depending on the allowed level of generalization. First, we establish explicit (analytic) characterizations of the relevance-complexity region for memoryless Gaussian sources and memoryless binary sources. Then, we derive a Blahut-Arimoto type algorithm that allows us to compute (an approximation of) the region for general discrete sources. Finally, an application example in the pattern classification problem is provided along with numerical results., Comment: Submitted to IEEE Information Theory Workshop 2021

This paper studies the problem of discriminating two multivariate Gaussian distributions in a distributed manner. Specifically, it characterizes in a special case the optimal typeII error exponent as a function of the available communication rate. As a side-result, the paper also presents the optimal type-II error exponent of a slight generalization of the hypothesis testing against conditional independence problem where the marginal distributions under the two hypotheses can be different., Comment: To appear in 2020 IEEE International Symposium on Information Theory, ISIT'20

This tutorial paper focuses on the variants of the bottleneck problem taking an information theoretic perspective and discusses practical methods to solve it, as well as its connection to coding and learning aspects. The intimate connections of this setting to remote source-coding under logarithmic loss distortion measure, information combining, common reconstruction, the Wyner-Ahlswede-Korner problem, the efficiency of investment information, as well as, generalization, variational inference, representation learning, autoencoders, and others are highlighted. We discuss its extension to the distributed information bottleneck problem with emphasis on the Gaussian model and highlight the basic connections to the uplink Cloud Radio Access Networks (CRAN) with oblivious processing. For this model, the optimal trade-offs between relevance (i.e., information) and complexity (i.e., rates) in the discrete and vector Gaussian frameworks is determined. In the concluding outlook, some interesting problems are mentioned such as the characterization of the optimal inputs ("features") distributions under power limitations maximizing the "relevance" for the Gaussian information bottleneck, under "complexity" constraints., Comment: To be published in Entropy as part of the Special Issue Information Theory for Data Communications and Processing. 51 pages. arXiv admin note: text overlap with arXiv:1807.04193

A single-sensor two-detectors system is considered where the sensor communicates with both detectors and Detector 1 communicates with Detector 2, all over noise-free rate-limited links. The sensor and both detectors observe discrete memoryless source sequences whose joint probability mass function depends on a binary hypothesis. The goal at each detector is to guess the binary hypothesis in a way that, for increasing observation lengths, the probability of error under one of the hypotheses decays to zero with largest possible exponential decay, whereas the probability of error under the other hypothesis can decay to zero arbitrarily slow. For the setting with zero-rate communication on both links, we exactly characterize the set of possible exponents and the gain brought up by cooperation, in function of the number of bits that are sent over the two links. Notice that, for this setting, tradeoffs between the exponents achieved at the two detectors arise only in few particular cases. In all other cases, each detector achieves the same performance as if it were the only detector in the system. For the setting with positive communication rates from the sensor to the detectors, we characterize the set of all possible exponents in a special case of testing against independence. In this case the cooperation link allows Detector~2 to increase its Type-II error exponent by an amount that is equal to the exponent attained at Detector 1. We also provide a general inner bound on the set of achievable error exponents. For most cases it shows a tradeoff between the two exponents., Comment: submitted to IEEE Transactions on Information Theory, 30 pages, 5 figures

In this paper, we develop an unsupervised generative clustering framework that combines the Variational Information Bottleneck and the Gaussian Mixture Model. Specifically, in our approach, we use the Variational Information Bottleneck method and model the latent space as a mixture of Gaussians. We derive a bound on the cost function of our model that generalizes the Evidence Lower Bound (ELBO) and provide a variational inference type algorithm that allows computing it. In the algorithm, the coders' mappings are parametrized using neural networks, and the bound is approximated by Monte Carlo sampling and optimized with stochastic gradient descent. Numerical results on real datasets are provided to support the efficiency of our method., Comment: accepted for publication in Entropy, Special Issue on Information Theory for Data Communications and Processing

We study a class of distributed hypothesis testing against conditional independence problems. Under the criterion that stipulates minimization of the Type II error rate subject to a (constant) upper bound $\epsilon$ on the Type I error rate, we characterize the set of encoding rates and exponent for both discrete memoryless and memoryless vector Gaussian settings., Comment: Submitted for publication to the IEEE Information Theory Workshop, ITW 2019. arXiv admin note: substantial text overlap with arXiv:1811.03933

In this paper, we study the vector Gaussian Chief Executive Officer (CEO) problem under logarithmic loss distortion measure. Specifically, $K \geq 2$ agents observe independently corrupted Gaussian noisy versions of a remote vector Gaussian source, and communicate independently with a decoder or CEO over rate-constrained noise-free links. The CEO wants to reconstruct the remote source to within some prescribed distortion level where the incurred distortion is measured under the logarithmic loss penalty criterion. We find an explicit characterization of the rate-distortion region of this model. For the proof of this result, we obtain an outer bound on the region of the vector Gaussian CEO problem by means of a technique that relies on the de Bruijn identity and the properties of Fisher information. The approach is similar to Ekrem-Ulukus outer bounding technique for the vector Gaussian CEO problem under quadratic distortion measure, for which it was there found generally non-tight; but it is shown here to yield a complete characterization of the region for the case of logarithmic loss measure. Also, we show that Gaussian test channels with time-sharing exhaust the Berger-Tung inner bound, which is optimal. Furthermore, we also show that the established result under logarithmic loss provides an outer bound for a quadratic vector Gaussian CEO problem with determinant constraint, for which we characterize the optimal rate-distortion region., Comment: This paper was accepted at the IEEE Information Theory Workshop (ITW), 2018. 5 pages, 1 figure. arXiv admin note: substantial text overlap with arXiv:1811.03933

We study the vector Gaussian Chief Executive Officer (CEO) problem under logarithmic loss distortion measure. Specifically, $K \geq 2$ agents observe independently corrupted Gaussian noisy versions of a remote vector Gaussian source, and communicate independently with a decoder or CEO over rate-constrained noise-free links. The CEO also has its own Gaussian noisy observation of the source and wants to reconstruct the remote source to within some prescribed distortion level where the incurred distortion is measured under the logarithmic loss penalty criterion. We find an explicit characterization of the rate-distortion region of this model. The result can be seen as the counterpart to the vector Gaussian setting of that by Courtade-Weissman which provides the rate-distortion region of the model in the discrete memoryless setting. For the proof of this result, we obtain an outer bound by means of a technique that relies on the de Bruijn identity and the properties of Fisher information. The approach is similar to Ekrem-Ulukus outer bounding technique for the vector Gaussian CEO problem under quadratic distortion measure, for which it was there found generally non-tight; but it is shown here to yield a complete characterization of the region for the case of logarithmic loss measure. Also, we show that Gaussian test channels with time-sharing exhaust the Berger-Tung inner bound, which is optimal. Furthermore, application of our results allows us to find the complete solutions of two related problems: a quadratic vector Gaussian CEO problem with determinant constraint and the vector Gaussian distributed Information Bottleneck problem. Finally, we develop Blahut-Arimoto type algorithms that allow to compute numerically the regions provided in this paper, for both discrete and Gaussian models. We illustrate the efficiency of our algorithms through some numerical examples., Comment: accepted for publication in IEEE Transactions on Information Theory

A detection system with a single sensor and two detectors is considered, where each of the terminals observes a memoryless source sequence, the sensor sends a message to both detectors and the first detector sends a message to the second detector. Communication of these messages is assumed to be error-free but rate-limited. The joint probability mass function (pmf) of the source sequences observed at the three terminals depends on an $\mathsf{M}$-ary hypothesis $(\mathsf{M} \geq 2)$, and the goal of the communication is that each detector can guess the underlying hypothesis. Detector $k$, $k=1,2$, aims to maximize the error exponent \textit{under hypothesis} $i_k$, $i_k \in \{1,\ldots,\mathsf{M}\}$, while ensuring a small probability of error under all other hypotheses. We study this problem in the case in which the detectors aim to maximize their error exponents under the \textit{same} hypothesis (i.e., $i_1=i_2$) and in the case in which they aim to maximize their error exponents under \textit{distinct} hypotheses (i.e., $i_1 \neq i_2$). For the setting in which $i_1=i_2$, we present an achievable exponents region for the case of positive communication rates, and show that it is optimal for a specific case of testing against independence. We also characterize the optimal exponents region in the case of zero communication rates. For the setting in which $i_1 \neq i_2$, we characterize the optimal exponents region in the case of zero communication rates., Comment: 7 pages, 1 figure to be published in 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

The problem of distributed representation learning is one in which multiple sources of information $X_1,\ldots,X_K$ are processed separately so as to learn as much information as possible about some ground truth $Y$. We investigate this problem from information-theoretic grounds, through a generalization of Tishby's centralized Information Bottleneck (IB) method to the distributed setting. Specifically, $K$ encoders, $K \geq 2$, compress their observations $X_1,\ldots,X_K$ separately in a manner such that, collectively, the produced representations preserve as much information as possible about $Y$. We study both discrete memoryless (DM) and memoryless vector Gaussian data models. For the discrete model, we establish a single-letter characterization of the optimal tradeoff between complexity (or rate) and relevance (or information) for a class of memoryless sources (the observations $X_1,\ldots,X_K$ being conditionally independent given $Y$). For the vector Gaussian model, we provide an explicit characterization of the optimal complexity-relevance tradeoff. Furthermore, we develop a variational bound on the complexity-relevance tradeoff which generalizes the evidence lower bound (ELBO) to the distributed setting. We also provide two algorithms that allow to compute this bound: i) a Blahut-Arimoto type iterative algorithm which enables to compute optimal complexity-relevance encoding mappings by iterating over a set of self-consistent equations, and ii) a variational inference type algorithm in which the encoding mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms developed in this paper., Comment: 35 pages, 10 figures, submitted for possible publication

We study the transmission over a network in which users send information to a remote destination through relay nodes that are connected to the destination via finite-capacity error-free links, i.e., a cloud radio access network. The relays are constrained to operate without knowledge of the users' codebooks, i.e., they perform oblivious processing. The destination, or central processor, however, is informed about the users' codebooks. We establish a single-letter characterization of the capacity region of this model for a class of discrete memoryless channels in which the outputs at the relay nodes are independent given the users' inputs. We show that both relaying \`a-la Cover-El Gamal, i.e., compress-and-forward with joint decompression and decoding, and "noisy network coding", are optimal. The proof of the converse part establishes, and utilizes, connections with the Chief Executive Officer (CEO) source coding problem under logarithmic loss distortion measure. Extensions to general discrete memoryless channels are also investigated. In this case, we establish inner and outer bounds on the capacity region. For memoryless Gaussian channels within the studied class of channels, we characterize the capacity region when the users are constrained to time-share among Gaussian codebooks. Furthermore, we also discuss the suboptimality of separate decompression-decoding and the role of time-sharing., Comment: Accepted to IEEE Transactions on Information Theory

We study the problem of distributed information bottleneck, in which multiple encoders separately compress their observations in a manner such that, collectively, the compressed signals preserve as much information as possible about another signal. The model generalizes Tishby's centralized information bottleneck method to the setting of multiple distributed encoders. We establish single-letter characterizations of the information-rate region of this problem for both i) a class of discrete memoryless sources and ii) memoryless vector Gaussian sources. Furthermore, assuming a sum constraint on rate or complexity, for both models we develop Blahut-Arimoto type iterative algorithms that allow to compute optimal information-rate trade-offs, by iterating over a set of self-consistent equations., Comment: Submitted to the 2018 International Zurich Seminar on Information and Communication (IZS)

In this paper, we present iterative algorithms that numerically compute the rate-distortion regions of two problems: the two-encoder multiterminal source coding problem and the Chief Executive Officer (CEO) problem, both under logarithmic loss distortion measure.With the clear connection of these models with the distributed information bottleneck method, the proposed algorithms may find usefulness in a variety of applications, such as clustering, pattern recognition and learning. We illustrate the efficiency of our algorithms through some numerical examples., Comment: IEEE Information Theory Workshop (ITW) 2017. Accepted for publication

In this work, we investigate two source coding models, a \emph{Helper} problem and a \emph{Gray-Wyner} problem, under equivocation constraints. Specifically, in the Helper problem, an encoder communicates with a legitimate receiver through noise-free rate-limited public and private links; and an external passive eavesdropper intercepts every information that is sent on the public link. We study two classes of this model: i) when a pair of arbitrarily correlated discrete memoryless sources is to be encoded such that one component has to be recovered lossily at the legitimate receiver while the equivocation about both components at the eavesdropper must be maintained no smaller than some prescribed level; and ii) when the legitimate receiver reproduces both components, one of which, that is recovered losslessly, has to be concealed from the eavesdropper to some equivocation level. For both classes problems, we establish single-letter characterizations of optimal rate-distortion-equivocation tradeoffs in the discrete memoryless case. Next, we extend our results to the case of two legitimate receivers, i.e., Gray-Wyner network with equivocation constraints. Here, two legitimate receivers are connected to the encoder each through a dedicated error-free private link as well as a common error-free public link; and an external passive eavesdropper overhears on the public link. We study two classes of this model that are extensions of the aforementioned instances of Helper problems to the case of two receivers. For each of the two classes, we establish a single-letter characterization of the optimal rate-distortion-equivocation region. Throughout the paper, the analysis sheds light on the role of the private links, and we illustrate the results by computing them for some binary examples. Also, we make some meaningful connections, e.g., with problems of secret-sharing and encryption., Comment: Submitted to IEEE Transactions on Information Theory

In this work, we establish a full single-letter characterization of the rate-distortion region of an instance of the Gray-Wyner model with side information at the decoders. Specifically, in this model an encoder observes a pair of memoryless, arbitrarily correlated, sources $(S^n_1,S^n_2)$ and communicates with two receivers over an error-free rate-limited link of capacity $R_0$, as well as error-free rate-limited individual links of capacities $R_1$ to the first receiver and $R_2$ to the second receiver. Both receivers reproduce the source component $S^n_2$ losslessly; and Receiver $1$ also reproduces the source component $S^n_1$ lossily, to within some prescribed fidelity level $D_1$. Also, Receiver $1$ and Receiver $2$ are equipped respectively with memoryless side information sequences $Y^n_1$ and $Y^n_2$. Important in this setup, the side information sequences are arbitrarily correlated among them, and with the source pair $(S^n_1,S^n_2)$; and are not assumed to exhibit any particular ordering. Furthermore, by specializing the main result to two Heegard-Berger models with successive refinement and scalable coding, we shed light on the roles of the common and private descriptions that the encoder should produce and what they should carry optimally. We develop intuitions by analyzing the developed single-letter optimal rate-distortion regions of these models, and discuss some insightful binary examples., Comment: Submitted to IEEE Transactions on Information Theory

We study the problem of receive beamforming in uplink cascade multiple-input multiple-output (MIMO) systems as an instance of that of cascade multiterminal source coding for lossy function computation. Using this connection, we develop two coding schemes for the second and show that their application leads to beamforming schemes for the first. In the first coding scheme, each terminal in the cascade sends a description of the source that it observes; the decoder reconstructs all sources, lossily, and then computes an estimate of the desired function. This scheme improves upon standard routing in that every terminal only compresses the innovation of its source w.r.t. the descriptions that are sent by the previous terminals in the cascade. In the second scheme, the desired function is computed gradually in the cascade network, and each terminal sends a finer description of it. In the context of uplink cascade MIMO systems, the application of these two schemes leads to centralized receive-beamforming and distributed receive-beamforming, respectively. Numerical results illustrate the performance of the proposed methods and show that they outperform standard routing., Comment: Submitted to IEEE Transactions on Communications

This work investigates the downlink of a cloud radio access network (C-RAN) in which a central processor communicates with two mobile users through two base stations (BSs). The BSs act as relay nodes and cooperate with each other through error-free rate-limited links. We develop and analyze two coding schemes for this scenario. The first coding scheme is based on Liu-Kang scheme for C-RANs without BS cooperation; and extends it to scenarios allowing conferencing between the BSs. Among few other features, our new coding scheme enables arbitrary correlation among the auxiliary codewords that are recovered by the BSs. It also introduces common codewords to be described to both BSs. For the analysis of this coding scheme, we extend the multivariate covering lemma to non-Cartesian product sets, thereby correcting an erroneous application of this lemma in Liu-Kang's related work. We highlight key aspects of this scheme by studying three important instances of it. The second coding scheme extends the so-called compression scheme that was originally developed for memoryless Gaussian C-RANs without BS cooperation to general discrete memoryless C-RANs with BS cooperation. We show that this scheme subsumes the original compression scheme when applied to memoryless Gaussian C-RAN models. In the analysis of this scheme, we also highlight important connections with the so-called distributed decode--forward scheme, and refine the approximate capacity of a general $N$-BS $L$-user C-RAN model in the memoryless Gaussian case., Comment: 25 pages, 18 figures, submitted to IEEE Transactions on Information Theory

We study the transmission over a cloud radio access network in which multiple base stations (BS) are connected to a central processor (CP) via finite-capacity backhaul links. We propose two lattice-based coding schemes. In the first scheme, the base stations decode linear combinations of the transmitted messages, in the spirit of compute-and-forward (CoF), but differs from it essentially in that the decoded equations are remapped to linear combinations of the channel input symbols, sent compressed in a lossy manner to the central processor, and are not required to be linearly independent. Also, by opposition to the standard CoF, an appropriate multi-user decoder is utilized to recover the sent messages. The second coding scheme generalizes the first one by also allowing, at each relay node, a joint compression of the decoded equation and the received signal. Both schemes apply in general, but are more suited for situations in which there are more users than base stations. We show that both schemes can outperform standard CoF and successive Wyner-Ziv schemes in certain regimes, and illustrate the gains through some numerical examples., Comment: Submitted to IEEE Transactions on Communications

In this work, we investigate an instance of the Heegard-Berger problem with two sources and arbitrarily correlated side information sequences at two decoders, in which the reconstruction sets at the decoders are degraded. Specifically, two sources are to be encoded in a manner that one of the two is reproduced losslessly by both decoders, and the other is reproduced to within some prescribed distortion level at one of the two decoders. We establish a single-letter characterization of the rate-distortion function for this model. The investigation of this result in some special cases also sheds light on the utility of joint compression of the two sources. Furthermore, we also generalize our result to the setting in which the source component that is to be recovered by both users is reconstructed in a lossy fashion, under the requirement that all terminals (i.e., the encoder and both decoders) can share an exact copy of the compressed version of this source component, i.e., a common encoder-decoders reconstruction constraint. For this model as well, we establish a single-letter characterization of the associated rate-distortion function., Comment: Submitted to IEEE Trans. on Information Theory

We study the problem of secure transmission over a Gaussian multi-input single-output (MISO) two receiver channel with an external eavesdropper, under the assumption that the state of the channel which is available to each receiver is conveyed either perfectly ($P$) or with delay ($D$) to the transmitter. Denoting by $S_1$, $S_2$, and $S_3$ the channel state information at the transmitter (CSIT) of user 1, user 2, and eavesdropper, respectively, the overall CSIT can then alternate between eight possible states, i.e., $(S_1,S_2,S_3) \in \{P,D\}^3$. We denote by $\lambda_{S_1 S_2 S_3}$ the fraction of time during which the state $S_1S_2S_3$ occurs. Under these assumptions, we first consider the Gaussian MISO wiretap channel and characterize the secure degrees of freedom (SDoF). Next, we consider the general multi-receiver setup and characterize the SDoF region of fixed hybrid states $PPD$, $PDP$, and $DDP$. We then focus our attention on the symmetric case in which $\lambda_{PDD}=\lambda_{DPD}$. For this case, we establish bounds on SDoF region. The analysis reveals that alternating CSIT allows synergistic gains in terms of SDoF; and shows that, by opposition to encoding separately over different states, joint encoding across the states enables strictly better secure rates. Furthermore, we specialize our results for the two receivers channel with an external eavesdropper to the two-user broadcast channel. We show that, the synergistic gains in terms of SDoF by alternating CSIT is not restricted to multi-receiver wiretap channels; and, can also be harnessed under broadcast setting., Comment: To Appear in IEEE Transactions on Information Forensics and Security

In this paper, we consider a system in which multiple users communicate with a destination with the help of multiple half-duplex relays. Based on the compute-and-forward scheme, each relay, instead of decoding the users' messages, decodes an integer-valued linear combination that relates the transmitted messages. Then, it forwards the linear combination towards the destination. Given these linear combinations, the destination may or may not recover the transmitted messages since the linear combinations are not always full rank. Therefore, we propose an algorithm where we optimize the precoding factor at the users such that the probability that the equations are full rank is increased and that the transmission rate is maximized. We show, through some numerical examples, the effectiveness of our algorithm and the advantage of performing precoding allocation at the users. Also, we show that this scheme can outperform standard relaying techniques in certain regimes.

In this paper, we first study a two-user interference channel with generalized feedback. We establish an inner bound on its capacity region. The coding scheme that we employ for the inner bound is based on an appropriate combination of Han-Kobayash rate splitting and compress-and-forward at the senders. Each sender compresses the channel output that is observes using a compression scheme that is \`a-la Lim et al. noisy network coding and Avestimeher et al. quantize-map-and-forward. Next, we study an injective deterministic model in which the senders obtain output feedback only intermittently. Specializing the coding scheme of the model with generalized feedback to this scenario, we obtain useful insights onto effective ways of combining noisy network coding with interference alignment techniques. We also apply our results to linear deterministic interference channels with intermittent feedback., Comment: To appear in Proc. of the 2014 IEEE International Symposium on Information Theory, 6 pages, 2 figures

We consider a cooperative two-user multiaccess channel in which the transmission is controlled by a random state. Both encoders transmit a common message and, one of the encoders also transmits an individual message. We study the capacity region of this communication model for different degrees of availability of the states at the encoders, causally or strictly causally. In the case in which the states are revealed causally to both encoders but not to the decoder we find an explicit characterization of the capacity region in the discrete memoryless case. In the case in which the states are revealed only strictly causally to both encoders, we establish inner and outer bounds on the capacity region. The outer bound is non-trivial, and has a relatively simple form. It has the advantage of incorporating only one auxiliary random variable. We then introduce a class of cooperative multiaccess channels with states known strictly causally at both encoders for which the inner and outer bounds agree; and so we characterize the capacity region for this class. In this class of channels, the state can be obtained as a deterministic function of the channel inputs and output. We also study the model in which the states are revealed, strictly causally, in an asymmetric manner, to only one encoder. Throughout the paper, we discuss a number of examples; and compute the capacity region of some of these examples. The results shed more light on the utility of delayed channel state information for increasing the capacity region of state-dependent cooperative multiaccess channels; and tie with recent progress in this framework., Comment: 54 pages. To appear in IEEE Transactions on Information Theory. arXiv admin note: substantial text overlap with arXiv:1201.3278

This paper studies Gaussian Two-Way Relay Channel where two communication nodes exchange messages with each other via a relay. It is assumed that all nodes operate in half duplex mode without any direct link between the communication nodes. A compress-and-forward relaying strategy using nested lattice codes is first proposed. Then, the proposed scheme is improved by performing a layered coding : a common layer is decoded by both receivers and a refinement layer is recovered only by the receiver which has the best channel conditions. The achievable rates of the new scheme are characterized and are shown to be higher than those provided by the decode-and-forward strategy in some regions., Comment: 27 pages, 13 figures, Submitted to IEEE Transactions on Wireless Communications (October 2013)

We investigate the problem of secure transmission over a two-user multi-input multi-output (MIMO) X-channel in which channel state information is provided with one-unit delay to both transmitters (CSIT), and each receiver feeds back its channel output to a different transmitter. We refer to this model as MIMO X-channel with asymmetric output feedback and delayed CSIT. The transmitters are equipped with M-antennas each, and the receivers are equipped with N-antennas each. For this model, accounting for both messages at each receiver, we characterize the optimal sum secure degrees of freedom (SDoF) region. We show that, in presence of asymmetric output feedback and delayed CSIT, the sum SDoF region of the MIMO X-channel is same as the SDoF region of a two-user MIMO BC with 2M-antennas at the transmitter, N-antennas at each receiver and delayed CSIT. This result shows that, upon availability of asymmetric output feedback and delayed CSIT, there is no performance loss in terms of sum SDoF due to the distributed nature of the transmitters. Next, we show that this result also holds if only output feedback is conveyed to the transmitters, but in a symmetric manner, i.e., each receiver feeds back its output to both transmitters and no CSIT. We also study the case in which only asymmetric output feedback is provided to the transmitters, i.e., without CSIT, and derive a lower bound on the sum SDoF for this model. Furthermore, we specialize our results to the case in which there are no security constraints. In particular, similar to the setting with security constraints, we show that the optimal sum DoF region of the (M,M,N,N)--MIMO X-channel with asymmetric output feedback and delayed CSIT is same as the DoF region of a two-user MIMO BC with 2M-antennas at the transmitter, N-antennas at each receiver, and delayed CSIT. We illustrate our results with some numerical examples., Comment: To Appear in IEEE Transactions on Information Forensics and Security

The communication scenario under consideration in this paper corresponds to a multiuser channel with side information and consists of a broadcast channel with two legitimate receivers and an eavesdropper. Mainly, the results obtained are as follows. First, an achievable rate region is provided for the (general) case of discrete-input discrete-output channels, generalizing existing results. Second, the obtained theorem is used to derive achievable transmission rates for two practical cases of Gaussian channels. It is shown that known perturbations can enlarge the rate region of broadcast wiretap channels with side information and having side information at the decoder as well can increase the secrecy rate of channels with side information. Third, we establish for the first time an explicit connection between multiuser channels and observation structures in dynamic games. In this respect, we show how to exploit the proved achievability theorem (discrete case) to derive a communication-compatible upper bound on the minmax level of a player.

We consider a system in which two users communicate with a destination with the help of a half-duplex relay. Based on the compute-and-forward scheme, we develop and evaluate the performance of coding strategies that are of network coding spirit. In this framework, instead of decoding the users' information messages, the destination decodes two integer-valued linear combinations that relate the transmitted codewords. Two decoding schemes are considered. In the first one, the relay computes one of the linear combinations and then forwards it to the destination. The destination computes the other linear combination based on the direct transmissions. In the second one, accounting for the side information available at the destination through the direct links, the relay compresses what it gets using Wyner-Ziv compression and conveys it to the destination. The destination then computes the two linear combinations, locally. For both coding schemes, we discuss the design criteria, and derive the allowed symmetric-rate. Next, we address the power allocation and the selection of the integer-valued coefficients to maximize the offered symmetric-rate; an iterative coordinate descent method is proposed. The analysis shows that the first scheme can outperform standard relaying techniques in certain regimes, and the second scheme, while relying on feasible structured lattice codes, can at best achieve the same performance as regular compress-and-forward for the multiaccess relay network model that we study. The results are illustrated through some numerical examples.

We study a special case of Willems's two-user multi-access channel with partially cooperating encoders from a security perspective. This model differs from Willems's setup in that only one encoder, Encoder 1, is allowed to conference; Encoder 2 does not transmit any message, and there is an additional passive eavesdropper from whom the communication should be kept secret. For the discrete memoryless (DM) case, we establish inner and outer bounds on the capacity-equivocation region. The inner bound is based on a combination of Willems's coding scheme, noise injection and additional binning that provides randomization for security. For the memoryless Gaussian model, we establish lower and upper bounds on the secrecy capacity. We also show that, under certain conditions, these bounds agree in some extreme cases of cooperation between the encoders. We illustrate our results through some numerical examples., Comment: To appear in IEEE Transactions on Information Forensics and Security

We consider a two-user state-dependent multiaccess channel in which the states of the channel are known non-causally to one of the encoders and only strictly causally to the other encoder. Both encoders transmit a common message and, in addition, the encoder that knows the states non-causally transmits an individual message. We find explicit characterizations of the capacity region of this communication model in both discrete memoryless and memoryless Gaussian cases. The analysis also reveals optimal ways of exploiting the knowledge of the state only strictly causally at the encoder that sends only the common message when such a knowledge is beneficial. The encoders collaborate to convey to the decoder a lossy version of the state, in addition to transmitting the information messages through a generalized Gel'fand-Pinsker binning. Particularly important in this problem are the questions of 1) optimal ways of performing the state compression and 2) whether or not the compression indices should be decoded uniquely. We show that both compression \`a-la noisy network coding, i.e., with no binning, and compression using Wyner-Ziv binning are optimal. The scheme that uses Wyner-Ziv binning shares elements with Cover and El Gamal original compress-and-forward, but differs from it mainly in that backward decoding is employed instead of forward decoding and the compression indices are not decoded uniquely. Finally, by exploring the properties of our outer bound, we show that, although not required in general, the compression indices can in fact be decoded uniquely essentially without altering the capacity region, but at the expense of larger alphabets sizes for the auxiliary random variables., Comment: Submitted for publication to the 2012 IEEE International Symposium on Information Theory, 5 pages, 1 figure

We consider a two-user state-dependent multiaccess channel in which the states of the channel are known non-causally to one of the encoders and only strictly causally to the other encoder. Both encoders transmit a common message and, in addition, the encoder that knows the states non-causally transmits an individual message. We find explicit characterizations of the capacity region of this communication model in both discrete memoryless (DM) and memoryless Gaussian cases. In particular the capacity region analysis demonstrates the utility of the knowledge of the states only strictly causally at the encoder that sends only the common message in general. More specifically, in the DM setting we show that such a knowledge is beneficial and increases the capacity region in general. In the Gaussian setting, we show that such a knowledge does not help, and the capacity is same as if the states were completely unknown at the encoder that sends only the common message. The analysis also reveals optimal ways of exploiting the knowledge of the state only strictly causally at the encoder that sends only the common message when such a knowledge is beneficial. The encoders collaborate to convey to the decoder a lossy version of the state, in addition to transmitting the information messages through a generalized Gel'fand-Pinsker binning. Particularly important in this problem are the questions of 1) optimal ways of performing the state compression and 2) whether or not the compression indices should be decoded uniquely. We show that both compression \`a-la noisy network coding, i.e., with no binning and non-unique decoding, and compression using Wyner-Ziv binning with backward decoding and non-unique or unique decoding are optimal., Comment: Submitted to the IEEE Transactions on Information Theory, 38 pages, 2 figures

We consider a two-user state-dependent multiaccess channel in which the states of the channel are known non-causally to one of the encoders and only strictly causally to the other encoder. Both encoders transmit a common message and, in addition, the encoder that knows the states non-causally transmits an individual message. We study the capacity region of this communication model. In the discrete memoryless case, we establish inner and outer bounds on the capacity region. Although the encoder that sends both messages knows the states fully, we show that the strictly causal knowledge of these states at the other encoder can be beneficial for this encoder, and in general enlarges the capacity region. Furthermore, we find an explicit characterization of the capacity in the case in which the two encoders transmit only the common message. In the Gaussian case, we characterize the capacity region for the model with individual message as well. Our converse proof in this case shows that, for this model, strictly causal knowledge of the state at one of the encoders does not increase capacity if the other is informed non-causally, a result which sheds more light on the utility of conveying a compressed version of the state to the decoder in recent results by Lapidoth and Steinberg on a multiacess model with only strictly causal state at both encoders and independent messages., Comment: 5 pages, to appear in the 2011 IEEE International Symposium on Information Theory

We consider a three-terminal state-dependent relay channel with the channel state available non-causally at only the source. Such a model may be of interest for node cooperation in the framework of cognition, i.e., collaborative signal transmission involving cognitive and non-cognitive radios. We study the capacity of this communication model. One principal problem is caused by the relay's not knowing the channel state. For the discrete memoryless (DM) model, we establish two lower bounds and an upper bound on channel capacity. The first lower bound is obtained by a coding scheme in which the source describes the state of the channel to the relay and destination, which then exploit the gained description for a better communication of the source's information message. The coding scheme for the second lower bound remedies the relay's not knowing the states of the channel by first computing, at the source, the appropriate input that the relay would send had the relay known the states of the channel, and then transmitting this appropriate input to the relay. The relay simply guesses the sent input and sends it in the next block. The upper bound is non trivial and it accounts for not knowing the state at the relay and destination. For the general Gaussian model, we derive lower bounds on the channel capacity by exploiting ideas in the spirit of those we use for the DM model; and we show that these bounds are optimal for small and large noise at the relay irrespective to the strength of the interference. Furthermore, we also consider a special case model in which the source input has two components one of which is independent of the state. We establish a better upper bound for both DM and Gaussian cases and we also characterize the capacity in a number of special cases., Comment: Submitted to the IEEE Transactions on Information Theory, 54 pages, 6 figures

We investigate the problem of secure communication over parallel relay channel in the presence of a passive eavesdropper. We consider a four terminal relay-eavesdropper channel which consists of multiple relay-eavesdropper channels as subchannels. For the discrete memoryless model, we establish outer and inner bounds on the rate-equivocation region. The inner bound allows mode selection at the relay. For each subchannel, secure transmission is obtained through one of two coding schemes at the relay: decoding-and-forwarding the source message or confusing the eavesdropper through noise injection. For the Gaussian memoryless channel, we establish lower and upper bounds on the perfect secrecy rate. Furthermore, we study a special case in which the relay does not hear the source and show that under certain conditions the lower and upper bounds coincide. The results established for the parallel Gaussian relay-eavesdropper channel are then applied to study the fading relay-eavesdropper channel. Analytical results are illustrated through some numerical examples., Comment: To Appear in IEEE Transactions on Information Forensics and Security

We study a four terminal parallel relay-eavesdropper channel which consists of multiple independent relay-eavesdropper channels as subchannels. For the discrete memoryless case, we establish inner and outer bounds on the rate-equivocation region. For each subchannel, secure transmission is obtained through one of the two coding schemes at the relay: decoding-and-forwarding the source message or confusing the eavesdropper through noise injection. The inner bound allows relay mode selection. For the Gaussian model we establish lower and upper bounds on the perfect secrecy rate. We show that the bounds meet in some special cases, including when the relay does not hear the source. We illustrate the analytical results through some numerical examples., Comment: 8 pages, Presented at the Forty-Eighth Annual Allerton Conference on Communication, Control, and Computing, September 29 - October 1, 2010, Monticello, IL, USA

We consider a three-terminal state-dependent relay channel with the channel state available non-causally at only the source. Such a model may be of interest for node cooperation in the framework of cognition, i.e., collaborative signal transmission involving cognitive and non-cognitive radios. We study the capacity of this communication model. One principal problem in this setup is caused by the relay's not knowing the channel state. In the discrete memoryless (DM) case, we establish lower bounds on channel capacity. For the Gaussian case, we derive lower and upper bounds on the channel capacity. The upper bound is strictly better than the cut-set upper bound. We show that one of the developed lower bounds comes close to the upper bound, asymptotically, for certain ranges of rates., Comment: 5 pages, submitted to 2010 IEEE International Symposium on Information Theory

We consider a three-terminal state-dependent relay channel with the channel state noncausally available at only the relay. Such a model may be useful for designing cooperative wireless networks with some terminals equipped with cognition capabilities, i.e., the relay in our setup. In the discrete memoryless (DM) case, we establish lower and upper bounds on channel capacity. The lower bound is obtained by a coding scheme at the relay that uses a combination of codeword splitting, Gel'fand-Pinsker binning, and decode-and-forward relaying. The upper bound improves upon that obtained by assuming that the channel state is available at the source, the relay, and the destination. For the Gaussian case, we also derive lower and upper bounds on the capacity. The lower bound is obtained by a coding scheme at the relay that uses a combination of codeword splitting, generalized dirty paper coding, and decode-and-forward relaying; the upper bound is also better than that obtained by assuming that the channel state is available at the source, the relay, and the destination. In the case of degraded Gaussian channels, the lower bound meets with the upper bound for some special cases, and, so, the capacity is obtained for these cases. Furthermore, in the Gaussian case, we also extend the results to the case in which the relay operates in a half-duplex mode., Comment: 62 pages. To appear in IEEE Transactions on Information Theory

We consider a two-user state-dependent multiaccess channel in which only one of the encoders is informed, non-causally, of the channel states. Two independent messages are transmitted: a common message transmitted by both the informed and uninformed encoders, and an individual message transmitted by only the uninformed encoder. We derive inner and outer bounds on the capacity region of this model in the discrete memoryless case as well as the Gaussian case. Further, we show that the bounds for the Gaussian case are tight in some special cases., Comment: 5 pages, Proc. of IEEE International Symposium on Information theory, ISIT 2009, Seoul, Korea