Search

In \cite{butman1976} the linear coding scheme is applied, $X_t =g_t\Big(\Theta - {\bf E}\Big\{\Theta\Big|Y^{t-1}, V_0=v_0\Big\}\Big)$, $t=2,\ldots,n$, $X_1=g_1\Theta$, with $\Theta: \Omega \to {\mathbb R}$, a Gaussian random variable, to derive a lower bound on the feedback rate, for additive Gaussian noise (AGN) channels, $Y_t=X_t+V_t, t=1, \ldots, n$, where $V_t$ is a Gaussian autoregressive (AR) noise, and $\kappa \in [0,\infty)$ is the total transmitter power. For the unit memory AR noise, with parameters $(c, K_W)$, where $c\in [-1,1]$ is the pole and $K_W$ is the variance of the Gaussian noise, the lower bound is $C^{L,B} =\frac{1}{2} \log \chi^2$, where $\chi =\lim_{n\longrightarrow \infty} \chi_n$ is the positive root of $\chi^2=1+\Big(1+ \frac{|c|}{\chi}\Big)^2 \frac{\kappa}{K_W}$, and the sequence $\chi_n \triangleq \Big|\frac{g_n}{g_{n-1}}\Big|, n=2, 3, \ldots,$ satisfies a certain recursion, and conjectured that $C^{L,B}$ is the feedback capacity. In this correspondence, it is observed that the nontrivial lower bound $C^{L,B}=\frac{1}{2} \log \chi^2$ such that $\chi >1$, necessarily implies the scaling coefficients of the feedback code, $g_n$, $n=1,2, \ldots$, grow unbounded, in the sense that, $\lim_{n\longrightarrow\infty}|g_n| =+\infty$. The unbounded behaviour of $g_n$ follows from the ratio limit theorem of a sequence of real numbers, and it is verified by simulations. It is then concluded that such linear codes are not practical, and fragile with respect to a mismatch between the statistics of the mathematical model of the channel and the real statistics of the channel. In particular, if the error is perturbed by $\epsilon_n>0$ no matter how small, then $X_n =g_t\Big(\Theta - {\bf E}\Big\{\Theta\Big|Y^{t-1}, V_0=v_0\Big\}\Big)+g_n \epsilon_n$, and $|g_n|\epsilon_n \longrightarrow \infty$, as $n \longrightarrow \infty$., Comment: 18 pages, 3 figures

The main objective of this paper is to derive a new sequential characterization of the Cover and Pombra \cite{cover-pombra1989} characterization of the $n-$finite block or transmission feedback information ($n$-FTFI) capacity, which clarifies several issues of confusion and incorrect interpretation of results in literature. The optimal channel input processes of the new equivalent sequential characterizations are expressed as functionals of a sufficient statistic and a Gaussian orthogonal innovations process. From the new representations follows that the Cover and Pombra characterization of the $n-$FTFI capacity is expressed as a functional of two generalized matrix difference Riccati equations (DRE) of filtering theory of Gaussian systems. This contradicts results which are redundant in the literature, and illustrates the fundamental complexity of the feedback capacity formula., Comment: 6 pages. arXiv admin note: substantial text overlap with arXiv:2010.06226

In this paper we derive closed-form formulas of feedback capacity and nonfeedback achievable rates, for Additive Gaussian Noise (AGN) channels driven by nonstationary autoregressive moving average (ARMA) noise (with unstable one poles and zeros), based on time-invariant feedback codes and channel input distributions. From the analysis and simulations follows the surprising observations, (i) the use of time-invariant channel input distributions gives rise to multiple regimes of capacity that depend on the parameters of the ARMA noise, which may or may not use feedback, (ii) the more unstable the pole (resp. zero) of the ARMA noise the higher (resp. lower) the feedback capacity, (iii) certain conditions, known as detectability and stabilizability are necessary and sufficient to ensure the feedback capacity formulas and nonfeedback achievable rates {\it are independent of the initial state of the ARMA noise}. Another surprizing observation is that Kim's \cite{kim2010} characterization of feedback capacity which is developed for stable ARMA noise, if applied to the unstable ARMA noise, gives a lower value of feedback capacity compared to our feedback capacity formula., Comment: 6 pages, 3 figures

In the recent paper [1] it is shown, via an application example, that the Cover and Pombra [2] "characterization of the $n-$block or transmission" feedback capacity formula, of additive Gaussian noise (AGN) channels, is the subject of much confusion in the literature, with redundant incorrect results. The main objective of this paper is to clarify the main points of confusion and remove any further ambiguity. The first part of the paper applies time-domain methods, to derive for a first time, equivalent sequential characterizations of the Cover and Pombra characterization of feedback capacity of AGN channels driven by nonstationary and nonergodic Gaussian noise. The optimal channel input processes of the new equivalent sequential characterizations are expressed as functionals of a sufficient statistic and a Gaussian orthogonal innovations process. From the new representations follows that the Cover and Pombra $n-$block capacity formula is expressed as a functional of two generalized matrix difference Riccati equations (DRE) of filtering theory of Gaussian systems, contrary to results that appeared in the literature. In the second part of the paper the existence of the asymptotic limit of the $n-$block feedback capacity formula is shown to be equivalent to the convergence properties of solutions of the two generalized DREs. Further, necessary and or sufficient conditions are identified for existence of the asymptotic limits, for stable and unstable Gaussian noise, when the optimal input distributions are time-invariant, but not necessarily stationary. The paper contains an in depth analysis, with examples, of the specific technical issues, which are overlooked in past literature [3-7], that studied the AGN channel of [2], for stationary noises.

The capacity of additive Gaussian noise (AGN) channels, $Y_t=X_t+V_t, t=1, \ldots, n$, $\frac{1}{n} {\bf E}\big\{\sum_{t=1}^n |X_t|^2 \big\}\leq \kappa, \kappa \in [0,\infty)$, with time-invariant channel input feedback strategies, is characterized and conditions are identified for entropy rates, and limit of average power to exist, when the noise is described by {\it stable and unstable} autoregressive models, AR$(c)$, $V_t=cV_{t-1}+ W_t, V_0=v_0, t=1, \ldots, n$, where $c\in (-\infty,\infty)$, $W_t, t=1,\ldots, n$, is a zero mean, variance $K_W$, independent Gaussian sequence, independent of $V_0$. For stable AR$(c), c\in (-1,1)$ the conditions are necessary and sufficient for asymptotic stationarity of the processes $(X_t, Y_t), t=1, 2, \ldots$. New closed form capacity formulas and lower bounds are derived, for the AR$(c), c\in (-\infty,\infty)$ noise, which are fundamentally different from existing formulas in the literature, and illustrate multiple regimes of capacity, as a function of the parameters $(c,K_W,\kappa)$, as follows.\\ 1) feedback increases capacity for the regime, $c^2 \in (1, \infty),$ for $\kappa > \frac{K_W\big(1+\sqrt{4c^2-3}\big)}{2\big(c^2-1\big)^2}$, \\ 2) feedback does not increase capacity for the regime $c^2 \in (1, \infty)$, for $\kappa \leq \frac{K_W\big(1+\sqrt{4c^2-3}\big)}{2\big(c^2-1\big)^2}$, and \\ 3) feedback does not increase capacity for the regime $c \in [-1,1]$, for $ \kappa \in [0,\infty)$., Comment: submitted to IEEE Transactions on Information Theory (IT-19-0520)

Motivated by the demand of reliable and low latency communications, we employ tools from information theory, stochastic processes and queueing theory, in order to provide a comprehensive framework regarding the analysis of a Time Division Multiple Access (TDMA) network with bursty traffic, in the finite blocklength regime. Specifically, we re-examine the stability conditions of a noncooperative TDMA multiple access channel, evaluate the optimal throughput, and identify the optimal trade-off between data packet size and latency. The evaluation is performed both numerically and via the proposed approximations that result in closed form expressions. Then, we examine the stability conditions and the performance of the Multiple Access Relay Channel with TDMA scheduling, subject to finite blocklength constraints, by applying a cognitive cooperation protocol that assumes relaying is enabled when sources are idle. Finally, we propose the novel Batch-And-Forward (BAF) strategy, that can significantly enhance the performance of cooperative networks in the finite blocklength regime, as well as reduce the requirement in metadata. The BAF strategy is quite versatile, thus, it can be embedded in existing cooperative protocols, without imposing additional complexity on the overall scheme., Comment: Submitted to IEEE Transactions on Communications - TCOM-TPS-19-1051

We study finite alphabet channels with Unit Memory on the previous Channel Outputs called UMCO channels. We identify necessary and sufficient conditions, to test whether the capacity achieving channel input distributions with feedback are time-invariant, and whether feedback capacity is characterized by single letter, expressions, similar to that of memoryless channels. The method is based on showing that a certain dynamic programming equation, which in general, is a nested optimization problem over the sequence of channel input distributions, reduces to a non-nested optimization problem. Moreover, for UMCO channels, we give a simple expression for the ML error exponent, and we identify sufficient conditions to test whether feedback does not increase capacity. We derive similar results, when transmission cost constraints are imposed. We apply the results to a special class of the UMCO channels, the Binary State Symmetric Channel (BSSC) with and without transmission cost constraints, to show that the optimization problem of feedback capacity is non-nested, the capacity achieving channel input distribution and the corresponding channel output transition probability distribution are time-invariant, and feedback capacity is characterized by a single letter formulae, precisely as Shannon's single letter characterization of capacity of memoryless channels. Then we derive closed form expressions for the capacity achieving channel input distribution and feedback capacity. We use the closed form expressions to evaluate an error exponent for ML decoding., Comment: submitted to IEEE Transactions on Information Theory, Paper no. IT-16-0909

We derive sequential necessary and sufficient conditions for any channel input conditional distribution ${\cal P}_{0,n}\triangleq\{P_{X_t|X^{t-1},Y^{t-1}}:~t=0,\ldots,n\}$ to maximize the finite-time horizon directed information defined by $$C^{FB}_{X^n \rightarrow Y^n} \triangleq \sup_{{\cal P}_{0,n}} I(X^n\rightarrow{Y^n}),~~~ I(X^n \rightarrow Y^n) =\sum_{t=0}^n{I}(X^t;Y_t|Y^{t-1})$$ for channel distributions $\{P_{Y_t|Y^{t-1},X_t}:~t=0,\ldots,n\}$ and $\{P_{Y_t|Y_{t-M}^{t-1},X_t}:~t=0,\ldots,n\}$, where $Y^t\triangleq\{Y_0,\ldots,Y_t\}$ and $X^t\triangleq\{X_0,\ldots,X_t\}$ are the channel input and output random processes, and $M$ is a finite nonnegative integer. \noi We apply the necessary and sufficient conditions to application examples of time-varying channels with memory and we derive recursive closed form expressions of the optimal distributions, which maximize the finite-time horizon directed information. Further, we derive the feedback capacity from the asymptotic properties of the optimal distributions by investigating the limit $$C_{X^\infty \rightarrow Y^\infty}^{FB} \triangleq \lim_{n \longrightarrow \infty} \frac{1}{n+1} C_{X^n \rightarrow Y^n}^{FB}$$ without any \'a priori assumptions, such as, stationarity, ergodicity or irreducibility of the channel distribution. The necessary and sufficient conditions can be easily extended to a variety of channels with memory, beyond the ones considered in this paper., Comment: 57 pages, 9 figures, part of the paper was accepted for publication in the proceedings of the IEEE International Symposium on Information Theory (ISIT), Barcelona, Spain 10-15 July, 2016 (Date of submission of the conference paper: 25/1/2016)

A methodology is developed to realized optimal channel input conditional distributions, which maximize the finite-time horizon directed information, for channels with memory and feedback, by information lossless randomized strategies. The methodology is applied to general Time-Varying Multiple Input Multiple Output (MIMO) Gaussian Linear Channel Models (G-LCMs) with memory, subject to average transmission cost constraints of quadratic form. The realizations of optimal distributions by randomized strategies are shown to exhibit a decomposion into a deterministic part and a random part. The decomposition reveals the dual role of randomized strategies, to control the channel output process and to transmit new information over the channels. Moreover, a separation principle is shown between the computation of the optimal deterministic part and the random part of the randomized strategies. The dual role of randomized strategies generalizes the Linear-Quadratic-Gaussian (LQG) stochastic optimal control theory to directed information pay-offs. The characterizations of feedback capacity are obtained from the per unit time limits of finite-time horizon directed information, without imposing \'a priori assumptions, such as, stability of channel models or ergodicity of channel input and output processes. For time-invariant MIMO G-LCMs with memory, it is shown that whether feedback increases capacity, is directly related to the channel parameters and the transmission cost function, through the solutions of Riccati matrix equations, and moreover for unstable channels, feedback capacity is non-zero, provided the power exceeds a critical level., Comment: Submitted to the IEEE Transactions on Information Theory, Mar. 2016, paper IT-16-0166

For any class of channel conditional distributions, with finite memory dependence on channel input RVs $A^n {\stackrel{\triangle}{=}} \{A_i: i=0, \ldots, n\}$ or channel output RVs $B^n {\stackrel{\triangle}{=}} \{B_i: i=0, \ldots, n\}$ or both, we characterize the sets of channel input distributions, which maximize directed information defined by $ I(A^n \rightarrow B^n) {\stackrel{\triangle}{=}} \sum_{i=0}^n I(A^i;B_i|B^{i-1}) $ and we derive the corresponding expressions, called "characterizations of Finite Transmission Feedback Information (FTFI) capacity". The main theorems state that optimal channel input distributions occur in subsets ${\cal P}_{[0,n]}^{CI}\subseteq {\cal P}_{[0,n]} {\stackrel{\triangle}{=}}\big\{ {\bf P}_{A_i|A^{i-1}, B^{i-1}}: i=0, \ldots, n\big\}$, which satisfy conditional independence on past information. We derive similar characterizations, when general transmission cost constraints are imposed. Moreover, we also show that the structural properties apply to general nonlinear and linear autoregressive channel models defined by discrete-time recursions on general alphabet spaces, and driven by arbitrary distributed noise processes. We derive these structural properties by invoking stochastic optimal control theory and variational equalities of directed information, to identify tight upper bounds on $I(A^n \rightarrow B^n)$, which are achievable over subsets of conditional distributions ${\cal P}_{[0,n]}^{CI} \subseteq {\cal P}_{[0,n]}$, which satisfy conditional independence and they are specified by the dependence of channel distributions and transmission cost functions on inputs and output symbols. We apply the characterizations to recursive Multiple Input Multiple Output Gaussian Linear Channel Models with limited memory and we show a separation principle between the computation of the elements of the optimal strategies., Comment: Submitted to the IEEE Transactions on Information Theory,Feb. 2016, paper IT-16-0120

The Finite Transmission Feedback Information (FTFI) capacity is characterized for any class of channel conditional distributions $\big\{{\bf P}_{B_i|B^{i-1}, A_i} :i=0, 1, \ldots, n\big\}$ and $\big\{ {\bf P}_{B_i|B_{i-M}^{i-1}, A_i} :i=0, 1, \ldots, n\big\}$, where $M$ is the memory of the channel, $B^n {\stackrel{\triangle}{=}} \{B_j: j=\ldots, 0,1, \ldots, n\}$ are the channel outputs and $A^n{\stackrel{\triangle}{=}} \{A_j: j=\ldots, 0,1, \ldots, n\}$ are the channel inputs. The characterizations of FTFI capacity, are obtained by first identifying the information structures of the optimal channel input conditional distributions ${\cal P}_{[0,n]} {\stackrel{\triangle}{=}} \big\{ {\bf P}_{A_i|A^{i-1}, B^{i-1}}: i=0, \ldots, n\big\}$, which maximize directed information. The main theorem states, for any channel with memory $M$, the optimal channel input conditional distributions occur in the subset satisfying conditional independence $\stackrel{\circ}{\cal P}_{[0,n]}{\stackrel{\triangle}{=}} \big\{ {\bf P}_{A_i|A^{i-1}, B^{i-1}}= {\bf P}_{A_i|B_{i-M}^{i-1}}: i=1, \ldots, n\big\}$, and the characterization of FTFI capacity is given by $C_{A^n \rightarrow B^n}^{FB, M} {\stackrel{\triangle}{=}} \sup_{ \stackrel{\circ}{\cal P}_{[0,n]} } \sum_{i=0}^n I(A_i; B_i|B_{i-M}^{i-1}) $. The methodology utilizes stochastic optimal control theory and a variational equality of directed information, to derive upper bounds on $I(A^n \rightarrow B^n)$, which are achievable over specific subsets of channel input conditional distributions ${\cal P}_{[0,n]}$, which are characterized by conditional independence. For any of the above classes of channel distributions and transmission cost functions, a direct analogy, in terms of conditional independence, of the characterizations of FTFI capacity and Shannon's capacity formulae of Memoryless Channels is identified., Comment: Submitted to IEEE Transactions on Information Theory, IT-15-0963

This paper investigates applications of nonanticipative Rate Distortion Function (RDF) in a) zero-delay Joint Source-Channel Coding (JSCC) design based on average and excess distortion probability, b) in bounding the Optimal Performance Theoretically Attainable (OPTA) by noncausal and causal codes, and computing the Rate Loss (RL) of zero-delay and causal codes with respect to noncausal codes. These applications are described using two running examples, the Binary Symmetric Markov Source with parameter p, (BSMS(p)) and the multidimensional partially observed Gaussian-Markov source. For the multidimensional Gaussian-Markov source with square error distortion, the solution of the nonanticipative RDF is derived, its operational meaning using JSCC design via a noisy coding theorem is shown by providing the optimal encoding-decoding scheme over a vector Gaussian channel, and the RL of causal and zero-delay codes with respect to noncausal codes is computed. For the BSMS(p) with Hamming distortion, the solution of the nonanticipative RDF is derived, the RL of causal codes with respect to noncausal codes is computed, and an uncoded noisy coding theorem based on excess distortion probability is shown. The information nonanticipative RDF is shown to be equivalent to the nonanticipatory epsilon-entropy, which corresponds to the classical RDF with an additional causality or nonanticipative condition imposed on the optimal reproduction conditional distribution., Comment: 34 pages, 12 figures, part of this paper was accepted for publication in IEEE International Symposium on Information Theory (ISIT), 2014 and in book Coordination Control of Distributed Systems of series Lecture Notes in Control and Information Sciences, 2015

We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the channel state is the modulo2 addition of the current channel input and the previous channel output. We derive closed form expressions for the capacity and corresponding channel input distribution, of this BSSC with and without feedback and transmission cost. We also show that the capacity of the BSSC is not increased by feedback, and it is achieved by a first order symmetric Markov process.

The objective of this paper is to further investigate various applications of information Nonanticipative Rate Distortion Function (NRDF) by discussing two working examples, the Binary Symmetric Markov Source with parameter $p$ (BSMS($p$)) with Hamming distance distortion, and the multidimensional partially observed Gaussian-Markov source. For the BSMS($p$), we give the solution to the NRDF, and we use it to compute the Rate Loss (RL) of causal codes with respect to noncausal codes. For the multidimensional Gaussian-Markov source, we give the solution to the NRDF, we show its operational meaning via joint source-channel matching over a vector of parallel Gaussian channels, and we compute the RL of causal and zero-delay codes with respect to noncausal codes., Comment: 5 pages, 3 figures, accepted for publication in IEEE International Symposium on Information Theory (ISIT) proceedings, 2014

In this paper we analyze the probabilistic matching of sources with memory to channels with memory so that symbol-by-symbol code with memory without anticipation are optimal, with respect to an average distortion and excess distortion probability. We show achievability of such a symbolby- symbol code with memory without anticipation, and we show matching for the Binary Symmetric Markov source (BSMS(p)) over a first-order symmetric channel with a cost constraint., Comment: 6 pages. arXiv admin note: substantial text overlap with arXiv:1304.6528

In this paper we invoke a nonanticipative information Rate Distortion Function (RDF) for sources with memory, and we analyze its importance in probabilistic matching of the source to the channel so that transmission of a symbol-by-symbol code with memory without anticipation is optimal, with respect to an average distortion and excess distortion probability. We show achievability of the symbol-by-symbol code with memory without anticipation, and we evaluate the probabilistic performance of the code for a Markov source., Comment: 5 pages, 3 figures

In this paper we introduce a definition for nonanticipative Rate Distortion Function (RDF) on abstract alphabets, and we invoke weak convergence of probability measures to show various of its properties, such as, existence of the optimal reproduction conditional distribution, compactness of the fidelity set, lower semicontinuity of the RDF functional, etc. Further, we derive the closed form expression of the optimal nonstationary reproduction distribution. This expression is computed recursively backward in time. Throughout the paper we point out an operational meaning of the nonanticipative RDF by recalling the coding theorem derive in \cite{tatikonda2000}, and we state relations to Gorbunov-Pinsker's nonanticipatory $\epsilon-$entropy \cite{gorbunov-pinsker}., Comment: 5 pages, submitted to ISIT 2013

The relation between rate distortion function (RDF) and Bayesian filtering theory is discussed. The relation is established by imposing a causal or realizability constraint on the reconstruction conditional distribution of the RDF, leading to the definition of a causal RDF. Existence of the optimal reconstruction distribution of the causal RDF is shown using the topology of weak convergence of probability measures. The optimal non-stationary causal reproduction conditional distribution of the causal RDF is derived in closed form; it is given by a set of recursive equations which are computed backward in time. The realization of causal RDF is described via the source-channel matching approach, while an example is briefly discussed to illustrate the concepts., Comment: 5 pages, 3 figures, 1 table, 1 graph, submitted to Information Theory Workshop 2012

A causal rate distortion function with a general fidelity criterion is formulated on abstract alphabets and a coding theorem is derived. Existence of the minimizing kernel is shown using the topology of weak convergence of probability measures. The optimal reconstruction kernel is derived, which is causal, and certain properties of the causal rate distortion function are presented., Comment: 5 pages, Submitted to Internation Symposium on Information Theory(ISIT) 2012

A Causal rate distortion function with a general fidelity criterion is formulated on abstract alphabets and the optimal reconstruction kernel is derived, which consists of a product of causal kernels. In the process, general abstract spaces are introduced to show existence of the minimizing kernel using weak*-convergence. Certain properties of the causal rate distortion function are presented.