Search

The bandwidth of a bandlimited signal is an important number that is relevant in many applications and concepts. For example, according to the Shannon sampling theorem, the bandwidth determines the minimum sampling rate that is required for a perfect reconstruction. In this paper we consider bandlimited signals with finite energy and bandlimited signals that are absolutely integrable and analyze whether the bandwidth of these signals can be determined algorithmically. We employ the concept of Turing computability, a theoretical model that describes the fundamental limits of what can be solved algorithmically on a digital hardware, and ask if, for a given computable bandlimited signal, it is possible to compute its bandwidth on a Turing machine. We show that this is not possible in general, because there exist computable bandlimited signals for which the bandwidth is a non-computable real number. Even the weaker question if the bandwidth of a given signal is smaller than a predefined value cannot be always answered algorithmically. Further, we prove that in the case where the bandwidth in not computable, it is even impossible to algorithmically determine a sequence of upper bounds that converges to the actual bandwidth of the signal. As a positive result, we show that the set of signals whose bandwidth is larger than some given value is semi-decidable. [ABSTRACT FROM AUTHOR]

The capacity of noisy quantum channels characterizes the highest rate at which information can be reliably transmitted and it is therefore of practical as well as fundamental importance. Capacities of classical channels are computed using alternating optimization schemes, called Blahut-Arimoto algorithms. In this work, we generalize classical Blahut-Arimoto algorithms to the quantum setting. In particular, we give efficient iterative schemes to compute the capacity of channels with classical input and quantum output, the quantum capacity of less noisy channels, the thermodynamic capacity of quantum channels, as well as the entanglement-assisted capacity of quantum channels. We give rigorous a priori and a posteriori bounds on the estimation error by employing quantum entropy inequalities and demonstrate fast convergence of our algorithms in numerical experiments. [ABSTRACT FROM AUTHOR]

Motivated by applications in social and biological network analysis we introduce a new form of agnostic clustering termed motif correlation clustering, which aims to minimize the cost of clustering errors associated with both edges and higher-order network structures. The problem may be succinctly described as follows: Given a complete graph $G$ , partition the vertices of the graph so that certain predetermined “important” subgraphs mostly lie within the same cluster, while “less relevant” subgraphs are allowed to lie across clusters. Our contributions are as follows: We first introduce several variants of motif correlation clustering and then show that these clustering problems are NP-hard. We then proceed to describe polynomial-time clustering algorithms that provide constant approximation guarantees for the problems at hand. Despite following the frequently used LP relaxation and rounding procedure, the algorithms involve a sophisticated and carefully designed neighborhood growing step that combines information about both edges and motifs. We conclude with several examples illustrating the performance of the developed algorithms on synthetic and real networks. [ABSTRACT FROM AUTHOR]

Index coding, a source coding problem over broadcast channels, has been a subject of both theoretical and practical interests since its introduction (by Birk and Kol, 1998). In short, the problem can be defined as follows: there is an input $P \triangleq (p_{1}, {\dots }, p_{n})$ , a set of $n$ clients who each desire a single entry $p_{i}$ of the input, and a broadcaster whose goal is to send as few messages as possible to all clients so that each one can recover its desired entry. Additionally, each client has some predetermined “side information,” corresponding to the certain entries of the input $P$ , which we represent as the “side information graph” $ \mathcal {G}$. The graph $ \mathcal {G}$ has a vertex $v_{i}$ for client $i$ and a directed edge $(v_{i}, v_{j})$ , indicating that client $i$ knows the $j$ th entry of the input. Given a fixed side information graph $ \mathcal {G}$ , we are interested in determining or approximating the “broadcast rate” of index coding on the graph, i.e., the least number of messages the broadcaster can transmit so that every client recovers its desired information. The complexity of determining this broadcast rate in the most general case is open, and the best-known approximations are barely better than the trivial $O(n)$ -approximation corresponding to sending each client their information directly without performing any coding. Using index coding schemes based on linear programs (LPs), we take a two-pronged approach to approximating the broadcast rate. First, extending earlier work on planar graphs, we focus on approximating the broadcast rate for special graph families, such as graphs with the small chromatic number and disk graphs. In certain cases, we are able to show that simple LP-based schemes give constant-factor approximations of the broadcast rate, which seem extremely difficult to obtain in the general case. Second, we provide several LP-based schemes for the general case, which are not constant-factor approximations, but which strictly improve on the best-known schemes. These can be viewed as both a strengthening of the constant-factor approximations proven for special graph families (as these schemes strictly improve on those which we prove are good approximations), as well as another tool that can be used either in practice or in future theoretical analyses. [ABSTRACT FROM AUTHOR]

The two-receiver broadcast packet erasure channel with feedback and memory is studied. Memory is modeled using a finite-state Markov chain representing a channel state. Two scenarios are considered: 1) when the transmitter has causal knowledge of the channel state (i.e., the state is visible) and 2) when the channel state is unknown at the transmitter, but observations of it are available at the transmitter through feedback (i.e., the state is hidden). In both scenarios, matching outer and inner bounds on the rates of communication are derived and the capacity region is determined. It is shown that similar results carry over to channels with memory and delayed feedback and memoryless compound channels with feedback. When the state is visible, the capacity region has a single-letter characterization and is in terms of a linear program. Two optimal coding schemes are devised that use feedback to keep track of the sent/received packets via a network of queues: a probabilistic scheme and a deterministic backpressure-like algorithm. The former bases its decisions solely on the past channel state information and the latter follows a max-weight queue-based policy. The performance of the algorithms is analyzed using the frameworks of rate stability in networks of queues, max-flow min-cut duality in networks, and finite-horizon Lyapunov drift analysis. When the state is hidden, the capacity region does not have a single-letter characterization and is, in this sense, uncomputable. Approximations of the capacity region are provided and two optimal coding algorithms are outlined. The first algorithm is a probabilistic coding scheme that bases its decisions on the past $L$ acknowledgments and its achievable rate region approaches the capacity region exponentially fast in $L$. The second algorithm is a backpressure-like algorithm that performs optimally in the long run. [ABSTRACT FROM AUTHOR]

In a network of $n$ nodes (modeled as a digraph), the goal of a perfectly secret message transmission (PSMT) protocol is to replicate sender’s message $m$ at the receiver’s end without revealing any information about $m$ to a computationally unbounded adversary that eavesdrops on any $t$ nodes. The adversary may be mobile too that is, it may eavesdrop on a different set of $t$ nodes in different rounds. We prove a necessary and sufficient condition on the synchronous network for the existence of $r$ -round PSMT protocols, for any given $r > 0$ ; further, we show that round-optimality is achieved without trading-off the communication complexity; specifically, our protocols have an overall communication complexity of $O(n)$ elements of a finite field to perfectly transmit one field element. Apart from optimality/scalability, two interesting implications of our results are: 1) adversarial mobility does not affect its tolerability: PSMT tolerating a static $t$ -adversary is possible if and only if PSMT tolerating mobile $t$ -adversary is possible; and 2) mobility does not affect the round optimality: the fastest PSMT protocol tolerating a static $t$ -adversary is not faster than the one tolerating a mobile $t$ -adversary. [ABSTRACT FROM AUTHOR]

A central question in information theory is to determine the maximum success probability that can be achieved in sending a fixed number of messages over a noisy channel. This was first studied in the pioneering work of Shannon, who established a simple expression characterizing this quantity in the limit of multiple independent uses of the channel. Here, we consider the general setting with only one use of the channel. We observe that the maximum success probability can be expressed as the maximum value of a submodular function. Using this connection, we establish the following results: 1) There is a simple greedy polynomial-time algorithm that computes a code achieving a (1-e^-1) -approximation of the maximum success probability. The factor (1-e^-1) can be improved arbitrarily close to 1 at the cost of slightly reducing the number of messages to be sent. Moreover, it is NP-hard to obtain an approximation ratio strictly better than (1-e^-1) for the problem of computing the maximum success probability. 2) Shared quantum entanglement between the sender and the receiver can increase the success probability by a factor of at most (1/(1-e^-1)) . In addition, this factor is tight if one allows an arbitrary non-signaling box between the sender and the receiver. 3) We give tight bounds on the one-shot performance of the meta-converse of Polyanskiy-Poor-Verdú. [ABSTRACT FROM AUTHOR]

Impulsive noise features in many modern communication systems—ranging from wireless to molecular—and is often modeled by the $\alpha $ -stable distribution. At present, the capacity of $\alpha $ -stable noise channels is not well understood, with the exception of Cauchy noise ( $\alpha = 1$ ) with a logarithmic constraint and Gaussian noise ( $\alpha = 2$ ) with a power constraint. In this paper, we consider additive symmetric $\alpha $ -stable noise channels with $\alpha \in (1,2]$ . We derive bounds for the capacity with an absolute moment constraint. We then compare our bounds with a numerical approximation via the Blahut–Arimoto algorithm, which provides insight into the effect of noise parameters on the bounds. In particular, we find that our lower bound is in good agreement with the numerical approximation for $\alpha $ near 2. [ABSTRACT FROM AUTHOR]

Finite precision approximations of discrete probability distributions are considered, applicable for distribution synthesis, e.g., probabilistic shaping. Two algorithms are presented that find the optimal $M$ -type approximation Q$ of a distribution P$ in terms of the variational distance and the informational divergence $D(Q\Vert P)$ . Bounds on the approximation errors are derived and shown to be asymptotically tight. Several examples illustrate that the variational distance optimal approximation can be quite different from the informational divergence optimal approximation. [ABSTRACT FROM PUBLISHER]

A joint transmitter–receiver energy harvesting model is considered, where both the transmitter and the receiver are powered by random (renewable) energy source. Given a fixed number of bits, the problem is to find the optimal transmission power profile at the transmitter and ON–OFF profile at the receiver to minimize the transmission time. With infinite capacity at both the transmitter and the receiver, the optimal offline and optimal online policies are derived. The optimal online policy is shown to be two-competitive in the arbitrary input case. With finite battery capacities at both ends, only random energy arrival sequence with given distribution is considered, for which an online policy with bounded expected competitive ratio is proposed. [ABSTRACT FROM PUBLISHER]

The usefulness of Gabor frames depends on the easy computability of a suitable dual window. This question is addressed under several aspects. Several versions of Schulz’s iterative algorithm for the approximation of the canonical dual window are analyzed for their numerical stability. For Gabor frames with totally positive windows or with exponential B-splines, a direct algorithm yields a family of exact dual windows with compact support. It is shown that these dual windows converge exponentially fast to the canonical dual window. [ABSTRACT FROM AUTHOR]

We propose a graphical model for representing networks of stochastic processes, the minimal generative model graph. It is based on reduced factorizations of the joint distribution over time. We show that under appropriate conditions, it is unique and consistent with another type of graphical model, the directed information graph, which is based on a generalization of Granger causality. We demonstrate how directed information quantifies Granger causality in a particular sequential prediction setting. We also develop efficient methods to estimate the topological structure from data that obviate estimating the joint statistics. One algorithm assumes upper bounds on the degrees and uses the minimal dimension statistics necessary. In the event that the upper bounds are not valid, the resulting graph is nonetheless an optimal approximation in terms of Kullback-Leibler (KL) divergence. Another algorithm uses near-minimal dimension statistics when no bounds are known, but the distribution satisfies a certain criterion. Analogous to how structure learning algorithms for undirected graphical models use mutual information estimates, these algorithms use directed information estimates. We characterize the sample-complexity of two plug-in directed information estimators and obtain confidence intervals. For the setting when point estimates are unreliable, we propose an algorithm that uses confidence intervals to identify the best approximation that is robust to estimation error. Last, we demonstrate the effectiveness of the proposed algorithms through the analysis of both synthetic data and real data from the Twitter network. In the latter case, we identify which news sources influence users in the network by merely analyzing tweet times. [ABSTRACT FROM AUTHOR]

This paper concerns the rates of power law growth of mutual information computed for a stationary measure or for a universal code. The rates are called Hilberg exponents, and four such quantities are defined for each measure and each code: two random exponents and two expected exponents. A particularly interesting case arises for the conditional algorithmic mutual information. In this case, the random Hilberg exponents are almost surely constant on ergodic sources and are bounded by the expected Hilberg exponents. This property is the second-order analog of the Shannon–McMillan–Breiman theorem, proved without invoking the ergodic theorem. It carries over to Hilberg exponents for the underlying probability measure via Shannon–Fano coding and Barron inequality. Moreover, the expected Hilberg exponents can be linked for different universal codes. Namely, if one code dominates another, the expected Hilberg exponents are greater for the former than for the latter. This paper is concluded by an evaluation of Hilberg exponents for certain sources, such as the mixture Bernoulli process and the Santa Fe processes. [ABSTRACT FROM PUBLISHER]

Inspired by ideas from the field of stochastic approximation, we propose a randomized algorithm to compute the capacity of a finite-state channel with a Markovian input. When the mutual information rate of the channel is concave with respect to the chosen parameterization, the proposed algorithm proves to be convergent to the capacity of the channel almost surely with the derived convergence rate. We also discuss the convergence behavior of the algorithm without the concavity assumption. [ABSTRACT FROM PUBLISHER]

The use of normal bases for representing elements in a binary field is attractive in some applications because it is easy to perform squaring operations in hardware. In such cases, the costs of implementing the multiplication operation become a primary concern. We present new algorithms for reducing the space complexity of Gaussian normal basis multipliers over binary fields GF(2^m) , where $m$ is odd. Compared with previous results, our approach incurs no additional costs in time complexity, and achieves improvements in space complexity over a wide range of finite fields and digit sizes. For the binary fields specified in the NIST FIPS 186-3 elliptic curve digital signature algorithm standards document, our algorithms reduce by 16% (respectively, 27%) the number of XOR gates needed for the implementation of a digit-level parallel-input parallel-output multiplier over a 163-bit (respectively, 409 bit) binary field. [ABSTRACT FROM AUTHOR]

We propose an iterative method for approximately computing the capacity of discrete memoryless channels, possibly under additional constraints on the input distribution. Based on duality of convex programming, we derive explicit upper and lower bounds for the capacity. The presented method requires O(M^2 N \sqrt \log N/\varepsilon ) to provide an estimate of the capacity to within $\varepsilon $ , where $N$ and $M$ denote the input and output alphabet size; a single iteration has a complexity $O(M N)$ . We also show how to approximately compute the capacity of memoryless channels having a bounded continuous input alphabet and a countable output alphabet under some mild assumptions on the decay rate of the channel’s tail. It is shown that discrete-time Poisson channels fall into this problem class. As an example, we compute sharp upper and lower bounds for the capacity of a discrete-time Poisson channel with a peak-power input constraint. [ABSTRACT FROM AUTHOR]

The nascent field of compressed sensing is founded on the fact that high-dimensional signals with simple structure can be recovered accurately from just a small number of randomized samples. Several specific kinds of structures have been explored in the literature, from sparsity and group sparsity to low-rankness. However, two fundamental questions have been left unanswered. What are the general abstract meanings of structure and simplicity? Do there exist universal algorithms for recovering such simple structured objects from fewer samples than their ambient dimension? In this paper, we address these two questions. Using algorithmic information theory tools such as the Kolmogorov complexity, we provide a unified definition of structure and simplicity. Leveraging this new definition, we develop and analyze an abstract algorithm for signal recovery motivated by Occam's Razor. Minimum complexity pursuit (MCP) requires approximately 2\kappa randomized samples to recover a signal of complexity \kappa and ambient dimension n. We also discuss the performance of the MCP in the presence of measurement noise and with approximately simple signals. [ABSTRACT FROM PUBLISHER]

The table of contents for the November 2013 issue of "IEEE Transactions on Information Theory" is presented.

For an ordered set, consider the model of distributions P for which an element that precedes another element is considered the more significant one in the sense that the implication a\leq b\Rightarrow P(a)\geq P(b) holds. It will be shown that if the ordered set is a finite co-tree, then the universal predictor for the model or, equivalently, the corresponding universal code, can be determined exactly via an algorithm of low complexity. Natural relations to problems on the computation of capacity and on the determination of information projections are established. More surprisingly, a direct connection to a problem of isotone regression also appears possible. [ABSTRACT FROM AUTHOR]

We consider the problem of finding high-dimensional approximate nearest neighbors. We introduce an old style probabilistic formulation instead of the more general locality sensitive hashing (LSH) formulation, and show that at least for sparse problems it recognizes much more efficient algorithms than the sparseness destroying LSH random projections. Efficient algorithms for homogeneous (all coordinates have the same probability distribution) problems are well known, the most famous reference being the work by Broder in 1998. The main theme of this paper is to find its “best” generalization to heterogeneous (different coordinate probabilities) problems. We find a practical algorithm which is asymptotically best in a wide natural class of algorithms. Readers interested in the more complicated very best (at least up to date) can look up our previous work in 2010. The analysis of our algorithms reveals that its complexity is governed by an information like function, which we call “small leaves bucketing forest information.” Any doubts whether it is “information” are dispelled by the aforementioned work. [ABSTRACT FROM PUBLISHER]

A system with an n-dimensional state vector and a controller and an actor is considered. The controller has complete information about the system state, and reveals a certain “minimum” amount of information to the actor. The actor takes certain actions based on the information the controller reveals, and the actions fetch certain utilities for each entity. Both the controller and actor seek to maximize their individual utilities by respectively selecting the information to reveal and the actions to adopt. This decision problem forms the basis of several technical and social systems, and can be formulated as a signaling game. It is shown that the Perfect Bayesian Equilibrium of this game has several counterintuitive properties and can be obtained as a saddle point of a different two person zero sum game. The computation time for saddle points using standard linear programs however turns out to be superexponential in n, which leads to computational intractability even for moderate n. Algorithms for computing saddle point policies using a computation time that is exponential in n are presented. Finally, simple linear time computable policies that approximate the saddle-point policies within guaranteeable approximation ratios are obtained. [ABSTRACT FROM PUBLISHER]