Search

The information bottleneck (IB) problem is a widely studied framework in machine learning for extracting compressed features that are informative for downstream tasks. However, current approaches to solving the IB problem rely on a heuristic tuning of hyperparameters, offering no guarantees that the learned features satisfy information-theoretic constraints. In this work, we introduce a statistically valid solution to this problem, referred to as IB via multiple hypothesis testing (IB-MHT), which ensures that the learned features meet the IB constraints with high probability, regardless of the size of the available dataset. The proposed methodology builds on Pareto testing and learn-then-test (LTT), and it wraps around existing IB solvers to provide statistical guarantees on the IB constraints. We demonstrate the performance of IB-MHT on classical and deterministic IB formulations, including experiments on distillation of language models. The results validate the effectiveness of IB-MHT in outperforming conventional methods in terms of statistical robustness and reliability.

Temporal Graph Neural Networks, a new and trending area of machine learning, suffers from a lack of formal analysis. In this paper, information theory is used as the primary tool to provide a framework for the analysis of temporal GNNs. For this reason, the concept of information bottleneck is used and adjusted to be suitable for a temporal analysis of such networks. To this end, a new definition for Mutual Information Rate is provided, and the potential use of this new metric in the analysis of temporal GNNs is studied., Comment: To be presented at Information Theory Workshop 2024

The increasing adoption of Artificial Intelligence (AI) in engineering problems calls for the development of calibration methods capable of offering robust statistical reliability guarantees. The calibration of black box AI models is carried out via the optimization of hyperparameters dictating architecture, optimization, and/or inference configuration. Prior work has introduced learn-then-test (LTT), a calibration procedure for hyperparameter optimization (HPO) that provides statistical guarantees on average performance measures. Recognizing the importance of controlling risk-aware objectives in engineering contexts, this work introduces a variant of LTT that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem.

Measuring the complexity of tree structures can be beneficial in areas that use tree data structures for storage, communication, and processing purposes. This complexity can then be used to compress tree data structures to their information-theoretic limit. Additionally, the lack of models for random generation of trees is very much felt in mathematical modeling of trees and graphs. In this paper, a number of existing tree generation models such as simply generated trees are discussed, and their information content is analysed by means of information theory and Shannon's entropy. Subsequently, a new model for generating trees based on practical appearances of trees is introduced, and an upper bound for its entropy is calculated. This model is based on selecting a random tree from possible spanning trees of graphs, which is what happens often in practice. Moving on to tree compression, we find approaches to universal tree compression of the discussed models. These approaches first transform a tree into a sequence of symbols, and then apply a dictionary-based compression method. Conditions for the universality of these method are then studied and analysed.

Routing is one of the critical and ongoing challenges in Wireless Sensor Networks. The main challenge has always been to have a routing protocol that reduces the communication overhead, hence saving the energy of the sensors in the network. Hierarchical routing protocols are known to be the most energy-efficient routing protocols for Wireless Sensor Networks. In this paper, a more generalized hierarchical routing protocol is introduced for Wireless Sensor Network, which is based on tree data structures. The clustering in the proposed protocol has the format of a general tree, which is constructed in an adaptive manner based on the distance of the sensors. Results show that the proposed tree-based protocol introduces a significant benefit in energy consumption and lifetime of the network over the existing hierarchical approaches.

Routing tables in ad hoc and wireless routing protocols can be represented using rooted trees. The constant need for communication and storage of these trees in routing protocols demands an efficient rooted tree coding algorithm. This efficiency is defined in terms of the average code length, and the optimality of the algorithm is measured by comparing the average code length with the entropy of the source. In this work, TreeExplorer is introduced as an easy-to-implement and nearly optimal algorithm for coding rooted tree structures. This method utilizes the number of leaves of the tree as an indicator for choosing the best method of coding. We show how TreeExplorer can improve existing routing protocols for ad hoc and wireless systems, which normally entails a significant communication overhead.

A novel Markovian network evolution model is introduced and analysed by means of information theory. It will be proved that the model, called Network Evolution Chain, is a stationary and ergodic stochastic process. Therefore, the Asymptotic Equipartition Property can be applied to it. The model's entropy rate and typical sequences are also explored. Extracting particular information from the network and methods to simulate network evolution in the continuous time domain are discussed. Additionally, the Erdos-Renyi Network Evolution Chain is introduced as a subset of our model with the additional property of its stationary distribution matching the Erdos-Renyi random graph model. The stationary distributions of nodes and graphs are calculated for this subset alongside its entropy rate. The simulation results at the end of the paper back up the proved theorems and calculated values.

A novel compression scheme for compressing the outcome of $n$ independent Bernoulli trials is introduced and analysed. The resulting algorithm, BernoulliZip, is a fast and near-optimal method to produce prefix codes for a Bernoulli process. BernoulliZip's main principle is to first represent the number of 1s in the sequence and then specify the sequence. The application of BernoulliZip on compressing Erdos-Renyi graphs is explored., Comment: Submitted to ITW 2021