Search

Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while decoder-only models are resistant to a comparable degree of compression, decoders can be effectively sliced for a significant reduction in training time., Comment: This paper has been accepted for publication in NAACL 2024 (Industry Track)

Optimal transport is a framework for comparing measures whereby a cost is incurred for transporting one measure to another. Recent works have aimed to improve optimal transport plans through the introduction of various forms of structure. We introduce novel order constraints into the optimal transport formulation to allow for the incorporation of structure. We define an efficient method for obtaining explainable solutions to the new formulation that scales far better than standard approaches. The theoretical properties of the method are provided. We demonstrate experimentally that order constraints improve explainability using the e-SNLI (Stanford Natural Language Inference) dataset that includes human-annotated rationales as well as on several image color transfer examples., Comment: To appear in Proceedings of ICML 2022. Main Paper + Supplementary

Index coding studies multiterminal source-coding problems where a set of receivers are required to decode multiple (possibly different) messages from a common broadcast, and they each know some messages a priori. In this paper, at the receiver end, we consider a special setting where each receiver knows only one message a priori, and each message is known to only one receiver. At the broadcasting end, we consider a generalized setting where there could be multiple senders, and each sender knows a subset of the messages. The senders collaborate to transmit an index code. This work looks at minimizing the number of total coded bits the senders are required to transmit. When there is only one sender, we propose a pruning algorithm to find a lower bound on the optimal (i.e., the shortest) index codelength, and show that it is achievable by linear index codes. When there are two or more senders, we propose an appending technique to be used in conjunction with the pruning technique to give a lower bound on the optimal index codelength; we also derive an upper bound based on cyclic codes. While the two bounds do not match in general, for the special case where no two distinct senders know any message in common, the bounds match, giving the optimal index codelength. The results are expressed in terms of strongly connected components in directed graphs that represent the index-coding problems., Comment: Author final manuscript

We focus on the following instance of an index coding problem, where a set of receivers are required to decode multiple messages, whilst each knows one of the messages a priori. In particular, here we consider a generalized setting where they are multiple senders, each sender only knows a subset of messages, and all senders are required to collectively transmit the index code. For a single sender, Ong and Ho (ICC, 2012) have established the optimal index codelength, where the lower bound was obtained using a pruning algorithm. In this paper, the pruning algorithm is simplified, and used in conjunction with an appending technique to give a lower bound to the multi-sender case. An upper bound is derived based on network coding. While the two bounds do not match in general, for the special case where no two senders know any message bit in common, the bounds match, giving the optimal index codelength. The results are derived based on graph theory, and are expressed in terms of strongly connected components., Comment: This is an extended version of the same-titled paper accepted and to be presented at the IEEE International Symposium on Information Theory (ISIT), Istanbul, in July 2013

In another related work, U-statistics were used for non-asymptotic "average-case" analysis of random compressed sensing matrices. In this companion paper the same analytical tool is adopted differently - here we perform non-asymptotic "worst-case" analysis. Simple union bounds are a natural choice for "worst-case" analyses, however their tightness is an issue (and questioned in previous works). Here we focus on a theoretical U-statistical result, which potentially allows us to prove that these union bounds are tight. To our knowledge, this kind of (powerful) result is completely new in the context of CS. This general result applies to a wide variety of parameters, and is related to (Stein-Chen) Poisson approximation. In this paper, we consider i) restricted isometries, and ii) mutual coherence. For the bounded case, we show that k-th order restricted isometry constants have tight union bounds, when the measurements m = \mathcal{O}(k (1 + \log(n/k))). Here we require the restricted isometries to grow linearly in k, however we conjecture that this result can be improved to allow them to be fixed. Also, we show that mutual coherence (with the standard estimate \sqrt{(4\log n)/m}) have very tight union bounds. For coherence, the normalization complicates general discussion, and we consider only Gaussian and Bernoulli cases here., Comment: 12 pages. Submitted to IEEE Transactions on Signal Processing

Hoeffding's U-statistics model combinatorial-type matrix parameters (appearing in CS theory) in a natural way. This paper proposes using these statistics for analyzing random compressed sensing matrices, in the non-asymptotic regime (relevant to practice). The aim is to address certain pessimisms of "worst-case" restricted isometry analyses, as observed by both Blanchard & Dossal, et. al. We show how U-statistics can obtain "average-case" analyses, by relating to statistical restricted isometry property (StRIP) type recovery guarantees. However unlike standard StRIP, random signal models are not required; the analysis here holds in the almost sure (probabilistic) sense. For Gaussian/bounded entry matrices, we show that both l1-minimization and LASSO essentially require on the order of k \cdot [\log((n-k)/u) + \sqrt{2(k/n) \log(n/k)}] measurements to respectively recover at least 1-5u fraction, and 1-4u fraction, of the signals. Noisy conditions are considered. Empirical evidence suggests our analysis to compare well to Donoho & Tanner's recent large deviation bounds for l0/l1-equivalence, in the regime of block lengths 1000-3000 with high undersampling (50-150 measurements); similar system sizes are found in recent CS implementation. In this work, it is assumed throughout that matrix columns are independently sampled., Comment: 12 pages. 3 pages supplementary material. Submitted to IEEE Trans. Signal Processing

There is recent interest in compressing data sets for non-sequential settings, where lack of obvious orderings on their data space, require notions of data equivalences to be considered. For example, Varshney & Goyal (DCC, 2006) considered multiset equivalences, while Choi & Szpankowski (IEEE Trans. IT, 2012) considered isomorphic equivalences in graphs. Here equivalences are considered under a relatively broad framework - finite-dimensional, non-sequential data spaces with equivalences under group action, for which analogues of two well-studied embedding theorems are derived: the Whitney embedding theorem and the Johnson-Lindenstrauss lemma. Only the canonical data points need to be carefully embedded, each such point representing a set of data points equivalent under group action. Two-step embeddings are considered. First, a group invariant is applied to account for equivalences, and then secondly, a linear embedding takes it down to low-dimensions. Our results require hypotheses on discriminability of the applied invariant, such notions related to seperating invariants (Dufresne, 2008), and completeness in pattern recognition (Kakarala, 1992). In the latter theorem, the embedding complexity depends on the size of the canonical part, which may be significantly smaller than the whole data set, up to a factor equal to the size the group., Comment: 10 page extended abstract plus two sets of supplementary material. 1 figure. Preliminary report

Recent interest on permutation rank modulation shows the Kendall tau metric as an important distance metric. This note documents our first efforts to obtain upper bounds on optimal code sizes (for said metric) ala Delsarte's approach. For the Hamming metric, Delsarte's seminal work on powerful linear programming (LP) bounds have been extended to permutation codes, via association scheme theory. For the Kendall tau metric, the same extension needs the more general theory of coherent configurations, whereby the optimal code size problem can be formulated as an extremely huge semidefinite programming (SDP) problem. Inspired by recent algebraic techniques for solving SDP's, we consider the dual problem, and propose an LP to search over a subset of dual feasible solutions. We obtain modest improvement over a recent Singleton bound due to Barg and Mazumdar. We regard this work as a starting point, towards fully exploiting the power of Delsarte's method, which are known to give some of the best bounds in the context of binary codes., Comment: IEEE Intl. Symp. on Inform. Theory 2012. Final Version

The max-log-map (MLM) receiver is an approximated version of the well-known, Bahl-Cocke-Jelinek-Raviv (BCJR) algorithm. The MLM algorithm is attractive due to its implementation simplicity. In practice, sliding-window implementations are preferred; these practical implementations consider truncated signaling neighborhoods around each transmission time instant. In this paper, we consider sliding-window MLM receivers, where for any integer m, the MLM detector is truncated to a length- m signaling neighborhood. For any number n of chosen times instants, we derive exact expressions for both i) the joint distribution of the MLM symbol reliabilities, and ii) the joint probability of the erroneous MLM symbol detections. We show that the obtained expressions can be efficiently evaluated using Monte-Carlo techniques. Our proposed method is efficient; the most computationally expensive operation (in each Monte-Carlo trial) is an eigenvalue decomposition of a size 2mn by 2mn matrix. Practical truncation lengths can be easily handled. Finally, our proposed method is extremely general, and various scenarios such as correlated noise distributions, modulation coding, etc. may be easily accommodated., Comment: 17 pages, 11 figures