Search

Large language models (LLMs) have excelled in various applications, yet serving them at scale is challenging due to their substantial resource demands and high latency. Our real-world studies reveal that over 60% of user requests to LLMs have semantically similar counterparts, suggesting the potential for knowledge sharing among requests. However, naively caching and reusing past responses leads to large quality degradation. In this paper, we introduce EchoLM, an in-context caching system that leverages historical requests as examples to guide response generation, enabling selective offloading of requests to more efficient LLMs. However, enabling this real-time knowledge transfer leads to intricate tradeoffs between response quality, latency, and system throughput at scale. For a new request, EchoLM identifies similar, high-utility examples and efficiently prepends them to the input for better response. At scale, EchoLM adaptively routes requests to LLMs of varying capabilities, accounting for response quality and serving loads. EchoLM employs a cost-aware cache replay mechanism to improve example quality and coverage offline, maximizing cache utility and runtime efficiency. Evaluations on millions of open-source requests demonstrate that EchoLM has a throughput improvement of 1.4-5.9x while reducing latency by 28-71% without hurting response quality on average.

The capabilities of multi-antenna technology have recently been significantly enhanced by the proliferation of extra large array architectures. The high dimensionality of these systems implies that communications take place in the nearfield regime, which poses some questions as to their effective perfomrance even under simple line of sight configurations. In order to study these limitations, a uniform linear array (ULA) is considered here, the elements of which are three infinitesimal dipoles transmitting different signals in the three spatial dimensions. The receiver consists of a single element with three orthogonal infinitesimal dipoles and full channel state information is assumed to be available at both ends. A capacity analysis is presented when the number of elements of the ULA increases without bound while the interelement distance converges to zero, so that the total aperture length is kept asymptotically fixed. In particular, the total number of available spatial eigenmodes is shown to depend crucially on the receiver position in space, and closed form expressions are provided for the different achievability regions. From the analysis it can be concluded that the use of three orthogonal polarizations at the transmitter guarantees the almost universal availability of two spatial streams, whereas the use of only two polarizations results in a more extensive region where maximum multiplexing gain is available., Comment: Submitted to IEEE Open Journal on Signal Processing

Streamlining the deployment of Deep Neural Networks (DNNs) on heterogeneous edge platforms, coupling within the same micro-controller unit (MCU) instruction processors and hardware accelerators for tensor computations, is becoming one of the crucial challenges of the TinyML field. The best-performing DNN compilation toolchains are usually deeply customized for a single MCU family, and porting to a different heterogeneous MCU family implies labor-intensive re-development of almost the entire compiler. On the opposite side, retargetable toolchains, such as TVM, fail to exploit the capabilities of custom accelerators, resulting in the generation of general but unoptimized code. To overcome this duality, we introduce MATCH, a novel TVM-based DNN deployment framework designed for easy agile retargeting across different MCU processors and accelerators, thanks to a customizable model-based hardware abstraction. We show that a general and retargetable mapping framework enhanced with hardware cost models can compete with and even outperform custom toolchains on diverse targets while only needing the definition of an abstract hardware model and a SoC-specific API. We tested MATCH on two state-of-the-art heterogeneous MCUs, GAP9 and DIANA. On the four DNN models of the MLPerf Tiny suite MATCH reduces inference latency by up to 60.88 times on DIANA, compared to using the plain TVM, thanks to the exploitation of the on-board HW accelerator. Compared to HTVM, a fully customized toolchain for DIANA, we still reduce the latency by 16.94%. On GAP9, using the same benchmarks, we improve the latency by 2.15 times compared to the dedicated DORY compiler, thanks to our heterogeneous DNN mapping approach that synergically exploits the DNN accelerator and the eight-cores cluster available on board., Comment: 13 pages, 11 figures, 4 tables

Generative AI is revolutionizing content creation and has the potential to enable real-time, personalized educational experiences. We investigated the effectiveness of converting textbook chapters into AI-generated podcasts and explored the impact of personalizing these podcasts for individual learner profiles. We conducted a 3x3 user study with 180 college students in the United States, comparing traditional textbook reading with both generalized and personalized AI-generated podcasts across three textbook subjects. The personalized podcasts were tailored to students' majors, interests, and learning styles. Our findings show that students found the AI-generated podcast format to be more enjoyable than textbooks and that personalized podcasts led to significantly improved learning outcomes, although this was subject-specific. These results highlight that AI-generated podcasts can offer an engaging and effective modality transformation of textbook material, with personalization enhancing content relevance. We conclude with design recommendations for leveraging AI in education, informed by student feedback.

We prove that if an orientable 3-manifold $M$ admits a complete Riemannian metric whose scalar curvature is positive and has a subquadratic decay at infinity, then it decomposes as a (possibly infinite) connected sum of spherical manifolds and $\mathbb{S}^2 \times \mathbb{S}^1$ summands. This generalises a theorem of Gromov and Wang by using a different, more topological, approach. As a result, the manifold $M$ carries a complete Riemannian metric of uniformly positive scalar curvature, which partially answers a conjecture of Gromov. More generally, the topological decomposition holds without any scalar curvature assumption under a weaker condition on the filling discs of closed curves in the universal cover based on the notion of fill radius. Moreover, the decay rate of the scalar curvature is optimal in this decomposition theorem. Indeed, the manifold $\mathbb{R}^2 \times \mathbb{S}^1$ supports a complete metric of positive scalar curvature with exactly quadratic decay, but does not admit a decomposition as a connected sum., Comment: 20 pages, 7 figures

GPGPU execution analysis has always been tied to closed-source, proprietary benchmarking tools that provide high-level, non-exhaustive, and/or statistical information, preventing a thorough understanding of bottlenecks and optimization possibilities. Open-source hardware platforms offer opportunities to overcome such limits and co-optimize the full {hardware-mapping-algorithm} compute stack. Yet, so far, this has remained under-explored. In this work, we exploit micro-architecture parameter analysis to develop a hardware-aware, runtime mapping technique for OpenCL kernels on the open Vortex RISC-V GPGPU. Our method is based on trace observations and targets optimal hardware resource utilization to achieve superior performance and flexibility compared to hardware-agnostic mapping approaches. The technique was validated on different architectural GPU configurations across several OpenCL kernels. Overall, our approach significantly enhances the performance of the open-source Vortex GPGPU, contributing to unlocking its potential and usability.

Optimal deployment of deep neural networks (DNNs) on state-of-the-art Systems-on-Chips (SoCs) is crucial for tiny machine learning (TinyML) at the edge. The complexity of these SoCs makes deployment non-trivial, as they typically contain multiple heterogeneous compute cores with limited, programmer-managed memory to optimize latency and energy efficiency. We propose HTVM - a compiler that merges TVM with DORY to maximize the utilization of heterogeneous accelerators and minimize data movements. HTVM allows deploying the MLPerf(TM) Tiny suite on DIANA, an SoC with a RISC-V CPU, and digital and analog compute-in-memory AI accelerators, at 120x improved performance over plain TVM deployment., Comment: Presented at DAC2023. Open-source code is available at https://github.com/KULeuven-MICAS/htvm

Conversational systems are crucial for human-computer interaction, managing complex dialogues by identifying threads and prioritising responses. This is especially vital in multi-party conversations, where precise identification of threads and strategic response prioritisation ensure efficient dialogue management. To address these challenges an end-to-end model that identifies threads and prioritises their response generation based on the importance was developed, involving a systematic decomposition of the problem into discrete components - thread detection, prioritisation, and performance optimisation which was meticulously analysed and optimised. These refined components seamlessly integrate into a unified framework, in conversational systems. Llama2 7b is used due to its high level of generalisation but the system can be updated with any open source Large Language Model(LLM). The computational capabilities of the Llama2 model was augmented by using fine tuning methods and strategic prompting techniques to optimise the model's performance, reducing computational time and increasing the accuracy of the model. The model achieves up to 10x speed improvement, while generating more coherent results compared to existing models., Comment: 6 pages, 4 figures, submitted to 2024 IEEE International Conference on Signal Processing and Communications (SPCOM)

We present Prompt Cache, an approach for accelerating inference for large language models (LLM) by reusing attention states across different LLM prompts. Many input prompts have overlapping text segments, such as system messages, prompt templates, and documents provided for context. Our key insight is that by precomputing and storing the attention states of these frequently occurring text segments on the inference server, we can efficiently reuse them when these segments appear in user prompts. Prompt Cache employs a schema to explicitly define such reusable text segments, called prompt modules. The schema ensures positional accuracy during attention state reuse and provides users with an interface to access cached states in their prompt. Using a prototype implementation, we evaluate Prompt Cache across several LLMs. We show that Prompt Cache significantly reduce latency in time-to-first-token, especially for longer prompts such as document-based question answering and recommendations. The improvements range from 8x for GPU-based inference to 60x for CPU-based inference, all while maintaining output accuracy and without the need for model parameter modifications., Comment: To appear at MLSys 2024

The need to execute Deep Neural Networks (DNNs) at low latency and low power at the edge has spurred the development of new heterogeneous Systems-on-Chips (SoCs) encapsulating a diverse set of hardware accelerators. How to optimally map a DNN onto such multi-accelerator systems is an open problem. We propose ODiMO, a hardware-aware tool that performs a fine-grain mapping across different accelerators on-chip, splitting individual layers and executing them in parallel, to reduce inference energy consumption or latency, while taking into account each accelerator's quantization precision to maintain accuracy. Pareto-optimal networks in the accuracy vs. energy or latency space are pursued for three popular dataset/DNN pairs, and deployed on the DIANA heterogeneous ultra-low power edge AI SoC. We show that ODiMO reduces energy/latency by up to 33%/31% with limited accuracy drop (-0.53%/-0.32%) compared to manual heuristic mappings., Comment: Accepted at 2023 ACM/IEEE International Symposium on Low Power Electronics and Design (ISLPED)

In many software systems, heuristics are used to make decisions - such as cache eviction, task scheduling, and information presentation - that have a significant impact on overall system behavior. While machine learning may outperform these heuristics, replacing existing heuristics in a production system safely and reliably can be prohibitively costly. We present SmartChoices, a novel approach that reduces the cost to deploy production-ready ML solutions for contextual bandits problems. SmartChoices' interface cleanly separates problem formulation from implementation details: engineers describe their use case by defining datatypes for the context, arms, and feedback that are passed to SmartChoices APIs, while SmartChoices manages encoding & logging data and training, evaluating & deploying policies. Our implementation codifies best practices, is efficient enough for use in low-level applications, and provides valuable production features off the shelf via a shared library. Overall, SmartChoices enables non-experts to rapidly deploy production-ready ML solutions by eliminating many sources of technical debt common to ML systems. Engineers have independently used SmartChoices to improve a wide range of software including caches, batch processing workloads, and UI layouts, resulting in better latency, throughput, and click-through rates., Comment: Accepted as a workshop paper at the Machine Learning for Systems Workshop at NeurIPS 2023

In this work, we demonstrate the viability of using federated learning to successfully predict energy consumption as well as solar production for all households within a certain network using low-power and low-space consuming embedded devices. We also demonstrate our prediction performance improving over time without the need for sharing private consumer energy data. We simulate a system with four nodes using data for one year to show this.

In today's era of smart cyber-physical systems, Deep Neural Networks (DNNs) have become ubiquitous due to their state-of-the-art performance in complex real-world applications. The high computational complexity of these networks, which translates to increased energy consumption, is the foremost obstacle towards deploying large DNNs in resource-constrained systems. Fixed-Point (FP) implementations achieved through post-training quantization are commonly used to curtail the energy consumption of these networks. However, the uniform quantization intervals in FP restrict the bit-width of data structures to large values due to the need to represent most of the numbers with sufficient resolution and avoid high quantization errors. In this paper, we leverage the key insight that (in most of the scenarios) DNN weights and activations are mostly concentrated near zero and only a few of them have large magnitudes. We propose CoNLoCNN, a framework to enable energy-efficient low-precision deep convolutional neural network inference by exploiting: (1) non-uniform quantization of weights enabling simplification of complex multiplication operations; and (2) correlation between activation values enabling partial compensation of quantization errors at low cost without any run-time overheads. To significantly benefit from non-uniform quantization, we also propose a novel data representation format, Encoded Low-Precision Binary Signed Digit, to compress the bit-width of weights while ensuring direct use of the encoded weight for processing using a novel multiply-and-accumulate (MAC) unit design., Comment: 8 pages, 15 figures, 2 tables

Face verification systems aim to validate the claimed identity using feature vectors and distance metrics. However, no attempt has been made to bypass such a system using generated images that are constrained by the same feature vectors. In this work, we train StarGAN v2 to generate diverse images based on a human user, that have similar feature vectors yet qualitatively look different. We then demonstrate a proof of concept on a custom face verification system and verify our claims by demonstrating the same proof of concept in a black box setting on dating applications that utilize similar face verification systems.

This paper outlines a social robot platform for providing health information. In comparison with previous findings for accessing information online, the use of a social robot may affect which factors users consider important when evaluating the trustworthiness of health information provided., Comment: 2 pages

Solar X-ray Monitor (XSM) instrument of India's Chandrayaan-2 lunar mission carries out broadband spectroscopy of the Sun in soft X-rays. XSM, with its unique features such as low background, high time cadence, and high spectral resolution, provides the opportunity to characterize transient and quiescent X-ray emission from the Sun even during low activity periods. It records the X-ray spectrum at one-second cadence, and the data recorded on-board are downloaded at regular intervals along with that of other payloads. During ground pre-processing, the XSM data is segregated, and the level-0 data is made available for higher levels of processing at the Payload Operations Center (POC). XSM Data Analysis Software (XSMDAS) is developed to carry out the processing of the level-0 data to higher levels and to generate calibrated light curves and spectra for user-defined binning parameters such that it is suitable for further scientific analysis. A front-end for the XSMDAS named XSM Quick Look Display (XSMQLD) is also developed to facilitate a first look at the data without applying calibration. XSM Data Management-Monitoring System (XSMDMS) is designed to carry out automated data processing at the POC and to maintain an SQLite database with relevant information on the data sets and an internal web application for monitoring data quality and instrument health. All XSM raw and calibrated data products are in FITS format, organized into day-wise files, and the data archive follows Planetary Data System-4 (PDS4) standards. The XSM data will be made available after a lock-in period along with the XSM Data Analysis Software from ISRO Science Data Archive (ISDA) at Indian Space Science Data Center(ISSDC). Here we discuss the design and implementation of all components of the software for the XSM data processing and the contents of the XSM data archive., Comment: Submitted to Astronomy and Computing, 13 pages, 9 figures

We present SmartChoices, an approach to making machine learning (ML) a first class citizen in programming languages which we see as one way to lower the entrance cost to applying ML to problems in new domains. There is a growing divide in approaches to building systems: on the one hand, programming leverages human experts to define a system while on the other hand behavior is learned from data in machine learning. We propose to hybridize these two by providing a 3-call API which we expose through an object called SmartChoice. We describe the SmartChoices-interface, how it can be used in programming with minimal code changes, and demonstrate that it is an easy to use but still powerful tool by demonstrating improvements over not using ML at all on three algorithmic problems: binary search, QuickSort, and caches. In these three examples, we replace the commonly used heuristics with an ML model entirely encapsulated within a SmartChoice and thus requiring minimal code changes. As opposed to previous work applying ML to algorithmic problems, our proposed approach does not require to drop existing implementations but seamlessly integrates into the standard software development workflow and gives full control to the software developer over how ML methods are applied. Our implementation relies on standard Reinforcement Learning (RL) methods. To learn faster, we use the heuristic function, which they are replacing, as an initial function. We show how this initial function can be used to speed up and stabilize learning while providing a safety net that prevents performance to become substantially worse -- allowing for a safe deployment in critical applications in real life., Comment: published at the Reinforcement Learning for Real Life (RL4RealLife) Workshop in the 36th International Conference on Machine Learning (ICML), Long Beach, California, USA, 2019

Watermarking techniques have been proposed during the last 10 years as an approach to trace network flows for intrusion detection purposes. These techniques aim to impress a hidden signature on a traffic flow. A central property of network flow watermarking is invisibility, i.e., the ability to go unidentified by an unauthorized third party. Although widely sought after, the development of an invisible watermark is a challenging task that has not yet been accomplished. In this paper we take a step forward in addressing the invisibility problem with DROPWAT, an active network flow watermarking technique developed for tracing Internet flows directed to the staging server that is the final destination in a data exfiltration attack, even in the presence of several intermediate stepping stones or an anonymous network. DROPWAT is a timing-based technique that indirectly modifies interpacket delays by exploiting network reaction to packet loss. We empirically demonstrate that the watermark embedded by means of DROPWAT is invisible to a third party observing the watermarked traffic. We also validate DROPWAT and analyze its performance in a controlled experimental framework involving the execution of a series of experiments on the Internet, using Web proxy servers as stepping stones executed on several instances in Amazon Web Services, as well as the TOR anonymous network in the place of the stepping stones. Our results show that the detection algorithm is able to identify an embedded watermark achieving over 95% accuracy while being invisible.

Field trials of a 4 meter long, 180 kilogram, unmanned surface vehicle (USV) have been conducted to evaluate the performance of station-keeping heading and position controllers in an outdoor marine environment disturbed by wind and current. The USV has a twin hull configuration and a custom-designed propulsion system, which consists of two azimuthing thrusters, one for each hull. Nonlinear proportional derivative, backstepping and sliding mode feedback controllers were tested in winds of about 4-5 knots, with and without wind feedforward control. The controllers were tested when the longitudinal axis of the USV was aligned with the mean wind direction and when the longitudinal axis was perpendicular to the mean wind direction. It was found that the sliding mode controller performed best overall and that the addition of wind feedforward control did not significantly improve its effectiveness. However, wind feedforward control did substantially improve the performance of the proportional derivative and backstepping controllers when the mean wind direction was perpendicular to the longitudinal axis of the USV. An analysis of the length scales present in the power spectrum of the turbulent speed fluctuations in the wind suggests that a single anemometer is sufficient to characterize the speed and direction of the wind acting on the USV.

Language in social media is mostly driven by new words and spellings that are constantly entering the lexicon thereby polluting it and resulting in high deviation from the formal written version. The primary entities of such language are the out-of-vocabulary (OOV) words. In this paper, we study various sociolinguistic properties of the OOV words and propose a classification model to categorize them into at least six categories. We achieve 81.26% accuracy with high precision and recall. We observe that the content features are the most discriminative ones followed by lexical and context features., Comment: 4 pages, 1 figure, CSCW '16

Urban environments develop complex, non-obvious structures that are often hard to represent in the form of maps or guides. Finding the right place to go often requires intimate familiarity with the location in question and cannot easily be deduced by visitors. In this work, we exploit large-scale samples of usage information, in the form of mobile phone traces and geo-tagged Twitter messages in order to automatically explore and annotate city maps via kernel density estimation. Our experiments are based on one year's worth of mobile phone activity collected by Nokia's Mobile Data Challenge (MDC). We show that usage information can be a strong predictor of semantic place categories, allowing us to automatically annotate maps based on the behavior of the local user base., Comment: 13 pages, 1 figure, 1 table

The paper considers functional linear regression, where scalar responses $Y_1,\ldots,Y_n$ are modeled in dependence of i.i.d. random functions $X_1,\ldots,X_n$. We study a generalization of the classical functional linear regression model. It is assumed that there exists an unknown number of "points of impact," that is, discrete observation times where the corresponding functional values possess significant influences on the response variable. In addition to estimating a functional slope parameter, the problem then is to determine the number and locations of points of impact as well as corresponding regression coefficients. Identifiability of the generalized model is considered in detail. It is shown that points of impact are identifiable if the underlying process generating $X_1,\ldots,X_n$ possesses "specific local variation." Examples are well-known processes like the Brownian motion, fractional Brownian motion or the Ornstein-Uhlenbeck process. The paper then proposes an easily implementable method for estimating the number and locations of points of impact. It is shown that this number can be estimated consistently. Furthermore, rates of convergence for location estimates, regression coefficients and the slope parameter are derived. Finally, some simulation results as well as a real data application are presented., Comment: Published at http://dx.doi.org/10.1214/15-AOS1323 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Missing values in datasets are a well-known problem and there are quite a lot of R packages offering imputation functions. But while imputation in general is well covered within R, it is hard to find functions for imputation of univariate time series. The problem is, most standard imputation techniques can not be applied directly. Most algorithms rely on inter-attribute correlations, while univariate time series imputation needs to employ time dependencies. This paper provides an overview of univariate time series imputation in general and an in-detail insight into the respective implementations within R packages. Furthermore, we experimentally compare the R functions on different time series using four different ratios of missing data. Our results show that either an interpolation with seasonal kalman filter from the zoo package or a linear interpolation on seasonal loess decomposed data from the forecast package were the most effective methods for dealing with missing data in most of the scenarios assessed in this paper.

The "developmental hourglass" describes a pattern of increasing morphological divergence towards earlier and later embryonic development, separated by a period of significant conservation across distant species (the "phylotypic stage"). Recent studies have found evidence in support of the hourglass effect at the genomic level. For instance, the phylotypic stage expresses the oldest and most conserved transcriptomes. However, the regulatory mechanism that causes the hourglass pattern remains an open question. Here, we use an evolutionary model of regulatory gene interactions during development to identify the conditions under which the hourglass effect can emerge in a general setting. The model focuses on the hierarchical gene regulatory network that controls the developmental process, and on the evolution of a population under random perturbations in the structure of that network. The model predicts, under fairly general assumptions, the emergence of an hourglass pattern in the structure of a temporal representation of the underlying gene regulatory network. The evolutionary age of the corresponding genes also follows an hourglass pattern, with the oldest genes concentrated at the hourglass waist. The key behind the hourglass effect is that developmental regulators should have an increasingly specific function as development progresses. Analysis of developmental gene expression profiles from Drosophila melanogaster and Arabidopsis thaliana provide consistent results with our theoretical predictions.

The paper considers linear regression problems where the number of predictor variables is possibly larger than the sample size. The basic motivation of the study is to combine the points of view of model selection and functional regression by using a factor approach: it is assumed that the predictor vector can be decomposed into a sum of two uncorrelated random components reflecting common factors and specific variabilities of the explanatory variables. It is shown that the traditional assumption of a sparse vector of parameters is restrictive in this context. Common factors may possess a significant influence on the response variable which cannot be captured by the specific effects of a small number of individual variables. We therefore propose to include principal components as additional explanatory variables in an augmented regression model. We give finite sample inequalities for estimates of these components. It is then shown that model selection procedures can be used to estimate the parameters of the augmented model, and we derive theoretical properties of the estimators. Finite sample performance is illustrated by a simulation study., Comment: Published in at http://dx.doi.org/10.1214/11-AOS905 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

A high level of diversity has already been observed among the planets of our own Solar System. As such, one expects extrasolar planets to present a wide range of distinctive features, therefore the characterisation of Earth- and super Earth-like planets is becoming of key importance in scientific research. The SEARCH (Spectropolarimetric Exoplanet AtmospheRe CHaracerisation) mission proposal of this paper represents one possible approach to realising these objectives. The mission goals of SEARCH include the detailed characterisation of a wide variety of exoplanets, ranging from terrestrial planets to gas giants. More specifically, SEARCH will determine atmospheric properties such as cloud coverage, surface pressure and atmospheric composition, and may also be capable of identifying basic surface features. To resolve a planet with a semi major axis of down to 1.4AU and 30pc distant SEARCH will have a mirror system consisting of two segments, with elliptical rim, cut out of a parabolic mirror. This will yield an effective diameter of 9 meters along one axis. A phase mask coronagraph along with an integral spectrograph will be used to overcome the contrast ratio of star to planet light. Such a mission would provide invaluable data on the diversity present in extrasolar planetary systems and much more could be learned from the similarities and differences compared to our own Solar System. This would allow our theories of planetary formation, atmospheric accretion and evolution to be tested, and our understanding of regions such as the outer limit of the Habitable Zone to be further improved., Comment: 23 pages, accepted for publication in Experimental Astronomy