Search

Large language models (LLMs) have excelled in various applications, yet serving them at scale is challenging due to their substantial resource demands and high latency. Our real-world studies reveal that over 60% of user requests to LLMs have semantically similar counterparts, suggesting the potential for knowledge sharing among requests. However, naively caching and reusing past responses leads to large quality degradation. In this paper, we introduce EchoLM, an in-context caching system that leverages historical requests as examples to guide response generation, enabling selective offloading of requests to more efficient LLMs. However, enabling this real-time knowledge transfer leads to intricate tradeoffs between response quality, latency, and system throughput at scale. For a new request, EchoLM identifies similar, high-utility examples and efficiently prepends them to the input for better response. At scale, EchoLM adaptively routes requests to LLMs of varying capabilities, accounting for response quality and serving loads. EchoLM employs a cost-aware cache replay mechanism to improve example quality and coverage offline, maximizing cache utility and runtime efficiency. Evaluations on millions of open-source requests demonstrate that EchoLM has a throughput improvement of 1.4-5.9x while reducing latency by 28-71% without hurting response quality on average.

The capabilities of multi-antenna technology have recently been significantly enhanced by the proliferation of extra large array architectures. The high dimensionality of these systems implies that communications take place in the nearfield regime, which poses some questions as to their effective perfomrance even under simple line of sight configurations. In order to study these limitations, a uniform linear array (ULA) is considered here, the elements of which are three infinitesimal dipoles transmitting different signals in the three spatial dimensions. The receiver consists of a single element with three orthogonal infinitesimal dipoles and full channel state information is assumed to be available at both ends. A capacity analysis is presented when the number of elements of the ULA increases without bound while the interelement distance converges to zero, so that the total aperture length is kept asymptotically fixed. In particular, the total number of available spatial eigenmodes is shown to depend crucially on the receiver position in space, and closed form expressions are provided for the different achievability regions. From the analysis it can be concluded that the use of three orthogonal polarizations at the transmitter guarantees the almost universal availability of two spatial streams, whereas the use of only two polarizations results in a more extensive region where maximum multiplexing gain is available., Comment: Submitted to IEEE Open Journal on Signal Processing

Streamlining the deployment of Deep Neural Networks (DNNs) on heterogeneous edge platforms, coupling within the same micro-controller unit (MCU) instruction processors and hardware accelerators for tensor computations, is becoming one of the crucial challenges of the TinyML field. The best-performing DNN compilation toolchains are usually deeply customized for a single MCU family, and porting to a different heterogeneous MCU family implies labor-intensive re-development of almost the entire compiler. On the opposite side, retargetable toolchains, such as TVM, fail to exploit the capabilities of custom accelerators, resulting in the generation of general but unoptimized code. To overcome this duality, we introduce MATCH, a novel TVM-based DNN deployment framework designed for easy agile retargeting across different MCU processors and accelerators, thanks to a customizable model-based hardware abstraction. We show that a general and retargetable mapping framework enhanced with hardware cost models can compete with and even outperform custom toolchains on diverse targets while only needing the definition of an abstract hardware model and a SoC-specific API. We tested MATCH on two state-of-the-art heterogeneous MCUs, GAP9 and DIANA. On the four DNN models of the MLPerf Tiny suite MATCH reduces inference latency by up to 60.88 times on DIANA, compared to using the plain TVM, thanks to the exploitation of the on-board HW accelerator. Compared to HTVM, a fully customized toolchain for DIANA, we still reduce the latency by 16.94%. On GAP9, using the same benchmarks, we improve the latency by 2.15 times compared to the dedicated DORY compiler, thanks to our heterogeneous DNN mapping approach that synergically exploits the DNN accelerator and the eight-cores cluster available on board., Comment: 13 pages, 11 figures, 4 tables

Generative AI is revolutionizing content creation and has the potential to enable real-time, personalized educational experiences. We investigated the effectiveness of converting textbook chapters into AI-generated podcasts and explored the impact of personalizing these podcasts for individual learner profiles. We conducted a 3x3 user study with 180 college students in the United States, comparing traditional textbook reading with both generalized and personalized AI-generated podcasts across three textbook subjects. The personalized podcasts were tailored to students' majors, interests, and learning styles. Our findings show that students found the AI-generated podcast format to be more enjoyable than textbooks and that personalized podcasts led to significantly improved learning outcomes, although this was subject-specific. These results highlight that AI-generated podcasts can offer an engaging and effective modality transformation of textbook material, with personalization enhancing content relevance. We conclude with design recommendations for leveraging AI in education, informed by student feedback.

We prove that if an orientable 3-manifold $M$ admits a complete Riemannian metric whose scalar curvature is positive and has a subquadratic decay at infinity, then it decomposes as a (possibly infinite) connected sum of spherical manifolds and $\mathbb{S}^2 \times \mathbb{S}^1$ summands. This generalises a theorem of Gromov and Wang by using a different, more topological, approach. As a result, the manifold $M$ carries a complete Riemannian metric of uniformly positive scalar curvature, which partially answers a conjecture of Gromov. More generally, the topological decomposition holds without any scalar curvature assumption under a weaker condition on the filling discs of closed curves in the universal cover based on the notion of fill radius. Moreover, the decay rate of the scalar curvature is optimal in this decomposition theorem. Indeed, the manifold $\mathbb{R}^2 \times \mathbb{S}^1$ supports a complete metric of positive scalar curvature with exactly quadratic decay, but does not admit a decomposition as a connected sum., Comment: 20 pages, 7 figures

GPGPU execution analysis has always been tied to closed-source, proprietary benchmarking tools that provide high-level, non-exhaustive, and/or statistical information, preventing a thorough understanding of bottlenecks and optimization possibilities. Open-source hardware platforms offer opportunities to overcome such limits and co-optimize the full {hardware-mapping-algorithm} compute stack. Yet, so far, this has remained under-explored. In this work, we exploit micro-architecture parameter analysis to develop a hardware-aware, runtime mapping technique for OpenCL kernels on the open Vortex RISC-V GPGPU. Our method is based on trace observations and targets optimal hardware resource utilization to achieve superior performance and flexibility compared to hardware-agnostic mapping approaches. The technique was validated on different architectural GPU configurations across several OpenCL kernels. Overall, our approach significantly enhances the performance of the open-source Vortex GPGPU, contributing to unlocking its potential and usability.

Optimal deployment of deep neural networks (DNNs) on state-of-the-art Systems-on-Chips (SoCs) is crucial for tiny machine learning (TinyML) at the edge. The complexity of these SoCs makes deployment non-trivial, as they typically contain multiple heterogeneous compute cores with limited, programmer-managed memory to optimize latency and energy efficiency. We propose HTVM - a compiler that merges TVM with DORY to maximize the utilization of heterogeneous accelerators and minimize data movements. HTVM allows deploying the MLPerf(TM) Tiny suite on DIANA, an SoC with a RISC-V CPU, and digital and analog compute-in-memory AI accelerators, at 120x improved performance over plain TVM deployment., Comment: Presented at DAC2023. Open-source code is available at https://github.com/KULeuven-MICAS/htvm

Conversational systems are crucial for human-computer interaction, managing complex dialogues by identifying threads and prioritising responses. This is especially vital in multi-party conversations, where precise identification of threads and strategic response prioritisation ensure efficient dialogue management. To address these challenges an end-to-end model that identifies threads and prioritises their response generation based on the importance was developed, involving a systematic decomposition of the problem into discrete components - thread detection, prioritisation, and performance optimisation which was meticulously analysed and optimised. These refined components seamlessly integrate into a unified framework, in conversational systems. Llama2 7b is used due to its high level of generalisation but the system can be updated with any open source Large Language Model(LLM). The computational capabilities of the Llama2 model was augmented by using fine tuning methods and strategic prompting techniques to optimise the model's performance, reducing computational time and increasing the accuracy of the model. The model achieves up to 10x speed improvement, while generating more coherent results compared to existing models., Comment: 6 pages, 4 figures, submitted to 2024 IEEE International Conference on Signal Processing and Communications (SPCOM)

We present Prompt Cache, an approach for accelerating inference for large language models (LLM) by reusing attention states across different LLM prompts. Many input prompts have overlapping text segments, such as system messages, prompt templates, and documents provided for context. Our key insight is that by precomputing and storing the attention states of these frequently occurring text segments on the inference server, we can efficiently reuse them when these segments appear in user prompts. Prompt Cache employs a schema to explicitly define such reusable text segments, called prompt modules. The schema ensures positional accuracy during attention state reuse and provides users with an interface to access cached states in their prompt. Using a prototype implementation, we evaluate Prompt Cache across several LLMs. We show that Prompt Cache significantly reduce latency in time-to-first-token, especially for longer prompts such as document-based question answering and recommendations. The improvements range from 8x for GPU-based inference to 60x for CPU-based inference, all while maintaining output accuracy and without the need for model parameter modifications., Comment: To appear at MLSys 2024

The need to execute Deep Neural Networks (DNNs) at low latency and low power at the edge has spurred the development of new heterogeneous Systems-on-Chips (SoCs) encapsulating a diverse set of hardware accelerators. How to optimally map a DNN onto such multi-accelerator systems is an open problem. We propose ODiMO, a hardware-aware tool that performs a fine-grain mapping across different accelerators on-chip, splitting individual layers and executing them in parallel, to reduce inference energy consumption or latency, while taking into account each accelerator's quantization precision to maintain accuracy. Pareto-optimal networks in the accuracy vs. energy or latency space are pursued for three popular dataset/DNN pairs, and deployed on the DIANA heterogeneous ultra-low power edge AI SoC. We show that ODiMO reduces energy/latency by up to 33%/31% with limited accuracy drop (-0.53%/-0.32%) compared to manual heuristic mappings., Comment: Accepted at 2023 ACM/IEEE International Symposium on Low Power Electronics and Design (ISLPED)

In many software systems, heuristics are used to make decisions - such as cache eviction, task scheduling, and information presentation - that have a significant impact on overall system behavior. While machine learning may outperform these heuristics, replacing existing heuristics in a production system safely and reliably can be prohibitively costly. We present SmartChoices, a novel approach that reduces the cost to deploy production-ready ML solutions for contextual bandits problems. SmartChoices' interface cleanly separates problem formulation from implementation details: engineers describe their use case by defining datatypes for the context, arms, and feedback that are passed to SmartChoices APIs, while SmartChoices manages encoding & logging data and training, evaluating & deploying policies. Our implementation codifies best practices, is efficient enough for use in low-level applications, and provides valuable production features off the shelf via a shared library. Overall, SmartChoices enables non-experts to rapidly deploy production-ready ML solutions by eliminating many sources of technical debt common to ML systems. Engineers have independently used SmartChoices to improve a wide range of software including caches, batch processing workloads, and UI layouts, resulting in better latency, throughput, and click-through rates., Comment: Accepted as a workshop paper at the Machine Learning for Systems Workshop at NeurIPS 2023

In this work, we demonstrate the viability of using federated learning to successfully predict energy consumption as well as solar production for all households within a certain network using low-power and low-space consuming embedded devices. We also demonstrate our prediction performance improving over time without the need for sharing private consumer energy data. We simulate a system with four nodes using data for one year to show this.

In today's era of smart cyber-physical systems, Deep Neural Networks (DNNs) have become ubiquitous due to their state-of-the-art performance in complex real-world applications. The high computational complexity of these networks, which translates to increased energy consumption, is the foremost obstacle towards deploying large DNNs in resource-constrained systems. Fixed-Point (FP) implementations achieved through post-training quantization are commonly used to curtail the energy consumption of these networks. However, the uniform quantization intervals in FP restrict the bit-width of data structures to large values due to the need to represent most of the numbers with sufficient resolution and avoid high quantization errors. In this paper, we leverage the key insight that (in most of the scenarios) DNN weights and activations are mostly concentrated near zero and only a few of them have large magnitudes. We propose CoNLoCNN, a framework to enable energy-efficient low-precision deep convolutional neural network inference by exploiting: (1) non-uniform quantization of weights enabling simplification of complex multiplication operations; and (2) correlation between activation values enabling partial compensation of quantization errors at low cost without any run-time overheads. To significantly benefit from non-uniform quantization, we also propose a novel data representation format, Encoded Low-Precision Binary Signed Digit, to compress the bit-width of weights while ensuring direct use of the encoded weight for processing using a novel multiply-and-accumulate (MAC) unit design., Comment: 8 pages, 15 figures, 2 tables

Face verification systems aim to validate the claimed identity using feature vectors and distance metrics. However, no attempt has been made to bypass such a system using generated images that are constrained by the same feature vectors. In this work, we train StarGAN v2 to generate diverse images based on a human user, that have similar feature vectors yet qualitatively look different. We then demonstrate a proof of concept on a custom face verification system and verify our claims by demonstrating the same proof of concept in a black box setting on dating applications that utilize similar face verification systems.

This paper outlines a social robot platform for providing health information. In comparison with previous findings for accessing information online, the use of a social robot may affect which factors users consider important when evaluating the trustworthiness of health information provided., Comment: 2 pages

Solar X-ray Monitor (XSM) instrument of India's Chandrayaan-2 lunar mission carries out broadband spectroscopy of the Sun in soft X-rays. XSM, with its unique features such as low background, high time cadence, and high spectral resolution, provides the opportunity to characterize transient and quiescent X-ray emission from the Sun even during low activity periods. It records the X-ray spectrum at one-second cadence, and the data recorded on-board are downloaded at regular intervals along with that of other payloads. During ground pre-processing, the XSM data is segregated, and the level-0 data is made available for higher levels of processing at the Payload Operations Center (POC). XSM Data Analysis Software (XSMDAS) is developed to carry out the processing of the level-0 data to higher levels and to generate calibrated light curves and spectra for user-defined binning parameters such that it is suitable for further scientific analysis. A front-end for the XSMDAS named XSM Quick Look Display (XSMQLD) is also developed to facilitate a first look at the data without applying calibration. XSM Data Management-Monitoring System (XSMDMS) is designed to carry out automated data processing at the POC and to maintain an SQLite database with relevant information on the data sets and an internal web application for monitoring data quality and instrument health. All XSM raw and calibrated data products are in FITS format, organized into day-wise files, and the data archive follows Planetary Data System-4 (PDS4) standards. The XSM data will be made available after a lock-in period along with the XSM Data Analysis Software from ISRO Science Data Archive (ISDA) at Indian Space Science Data Center(ISSDC). Here we discuss the design and implementation of all components of the software for the XSM data processing and the contents of the XSM data archive., Comment: Submitted to Astronomy and Computing, 13 pages, 9 figures