Search

Multi-modal models require aligned, shared embedding spaces. However, common CLIP-based approaches need large amounts of samples and do not natively support 3D or tabular data, both of which are crucial in the medical domain. To address these issues, we revisit CLIP-style alignment by training a domain-specific 3D foundation model as an image encoder and demonstrate that modality alignment is feasible with only 62 MRI scans. Our approach is enabled by a simple embedding accumulation strategy required for training in 3D, which scales the amount of negative pairs across batches in order to stabilize training. We perform a thorough evaluation of various design choices, including the choice of backbone and loss functions, and evaluate the proposed methodology on zero-shot classification and image-retrieval tasks. While zero-shot image-retrieval remains challenging, zero-shot classification results demonstrate that the proposed approach can meaningfully align the representations of 3D MRI with tabular data., Comment: 10 pages, 2 figures. To be published in ISBI 2025

We study the eigenvalues of Hodge--Laplace operators over bounded convex domains. Generalizing contributions by Guerini and Savo, we show that the Poincar\'e--Friedrichs constants of the Sobolev de~Rham complexes increase with the degree of the forms. We establish that the spectra of Hodge--Laplace operators converge when the domain is approximated by a sequence of convex domains up to a bi-Lipschitz deformation of the boundary. As preparatory work with independent relevance, we study the gauge function and the reciprocal radial function of convex sets, providing new proofs for Lipschitz estimates by Vre\'{c}ica and Toranzos for the reciprocal radial function and improving a Lipschitz estimate for the gauge function due to Beer., Comment: Comments welcome

We introduce and analyse a class of weighted Sobolev spaces with mixed weights on angular domains. The weights are based on both the distance to the boundary and the distance to the one vertex of the domain. Moreover, we show how the regularity of the Poisson equation can be analysed in the framework of these spaces by means of the Mellin transform, provided the integrability parameter equals two. Our main motivation comes from the study of stochastic partial differential equations and associated degenerate deterministic parabolic equations., Comment: 47 pages

Digital systems are growing in importance and computing hardware is growing more heterogeneous. Hardware design, however, remains laborious and expensive, in part due to the limitations of conventional hardware description languages (HDLs) like VHDL and Verilog. A longstanding research goal has been programming hardware like software, with high-level languages that can generate efficient hardware designs. This paper describes Kanagawa, a language that takes a new approach to combine the programmer productivity benefits of traditional High-Level Synthesis (HLS) approaches with the expressibility and hardware efficiency of Register-Transfer Level (RTL) design. The language's concise syntax, matched with a hardware design-friendly execution model, permits a relatively simple toolchain to map high-level code into efficient hardware implementations., Comment: Accepted to PLDI'24

This study identifies a gap in data-driven approaches to robot-centric pedestrian interactions and proposes a corresponding pipeline. The pipeline utilizes unsupervised learning techniques to identify patterns in interaction data of urban environments, specifically focusing on conflict scenarios. Analyzed features include the robot's and pedestrian's speed and contextual parameters such as proximity to intersections. They are extracted and reduced in dimensionality using Principal Component Analysis (PCA). Finally, K-means clustering is employed to uncover underlying patterns in the interaction data. A use case application of the pipeline is presented, utilizing real-world robot mission data from a mid-sized German city. The results indicate the need for enriching interaction representations with contextual information to enable fine-grained analysis and reasoning. Nevertheless, they also highlight the need for expanding the data set and incorporating additional contextual factors to enhance the robots situational awareness and interaction quality.

The limited dynamic range of the detector can impede coherent diffractive imaging (CDI) schemes from achieving diffraction-limited resolution. To overcome this limitation, a straightforward approach is to utilize high dynamic range (HDR) imaging through multi-exposure image fusion (MEF). This method involves capturing measurements at different exposure times, spanning from under to overexposure and fusing them into a single HDR image. The conventional MEF technique in ptychography typically involves subtracting the background noise, ignoring the saturated pixels and then merging the acquisitions. However, this approach is inadequate under conditions of low signal-to-noise ratio (SNR). Additionally, variations in illumination intensity significantly affect the phase retrieval process. To address these issues, we propose a Bayesian MEF modeling approach based on a modified Poisson distribution that takes the background and saturation into account. To infer the model parameters, the expectation-maximization (EM) algorithm is employed. As demonstrated with synthetic and experimental data, our approach outperforms the conventional MEF method, offering superior phase retrieval under challenging experimental conditions. This work underscores the significance of robust multi-exposure image fusion for ptychography, particularly in imaging shot-noise-dominated weakly scattering specimens or in cases where access to HDR detectors with high SNR is limited. Furthermore, the applicability of the Bayesian MEF approach extends beyond CDI to any imaging scheme that requires HDR treatment. Given this versatility, we provide the implementation of our algorithm as a Python package.

Managing rapidly growing decentralized gaming communities brings unique challenges at the nexus of cultural economics and technology. This paper introduces a streamlined analytical framework that utilizes Large Language Models (LLMs), in this instance open-access generative pre-trained transformer (GPT) models, offering an efficient solution with deeper insights into community dynamics. The framework aids moderators in identifying pseudonymous actor intent, moderating toxic behavior, rewarding desired actions to avoid unintended consequences of blockchain-based gaming, and gauging community sentiment as communities venture into metaverse platforms and plan for hypergrowth. This framework strengthens community controls, eases onboarding, and promotes a common moral mission across communities while reducing agency costs by 95 pct. Highlighting the transformative role of generative AI, the paper emphasizes its potential to redefine the cost of cultural production. It showcases the utility of GPTs in digital community management, expanding their implications in cultural economics and transmedia storytelling.

Large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model-SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. SeamlessM4T v2 provides the foundation on which our next two models are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one's voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. The contributions to this work are publicly released and accessible at https://github.com/facebookresearch/seamless_communication

We survey recent contributions to finite element exterior calculus on manifolds and surfaces within a comprehensive formalism for the error analysis of vector-valued partial differential equations on manifolds. Our primary focus is on uniformly bounded commuting projections on manifolds: these projections map from Sobolev de Rham complexes onto finite element de Rham complexes, commute with the differential operators, and satisfy uniform bounds in Lebesgue norms. They enable the Galerkin theory of Hilbert complexes for a large range of intrinsic finite element methods on manifolds. However, these intrinsic finite element methods are generally not computable and thus primarily of theoretical interest. This leads to our second point: estimating the geometric variational crime incurred by transitioning to computable approximate problems. Lastly, our third point addresses how to estimate the approximation error of the intrinsic finite element method in terms of the mesh size. If the solution is not continuous, then such an estimate is achieved via modified Cl\'ement or Scott-Zhang interpolants that facilitate a broken Bramble--Hilbert lemma., Comment: Contribution to ENUMATH Proceedings 2023. 8 pages

We develop commuting finite element projections over smooth Riemannian manifolds. This extension of finite element exterior calculus establishes the stability and convergence of finite element methods for the Hodge-Laplace equation on manifolds. The commuting projections use localized mollification operators, building upon a classical construction by de Rham. These projections are uniformly bounded on Lebesgue spaces of differential forms and map onto intrinsic finite element spaces defined with respect to an intrinsic smooth triangulation of the manifold. We analyze the Galerkin approximation error. Since practical computations use extrinsic finite element methods over approximate computational manifolds, we also analyze the geometric error incurred., Comment: Submitted. 31 pages

We construct finite element de~Rham complexes of higher and possibly non-uniform polynomial order in finite element exterior calculus (FEEC). Starting from the finite element differential complex of lowest-order, known as the complex of Whitney forms, we incrementally construct the higher-order complexes by adjoining exact local complexes associated to simplices. We define a commuting canonical interpolant. On the one hand, this research provides a base for studying $hp$-adaptive methods in finite element exterior calculus. On the other hand, our construction of higher-order spaces enables a new tool in numerical analysis which we call "partially localized flux reconstruction". One major application of this concept is in the area of equilibrated a~posteriori error estimators: we generalize the Braess-Sch\"oberl error estimator to edge elements of higher and possibly non-uniform order.

We show that the standard partition of unity subordinate to an open cover of a metric space has Lipschitz constant $\max(1,M-1)/\mathcal{L}$, where $\mathcal{L}$ is the Lebesgue number and $M$ is the multiplicity of the cover. If the metric space satisfies the approximate midpoint property, such as length spaces do, then the upper bound improves to $(M-1)/(2\mathcal{L})$. These Lipschitz estimates are optimal. We also address the Lipschitz analysis of $\ell^{p}$-generalizations of the standard partition of unity, their partial sums, and their categorical products. Lastly, we characterize metric spaces with Assouad--Nagata dimension $n$ as exactly those metric spaces for which every Lebesgue cover admits an open refinement with multiplicity $n+1$ while reducing the Lebesgue number by at most a constant factor., Comment: 20 pages, all comments welcome

We give a constructive proof for the following new collar theorem: every locally collared closed set that is paracompact in a Hausdorff space is collared. This includes the important special case of locally collared closed sets in paracompact Hausdorff spaces. Importantly, we use Stone's result that every open cover of a paracompact space has an open locally finite refinement which is the countable union of discrete families. Furthermore, in the LIP category, our construction yields collars that are locally bi-Lipschitz embeddings. If the initial data satisfy uniform estimates, then this collar is even bi-Lipschitz onto its image and we explicitly bound the constants. We also provide partitions of unity whose Lipschitz constants are bounded by the Lebesgue constant and the order of the cover., Comment: Feedback welcome

What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages? While recent breakthroughs in text-based models have pushed machine translation coverage beyond 200 languages, unified speech-to-speech translation models have yet to achieve similar strides. More specifically, conventional speech-to-speech translation systems rely on cascaded systems that perform translation progressively, putting high-performing unified systems out of reach. To address these gaps, we introduce SeamlessM4T, a single model that supports speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and automatic speech recognition for up to 100 languages. To build this, we used 1 million hours of open speech audio data to learn self-supervised speech representations with w2v-BERT 2.0. Subsequently, we created a multimodal corpus of automatically aligned speech translations. Filtered and combined with human-labeled and pseudo-labeled data, we developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation. Compared to strong cascaded models, SeamlessM4T improves the quality of into-English translation by 1.3 BLEU points in speech-to-text and by 2.6 ASR-BLEU points in speech-to-speech. Tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks compared to the current SOTA model. Critically, we evaluated SeamlessM4T on gender bias and added toxicity to assess translation safety. Finally, all contributions in this work are open-sourced and accessible at https://github.com/facebookresearch/seamless_communication

Vector search has emerged as the foundation for large-scale information retrieval and machine learning systems, with search engines like Google and Bing processing tens of thousands of queries per second on petabyte-scale document datasets by evaluating vector similarities between encoded query texts and web documents. As performance demands for vector search systems surge, accelerated hardware offers a promising solution in the post-Moore's Law era. We introduce \textit{FANNS}, an end-to-end and scalable vector search framework on FPGAs. Given a user-provided recall requirement on a dataset and a hardware resource budget, \textit{FANNS} automatically co-designs hardware and algorithm, subsequently generating the corresponding accelerator. The framework also supports scale-out by incorporating a hardware TCP/IP stack in the accelerator. \textit{FANNS} attains up to 23.0$\times$ and 37.2$\times$ speedup compared to FPGA and CPU baselines, respectively, and demonstrates superior scalability to GPUs, achieving 5.5$\times$ and 7.6$\times$ speedup in median and 95\textsuperscript{th} percentile (P95) latency within an eight-accelerator configuration. The remarkable performance of \textit{FANNS} lays a robust groundwork for future FPGA integration in data centers and AI supercomputers., Comment: 11 pages

The task of cultivating healthy communication in online communities becomes increasingly urgent, as gaming and social media experiences become progressively more immersive and life-like. We approach the challenge of moderating online communities by training student models using a large language model (LLM). We use zero-shot learning models to distill and expand datasets followed by a few-shot learning and a fine-tuning approach, leveraging open-access generative pre-trained transformer models (GPT) from OpenAI. Our preliminary findings suggest, that when properly trained, LLMs can excel in identifying actor intentions, moderating toxic comments, and rewarding positive contributions. The student models perform above-expectation in non-contextual assignments such as identifying classically toxic behavior and perform sufficiently on contextual assignments such as identifying positive contributions to online discourse. Further, using open-access models like OpenAI's GPT we experience a step-change in the development process for what has historically been a complex modeling task. We contribute to the information system (IS) discourse with a rapid development framework on the application of generative AI in content online moderation and management of culture in decentralized, pseudonymous communities by providing a sample model suite of industrial-ready generative AI models based on open-access LLMs.

Dataflow devices represent an avenue towards saving the control and data movement overhead of Load-Store Architectures. Various dataflow accelerators have been proposed, but how to efficiently schedule applications on such devices remains an open problem. The programmer can explicitly implement both temporal and spatial parallelism, and pipelining across multiple processing elements can be crucial to take advantage of the fast on-chip interconnect, enabling the concurrent execution of different program components. This paper introduces canonical task graphs, a model that enables streaming scheduling of task graphs over dataflow architectures. We show how a task graph can be statically analyzed to understand its steady-state behavior, and we use this information to partition it into temporally multiplexed components of spatially executed tasks. Results on synthetic and realistic workloads show how streaming scheduling can increase speedup and device utilization over a traditional scheduling approach.

We introduce a multilingual extension of the HOLISTICBIAS dataset, the largest English template-based taxonomy of textual people references: MULTILINGUALHOLISTICBIAS. This extension consists of 20,459 sentences in 50 languages distributed across all 13 demographic axes. Source sentences are built from combinations of 118 demographic descriptors and three patterns, excluding nonsensical combinations. Multilingual translations include alternatives for gendered languages that cover gendered translations when there is ambiguity in English. Our benchmark is intended to uncover demographic imbalances and be the tool to quantify mitigations towards them. Our initial findings show that translation quality for EN-to-XX translations is an average of 8 spBLEU better when evaluating with the masculine human reference compared to feminine. In the opposite direction, XX-to-EN, we compare the robustness of the model when the source input only differs in gender (masculine or feminine) and masculine translations are an average of almost 4 spBLEU better than feminine. When embedding sentences to a joint multilingual sentence representations space, we find that for most languages masculine translations are significantly closer to the English neutral sentences when embedded.

We develop projection operators onto finite element differential forms over simplicial meshes. Our projection is locally bounded in Lebesgue and Sobolev-Slobodeckij norms, uniformly with respect to mesh parameters. Moreover, it incorporates homogeneous boundary conditions and satisfies a local broken Bramble-Hilbert estimate. The construction principle includes the Ern-Guermond projection and a modified Cl\'ement-type interpolant with the projection property. The latter seems to be a new result even for Lagrange elements. This projection operator immediately enables an equivalence result on local- and global-best approximations. We combine techniques for the Scott-Zhang and Ern-Guermond projections and adopt the framework of finite element exterior calculus. We instantiate the abstract projection for Brezzi-Douglas-Marini, N\'ed\'elec, and Raviart-Thomas elements., Comment: 27 pages. Submitted. Comments welcome

Although high-level synthesis (HLS) tools have significantly improved programmer productivity over hardware description languages, developing for FPGAs remains tedious and error prone. Programmers must learn and implement a large set of vendor-specific syntax, patterns, and tricks to optimize (or even successfully compile) their applications, while dealing with ever-changing toolflows from the FPGA vendors. We propose a new way to develop, optimize, and compile FPGA programs. The Data-Centric parallel programming (DaCe) framework allows applications to be defined by their dataflow and control flow through the Stateful DataFlow multiGraph (SDFG) representation, capturing the abstract program characteristics, and exposing a plethora of optimization opportunities. In this work, we show how extending SDFGs with multi-level Library Nodes incorporates both domain-specific and platform-specific optimizations into the design flow, enabling knowledge transfer across application domains and FPGA vendors. We present the HLS-based FPGA code generation backend of DaCe, and show how SDFGs are code generated for either FPGA vendor, emitting efficient HLS code that is structured and annotated to implement the desired architecture.

We review Maxwell's equations and constitutive relations for 3D bianisotropic media in a generalized form: we consider all four variables and allow for nonzero polarization or magnetization, and also nonzero nonzero magnetic charge or current. After a discussion of general boundary conditions, we introduce a time-harmonic variational formulation of linear Maxwell's equations within 3D bianisotropic media in terms of the electric and magnetic fields. We showcase a finite element approximation of our variational formulation, using curl-conforming N\'ed\'elec edge elements of the first kind. Numerical examples illustrate the convergence of the method., Comment: Submitted

We describe higher-order chain rules for multivariate functions and tensor fields. We estimate Sobolev-Slobodeckij norms, Musielak-Orlicz norms, and the total variation seminorms of the higher derivatives of tensor fields after a change of variables and determine sufficient regularity conditions for the coordinate change. We also introduce a novel higher-order chain rule for composition chains of multivariate functions that is described via nested set partitions and generalized Bell polynomials; it is a natural extension of the Fa\`a di Bruno formula. Our discussion uses the coordinate-free language of tensor calculus and includes Fr\'echet-differentiable mappings between Banach spaces., Comment: Submitted

The time development of the excitation energy of molecules and clusters cooling by emission of thermal vibrational infrared radiation has been studied. The energy distributions and the photon emission rates develop into near-universal functions that can be characterized with only a few parameters, irrespective of the precise vibrational spectra and oscillator strengths of the systems. The photon emission constant and emitted power averaged over all thermally populated states vary linearly with total excitation energy with a small offset. The time developments of ensemble internal energy distributions are calculated with respect to their first two moments. For the derived linear dependence of the emission rate constant, these results are exact.