Search

We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user's image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models. All videos from this paper are available at https://go.fb.me/MovieGenResearchVideos.

Water hydrogen bonding is extremely versatile; approximately 20 ice structures and several types of clathrate hydrate structures have been identified. These crystalline water structures form at temperatures below room temperature and/or at high pressure. We used transmission electron microscopy to study a new crystalline water structure in a clathrate state that is prepared by sandwiching gas-supersaturated water between graphene layers under ambient conditions. In this clathrate state, water molecules form a three-dimensional hydrogen bonding network that encloses gas-filled cages 2-4 nm in size. We derived the crystalline water structure by recording and analyzing electron diffraction patterns and performing first-principles calculations. The structure consists purely of pentagonal rings and has a topology similar to that of water ice XVII. The study proposed a mechanism for the formation of the clathrate state. The present results improve the understanding of interactions among water and small nonpolar molecules and offer novel insights into the local structures of ambient liquid water.

Relative free energy calculations are now widely used in academia and industry, but the accuracy is often limited by poor sampling of the complexes conformational ensemble. To address this, we have developed a novel method termed Multi-Topology Replica Exchange of Expanded Ensembles (MT-REXEE). This method enables parallel expanded ensemble calculations, facilitating iterative relative free energy computations while allowing conformational exchange between parallel transformations. These iterative transformations are adaptable to any set of systems with a common backbone or central substructure. We demonstrate that the MT-REXEE method maintains thermodynamic cycle closure to the same extent as standard expanded ensemble for both solvation free energy and relative binding free energy. The transformations tested involve simple systems that incorporate diverse heavy atoms and multi-site perturbations of a small molecule core resembling multi-site $\lambda$ dynamics, without necessitating modifications to the MD code, which in our initial implementation is GROMACS. We outline a systematic approach for topology set-up and provide instructions on how to perform inter-replicate coordinate modifications. This work shows that MT-REEXE can be used to perform accurate and reproducible free energy estimates and prompts expansion to more complex test systems and other molecular dynamics simulation infrastructures.

We introduce MelodyFlow, an efficient text-controllable high-fidelity music generation and editing model. It operates on continuous latent representations from a low frame rate 48 kHz stereo variational auto encoder codec. Based on a diffusion transformer architecture trained on a flow-matching objective the model can edit diverse high quality stereo samples of variable duration, with simple text descriptions. We adapt the ReNoise latent inversion method to flow matching and compare it with the original implementation and naive denoising diffusion implicit model (DDIM) inversion on a variety of music editing prompts. Our results indicate that our latent inversion outperforms both ReNoise and DDIM for zero-shot test-time text-guided editing on several objective metrics. Subjective evaluations exhibit a substantial improvement over previous state of the art for music editing. Code and model weights will be publicly made available. Samples are available at https://melodyflow.github.io.

Generating realistic audio for human actions is important for many applications, such as creating sound effects for films or virtual reality games. Existing approaches implicitly assume total correspondence between the video and audio during training, yet many sounds happen off-screen and have weak to no correspondence with the visuals -- resulting in uncontrolled ambient sounds or hallucinations at test time. We propose a novel ambient-aware audio generation model, AV-LDM. We devise a novel audio-conditioning mechanism to learn to disentangle foreground action sounds from the ambient background sounds in in-the-wild training videos. Given a novel silent video, our model uses retrieval-augmented generation to create audio that matches the visual content both semantically and temporally. We train and evaluate our model on two in-the-wild egocentric video datasets, Ego4D and EPIC-KITCHENS, and we introduce Ego4D-Sounds -- 1.2M curated clips with action-audio correspondence. Our model outperforms an array of existing methods, allows controllable generation of the ambient sound, and even shows promise for generalizing to computer graphics game clips. Overall, our approach is the first to focus video-to-audio generation faithfully on the observed visual content despite training from uncurated clips with natural background sounds., Comment: Project page: https://vision.cs.utexas.edu/projects/action2sound. ECCV 2024 camera-ready version

As the scale of generative models continues to grow, efficient reuse and adaptation of pre-trained models have become crucial considerations. In this work, we propose Voicebox Adapter, a novel approach that integrates fine-grained conditions into a pre-trained Voicebox speech generation model using a cross-attention module. To ensure a smooth integration of newly added modules with pre-trained ones, we explore various efficient fine-tuning approaches. Our experiment shows that the LoRA with bias-tuning configuration yields the best performance, enhancing controllability without compromising speech quality. Across three fine-grained conditional generation tasks, we demonstrate the effectiveness and resource efficiency of Voicebox Adapter. Follow-up experiments further highlight the robustness of Voicebox Adapter across diverse data setups., Comment: Accepted by InterSpeech 2024

Generative multimodal content is increasingly prevalent in much of the content creation arena, as it has the potential to allow artists and media personnel to create pre-production mockups by quickly bringing their ideas to life. The generation of audio from text prompts is an important aspect of such processes in the music and film industry. Many of the recent diffusion-based text-to-audio models focus on training increasingly sophisticated diffusion models on a large set of datasets of prompt-audio pairs. These models do not explicitly focus on the presence of concepts or events and their temporal ordering in the output audio with respect to the input prompt. Our hypothesis is focusing on how these aspects of audio generation could improve audio generation performance in the presence of limited data. As such, in this work, using an existing text-to-audio model Tango, we synthetically create a preference dataset where each prompt has a winner audio output and some loser audio outputs for the diffusion model to learn from. The loser outputs, in theory, have some concepts from the prompt missing or in an incorrect order. We fine-tune the publicly available Tango text-to-audio model using diffusion-DPO (direct preference optimization) loss on our preference dataset and show that it leads to improved audio output over Tango and AudioLDM2, in terms of both automatic- and manual-evaluation metrics., Comment: Accepted at ACM MM 2024

Speech recognition and translation systems perform poorly on noisy inputs, which are frequent in realistic environments. Augmenting these systems with visual signals has the potential to improve robustness to noise. However, audio-visual (AV) data is only available in limited amounts and for fewer languages than audio-only resources. To address this gap, we present XLAVS-R, a cross-lingual audio-visual speech representation model for noise-robust speech recognition and translation in over 100 languages. It is designed to maximize the benefits of limited multilingual AV pre-training data, by building on top of audio-only multilingual pre-training and simplifying existing pre-training schemes. Extensive evaluation on the MuAViC benchmark shows the strength of XLAVS-R on downstream audio-visual speech recognition and translation tasks, where it outperforms the previous state of the art by up to 18.5% WER and 4.7 BLEU given noisy AV inputs, and enables strong zero-shot audio-visual ability with audio-only fine-tuning., Comment: ACL2024

The phase transition of confined fluids in mesoporous materials deviates from that of bulk fluids due to the interactions with the surrounding heterogeneous structure. For example, adsorbed fluids in metal-organic-frameworks (MOFs) have atypical phase characteristics such as capillary condensation and higher-order phase transitions due to a strong heterogeneous field. Considering a many-body problem in the presence of a nonuniform external field, we model the host-guest and guest-guest interactions in MOFs. To solve the three-dimensional Ising model, we use the mean-field theory to approximate the guest-guest interactions and Mayer's f-functions to describe the host-guest interactions in a unit cell. Later, using Hill's theory of nanothermodynamics, we define differential thermodynamic functions to understand the distribution of intensive properties and integral thermodynamic functions to explain the phase transition in confined fluids. The investigation reveals a distinct behavior where fluids confined in larger pores undergo a discontinuous (first-order) phase transition, whereas those confined in smaller pores experience a continuous (higher-order) phase transition. Furthermore, the results indicate that the free-energy barrier for phase transitions is lower in confined fluids than in bulk fluids giving rise to a lower condensation pressure relative to the bulk saturation pressure. Finally, the integral thermodynamic functions are succinctly presented in the form of a phase diagram, marking an initial step toward a more practical approach for understanding the phase behavior of confined fluids., Comment: 25 pages, 14 figures

Resistive switching is the fundamental process that triggers the sudden change of the electrical properties in solid-state devices under the action of intense electric fields. Despite its relevance for information processing, ultrafast electronics, neuromorphic devices, resistive memories and brain-inspired computation, the nature of the local stochastic fluctuations that drive the formation of metallic nuclei out of the insulating state has remained hidden. Here, using operando X-ray nano-imaging, we have captured the early-stages of resistive switching in a V2O3-based device under working conditions. V2O3 is a paradigmatic Mott material, which undergoes a first-order metal-to-insulator transition coupled to a lattice transformation that breaks the threefold rotational symmetry of the rhombohedral metal phase. We reveal a new class of volatile electronic switching triggered by nanoscale topological defects of the lattice order parameter of the insulating phase. Our results pave the way to the use of strain engineering approaches to manipulate topological defects and achieve the full control of the electronic Mott switching. The concept of topology-driven reversible electronic transition is of interest for a broad class of quantum materials, comprising transition metal oxides, chalcogenides and kagome metals, that exhibit first-order electronic transitions coupled to a symmetry-breaking order.

Surfaces (interfaces) dictate many physical and chemical properties of solid materials and adsorbates considerably affect these properties. Nitrogen molecules, which are the most abundant constituent in ambient air, are considered to be inert. Our study combining atomic force microscopy (AFM), X-ray photoemission spectroscopy (XPS), and thermal desorption spectroscopy (TDS) revealed that nitrogen and water molecules can self-assemble into two-dimensional domains, forming ordered stripe structures on graphitic surfaces in both water and ambient air. The stripe structures of this study were composed of approximately 90% and 10% water and nitrogen molecules, respectively, and survived in ultra-high vacuum (UHV) conditions at temperatures up to approximately 350 K. Because pure water molecules completely desorb from graphitic surfaces in a UHV at temperatures lower than 200 K, our results indicate that the incorporation of nitrogen molecules substantially enhanced the stability of the crystalline water hydrogen bonding network. Additional studies on interfacial gas hydrates can provide deeper insight into the mechanisms underlying formation of gas hydrates.

Audio is an essential part of our life, but creating it often requires expertise and is time-consuming. Research communities have made great progress over the past year advancing the performance of large scale audio generative models for a single modality (speech, sound, or music) through adopting more powerful generative models and scaling data. However, these models lack controllability in several aspects: speech generation models cannot synthesize novel styles based on text description and are limited on domain coverage such as outdoor environments; sound generation models only provide coarse-grained control based on descriptions like "a person speaking" and would only generate mumbling human voices. This paper presents Audiobox, a unified model based on flow-matching that is capable of generating various audio modalities. We design description-based and example-based prompting to enhance controllability and unify speech and sound generation paradigms. We allow transcript, vocal, and other audio styles to be controlled independently when generating speech. To improve model generalization with limited labels, we adapt a self-supervised infilling objective to pre-train on large quantities of unlabeled audio. Audiobox sets new benchmarks on speech and sound generation (0.745 similarity on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and unlocks new methods for generating audio with novel vocal and acoustic styles. We further integrate Bespoke Solvers, which speeds up generation by over 25 times compared to the default ODE solver for flow-matching, without loss of performance on several tasks. Our demo is available at https://audiobox.metademolab.com/

Dimensional confinement has shown to be an effective strategy to tune competing degrees of freedom in complex oxides. Here, we achieved atomic layered growth of trigonal vanadium sesquioxide (V2O3) by means of oxygen-assisted molecular beam epitaxy. This led to a series of high-quality epitaxial ultrathin V2O3 films down to unit cell thickness, enabling the study of the intrinsic electron correlations upon confinement. By electrical and optical measurements, we demonstrate a dimensional confinement-induced metal-insulator transition in these ultrathin films. We shed light on the Mott-Hubbard nature of this transition, revealing an abrupt vanishing of the quasiparticle weight as demonstrated by photoemission spectroscopy. Furthermore, we prove that dimensional confinement acts as an effective out-of-plane stress. This highlights the structural component of correlated oxides in a confined architecture, while opening an avenue to control both in-plane and out-of-plane lattice components by epitaxial strain and confinement, respectively.

In this paper, we show that a simple self-supervised pre-trained audio model can achieve comparable inference efficiency to more complicated pre-trained models with speech transformer encoders. These speech transformers rely on mixing convolutional modules with self-attention modules. They achieve state-of-the-art performance on ASR with top efficiency. We first show that employing these speech transformers as an encoder significantly improves the efficiency of pre-trained audio models as well. However, our study shows that we can achieve comparable efficiency with advanced self-attention solely. We demonstrate that this simpler approach is particularly beneficial with a low-bit weight quantization technique of a neural network to improve efficiency. We hypothesize that it prevents propagating the errors between different quantized modules compared to recent speech transformers mixing quantized convolution and the quantized self-attention modules., Comment: 5 pages; accepted to Self-supervision in Audio, Speech and Beyond (SASB) workshop in ICASSP24

Generative models have gained more and more attention in recent years for their remarkable success in tasks that required estimating and sampling data distribution to generate high-fidelity synthetic data. In speech, text-to-speech synthesis and neural vocoder are good examples where generative models have shined. While generative models have been applied to different applications in speech, there exists no general-purpose generative model that models speech directly. In this work, we take a step toward this direction by showing a single pre-trained generative model can be adapted to different downstream tasks with strong performance. Specifically, we pre-trained a generative model, named SpeechFlow, on 60k hours of untranscribed speech with Flow Matching and masked conditions. Experiment results show the pre-trained generative model can be fine-tuned with task-specific data to match or surpass existing expert models on speech enhancement, separation, and synthesis. Our work suggested a foundational model for generation tasks in speech can be built with generative pre-training., Comment: ICLR 2024

Due to the remarkable language understanding and generation abilities of large language models (LLMs), their use in educational applications has been explored. However, little work has been done on investigating the pedagogical ability of LLMs in helping students to learn mathematics. In this position paper, we discuss the challenges associated with employing LLMs to enhance students' mathematical problem-solving skills by providing adaptive feedback. Apart from generating the wrong reasoning processes, LLMs can misinterpret the meaning of the question, and also exhibit difficulty in understanding the given questions' rationales when attempting to correct students' answers. Three research questions are formulated., Comment: Accepted by EMNLP 2023 Findings

Speech and text are two major forms of human language. The research community has been focusing on mapping speech to text or vice versa for many years. However, in the field of language modeling, very little effort has been made to model them jointly. In light of this, we explore joint language modeling for speech units and text. Specifically, we compare different speech tokenizers to transform continuous speech signals into discrete units and use different methods to construct mixed speech-text data. We introduce automatic metrics to evaluate how well the joint LM mixes speech and text. We also fine-tune the LM on downstream spoken language understanding (SLU) tasks with different modalities (speech or text) and test its performance to assess the model's learning of shared representations. Our results show that by mixing speech units and text with our proposed mixing techniques, the joint LM improves over a speech-only baseline on SLU tasks and shows zero-shot cross-modal transferability., Comment: EMNLP findings 2023

Self-supervised learning (SSL) techniques have achieved remarkable results in various speech processing tasks. Nonetheless, a significant challenge remains in reducing the reliance on vast amounts of speech data for pre-training. This paper proposes to address this challenge by leveraging synthetic speech to augment a low-resource pre-training corpus. We construct a high-quality text-to-speech (TTS) system with limited resources using SSL features and generate a large synthetic corpus for pre-training. Experimental results demonstrate that our proposed approach effectively reduces the demand for speech data by 90% with only slight performance degradation. To the best of our knowledge, this is the first work aiming to enhance low-resource self-supervised learning in speech processing., Comment: ASRU 2023 SPARKS Workshop

Low driving voltage (Vpp), high-speed silicon microring modulator plays a critical role in energy-efficient optical interconnect and optical computing systems owing to its ultra-compact footprint and capability for on-chip wavelength-division multiplexing. However, existing silicon microring modulators usually require more than 2 V of Vpp, which is limited by the relatively weak plasma dispersion effect of silicon and the small capacitance density of the reversed PN-junction. Here we present a highly efficient metal-oxide semiconductor capacitor (MOSCAP) microring modulator through heterogeneous integration between silicon photonics and titanium-doped indium oxide, which is a high-mobility transparent conductive oxide (TCO) material with a strong plasma dispersion effect. The device is co-fabricated by Intel's photonics fab and TCO patterning processes at Oregon State University, which exhibits a high electro-optic modulation efficiency of 117 pm/V with a low VpiL of 0.12 Vcm, and consequently can be driven by an extremely low Vpp of 0.8 V. At a 11 GHz modulation bandwidth where the modulator is limited by the high parasitic capacitance, we obtained 25 Gb/s clear eye diagrams with energy efficiency of 53 fJ/bit and demonstrated 35 Gb/s open eyes with a higher driving voltage. Further optimization of the device is expected to increase the modulation bandwidth up to 52 GHz that can encode data at 100 Gb/s for next-generation, energy-efficient optical communication and computation with sub-volt driving voltage without using any high voltage swing amplifier., Comment: Main text: 16 pages, Supplementary Information: 5 pages, 11 figures, 2 tables

Boiling heat transfer is the basis of many commonly used cooling techniques. In cooling of electronic devices, for example, it is desirable to further miniaturize heat exchangers to achieve higher heat transfer, and thus it is necessary to understand boiling phenomena on shorter spatial and temporal scales. This is especially challenging at the nanometer scale because conventional imaging techniques cannot capture the dynamics of nanobubbles, owing to the Abbe diffraction limit. Here in this research, we utilize the nanopore Joule heating system that enables the generation of nanobubbles and simultaneous diagnosis of their nanosecond resolution dynamics using resistive pulse sensing. When a bias voltage is applied across a silicon nitride nanopore immersed in an aqueous salt solution, Joule heat is generated owing to the flow of ionic current. With increasing voltage, the Joule heating intensifies, and the temperature and entropy production in the pore increase. Our sensing results show that nanopore boiling follows the theory of minimum entropy production and attempts to settle to a minimum dissipative state. This results in two boiling bifurcations corresponding to the transition between different boiling states. These characteristics of nanopore boiling are represented by an "M"-shaped boiling curve, experimentally obtained from the Joule heat variation with the applied voltage. A theoretical framework is proposed to model the thermodynamics of nanopore bubbles and estimate the system dissipation which explains the four arms of the "M"-shaped boiling curve. The present study reveals that the utilization of nanopore boiling as a benchmark platform offers a valuable means for investigating the intricate boiling phenomenon and its correlation with nanoscale bubble dynamics. This would provide fundamental insights into the chaotic transition boiling regime, which is least understood., Comment: 12 pages, 4 figures

Generalized ensemble methods such as Hamiltonian replica exchange (HREX) and expanded ensemble (EE) have been shown effective in free energy calculations for various contexts, given their ability to circumvent free energy barriers via nonphysical pathways defined by states with different modified Hamiltonians. However, both HREX and EE methods come with drawbacks, such as limited flexibility in parameter specification, or the lack of parallelizability for more complicated applications. To address this challenge, we present the method of replica exchange of expanded ensembles (REXEE), which integrates the principles of HREX and EE methods by periodically exchanging coordinates of EE replicas sampling different yet overlapping sets of alchemical states. With the solvation free energy calculation of anthracene and binding free energy calculation of the CB7-10 binding complex, we show that the REXEE method achieves the same level of accuracy in free energy calculations as the HREX and EE methods, while offering enhanced flexibility and parallelizability. Additionally, we examined REXEE simulations with various setups to understand how different exchange frequencies and replica configurations influence the sampling efficiency in the fixed-weight phase and the weight convergence in the weight-updating phase. The REXEE approach can be further extended to support asynchronous parallelization schemes, allowing looser communications between larger numbers of loosely coupled processors such as cloud computing and therefore promising much more scalable and adaptive executions of alchemical free energy calculations. All algorithms for the REXEE method are available in the Python package ensemble_md, which offers an interface for REXEE simulation management without modifying the source code in GROMACS.

Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fashion and can therefore capture expressive aspects of speech that are hard to transcribe (prosody, voice styles, non-verbal vocalization). The adoption of these methods is still limited by the fact that most speech synthesis datasets are read, severely limiting spontaneity and expressivity. Here, we introduce Expresso, a high-quality expressive speech dataset for textless speech synthesis that includes both read speech and improvised dialogues rendered in 26 spontaneous expressive styles. We illustrate the challenges and potentials of this dataset with an expressive resynthesis benchmark where the task is to encode the input in low-bitrate units and resynthesize it in a target voice while preserving content and style. We evaluate resynthesis quality with automatic metrics for different self-supervised discrete encoders, and explore tradeoffs between quality, bitrate and invariance to speaker and style. All the dataset, evaluation metrics and baseline models are open source

Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in \url{https://voicebox.metademolab.com}., Comment: Accepted to NeurIPS 2023

Expanding the language coverage of speech technology has the potential to improve access to information for many more people. However, current speech technology is restricted to about one hundred languages which is a small fraction of the over 7,000 languages spoken around the world. The Massively Multilingual Speech (MMS) project increases the number of supported languages by 10-40x, depending on the task. The main ingredients are a new dataset based on readings of publicly available religious texts and effectively leveraging self-supervised learning. We built pre-trained wav2vec 2.0 models covering 1,406 languages, a single multilingual automatic speech recognition model for 1,107 languages, speech synthesis models for the same number of languages, as well as a language identification model for 4,017 languages. Experiments show that our multilingual speech recognition model more than halves the word error rate of Whisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data.

In this paper, we introduce self-distillation and online clustering for self-supervised speech representation learning (DinoSR) which combines masked language modeling, self-distillation, and online clustering. We show that these concepts complement each other and result in a strong representation learning model for speech. DinoSR first extracts contextualized embeddings from the input audio with a teacher network, then runs an online clustering system on the embeddings to yield a machine-discovered phone inventory, and finally uses the discretized tokens to guide a student network. We show that DinoSR surpasses previous state-of-the-art performance in several downstream tasks, and provide a detailed analysis of the model and the learned discrete units.

Self-supervised learning leverages unlabeled data effectively, improving label efficiency and generalization to domains without labeled data. While recent work has studied generalization to more acoustic/linguistic domains, languages, and modalities, these investigations are limited to single-source speech with one primary speaker in the recording. This paper presents Cocktail HuBERT, a self-supervised learning framework that generalizes to mixture speech using a masked pseudo source separation objective. This objective encourages the model to identify the number of sources, separate and understand the context, and infer the content of masked regions represented as discovered units. Cocktail HuBERT outperforms state-of-the-art results with 69% lower WER on multi-speaker ASR, 31% lower DER on diarization, and is competitive on single- and multi-speaker tasks from SUPERB., Comment: ICASSP 2023

We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages. It is fully transcribed and covers 6 English-to-X translation as well as 6 X-to-English translation directions. To the best of our knowledge, this is the first open benchmark for audio-visual speech-to-text translation and the largest open benchmark for multilingual audio-visual speech recognition. Our baseline results show that MuAViC is effective for building noise-robust speech recognition and translation models. We make the corpus available at https://github.com/facebookresearch/muavic.

Self-supervision has shown great potential for audio-visual speech recognition by vastly reducing the amount of labeled data required to build good systems. However, existing methods are either not entirely end-to-end or do not train joint representations of both modalities. In this paper, we introduce AV-data2vec which addresses these challenges and builds audio-visual representations based on predicting contextualized representations which has been successful in the uni-modal case. The model uses a shared transformer encoder for both audio and video and can combine both modalities to improve speech recognition. Results on LRS3 show that AV-data2vec consistently outperforms existing methods under all settings with the same amount of data and model size., Comment: 2023 ASRU