Search

The comprehension of text-rich visual scenes has become a focal point for evaluating Multi-modal Large Language Models (MLLMs) due to their widespread applications. Current benchmarks tailored to the scenario emphasize perceptual capabilities, while overlooking the assessment of cognitive abilities. To address this limitation, we introduce a Multimodal benchmark towards Text-rich visual scenes, to evaluate the Cognitive capabilities of MLLMs through visual reasoning and content-creation tasks (MCTBench). To mitigate potential evaluation bias from the varying distributions of datasets, MCTBench incorporates several perception tasks (e.g., scene text recognition) to ensure a consistent comparison of both the cognitive and perceptual capabilities of MLLMs. To improve the efficiency and fairness of content-creation evaluation, we conduct an automatic evaluation pipeline. Evaluations of various MLLMs on MCTBench reveal that, despite their impressive perceptual capabilities, their cognition abilities require enhancement. We hope MCTBench will offer the community an efficient resource to explore and enhance cognitive capabilities towards text-rich visual scenes., Comment: 12 pages, 5 figures, project page: https://github.com/xfey/MCTBench?tab=readme-ov-file

This work presents ParGo, a novel Partial-Global projector designed to connect the vision and language modalities for Multimodal Large Language Models (MLLMs). Unlike previous works that rely on global attention-based projectors, our ParGo bridges the representation gap between the separately pre-trained vision encoders and the LLMs by integrating global and partial views, which alleviates the overemphasis on prominent regions. To facilitate the effective training of ParGo, we collect a large-scale detail-captioned image-text dataset named ParGoCap-1M-PT, consisting of 1 million images paired with high-quality captions. Extensive experiments on several MLLM benchmarks demonstrate the effectiveness of our ParGo, highlighting its superiority in aligning vision and language modalities. Compared to conventional Q-Former projector, our ParGo achieves an improvement of 259.96 in MME benchmark. Furthermore, our experiments reveal that ParGo significantly outperforms other projectors, particularly in tasks that emphasize detail perception ability.

Tables contain factual and quantitative data accompanied by various structures and contents that pose challenges for machine comprehension. Previous methods generally design task-specific architectures and objectives for individual tasks, resulting in modal isolation and intricate workflows. In this paper, we present a novel large vision-language model, TabPedia, equipped with a concept synergy mechanism. In this mechanism, all the involved diverse visual table understanding (VTU) tasks and multi-source visual embeddings are abstracted as concepts. This unified framework allows TabPedia to seamlessly integrate VTU tasks, such as table detection, table structure recognition, table querying, and table question answering, by leveraging the capabilities of large language models (LLMs). Moreover, the concept synergy mechanism enables table perception-related and comprehension-related tasks to work in harmony, as they can effectively leverage the needed clues from the corresponding source perception embeddings. Furthermore, to better evaluate the VTU task in real-world scenarios, we establish a new and comprehensive table VQA benchmark, ComTQA, featuring approximately 9,000 QA pairs. Extensive quantitative and qualitative experiments on both table perception and comprehension tasks, conducted across various public benchmarks, validate the effectiveness of our TabPedia. The superior performance further confirms the feasibility of using LLMs for understanding visual tables when all concepts work in synergy. The benchmark ComTQA has been open-sourced at https://huggingface.co/datasets/ByteDance/ComTQA. The source code and model also have been released athttps://github.com/zhaowc-ustc/TabPedia., Comment: Accepted by NeurIPS 2024

Text-centric visual question answering (VQA) has made great strides with the development of Multimodal Large Language Models (MLLMs), yet open-source models still fall short of leading models like GPT4V and Gemini, partly due to a lack of extensive, high-quality instruction tuning data. To this end, we introduce a new approach for creating a massive, high-quality instruction-tuning dataset, Square-10M, which is generated using closed-source MLLMs. The data construction process, termed Square, consists of four steps: Self-Questioning, Answering, Reasoning, and Evaluation. Our experiments with Square-10M led to three key findings: 1) Our model, TextSquare, considerably surpasses open-source previous state-of-the-art Text-centric MLLMs and sets a new standard on OCRBench(62.2%). It even outperforms top-tier models like GPT4V and Gemini in 6 of 10 text-centric benchmarks. 2) Additionally, we demonstrate the critical role of VQA reasoning data in offering comprehensive contextual insights for specific questions. This not only improves accuracy but also significantly mitigates hallucinations. Specifically, TextSquare scores an average of 75.1% across four general VQA and hallucination evaluation datasets, outperforming previous state-of-the-art models. 3) Notably, the phenomenon observed in scaling text-centric VQA datasets reveals a vivid pattern: the exponential increase of instruction tuning data volume is directly proportional to the improvement in model performance, thereby validating the necessity of the dataset scale and the high quality of Square-10M.

The recent discovery of superconductivity and magnetism in trilayer rhombohedral graphene (RG) establishes an ideal, untwisted platform to study strong correlation electronic phenomena. However, the correlated effects in multilayer RG have received limited attention, and, particularly, the evolution of the correlations with increasing layer number remains an unresolved question. Here, we show the observation of layer-dependent electronic structures and correlations in RG multilayers from 3 to 9 layers by using scanning tunneling microscopy and spectroscopy. We explicitly determine layer-enhanced low-energy flat bands and interlayer coupling strength. The former directly demonstrates the further flattening of low-energy bands in thicker RG, and the later indicates the presence of varying interlayer interactions in RG multilayers. Moreover, we find significant splitting of the flat bands, ranging from ~50-80 meV, under liquid nitrogen temperature when they are partially filled, indicating the emergence of interaction-induced strongly correlated states. Particularly, the strength of the correlated states is notably enhanced in thicker RG and reaches its maximum in the six-layer, validating directly theoretical predictions and establishing abundant new candidates for strongly correlated systems. Our results provide valuable insights into the layer dependence of the electronic properties in RG, paving the way for investigating robust and highly accessible correlated phases in simpler systems., Comment: 21 pages, 4 figures

In recent years, raw video denoising has garnered increased attention due to the consistency with the imaging process and well-studied noise modeling in the raw domain. However, two problems still hinder the denoising performance. Firstly, there is no large dataset with realistic motions for supervised raw video denoising, as capturing noisy and clean frames for real dynamic scenes is difficult. To address this, we propose recapturing existing high-resolution videos displayed on a 4K screen with high-low ISO settings to construct noisy-clean paired frames. In this way, we construct a video denoising dataset (named as ReCRVD) with 120 groups of noisy-clean videos, whose ISO values ranging from 1600 to 25600. Secondly, while non-local temporal-spatial attention is beneficial for denoising, it often leads to heavy computation costs. We propose an efficient raw video denoising transformer network (RViDeformer) that explores both short and long-distance correlations. Specifically, we propose multi-branch spatial and temporal attention modules, which explore the patch correlations from local window, local low-resolution window, global downsampled window, and neighbor-involved window, and then they are fused together. We employ reparameterization to reduce computation costs. Our network is trained in both supervised and unsupervised manners, achieving the best performance compared with state-of-the-art methods. Additionally, the model trained with our proposed dataset (ReCRVD) outperforms the model trained with previous benchmark dataset (CRVD) when evaluated on the real-world outdoor noisy videos. Our code and dataset will be released after the acceptance of this work., Comment: 16 pages,15 figures

Epitaxial growth and controllable doping of wafer-scale single-crystal transition-metal dichalcogenides (TMDCs) are two central tasks for extending Moore's law beyond silicon. However, despite considerable efforts, addressing such crucial issues simultaneously under two-dimensional (2D) confinement is yet to be realized. Here we design an ingenious epitaxial strategy to synthesize record-breaking 4-inch single-crystal Fe-doped TMDCs monolayers on industry-compatible c-plane sapphire without miscut angle. In-depth characterizations and theoretical calculations reveal that the introduction of Fe significantly decreases the formation energy of parallel steps on sapphire surfaces and contributes to the edge-nucleation of unidirectional TMDCs domains (>99%). The ultrahigh electron mobility (~86 cm2 V -1 s-1) and remarkable on/off current ratio (~108) are discovered on 4-inch single-crystal Fe-MoS2 monolayers due to the ultralow contact resistance and perfect Ohmic contact with metal electrodes. This work represents a substantial leap in terms of bridging the synthesis and doping of wafer-scale single-crystal 2D semiconductors without the need for substrate miscut, which should promote the further device downscaling and extension of Moore's law., Comment: 17 pages, 5 figures

Magnetic insulators with strong perpendicular magnetic anisotropy (PMA) play a key role in exploring pure spin current phenomena and developing ultralow-dissipation spintronic devices, thereby it is highly desirable to develop new material platforms. Here we report epitaxial growth of La2/3Sr1/3MnO3 (LSMO)-SrIrO3 (SIO) composite oxide films (LSMIO) with different crystalline orientations fabricated by sequential two-target ablation process using pulsed laser deposition. The LSMIO films exhibit high crystalline quality with homogeneous mixture of LSMO and SIO at atomic level. Ferrimagnetic and insulating transport characteristics are observed, with the temperature-dependent electric resistivity well fitted by Mott variable-range-hopping model. Moreover, the LSMIO films show strong PMA. Through further constructing all perovskite oxide heterostructures of the ferrimagnetic insulator LSMIO and a strong spin-orbital coupled SIO layer, pronounced spin Hall magnetoresistance (SMR) and spin Hall-like anomalous Hall effect (SH-AHE) were observed. These results illustrate the potential application of the ferrimagnetic insulator LSMIO in developing all-oxide ultralow-dissipation spintronic devices.

Doping is of great importance to tailor the electrical properties of semiconductors. However, the present doping methodologies for organic semiconductors (OSCs) are either inefficient or can only apply to a small number of OSCs, seriously limiting their general application. Herein, we reveal a novel p-doping mechanism by investigating the interactions between the dopant trityl cation and poly(3-hexylthiophene) (P3HT). It is found that electrophilic attack of the trityl cations on thiophenes results in the formation of alkylated ions that induce electron transfer from neighboring P3HT chains, resulting in p-doping. This unique p-doping mechanism can be employed to dope various OSCs including those with high ionization energy (IE=5.8 eV). Moreover, this doping mechanism endows trityl cation with strong doping ability, leading to polaron yielding efficiency of 100 % and doping efficiency of over 80 % in P3HT. The discovery and elucidation of this novel doping mechanism not only points out that strong electrophiles are a class of efficient p-dopants for OSCs, but also provides new opportunities towards highly efficient doping of OSCs.

Unsupervised dialogue structure learning is an important and meaningful task in natural language processing. The extracted dialogue structure and process can help analyze human dialogue, and play a vital role in the design and evaluation of dialogue systems. The traditional dialogue system requires experts to manually design the dialogue structure, which is very costly. But through unsupervised dialogue structure learning, dialogue structure can be automatically obtained, reducing the cost of developers constructing dialogue process. The learned dialogue structure can be used to promote the dialogue generation of the downstream task system, and improve the logic and consistency of the dialogue robot's reply.In this paper, we propose a Bert-based unsupervised dialogue structure learning algorithm DSBERT (Dialogue Structure BERT). Different from the previous SOTA models VRNN and SVRNN, we combine BERT and AutoEncoder, which can effectively combine context information. In order to better prevent the model from falling into the local optimal solution and make the dialogue state distribution more uniform and reasonable, we also propose three balanced loss functions that can be used for dialogue structure learning. Experimental results show that DSBERT can generate a dialogue structure closer to the real structure, can distinguish sentences with different semantics and map them to different hidden states.

Topological polar vortices that are the electric analogues of magnetic objects, present great potential in applications of future nanoelectronics due to their nanometer size, anomalous dielectric response, and chirality. To enable the functionalities, it is prerequisite to manipulate the polar states and chirality by using external stimuli. Here, we probe the evolutions of polar state and chirality of polar vortices in PbTiO3/SrTiO3 superlattices under electric field by using atomically resolved in situ scanning transmission electron microscopy and phase-field simulations. We find that the adjacent clockwise and counterclockwise vortex usually have opposite chirality. The phase-field simulations suggest that the rotation reversal or axial polarization switching can lead to the chirality change. Guided by which, we experimentally validate that the vortex rotation direction can be changed by applying and subsequently removing of electric fields, offering a potential strategy to manipulate the vortex chirality. The revealed details of dynamic behavior for individual polar vortices at atomic scale and the proposed strategy for chirality manipulation provide fundamentals for future device applications.

In recent years, the supervised learning strategy for real noisy image denoising has been emerging and has achieved promising results. In contrast, realistic noise removal for raw noisy videos is rarely studied due to the lack of noisy-clean pairs for dynamic scenes. Clean video frames for dynamic scenes cannot be captured with a long-exposure shutter or averaging multi-shots as was done for static images. In this paper, we solve this problem by creating motions for controllable objects, such as toys, and capturing each static moment for multiple times to generate clean video frames. In this way, we construct a dataset with 55 groups of noisy-clean videos with ISO values ranging from 1600 to 25600. To our knowledge, this is the first dynamic video dataset with noisy-clean pairs. Correspondingly, we propose a raw video denoising network (RViDeNet) by exploring the temporal, spatial, and channel correlations of video frames. Since the raw video has Bayer patterns, we pack it into four sub-sequences, i.e RGBG sequences, which are denoised by the proposed RViDeNet separately and finally fused into a clean video. In addition, our network not only outputs a raw denoising result, but also the sRGB result by going through an image signal processing (ISP) module, which enables users to generate the sRGB result with their favourite ISPs. Experimental results demonstrate that our method outperforms state-of-the-art video and raw image denoising algorithms on both indoor and outdoor videos., Comment: CVPR2020 accepted paper

The ability to controllably manipulate the complex topological polar configurations, such as polar flux-closure via external stimuli, enables many applications in electromechanical devices and nanoelectronics including high-density information storage. Here, by using the atomically resolved in situ scanning transmission electron microscopy, we find that a polar flux-closure structure in PbTiO3/SrTiO3 superlattices films can be reversibly switched to ordinary mono ferroelectric c domain or a domain under electric field or stress. Specifically, the electric field initially drives the flux-closure move and breaks them to form intermediate a/c striped domains, while the mechanical stress firstly starts to squeeze the flux-closures to convert into small vortices at the interface and form a continues dipole wave. After the removal of the external stimuli, the flux-closure structure spontaneously returns. Our atomic study provides valuable insights into understanding the lattice-charge interactions and the competing interactions balance in these complex topological structures. Such reversible switching between the flux-closure and ordinary ferroelectric domains also provides the foundation for applications such as memories and sensors., Comment: 22 pages, 5figures

A space-time crystal has recently been observed in a superfluid Bose gas. Here we construct a variational model that allows us to describe from first principles the coupling between the radial breathing mode and the higher-order axial modes that underlies the observation of the space-time crystal. By comparing with numerical simulations we verify the validity of our variational Ansatz. From the model we determine the requirements for the observation of the space-time crystal and the Ising-like nature of the symmetry breaking involved. Also, we find the onset and growth rate of the space-time crystal, which can be compared to experiments., Comment: 10 pages, 6 figures, submission to Physical Review A

Time crystals are a phase of matter, for which the discrete time symmetry of the driving Hamiltonian is spontaneously broken. The breaking of discrete time symmetry has been observed in several experiments in driven spin systems. Here, we show the observation of a space-time crystal using ultra-cold atoms, where the periodic structure in both space and time are directly visible in the experimental images. The underlying physics in our superfluid can be described ab initio and allows for a clear identification of the mechanism that causes the spontaneous symmetry breaking. Our results pave the way for the usage of space-time crystals for the discovery of novel nonequilibrium phases of matter., Comment: 10 pages, 7 figures, article submission including Supplementary Materials

Nano-thick metallic transition metal dichalcogenides such as VS$_{2}$ are essential building blocks for constructing next-generation electronic and energy-storage applications, as well as for exploring unique physical issues associated with the dimensionality effect. However, such 2D layered materials have yet to be achieved through either mechanical exfoliation or bottom-up synthesis. Herein, we report a facile chemical vapor deposition route for direct production of crystalline VS$_{2}$ nanosheets with sub-10 nm thicknesses and domain sizes of tens of micrometers. The obtained nanosheets feature spontaneous superlattice periodicities and excellent electrical conductivities (~3$\times$10$^{3}$ S cm$^{-1}$), which has enabled a variety of applications such as contact electrodes for monolayer MoS$_{2}$ with contact resistances of ~1/4 to that of Ni/Au metals, and as supercapacitor electrodes in aqueous electrolytes showing specific capacitances as high as 8.6$\times$10$^{2}$ F g$^{-1}$. This work provides fresh insights into the delicate structure-property relationship and the broad application prospects of such metallic 2D materials., Comment: 23 pages, 5 figues

Photodetectors based on two dimensional materials have attracted growing interest. However, the sensitivity is still unsatisfactory even under high gate voltage. Here we demonstrate a MoS2 photodetector with a poly(vinylidene fluoride-trifluoroethylene) ferroelectric layer in place of the oxide layer in a traditional field effect transistor. The dark current of the photodetector is strongly suppressed by ferroelectric polarization. A high detectivity 2.21012 Jones) and photoresponsitivity (2570 A W) detector has been achieved under ZERO gate bias at a wavelength of 635 nm. Most strikingly, the band gap of few-layer MoS2 can be tuned by the ultra-high electrostatic field from the ferroelectric polarization. With this characteristic, photoresponse wavelengths of the photodetector are extended into the near infrared (0.85-1.55m). A ferroelectrics optoelectronics hybrid structure is an effective way to achieve high performance 2D electronic optoelectronic devices., Comment: 35pages,4 figures