Search

As language models continue to scale, Large Language Models (LLMs) have exhibited emerging capabilities in In-Context Learning (ICL), enabling them to solve language tasks by prefixing a few in-context demonstrations (ICDs) as context. Inspired by these advancements, researchers have extended these techniques to develop Large Multimodal Models (LMMs) with ICL capabilities. However, applying ICL usually faces two major challenges: 1) using more ICDs will largely increase the inference time and 2) the performance is sensitive to the selection of ICDs. These challenges are further exacerbated in LMMs due to the integration of multiple data types and the combinational complexity of multimodal ICDs. Recently, to address these challenges, some NLP studies introduce non-learnable In-Context Vectors (ICVs) which extract useful task information from ICDs into a single vector and then insert it into the LLM to help solve the corresponding task. However, although useful in simple NLP tasks, these non-learnable methods fail to handle complex multimodal tasks like Visual Question Answering (VQA). In this study, we propose Learnable In-Context VEctor (LIVE) to distill essential task information from demonstrations, improving ICL performance in LMMs. Experiments show that LIVE can significantly reduce computational costs while enhancing accuracy in VQA tasks compared to traditional ICL and other non-learnable ICV methods. The code is available at \url{https://github.com/ForJadeForest/LIVE-Learnable-In-Context-Vector}.

Large Vision-Language Models (LVLMs) are gaining traction for their remarkable ability to process and integrate visual and textual data. Despite their popularity, the capacity of LVLMs to generate precise, fine-grained textual descriptions has not been fully explored. This study addresses this gap by focusing on \textit{distinctiveness} and \textit{fidelity}, assessing how models like Open-Flamingo, IDEFICS, and MiniGPT-4 can distinguish between similar objects and accurately describe visual features. We proposed the Textual Retrieval-Augmented Classification (TRAC) framework, which, by leveraging its generative capabilities, allows us to delve deeper into analyzing fine-grained visual description generation. This research provides valuable insights into the generation quality of LVLMs, enhancing the understanding of multimodal language models. Notably, MiniGPT-4 stands out for its better ability to generate fine-grained descriptions, outperforming the other two models in this aspect. The code is provided at \url{https://anonymous.4open.science/r/Explore_FGVDs-E277}., Comment: 11 pages, 9 figures, 6 tables. For associated code, see https://anonymous.4open.science/r/Explore_FGVDs-E277

The comprehension of nonadiabatic dynamics in polyatomic systems relies heavily on the simultaneous advancements in theoretical and experimental domains. The gas-phase electron diffraction (GUED) technique has attracted widespread attention as a promising tool for observing the photochemical and photophysical features at all-atomic level with high temporal and spatial resolutions. In this work, the GUED spectra were predicted to perform a double-blind test of accuracy in excited-state simulation for cyclobutanone based on the trajectory surface hopping method, with respect to the benchmark data obtained by upcoming MeV-GUED experiments at the Stanfold Linear Accelerator Laboratory. The results show that the ultrafast nonadiabatic dynamics occurs in the photoinduced dynamics, and two C2 and C3 channels play dominant roles in the nonadiabatic reactions of cyclobutanone. The simulated UED signal can be directly interpreted by atomic movements, providing a unique view to monitor the time-dependent evolution of the molecular structure in the femtosecond dynamics.

Inspired by the success of Large Language Models in dealing with new tasks via In-Context Learning (ICL) in NLP, researchers have also developed Large Vision-Language Models (LVLMs) with ICL capabilities. However, when implementing ICL using these LVLMs, researchers usually resort to the simplest way like random sampling to configure the in-context sequence, thus leading to sub-optimal results. To enhance the ICL performance, in this study, we use Visual Question Answering (VQA) as case study to explore diverse in-context configurations to find the powerful ones. Additionally, through observing the changes of the LVLM outputs by altering the in-context sequence, we gain insights into the inner properties of LVLMs, improving our understanding of them. Specifically, to explore in-context configurations, we design diverse retrieval methods and employ different strategies to manipulate the retrieved demonstrations. Through exhaustive experiments on three VQA datasets: VQAv2, VizWiz, and OK-VQA, we uncover three important inner properties of the applied LVLM and demonstrate which strategies can consistently improve the ICL VQA performance. Our code is provided in: https://github.com/GaryJiajia/OFv2_ICL_VQA., Comment: 8 pages, 6 figures

Singlet fission (SF) is a very significant photophysical phenomenon and possesses potential applications. In this work, we try to give the rather detailed theoretical investigation of the SF process in the stacked polyacene dimer by combining the high-level quantum chemistry calculations, and the quantum dynamics simulations based on the tensor train decomposition method. Starting from the construction of the linear vibronic coupling model, we explore the pure electronic dynamics and the vibronic dynamics in the SF processes. The role of vibrational modes in nonadiabatic dynamics is addressed. The results show that the super-exchange mechanism mediated by the charge-transfer state is found in both pure electronic dynamics and the nonadiabatic dynamics. Particularly, the vibrational modes with the frequency resonance with the adiabatic energy gap play very import roles in the SF dynamics. This work not only provides a deep and detailed understanding of the SF process, but also verifies the efficiency of the tensor train decomposition method that can serve as the reference dynamics method to explore the dynamics behaviors of complex systems.

Accurately estimating the 3D pose and shape is an essential step towards understanding animal behavior, and can potentially benefit many downstream applications, such as wildlife conservation. However, research in this area is held back by the lack of a comprehensive and diverse dataset with high-quality 3D pose and shape annotations. In this paper, we propose Animal3D, the first comprehensive dataset for mammal animal 3D pose and shape estimation. Animal3D consists of 3379 images collected from 40 mammal species, high-quality annotations of 26 keypoints, and importantly the pose and shape parameters of the SMAL model. All annotations were labeled and checked manually in a multi-stage process to ensure highest quality results. Based on the Animal3D dataset, we benchmark representative shape and pose estimation models at: (1) supervised learning from only the Animal3D data, (2) synthetic to real transfer from synthetically generated images, and (3) fine-tuning human pose and shape estimation models. Our experimental results demonstrate that predicting the 3D shape and pose of animals across species remains a very challenging task, despite significant advances in human pose estimation. Our results further demonstrate that synthetic pre-training is a viable strategy to boost the model performance. Overall, Animal3D opens new directions for facilitating future research in animal 3D pose and shape estimation, and is publicly available., Comment: 11 pages, 5 figures, link to the dataset: https://xujiacong.github.io/Animal3D/

Natural language understanding (NLU) is an essential branch of natural language processing, which relies on representations generated by pre-trained language models (PLMs). However, PLMs primarily focus on acquiring lexico-semantic information, while they may be unable to adequately handle the meaning of constructions. To address this issue, we introduce construction grammar (CxG), which highlights the pairings of form and meaning, to enrich language representation. We adopt usage-based construction grammar as the basis of our work, which is highly compatible with statistical models such as PLMs. Then a HyCxG framework is proposed to enhance language representation through a three-stage solution. First, all constructions are extracted from sentences via a slot-constraints approach. As constructions can overlap with each other, bringing redundancy and imbalance, we formulate the conditional max coverage problem for selecting the discriminative constructions. Finally, we propose a relational hypergraph attention network to acquire representation from constructional information by capturing high-order word interactions among constructions. Extensive experiments demonstrate the superiority of the proposed model on a variety of NLU tasks., Comment: Long paper, accepted at the ACL 2023

We propose to Transform Scene Graphs (TSG) into more descriptive captions. In TSG, we apply multi-head attention (MHA) to design the Graph Neural Network (GNN) for embedding scene graphs. After embedding, different graph embeddings contain diverse specific knowledge for generating the words with different part-of-speech, e.g., object/attribute embedding is good for generating nouns/adjectives. Motivated by this, we design a Mixture-of-Expert (MOE)-based decoder, where each expert is built on MHA, for discriminating the graph embeddings to generate different kinds of words. Since both the encoder and decoder are built based on the MHA, as a result, we construct a homogeneous encoder-decoder unlike the previous heterogeneous ones which usually apply Fully-Connected-based GNN and LSTM-based decoder. The homogeneous architecture enables us to unify the training configuration of the whole model instead of specifying different training strategies for diverse sub-networks as in the heterogeneous pipeline, which releases the training difficulty. Extensive experiments on the MS-COCO captioning benchmark validate the effectiveness of our TSG. The code is in: https://github.com/GaryJiajia/TSG., Comment: 12 pages, 4 figures, has been accepted by ACL 2023 main conference

Grammatical Error Correction (GEC) has been broadly applied in automatic correction and proofreading system recently. However, it is still immature in Chinese GEC due to limited high-quality data from native speakers in terms of category and scale. In this paper, we present FCGEC, a fine-grained corpus to detect, identify and correct the grammatical errors. FCGEC is a human-annotated corpus with multiple references, consisting of 41,340 sentences collected mainly from multi-choice questions in public school Chinese examinations. Furthermore, we propose a Switch-Tagger-Generator (STG) baseline model to correct the grammatical errors in low-resource settings. Compared to other GEC benchmark models, experimental results illustrate that STG outperforms them on our FCGEC. However, there exists a significant gap between benchmark models and humans that encourages future models to bridge it., Comment: Long paper, accepted at the Findings of EMNLP 2022

The supervised machine learning (ML) approach is applied to realize the trajectory-based nonadiabatic dynamics within the framework of the symmetrical quasi-classical dynamics method based on the Meyer-Miller mapping Hamiltonian (MM-SQC). After the construction of the long short-term memory recurrent neural network (LSTM-RNN) model, it is used to perform the entire trajectory evolutions from initial sampling conditions. The proposed idea is proven to be reliable and accurate in the simulations of the dynamics of several site-exciton electron-phonon coupling models, which cover two-site and three-site systems with biased and unbiased energy levels, as well as include a few or many phonon modes. The LSTM-RNN approach also shows the powerful ability to obtain the accurate and stable results for the long-time evolutions. It indicates that the LSTM-RNN model perfectly captures of dynamical correction information in the trajectory evolution in the MM-SQC dynamics. Our work provides the possibility to employ the ML methods in the simulation of the trajectory-based nonadiabatic dynamic of complex systems with a large number of degrees of freedoms.

The machine learning approaches are applied in the dynamical simulation of open quantum systems. The long short-term memory recurrent neural network (LSTM-RNN) models are used to simulate the long-time quantum dynamics, which are built based on the key information of the short-time evolution. We employ various hyperparameter optimization methods, including the simulated annealing, Bayesian optimization with tree-structured parzen estimator and random search, to achieve the automatic construction and adjustment of the LSTM-RNN models. The implementation details of three hyperparameter optimization methods are examined, and among them the simulated annealing approach is strongly recommended due to its excellent performance. The uncertainties of the machine learning models are comprehensively analyzed by the combination of bootstrap sampling and Monte-Carlo dropout approaches, which give the prediction confidence of the LSTM-RNN models in the simulation of the open quantum dynamics. This work builds an effective machine learning approach to simulate the dynamics evolution of open quantum systems. In addition, the current study provides an efficient protocol to build the optimal neural networks and to estimate the trustiness of the machine learning models.

Aspect-level sentiment analysis aims to determine the sentiment polarity towards a specific target in a sentence. The main challenge of this task is to effectively model the relation between targets and sentiments so as to filter out noisy opinion words from irrelevant targets. Most recent efforts capture relations through target-sentiment pairs or opinion spans from a word-level or phrase-level perspective. Based on the observation that targets and sentiments essentially establish relations following the grammatical hierarchy of phrase-clause-sentence structure, it is hopeful to exploit comprehensive syntactic information for better guiding the learning process. Therefore, we introduce the concept of Scope, which outlines a structural text region related to a specific target. To jointly learn structural Scope and predict the sentiment polarity, we propose a hybrid graph convolutional network (HGCN) to synthesize information from constituency tree and dependency tree, exploring the potential of linking two syntax parsing methods to enrich the representation. Experimental results on four public datasets illustrate that our HGCN model outperforms current state-of-the-art baselines., Comment: 9 pages, 5 figures, 4 tables

We develop a broadly-applicable computational method for the automatic exploration of the bimolecular multi-reaction mechanism. The current methodology mainly involves the high-energy Born-Oppenheimer molecular dynamics (BOMD) simulation and the successive reaction pathway construction. Several computational tricks are introduced, which include the selection of the reactive regions based on the electronic-structure calculations and the employment of the virtual collision-dynamics simulations with monitoring atomic distance before BOMD. These prescreening steps largely reduce the number of trajectories in the BOMD simulations and significantly save computational cost. The hidden Markov model combined with modified atomic connectivity matrix is taken for the detection of reaction events in each BOMD trajectory. Starting from several geometries close to reaction events, the further intermediate optimization and transition-state searches are conducted. The proposed method allows us to build the complicated multi-reaction mechanism of medium-sized bimolecular systems automatically. Here we examine the feasibility and efficiency of the current method by its performance in searching the mechanisms of two prototype reactions in environmental science, which are the penicillin G anion + H2O and the penicillin G anion + OH radical reactions. The result indicates that the proposed theoretical method is a powerful protocol for the automatic searching of the bimolecular reaction mechanisms for medium-sized compounds.

The recurrent neural network with the long short-term memory cell (LSTM-NN) is employed to simulate the long-time dynamics of open quantum system. The bootstrap method is applied in the LSTM-NN construction and prediction, which provides a Monte-Carlo estimation of forecasting confidence interval. Within this approach, a large number of LSTM-NNs are constructed by resampling time-series sequences that were obtained from the early-stage quantum evolution given by numerically-exact multilayer multiconfigurational time-dependent Hartree method. The built LSTM-NN ensemble is used for the reliable propagation of the long-time quantum dynamics and the simulated result is highly consistent with the exact evolution. The forecasting uncertainty that partially reflects the reliability of the LSTM-NN prediction is also given. This demonstrates the bootstrap-based LSTM-NN approach is a practical and powerful tool to propagate the long-time quantum dynamics of open systems with high accuracy and low computational cost.

We carried out extensive studies to examine the performance of the fewest-switches surface hopping method in the description of the ultrafast intersystem crossing dynamic of various singlet-triplet (S-T) models by comparison with the results of the exact full quantum dynamics. Different implementation details and some derivative approaches were examined. As expected, it is better to perform the trajectory surface hopping calculations in the spin-adiabatic representation or by the local diabatization approach, instead of in the spin-diabatic representation. The surface hopping method provides reasonable results for the short-time dynamics in the S-T model with weak spin-orbital coupling (diabatic coupling), although it does not perform well in the models with strong spin-orbital coupling (diabatic coupling). When the system accesses the S-T potential energy crossing with rather high kinetic energy, the trajectory surface hopping method tends to produce the good description of the nonadiabatic intersystem crossing dynamics. The impact of the decoherence correction on the performance of the trajectory surface hopping is system dependent. It improves the result accuracy in many cases, while its influence may also be minor for other cases., Comment: Submitted to J. Chem. Phys