Search

Efficient state space models (SSMs), such as linear recurrent neural networks and linear attention variants, offer computational advantages over Transformers but struggle with tasks requiring long-range in-context retrieval-like text copying, associative recall, and question answering over long contexts. Previous efforts to address these challenges have focused on architectural modifications, often reintroducing computational inefficiencies. In this paper, we propose a novel training procedure, Birdie, that significantly enhances the in-context retrieval capabilities of SSMs without altering their architecture. Our approach combines bidirectional input processing with dynamic mixtures of specialized pre-training objectives, optimized via reinforcement learning. We introduce a new bidirectional SSM architecture that seamlessly transitions from bidirectional context processing to causal generation. Experimental evaluations demonstrate that Birdie markedly improves performance on retrieval-intensive tasks such as multi-number phone book lookup, long paragraph question-answering, and infilling. This narrows the performance gap with Transformers, while retaining computational efficiency. Our findings highlight the importance of training procedures in leveraging the fixed-state capacity of SSMs, offering a new direction to advance their capabilities. All code and pre-trained models are available at https://www.github.com/samblouir/birdie, with support for JAX and PyTorch., Comment: Accepted to EMNLP 2024 (Main Conference)

Given the huge size of the generic four-dimensional scalar potentials arising from the type II supergravities based on toroidal orientifolds, it is even hard to analytically solve the extremization conditions, and therefore the previous studies have been mainly focused on taking some numerical approaches. In this work, using the so-called {\it axionic flux polynomials} we demonstrate that the scalar potential and the extremization conditions can be simplified to a great extent, leading to the possibility of performing an analytic exploration of the flux landscape. In this regard, we consider the isotropic case of a type IIB model based on the standard ${\mathbb T}^6/({\mathbb Z}_2 \times {\mathbb Z}_2)$ orientifold having the three-form fluxes $F_3/H_3$ and the non-geometric $Q$-flux. This model results in around 300 terms in the scalar potential which depend on 6 moduli/axionic fields and 14 flux parameters. Considering that the axionic flux polynomials can take either zero or non-zero values results in the need of analyzing $2^{14}=16384$ candidate configurations, and we find that more than 16200 of those result in No-Go scenarios for Minkowskian/de-Sitter vacua. Based on our systematic exploration of non-tachyonic flux vacua, we present a detailed classification of such No-Go scenarios as well as the leftover ``undecided" configurations for which we could not conclude about the presence/absence of the stable Minkowskian/de-Sitter vacua., Comment: v3: 35 pages, 5 tables, Title changed, minor updates, references added, Published version

Monolingual data, being readily available in large quantities, has been used to upscale the scarcely available parallel data to train better models for automatic translation. Self-learning, where a model is made to learn from its output, is one approach to exploit such data. However, it has been shown that too much of this data can be detrimental to the performance of the model if the available parallel data is comparatively extremely low. In this study, we investigate whether the monolingual data can also be too little and if this reduction, based on quality, has any effect on the performance of the translation model. Experiments have shown that on English-German low-resource NMT, it is often better to select only the most useful additional data, based on quality or closeness to the domain of the test data, than utilizing all of the available data.

The aim of this paper is to study the weak convergence analysis of sequence of iterates generated by a three-operator splitting method of Davis and Yin incorporated with two-step inertial extrapolation for solving monotone inclusion problem involving the sum of two maximal monotone operators and a co-coercive operator in Hilbert spaces. Our results improve on the setbacks observed recently in the literature that one-step inertial Douglas-Rachford splitting method may fail to provide acceleration. Our convergence results also dispense with the summability conditions imposed on inertial parameters and the sequence of iterates assumed in recent results on multi-step inertial methods in the literature. Numerical illustrations from image restoration problem and Smoothly Clipped Absolute Deviation (SCAD) penalty problem are given to show the efficiency and advantage gained by incorporating two-step inertial extrapolation over one-step inertial extrapolation for three-operator splitting method.

Identity Management Systems (IdMs) have complemented how users are identified, authenticated, and authorised on e-services. Among the methods used for this purpose are traditional IdMs (isolated, centralised and federated) that mostly rely on identity providers (IdPs) to broker trust between a user and service-providers (SPs). An IdP also identifies and authenticates a user on-behalf of the SP, who then determines the authorisation of the user. In these processes, both SP and IdP collect, process or store private users' data, which can be prone to breach. One approach to address the data breach is to relieve the IdP, and return control and storage of personal data to the owner. Self-sovereign identity (SSI) was introduced as an IdM model to reduce the possibility of data breaches by offering control of personal data to the owner. SSI is a decentralised IdM, where the data owner has sovereign control of personal data stored in their digital wallet. Since SSI is an emerging technology, its components and methods require careful evaluation. This paper provides an evolution to IdMs and reviews the state-of-the-art SSI frameworks. We explored articles in the literature that reviewed blockchain solutions for General Data Protection Regulation (GDPR). We systematically searched recent SSI and blockchain proposals, evaluated the compliance of the retrieved documents with the GDPR privacy principles, and discussed their potentials, constraints, and limitations. This work identifies potential research gaps and opportunities.

Mathematical models and algorithms are an essential part of mathematical research data, as they are epistemically grounding numerical data. In order to represent models and algorithms as well as their relationship semantically to make this research data FAIR, two previously distinct ontologies were merged and extended, becoming a living knowledge graph. The link between the two ontologies is established by introducing computational tasks, as they occur in modeling, corresponding to algorithmic tasks. Moreover, controlled vocabularies are incorporated and a new class, distinguishing base quantities from specific use case quantities, was introduced. Also, both models and algorithms can now be enriched with metadata. Subject-specific metadata is particularly relevant here, such as the symmetry of a matrix or the linearity of a mathematical model. This is the only way to express specific workflows with concrete models and algorithms, as the feasible solution algorithm can only be determined if the mathematical properties of a model are known. We demonstrate this using two examples from different application areas of applied mathematics. In addition, we have already integrated over 250 research assets from applied mathematics into our knowledge graph., Comment: Preprint submitted to the 18th International Conference on Metadata and Semantics Research 2024

Traffic speed forecasting is an important task in intelligent transportation system management. The objective of much of the current computational research is to minimize the difference between predicted and actual speeds, but information modalities other than speed priors are largely not taken into account. In particular, though state of the art performance is achieved on speed forecasting with graph neural network methods, these methods do not incorporate information on roadway maintenance work zones and their impacts on predicted traffic flows; yet, the impacts of construction work zones are of significant interest to roadway management agencies, because they translate to impacts on the local economy and public well-being. In this paper, we build over the convolutional graph neural network architecture and present a novel ``Graph Convolutional Network for Roadway Work Zones" model that includes a novel data fusion mechanism and a new heterogeneous graph aggregation methodology to accommodate work zone information in spatio-temporal dependencies among traffic states. The model is evaluated on two data sets that capture traffic flows in the presence of work zones in the Commonwealth of Virginia. Extensive comparative evaluation and ablation studies show that the proposed model can capture complex and nonlinear spatio-temporal relationships across a transportation corridor, outperforming baseline models, particularly when predicting traffic flow during a workzone event., Comment: Traffic speed prediction, graph neural network, spatio-temporal correlation, hypergraph, work zone, maintenance downtime. arXiv admin note: text overlap with arXiv:2110.01535

We propose symbolic regression as a powerful tool for studying Beyond the Standard Model physics. As a benchmark model, we consider the so-called Constrained Minimal Supersymmetric Standard Model, which has a four-dimensional parameter space defined at the GUT scale. We provide a set of analytical expressions that reproduce three low-energy observables of interest in terms of the parameters of the theory: the Higgs mass, the contribution to the anomalous magnetic moment of the muon, and the cold dark matter relic density. To demonstrate the power of the approach, we employ the symbolic expressions in a global fits analysis to derive the posterior probability densities of the parameters, which are obtained extremely rapidly in comparison with conventional methods., Comment: 6 pages, 7 figures, for associated code and symbolic expressions see https://gitlab.com/miguel.romao/symbolic-regression-bsm

We introduce a detection algorithm for SAGBI basis in polynomial rings, analogous to a Gr\"obner basis detection algorithm previously proposed by Gritzmann and Sturmfels. We also present two accompanying software packages named SagbiGbDetection for Macaulay2 and Julia. Both packages allow the user to find one or more term orders for which a set of input polynomials form either Gr\"obner basis for the ideal they generate or a SAGBI basis for the subalgebra. Additionally, we investigate the computational complexity of homogeneous SAGBI detection and apply our implementation to several novel examples.

In this work, we propose some new Douglas-Rashford splitting algorithms for solving a class of generalized DC (difference of convex functions) in real Hilbert spaces. The proposed methods leverage the proximal properties of the nonsmooth component and a fasten control parameter which improves the convergence rate of the algorithms. We prove the convergence of these methods to the critical points of nonconvex optimization under reasonable conditions. We evaluate the performance and effectiveness of our methods through experimentation with three practical examples in machine learning. Our findings demonstrated that our methods offer efficiency in problem-solving and outperform state-of-the-art techniques like the DCA (DC Algorithm) and ADMM.

Many recently proposed gradient projection algorithms with inertial extrapolation step for solving quasi-variational inequalities in Hilbert spaces are proven to be strongly convergent with no linear rate given when the cost operator is strongly monotone and Lipschitz continuous. In this paper, our aim is to design an inertial type gradient projection algorithm for quasi-variational inequalities and obtain its linear rate of convergence. Therefore, our results fill in the gap for linear convergence results for inertial type gradient projection algorithms for quasi variational inequalities in Hilbert spaces. We perform numerical implementations of our proposed algorithm and give numerical comparisons with other related inertial type gradient projection algorithms for quasi variational inequalities in the literature.

Despite an extensive body of literature on deep learning optimization, our current understanding of what makes an optimization algorithm effective is fragmented. In particular, we do not understand well whether enhanced optimization translates to improved generalizability. Current research overlooks the inherent stochastic nature of stochastic gradient descent (SGD) and its variants, resulting in a lack of comprehensive benchmarking and insight into their statistical performance. This paper aims to address this gap by adopting a novel approach. Rather than solely evaluating the endpoint of individual optimization trajectories, we draw from an ensemble of trajectories to estimate the stationary distribution of stochastic optimizers. Our investigation encompasses a wide array of techniques, including SGD and its variants, flat-minima optimizers, and new algorithms we propose under the Basin Hopping framework. Through our evaluation, which encompasses synthetic functions with known minima and real-world problems in computer vision and natural language processing, we emphasize fair benchmarking under a statistical framework, comparing stationary distributions and establishing statistical significance. Our study uncovers several key findings regarding the relationship between training loss and hold-out accuracy, as well as the comparable performance of SGD, noise-enabled variants, and novel optimizers utilizing the BH framework. Notably, these algorithms demonstrate performance on par with flat-minima optimizers like SAM, albeit with half the gradient evaluations. We anticipate that our work will catalyze further exploration in deep learning optimization, encouraging a shift away from single-model approaches towards methodologies that acknowledge and leverage the stochastic nature of optimizers.

Numerous successes have been achieved in combating the COVID-19 pandemic, initially using various precautionary measures like lockdowns, social distancing, and the use of face masks. More recently, various vaccinations have been developed to aid in the prevention or reduction of the severity of the COVID-19 infection. Despite the effectiveness of the precautionary measures and the vaccines, there are several controversies that are massively shared on social media platforms like Twitter. In this paper, we explore the use of state-of-the-art transformer-based language models to study people's acceptance of vaccines in Nigeria. We developed a novel dataset by crawling multi-lingual tweets using relevant hashtags and keywords. Our analysis and visualizations revealed that most tweets expressed neutral sentiments about COVID-19 vaccines, with some individuals expressing positive views, and there was no strong preference for specific vaccine types, although Moderna received slightly more positive sentiment. We also found out that fine-tuning a pre-trained LLM with an appropriate dataset can yield competitive results, even if the LLM was not initially pre-trained on the specific language of that dataset.