Search

Transformers have become the de facto models of choice in machine learning, typically leading to impressive performance on many applications. At the same time, the architectural development in the transformer world is mostly driven by empirical findings, and the theoretical understanding of their architectural building blocks is rather limited. In contrast, Dense Associative Memory models or Modern Hopfield Networks have a well-established theoretical foundation, but have not yet demonstrated truly impressive practical results. We propose a transformer architecture that replaces the sequence of feedforward transformer blocks with a single large Associative Memory model. Our novel architecture, called Energy Transformer (or ET for short), has many of the familiar architectural primitives that are often used in the current generation of transformers. However, it is not identical to the existing architectures. The sequence of transformer layers in ET is purposely designed to minimize a specifically engineered energy function, which is responsible for representing the relationships between the tokens. As a consequence of this computational principle, the attention in ET is different from the conventional attention mechanism. In this work, we introduce the theoretical foundations of ET, explore it's empirical capabilities using the image completion task, and obtain strong quantitative results on the graph anomaly detection task.

Additional file 1. Detection results of clinical samples using LAMP-Microfluidic Chip.

Additional file 4. PCR primers of target genes.

Additional file 1: Fig. S1. O. pumila plant materials were used for metabolite profiling. Fig. S2. Analysis of gene expression patterns and phylogenic analysis of STRs, SLSs, LAMTs enzymes in CPT biosynthesis. Fig. S3. Protein sequence alignments mentioned in this article. Fig. S4. OpLAMT assay with secologanic acid and OpSTR competion expriments. Fig. S5. The standard curve of strictosidine and strictosidinic acid. Fig. S6. The SDS-PAGE gel of purified recombinant proteins used in the chemo-enzymatic synthesis of deuterium-labeled metabolites and biochemical assay. Fig. S7. Scheme of labeled substrates synthesis. Fig. S8. Chemoenzymatic synthesis of labeled substrates. Fig. S9. Detection of d4-strictosidinic acid and d4-strictosidine in the feeding experiments in O. pumila. Fig. S10. Metabolites detection of N. nimmoniana by LC-MS. Fig. S11. Phylogenetic relationship of DNA topoisomerase I in CPT-producing and non-producing species. Table S1. Relevant Compounds Detected in O. pumila plant and hairy root. Table S2. Identification of candidate CPT biosynthetic pathway genes in O. pumila as revealed by sequence identity with characterized genes from the pre-strictosidine biosynthetic pathways in Catharanthus roseus. Table S3. Identities and similarities among STRs from C. acuminata, N. nimmoniana, and O. pumila used in this work. Table S4. Kinetic parameters of OpSTR towards secologanin and secologanic acid. Table S5. Primers list used in this study.

Keyphrase extraction is the task of finding several interesting phrases in a text document, which provide a list of the main topics within the document. Most existing graph-based models use co-occurrence links as cohesion indicators to model the relationship of syntactic elements. However, a word may have different forms of expression within the document, and may have several synonyms as well. Simply using co-occurrence information cannot capture this information. In this paper, we enhance the graph-based ranking model by leveraging word embeddings as background knowledge to add semantic information to the inter-word graph. Our approach is evaluated on established benchmark datasets and empirical results show that the word embedding neighborhood information improves the model performance.

Additional file 2:. The original, uncropped SDS-PAGE of Fig. S6.