Search

Although language model (LM) agents are demonstrating growing potential in many domains, their success in cybersecurity has been limited due to simplistic design and the lack of fundamental features for this domain. We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges. EnIGMA introduces new Agent-Computer Interfaces (ACIs) to improve the success rate on CTF challenges. We establish the novel Interactive Agent Tool concept, which enables LM agents to run interactive command-line utilities essential for these challenges. Empirical analysis of EnIGMA on over 350 CTF challenges from three different benchmarks indicates that providing a robust set of new tools with demonstration of their usage helps the LM solve complex problems and achieves state-of-the-art results on the NYU CTF and Intercode-CTF benchmarks. Finally, we discuss insights on ACI design and agent behavior on cybersecurity tasks that highlight the need to adapt real-world tools for LM agents.

Language model (LM) agents are increasingly being used to automate complicated tasks in digital environments. Just as humans benefit from powerful software applications, such as integrated development environments, for complex tasks like software engineering, we posit that LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built interfaces to the software they use. We investigate how interface design affects the performance of language model agents. As a result of this exploration, we introduce SWE-agent: a system that facilitates LM agents to autonomously use computers to solve software engineering tasks. SWE-agent's custom agent-computer interface (ACI) significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs. We evaluate SWE-agent on SWE-bench and HumanEvalFix, achieving state-of-the-art performance on both with a pass@1 rate of 12.5% and 87.7%, respectively, far exceeding the previous state-of-the-art achieved with non-interactive LMs. Finally, we provide insight on how the design of the ACI can impact agents' behavior and performance., Comment: Code, data, and demo available at https://swe-agent.com

The use of computer technology to automate the enforcement of law is a promising alternative to simplify bureaucratic procedures. However, careless automation might result in an inflexible and dehumanise law enforcement system driven by algorithms that do not account for the particularities of individuals or minorities. In this paper, we argue that hybrid smart contracts deployed to monitor rather than to blindly enforce regulations can be used to add flexibility. Enforcement is a suitable alternative only when prevention is strictly necessary; however, we argue that in many situations a corrective approach based on monitoring is more flexible and suitable. To add more flexibility, the hybrid smart contract can be programmed to stop to request the intervention of a human or of a group of them when human judgement is needed., Comment: Eleven pages in total. Four pdf figures. Paper accepted for publication in Data and Policy Journal

Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere $1.96$% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous., Comment: Data, code, and leaderboard are available at https://www.swebench.com ICLR 2024, https://openreview.net/forum?id=VTF8yNQM66

Semantic textual similarity (STS), a cornerstone task in NLP, measures the degree of similarity between a pair of sentences, and has broad application in fields such as information retrieval and natural language understanding. However, sentence similarity can be inherently ambiguous, depending on the specific aspect of interest. We resolve this ambiguity by proposing a novel task called Conditional STS (C-STS) which measures sentences' similarity conditioned on an feature described in natural language (hereon, condition). As an example, the similarity between the sentences "The NBA player shoots a three-pointer." and "A man throws a tennis ball into the air to serve." is higher for the condition "The motion of the ball" (both upward) and lower for "The size of the ball" (one large and one small). C-STS's advantages are two-fold: (1) it reduces the subjectivity and ambiguity of STS and (2) enables fine-grained language model evaluation through diverse natural language conditions. We put several state-of-the-art models to the test, and even those performing well on STS (e.g. SimCSE, Flan-T5, and GPT-4) find C-STS challenging; all with Spearman correlation scores below 50. To encourage a more comprehensive evaluation of semantic similarity and natural language understanding, we make nearly 19K C-STS examples and code available for others to train and test their models., Comment: Published in EMNLP 2023

The widespread adoption of large language models such as ChatGPT and Bard has led to unprecedented demand for these technologies. The burgeoning cost of inference for ever-increasing model sizes coupled with hardware shortages has limited affordable access and poses a pressing need for efficiency approaches geared towards high throughput and performance. Multi-input multi-output (MIMO) algorithms such as data multiplexing, offer a promising solution with a many-fold increase in throughput by performing inference for multiple inputs at the cost of a single input. Yet these approaches are not currently performant enough to be deployed in modern systems. We change that by developing MUX-PLMs, a class of high throughput pre-trained language models (PLMs) trained with data multiplexing, that can be fine-tuned for any downstream task to yield high-throughput high-performance. Our novel multiplexing and demultiplexing modules proficiently entangle and disentangle inputs, and enable high-performance high throughput \muxplms{} that are competitive with vanilla PLMs while achieving 2x/5x inference speedup with only a $1-4\%$ drop on a broad suite of tasks.

We introduce CARETS, a systematic test suite to measure consistency and robustness of modern VQA models through a series of six fine-grained capability tests. In contrast to existing VQA test sets, CARETS features balanced question generation to create pairs of instances to test models, with each pair focusing on a specific capability such as rephrasing, logical symmetry or image obfuscation. We evaluate six modern VQA systems on CARETS and identify several actionable weaknesses in model comprehension, especially with concepts such as negation, disjunction, or hypernym invariance. Interestingly, even the most sophisticated models are sensitive to aspects such as swapping the order of terms in a conjunction or varying the number of answer choices mentioned in the question. We release CARETS to be used as an extensible tool for evaluating multi-modal model robustness., Comment: ACL 2022

We perform extended numerical studies to maximize the overall photon coupling efficiency of fiber-coupled quantum dot single-photon sources emitting in the near-infrared and telecom regime. Using the finite element method, we optimize the photon extraction and fiber-coupling efficiency of quantum dot single-photon sources based on micromesas, microlenses, circular Bragg grating cavities and micropillars. The numerical simulations which consider the entire system consisting of the quantum dot source itself, the coupling lens, and the single-mode fiber yield overall photon coupling efficiencies of up to 83%. Our work provides objectified comparability of different fiber-coupled single-photon sources and proposes optimized geometries for the realization of practical and highly efficient quantum dot single-photon sources.

In this paper, we introduce data multiplexing (DataMUX), a technique that enables deep neural networks to process multiple inputs simultaneously using a single compact representation. DataMUX demonstrates that neural networks are capable of generating accurate predictions over mixtures of inputs, resulting in increased throughput with minimal extra memory requirements. Our approach uses two key components -- 1) a multiplexing layer that performs a fixed linear transformation to each input before combining them to create a mixed representation of the same size as a single input, which is then processed by the base network, and 2) a demultiplexing layer that converts the base network's output back into independent representations before producing predictions for each input. We show the viability of DataMUX for different architectures (Transformers, and to a lesser extent MLPs and CNNs) across six different tasks spanning sentence classification, named entity recognition and image classification. For instance, DataMUX for Transformers can multiplex up to $20$x/$40$x inputs, achieving $11$x/$18$x increase in throughput with minimal absolute performance drops of $<2\%$ and $<4\%$ respectively on MNLI, a natural language inference task. We also provide a theoretical construction for multiplexing in self-attention networks and analyze the effect of various design elements in DataMUX., Comment: NeurIPS 2022

The hype about Bitcoin has overrated the potential of smart contracts deployed on-blockchains (on-chains) and underrated the potential of smart contracts deployed on-Trusted Third Parties (on-TTPs). As a result, current research and development in this field is focused mainly on smart contract applications that use on-chain smart contracts. We argue that there is a large class of smart contract applications where on-TTP smart contracts are a better alternative. The problem with on-chain smart contracts is that the fully decentralised model and indelible append-only data model followed by blockchains introduces several engineering problems that are hard to solve. In these situations, the inclusion of a TTP (assuming that the application can tolerate its inconveniences) instead of a blockchain to host the smart contract simplifies the problems and offers pragmatic solutions. The intention and contribution of this paper is to shed some light on this issue. We use a hypothetical use case of a car insurance application to illustrate technical problems that are easier to solve with on-TTP smart contracts than with on-chain smart contracts., Comment: The document consists of 14 pages and includes four figures, one table and 76 references. It the work of five authors. Carlos Molina-Jimenez is the corresponding and responsible author

We suggest the re-introduction of bartering to create a cryptocurrencyless, currencyless, and moneyless economy segment. We contend that a barter economy would benefit enterprises, individuals, governments and societies. For instance, the availability of an online peer-to-peer barter marketplace would convert ordinary individuals into potential traders of both tangible and digital items and services. For example, they will be able to barter files and data that they collect. Equally motivating, they will be able to barter and re-introduce to the economy items that they no longer need such as, books, garden tools, and bikes which are normally kept and wasted in garages and sheds. We argue that most of the pieces of technology needed for building a barter system are now available, including blockchains, smart contracts, cryptography, secure multiparty computations and fair exchange protocols. However, additional research is needed to refine and integrate the pieces together. We discuss potential research directions., Comment: The document consists of 22 pages in total, including references and two figures. The author list has five authors

We report the existence of persistently rotating azimuthons in media with self-focusing Kerr and absorption nonlinearities. The nonlinear loss is balanced by power influx from the peripheral reservoir stored in slowly decaying tails of the azimuthons. These modes are excited by a superposition of two Bessel beams with opposite vorticities, $\pm s$, and slightly different conicities. The excited azimuthon exhibits opposite vorticity in its center to that of the input Bessel-beam superposition due to spontaneous inversion of the topological charge in the course of the azimuthon formation. Unlike azimuthons in loss-free media, number $N$ of rotating intensity maxima and $s$ are not mutually independent, being related by $N=2s$. The robustness of the rotating azimuthons is enhanced in comparison to similar static dissipative patterns. They can be excited in almost any transparent material, in the range of intensities for which the nonlinear absorption, induced by multiphoton absorption, is relevant. Close to the ionization threshold, the rotating azimuthons reproduce recently observed helical filaments of light in air and CS$_{2}$., Comment: To be published in Phys. Rev. A. 10 pages, 11 figures

Given a bounded open subset $\Omega$ of $\mathbb R^n$, we establish the weak closure of the affine ball $B^{\mathcal A}_p(\Omega) = \{f \in W^{1,p}_0(\Omega):\ \mathcal E_p f \leq 1\}$ with respect to the affine functional $\mathcal E_pf$ introduced by Lutwak, Yang and Zhang in [43] as well as its compactness in $L^p(\Omega)$ for any $p \geq 1$. These points use strongly the celebrated Blaschke-Santal\'{o} inequality. As counterpart, we develop the basic theory of $p$-Rayleigh quotients in bounded domains, in the affine case, for $p\geq 1$. More specifically, we establish $p$-affine versions of the Poincar\'e inequality and some of their consequences. We introduce the affine invariant $p$-Laplace operator $\Delta_p^{\mathcal A} f$ defining the Euler-Lagrange equation of the minimization problem of the $p$-affine Rayleigh quotient. We also study its first eigenvalue $\lambda^{\mathcal A}_{1,p}(\Omega)$ which satisfies the corresponding affine Faber-Krahn inequality, this is that $\lambda^{\mathcal A}_{1,p}(\Omega)$ is minimized (among sets of equal volume) only when $\Omega$ is an ellipsoid. This point depends fundamentally on PDEs regularity analysis aimed at the operator $\Delta_p^{\mathcal A} f$. We also present some comparisons between affine and classical eigenvalues, including a result of rigidity through the characterization of equality cases for $p \geq 1$. All affine inequalities obtained are stronger and directly imply the classical ones., Comment: 30 pages, comments are welcome

If $K\subset\mathbb{R}^n$ is a convex body and $\Gamma_pK$ is the $p$-centroid body of $K$, the $L_p$ Busemann-Petty centroid inequality states that $\vol(\Gamma_pK) \geq \vol(K)$, with equality if and only if $K$ is an ellipsoid centered at the origin. In this work, we prove inequalities for a type of functional $r$-mixed volume for $1 \leq r < n$, and establish as a consequence, a functional version of the $L_p$ Busemann-Petty centroid inequality. \keywords{Convex body, Moment body, Busemann-Petty centroid} }, Comment: 14 pages. Comments are welcome