Search

Recent advancements in LLM-based web agents have introduced novel architectures and benchmarks showcasing progress in autonomous web navigation and interaction. However, most existing benchmarks prioritize effectiveness and accuracy, overlooking crucial factors like safety and trustworthiness which are essential for deploying web agents in enterprise settings. The risks of unsafe web agent behavior, such as accidentally deleting user accounts or performing unintended actions in critical business operations, pose significant barriers to widespread adoption. In this paper, we present ST-WebAgentBench, a new online benchmark specifically designed to evaluate the safety and trustworthiness of web agents in enterprise contexts. This benchmark is grounded in a detailed framework that defines safe and trustworthy (ST) agent behavior, outlines how ST policies should be structured and introduces the Completion under Policies metric to assess agent performance. Our evaluation reveals that current SOTA agents struggle with policy adherence and cannot yet be relied upon for critical business applications. Additionally, we propose architectural principles aimed at improving policy awareness and compliance in web agents. We open-source this benchmark and invite the community to contribute, with the goal of fostering a new generation of safer, more trustworthy AI agents. All code, data, environment reproduction resources, and video demonstrations are available at https://sites.google.com/view/st-webagentbench/home.

Predicting the next activity in an ongoing process is one of the most common classification tasks in the business process management (BPM) domain. It allows businesses to optimize resource allocation, enhance operational efficiency, and aids in risk mitigation and strategic decision-making. This provides a competitive edge in the rapidly evolving confluence of BPM and AI. Existing state-of-the-art AI models for business process prediction do not fully capitalize on available semantic information within process event logs. As current advanced AI-BPM systems provide semantically-richer textual data, the need for novel adequate models grows. To address this gap, we propose the novel SNAP method that leverages language foundation models by constructing semantic contextual stories from the process historical event logs and using them for the next activity prediction. We compared the SNAP algorithm with nine state-of-the-art models on six benchmark datasets and show that SNAP significantly outperforms them, especially for datasets with high levels of semantic content.

Business processes that involve AI-powered automation have been gaining importance and market share in recent years. These business processes combine the characteristics of classical business process management, goal-driven chatbots, conversational recommendation systems, and robotic process automation. In the new context, prescriptive process monitoring demands innovative approaches. Unfortunately, data logs from these new processes are still not available in the public domain. We describe the main challenges in this new domain and introduce a synthesized dataset that is based on an actual use case of intelligent process automation with chatbot orchestration. Using this dataset, we demonstrate crowd-wisdom and goal-driven approaches to prescriptive process monitoring., Comment: IJCAI 2022 Workshop on Process Management in the AI era (PMAI)

Accurately inferring underlying electrophysiological (EP) tissue properties from action potential recordings is expected to be clinically useful in the diagnosis and treatment of arrhythmias such as atrial fibrillation, but it is notoriously difficult to perform. We present EP-PINNs (Physics-Informed Neural Networks), a novel tool for accurate action potential simulation and EP parameter estimation, from sparse amounts of EP data. We demonstrate, using 1D and 2D in silico data, how EP-PINNs are able to reconstruct the spatio-temporal evolution of action potentials, whilst predicting parameters related to action potential duration (APD), excitability and diffusion coefficients. EP-PINNs are additionally able to identify heterogeneities in EP properties, making them potentially useful for the detection of fibrosis and other localised pathology linked to arrhythmias. Finally, we show EP-PINNs effectiveness on biological in vitro preparations, by characterising the effect of anti-arrhythmic drugs on APD using optical mapping data. EP-PINNs are a promising clinical tool for the characterisation and potential treatment guidance of arrhythmias.