Search

The aviation industry has experienced constant growth in air traffic since the deregulation of the U.S. airline industry in 1978. As a result, flight delays have become a major concern for airlines and passengers, leading to significant research on factors affecting flight delays such as departure, arrival, and total delays. Flight delays result in increased consumption of limited resources such as fuel, labor, and capital, and are expected to increase in the coming decades. To address the flight delay problem, this research proposes a hybrid approach that combines the feature of deep learning and classic machine learning techniques. In addition, several machine learning algorithms are applied on flight data to validate the results of proposed model. To measure the performance of the model, accuracy, precision, recall, and F1-score are calculated, and ROC and AUC curves are generated. The study also includes an extensive analysis of the flight data and each model to obtain insightful results for U.S. airlines.

Training data compositions for Large Language Models (LLMs) can significantly affect their downstream performance. However, a thorough data ablation study exploring large sets of candidate data mixtures is typically prohibitively expensive since the full effect is seen only after training the models; this can lead practitioners to settle for sub-optimal data mixtures. We propose an efficient method for approximating data ablations which trains individual models on subsets of a training corpus and reuses them across evaluations of combinations of subsets. In continued pre-training experiments, we find that, given an arbitrary evaluation set, the perplexity score of a single model trained on a candidate set of data is strongly correlated with perplexity scores of parameter averages of models trained on distinct partitions of that data. From this finding, we posit that researchers and practitioners can conduct inexpensive simulations of data ablations by maintaining a pool of models that were each trained on partitions of a large training corpus, and assessing candidate data mixtures by evaluating parameter averages of combinations of these models. This approach allows for substantial improvements in amortized training efficiency -- scaling only linearly with respect to new data -- by enabling reuse of previous training computation, opening new avenues for improving model performance through rigorous, incremental data assessment and mixing., Comment: EMNLP 2024. 17 pages

The Workflows Community Summit gathered 111 participants from 18 countries to discuss emerging trends and challenges in scientific workflows, focusing on six key areas: time-sensitive workflows, AI-HPC convergence, multi-facility workflows, heterogeneous HPC environments, user experience, and FAIR computational workflows. The integration of AI and exascale computing has revolutionized scientific workflows, enabling higher-fidelity models and complex, time-sensitive processes, while introducing challenges in managing heterogeneous environments and multi-facility data dependencies. The rise of large language models is driving computational demands to zettaflop scales, necessitating modular, adaptable systems and cloud-service models to optimize resource utilization and ensure reproducibility. Multi-facility workflows present challenges in data movement, curation, and overcoming institutional silos, while diverse hardware architectures require integrating workflow considerations into early system design and developing standardized resource management tools. The summit emphasized improving user experience in workflow systems and ensuring FAIR workflows to enhance collaboration and accelerate scientific discovery. Key recommendations include developing standardized metrics for time-sensitive workflows, creating frameworks for cloud-HPC integration, implementing distributed-by-design workflow modeling, establishing multi-facility authentication protocols, and accelerating AI integration in HPC workflow management. The summit also called for comprehensive workflow benchmarks, workflow-specific UX principles, and a FAIR workflow maturity model, highlighting the need for continued collaboration in addressing the complex challenges posed by the convergence of AI, HPC, and multi-facility research environments.

We present a quantum computational framework for SU(2) lattice gauge theory, leveraging continuous variables instead of discrete qubits to represent the infinite-dimensional Hilbert space of the gauge fields. We consider a ladder as well as a two-dimensional grid of plaquettes, detailing the use of gauge fixing to reduce the degrees of freedom and simplify the Hamiltonian. We demonstrate how the system dynamics, ground states, and energy gaps can be computed using the continuous-variable approach to quantum computing. Our results indicate that it is feasible to study non-Abelian gauge theories with continuous variables, providing new avenues for understanding the real-time dynamics of quantum field theories., Comment: 26 pages

In this article, we study a two-dimensional singularly perturbed parabolic equation of the convection-diffusion type, characterized by discontinuities in the source term and convection coefficient at a specific point in the domain. These discontinuities lead to the development of interior layers. To address these layers and ensure uniform convergence, we propose a hybrid monotone difference scheme that combines the central difference and midpoint upwind schemes for spatial discretization, applied on a piecewise-uniform Shishkin mesh. For temporal discretization, we employ the Crank-Nicolson method on a uniform mesh. The resulting scheme is proven to be uniformly convergent, order achieving almost two in space and two in time. Numerical experiments validate the theoretical error estimates, demonstrating superior accuracy and convergence when compared to existing methods., Comment: arXiv admin note: substantial text overlap with arXiv:2409.00354

Model-based planners and controllers are commonly used to solve complex manipulation problems as they can efficiently optimize diverse objectives and generalize to long horizon tasks. However, they are limited by the fidelity of their model which oftentimes leads to failures during deployment. To enable a robot to recover from such failures, we propose to use hierarchical reinforcement learning to learn a separate recovery policy. The recovery policy is triggered when a failure is detected based on sensory observations and seeks to take the robot to a state from which it can complete the task using the nominal model-based controllers. Our approach, called RecoveryChaining, uses a hybrid action space, where the model-based controllers are provided as additional \emph{nominal} options which allows the recovery policy to decide how to recover, when to switch to a nominal controller and which controller to switch to even with \emph{sparse rewards}. We evaluate our approach in three multi-step manipulation tasks with sparse rewards, where it learns significantly more robust recovery policies than those learned by baselines. Finally, we successfully transfer recovery policies learned in simulation to a physical robot to demonstrate the feasibility of sim-to-real transfer with our method., Comment: 8 pages, 9 figures

The pervasiveness of proprietary language models has raised privacy concerns for users' sensitive data, emphasizing the need for private inference (PI), where inference is performed directly on encrypted inputs. However, current PI methods face prohibitively higher communication and latency overheads, primarily due to nonlinear operations. In this paper, we present a comprehensive analysis to understand the role of nonlinearities in transformer-based decoder-only language models. We introduce AERO, a four-step architectural optimization framework that refines the existing LLM architecture for efficient PI by systematically removing nonlinearities such as LayerNorm and GELU and reducing FLOPs counts. For the first time, we propose a Softmax-only architecture with significantly fewer FLOPs tailored for efficient PI. Furthermore, we devise a novel entropy regularization technique to improve the performance of Softmax-only models. AERO achieves up to 4.23$\times$ communication and 1.94$\times$ latency reduction. We validate the effectiveness of AERO by benchmarking it against the state-of-the-art., Comment: 35 pages, 21 figures, and 9 tables. arXiv admin note: text overlap with arXiv:2410.09637

Recent works have shown that Federated Learning (FL) is vulnerable to backdoor attacks. Existing defenses cluster submitted updates from clients and select the best cluster for aggregation. However, they often rely on unrealistic assumptions regarding client submissions and sampled clients population while choosing the best cluster. We show that in realistic FL settings, state-of-the-art (SOTA) defenses struggle to perform well against backdoor attacks in FL. To address this, we highlight that backdoored submissions are adversarially biased and overconfident compared to clean submissions. We, therefore, propose an Adversarially Guided Stateful Defense (AGSD) against backdoor attacks on Deep Neural Networks (DNNs) in FL scenarios. AGSD employs adversarial perturbations to a small held-out dataset to compute a novel metric, called the trust index, that guides the cluster selection without relying on any unrealistic assumptions regarding client submissions. Moreover, AGSD maintains a trust state history of each client that adaptively penalizes backdoored clients and rewards clean clients. In realistic FL settings, where SOTA defenses mostly fail to resist attacks, AGSD mostly outperforms all SOTA defenses with minimal drop in clean accuracy (5% in the worst-case compared to best accuracy) even when (a) given a very small held-out dataset -- typically AGSD assumes 50 samples (<= 0.1% of the training data) and (b) no heldout dataset is available, and out-of-distribution data is used instead. For reproducibility, our code will be openly available at: https://github.com/hassanalikhatim/AGSD., Comment: 16 pages, Accepted at ACSAC 2024

LayerNorm is a critical component in modern large language models (LLMs) for stabilizing training and ensuring smooth optimization. However, it introduces significant challenges in mechanistic interpretability, outlier feature suppression, faithful signal propagation, and computational and communication complexity of private inference. This work explores desirable activation functions in normalization-free decoder-only LLMs. Contrary to the conventional preference for the GELU in transformer-based models, our empirical findings demonstrate an {\em opposite trend} -- ReLU significantly outperforms GELU in LayerNorm-free models, leading to an {\bf 8.2\%} perplexity improvement. We discover a key issue with GELU, where early layers experience entropic overload, leading to the under-utilization of the representational capacity of attention heads. This highlights that smoother activations like GELU are {\em ill-suited} for LayerNorm-free architectures, whereas ReLU's geometrical properties -- specialization in input space and intra-class selectivity -- lead to improved learning dynamics and better information retention in the absence of LayerNorm. This study offers key insights for optimizing transformer architectures where LayerNorm introduces significant challenges., Comment: NeurIPS 2024 Workshop on Attributing Model Behavior at Scale

We present multi-epoch optical spectropolarimetric and imaging polarimetric observations of the nearby Type II supernova (SN) 2023ixf discovered in M101 at a distance of 6.85 Mpc. The first imaging polarimetric observations were taken +2.33 days (60085.08 MJD) after the explosion, while the last imaging polarimetric data points (+73.19 and +76.19 days) were acquired after the fall from the light curve plateau. At +2.33 days there is strong evidence of circumstellar material (CSM) interaction in the spectra and the light curve. A significant level of polarization $P_r = 0.88\pm 0.06 \% $ seen during this phase indicates that this CSM is aspherical. We find that the polarization evolves with time toward the interstellar polarization level ($0.35\%$) during the photospheric phase, which suggests that the recombination photosphere is spherically symmetric. There is a jump in polarization ($P_r =0.65 \pm 0.08 \% $) at +73.19 days when the light curve falls from the plateau. This is a phase where polarimetric data is sensitive to non-spherical inner ejecta or a decrease in optical depth into the single scattering regime. We also present spectropolarimetric data that reveal line (de)polarization during most of the observed epochs. In addition, at +14.50 days we see an "inverse P Cygn" profile in the H and He line polarization, which clearly indicates the presence of asymmetrically distributed material overlying the photosphere. The overall temporal evolution of polarization is typical for Type II SNe, but the high level of polarization during the rising phase has only been observed in SN 2023ixf., Comment: 14 pages, 7 figures, submitted to ApJL, comments welcome

In this work, we propose a novel method (GLOV) enabling Large Language Models (LLMs) to act as implicit Optimizers for Vision-Langugage Models (VLMs) to enhance downstream vision tasks. Our GLOV meta-prompts an LLM with the downstream task description, querying it for suitable VLM prompts (e.g., for zero-shot classification with CLIP). These prompts are ranked according to a purity measure obtained through a fitness function. In each respective optimization step, the ranked prompts are fed as in-context examples (with their accuracies) to equip the LLM with the knowledge of the type of text prompts preferred by the downstream VLM. Furthermore, we also explicitly steer the LLM generation process in each optimization step by specifically adding an offset difference vector of the embeddings from the positive and negative solutions found by the LLM, in previous optimization steps, to the intermediate layer of the network for the next generation step. This offset vector steers the LLM generation toward the type of language preferred by the downstream VLM, resulting in enhanced performance on the downstream vision tasks. We comprehensively evaluate our GLOV on 16 diverse datasets using two families of VLMs, i.e., dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVa) models -- showing that the discovered solutions can enhance the recognition performance by up to 15.0% and 57.5% (3.8% and 21.6% on average) for these models., Comment: Code: https://github.com/jmiemirza/GLOV

Liver cirrhosis, the end stage of chronic liver disease, is characterized by extensive bridging fibrosis and nodular regeneration, leading to an increased risk of liver failure, complications of portal hypertension, malignancy and death. Early diagnosis and management of end-stage cirrhosis are significant clinical challenges. Magnetic resonance imaging (MRI) is a widely available, non-invasive imaging technique for cirrhosis assessment. However, the stage of liver fibrosis cannot be easily differentiated. Moreover, the fibrotic liver tissue (cirrhotic liver) causes significant change in liver enhancement, morphology and signal characteristics, which poses substantial challenges for the development of computer-aided diagnostic applications. Deep learning (DL) offers a promising solution for automatically segmenting and recognizing cirrhotic livers in MRI scans, potentially enabling fibrosis stage classification. However, the lack of datasets specifically focused on cirrhotic livers has hindered progress. CirrMRI600+ addresses this critical gap. This extensive dataset, the first of its kind, comprises 628 high-resolution abdominal MRI scans (310 T1-weighted and 318 T2-weighted, totaling nearly 40,000 slices) with annotated segmentation labels for cirrhotic livers. Unlike previous datasets, CirrMRI600+ specifically focuses on cirrhotic livers, capturing the complexities of this disease state. The link to the dataset is made publicly available at: https://osf.io/cuk24/. We also share 11 baseline deep learning segmentation methods used in our rigorous benchmarking experiments: https://github.com/NUBagciLab/CirrMRI600Plus.

Optimization methods are widely employed in deep learning to identify and mitigate undesired model responses. While gradient-based techniques have proven effective for image models, their application to language models is hindered by the discrete nature of the input space. This study introduces a novel optimization approach, termed the \emph{functional homotopy} method, which leverages the functional duality between model training and input generation. By constructing a series of easy-to-hard optimization problems, we iteratively solve these problems using principles derived from established homotopy methods. We apply this approach to jailbreak attack synthesis for large language models (LLMs), achieving a $20\%-30\%$ improvement in success rate over existing methods in circumventing established safe open-source models such as Llama-2 and Llama-3.

We use generative AI to extract managerial expectations about their economic outlook from over 120,000 corporate conference call transcripts. The overall measure, AI Economy Score, robustly predicts future economic indicators such as GDP growth, production, and employment, both in the short term and to 10 quarters. This predictive power is incremental to that of existing measures, including survey forecasts. Moreover, industry and firm-level measures provide valuable information about sector-specific and individual firm activities. Our findings suggest that managerial expectations carry unique insights about economic activities, with implications for both macroeconomic and microeconomic decision-making., Comment: 26 Pages, 3 Figures, 11 Tables

In this paper, we propose AutoDAN-Turbo, a black-box jailbreak method that can automatically discover as many jailbreak strategies as possible from scratch, without any human intervention or predefined scopes (e.g., specified candidate strategies), and use them for red-teaming. As a result, AutoDAN-Turbo can significantly outperform baseline methods, achieving a 74.3% higher average attack success rate on public benchmarks. Notably, AutoDAN-Turbo achieves an 88.5 attack success rate on GPT-4-1106-turbo. In addition, AutoDAN-Turbo is a unified framework that can incorporate existing human-designed jailbreak strategies in a plug-and-play manner. By integrating human-designed strategies, AutoDAN-Turbo can even achieve a higher attack success rate of 93.4 on GPT-4-1106-turbo., Comment: Pre-print. Project Page: https://autodans.github.io/AutoDAN-Turbo Code: https://github.com/SaFoLab-WISC/AutoDAN-Turbo

In 1951, G.I. Taylor modeled swimming microorganisms by hypothesizing an infinite sheet in 2D moving in a viscous medium due to a wave passing through it. This simple model not only captured the ability of microorganisms to swim due to the wavy motion of a flagella, but further development into the model captured the optimal nature of metachronal waves observed in ciliates. While the additional effects of nearby rigid boundaries and complex environments have been addressed, herein we explore the correction induced by a nearby soft boundary. Our simple model allows us to show that the magnitude of the swimming velocity gets modified near soft boundaries, and reduces for transverse waves while it increases for longitudinal waves. We further delve into the energetics of the process and the deformation of the corresponding soft boundary, highlighting the synchronization of the oscillations induced on the soft boundary with the waves passing through the sheet and the corresponding changes to the power exerted on the fluid.

Federated Learning (FL) offers a powerful strategy for training machine learning models across decentralized datasets while maintaining data privacy, yet domain shifts among clients can degrade performance, particularly in medical imaging tasks like polyp segmentation. This paper introduces a novel Frequency-Based Domain Generalization (FDG) framework, utilizing soft- and hard-thresholding in the Fourier domain to address these challenges. By applying soft- and hard-thresholding to Fourier coefficients, our method generates new images with reduced background noise and enhances the model's ability to generalize across diverse medical imaging domains. Extensive experiments demonstrate substantial improvements in segmentation accuracy and domain robustness over baseline methods. This innovation integrates frequency domain techniques into FL, presenting a resilient approach to overcoming domain variability in decentralized medical image analysis.

Personalized text-to-image diffusion models have grown popular for their ability to efficiently acquire a new concept from user-defined text descriptions and a few images. However, in the real world, a user may wish to personalize a model on multiple concepts but one at a time, with no access to the data from previous concepts due to storage/privacy concerns. When faced with this continual learning (CL) setup, most personalization methods fail to find a balance between acquiring new concepts and retaining previous ones -- a challenge that continual personalization (CP) aims to solve. Inspired by the successful CL methods that rely on class-specific information for regularization, we resort to the inherent class-conditioned density estimates, also known as diffusion classifier (DC) scores, for continual personalization of text-to-image diffusion models. Namely, we propose using DC scores for regularizing the parameter-space and function-space of text-to-image diffusion models, to achieve continual personalization. Using several diverse evaluation setups, datasets, and metrics, we show that our proposed regularization-based CP methods outperform the state-of-the-art C-LoRA, and other baselines. Finally, by operating in the replay-free CL setup and on low-rank adapters, our method incurs zero storage and parameter overhead, respectively, over the state-of-the-art., Comment: Work under review, 26 pages of manuscript

In this paper, we present an empirical study of typical spatial augmentation techniques used in self-supervised representation learning methods (both contrastive and non-contrastive), namely random crop and cutout. Our contributions are: (a) we dissociate random cropping into two separate augmentations, overlap and patch, and provide a detailed analysis on the effect of area of overlap and patch size to the accuracy on down stream tasks. (b) We offer an insight into why cutout augmentation does not learn good representation, as reported in earlier literature. Finally, based on these analysis, (c) we propose a distance-based margin to the invariance loss for learning scene-centric representations for the downstream task on object-centric distribution, showing that as simple as a margin proportional to the pixel distance between the two spatial views in the scence-centric images can improve the learned representation. Our study furthers the understanding of the spatial augmentations, and the effect of the domain-gap between the training augmentations and the test distribution., Comment: Accepted in ECCV 2024 Workshop on Out-of-distribution generalization in computer vision (OOD-CV)

We create a firm-level ChatGPT investment score, based on conference calls, that measures managers' anticipated changes in capital expenditures. We validate the score with interpretable textual content and its strong correlation with CFO survey responses. The investment score predicts future capital expenditure for up to nine quarters, controlling for Tobin's $q$ and other determinants, implying the investment score provides incremental information about firms' future investment opportunities. The investment score also separately forecasts future total, intangible, and R\&D investments. Consistent with theoretical predictions, high-investment-score firms experience significant positive short-term returns upon disclosure, and negative long-run future abnormal returns. We demonstrate ChatGPT's applicability to measure other policies, such as dividends and employment., Comment: 34 pages, 5 Figures, 18 Tables