Search

We introduce AXS (Astronomy eXtensions for Spark), a scalable open-source astronomical data analysis framework built on Apache Spark, a state-of-the-art industry-standard engine for big data processing. In the age when the most challenging questions of the day demand repeated, complex processing of large information-rich tabular datasets, scalable and stable tools that are easy to use by domain practitioners are crucial. Building on capabilities present in Spark, AXS enables querying and analyzing almost arbitrarily large astronomical catalogs using familiar Python/AstroPy concepts, DataFrame APIs, and SQL statements. AXS supports complex analysis workflows with astronomy-specific operations such as spatial selection or on-line cross-matching. Special attention has been given to usability, from conda packaging to enabling ready-to- use cloud deployments. AXS is regularly used within the University of Washington's DIRAC Institute, enabling the analysis of ZTF (Zwicky Transient Facility) and other datasets. As an example, AXS is able to cross-match Gaia DR2 (1.7 billion rows) and SDSS (710 million rows) in 25 seconds, with the data of interest (photometry) being passed to Python routines for further processing. Here, we will present current AXS capabilities, give an overview of future plans, and discuss some implications to analysis of LSST and similarly sized datasets. The long-term goal of AXS is to enable petascale catalog and stream analyses by individual researchers and groups.

Ever since the Galactic Center Arc was resolved into its component filaments a decade ago, it has been clear that its linear structure arises from the influence of a strong magnetic field. However, the origin and nature of the contributory phenomena have remained elusive. Since what is seen is synchrotron emission from relativistic particles, of prime interest is a knowledge of the acceleration mechanism involved. Interferometric imaging of the molecular gas in the vicinity of the Arc has now provided a tantalizing clue to the Arc’s origin: molecular clumps coinciding with the endpoints of a number of the Arc’s filaments point to these clumps as the source of the relativistic particles. This suggests that as dense molecular clumps course through the ambient magnetic field at the Galactic Center, magnetic energy is liberated in their leading layers via field reconnection, precipitating rapid acceleration of free charges to high energy.