Search

The Vera C. Rubin Observatory will produce an unprecedented astronomical data set for studies of the deep and dynamic universe. Its Legacy Survey of Space and Time (LSST) will image the entire southern sky every three to four days and produce tens of petabytes of raw image data and associated calibration data over the course of the experiment's run. More than 20 terabytes of data must be stored every night, and annual campaigns to reprocess the entire dataset since the beginning of the survey will be conducted over ten years. The Production and Distributed Analysis (PanDA) system was evaluated by the Rubin Observatory Data Management team and selected to serve the Observatory's needs due to its demonstrated scalability and flexibility over the years, for its Directed Acyclic Graph (DAG) support, its support for multi-site processing, and its highly scalable complex workflows via the intelligent Data Delivery Service (iDDS). PanDA is also being evaluated for prompt processing where data must be processed within 60 seconds after image capture. This paper will briefly describe the Rubin Data Management system and its Data Facilities (DFs). Finally, it will describe in depth the work performed in order to integrate the PanDA system with the Rubin Observatory to be able to run the Rubin Science Pipelines using PanDA., Comment: 8 pages, 3 figures, 26th International Conference on Computing in High Energy & Nuclear Physics

The Vera C. Rubin Observatory is preparing to execute the most ambitious astronomical survey ever attempted, the Legacy Survey of Space and Time (LSST). Currently the final phase of construction is under way in the Chilean Andes, with the Observatory's ten-year science mission scheduled to begin in 2025. Rubin's 8.4-meter telescope will nightly scan the southern hemisphere collecting imagery in the wavelength range 320-1050 nm covering the entire observable sky every 4 nights using a 3.2 gigapixel camera, the largest imaging device ever built for astronomy. Automated detection and classification of celestial objects will be performed by sophisticated algorithms on high-resolution images to progressively produce an astronomical catalog eventually composed of 20 billion galaxies and 17 billion stars and their associated physical properties. In this article we present an overview of the system currently being constructed to perform data distribution as well as the annual campaigns which reprocess the entire image dataset collected since the beginning of the survey. These processing campaigns will utilize computing and storage resources provided by three Rubin data facilities (one in the US and two in Europe). Each year a Data Release will be produced and disseminated to science collaborations for use in studies comprising four main science pillars: probing dark matter and dark energy, taking inventory of solar system objects, exploring the transient optical sky and mapping the Milky Way. Also presented is the method by which we leverage some of the common tools and best practices used for management of large-scale distributed data processing projects in the high energy physics and astronomy communities. We also demonstrate how these tools and practices are utilized within the Rubin project in order to overcome the specific challenges faced by the Observatory., Comment: 8 pages, 2 figures, 26th International Conference on Computing in High Energy & Nuclear Physics

The astronomical community is grappling with the increasing volume and complexity of data produced by modern telescopes, due to difficulties in reducing, accessing, analyzing, and combining archives of data. To address this challenge, we propose the establishment of a coordinating body, an "entity," with the specific mission of enhancing the interoperability, archiving, distribution, and production of both astronomical data and software. This report is the culmination of a workshop held in February 2023 on the Future of Astronomical Data Infrastructure. Attended by 70 scientists and software professionals from ground-based and space-based missions and archives spanning the entire spectrum of astronomical research, the group deliberated on the prevailing state of software and data infrastructure in astronomy, identified pressing issues, and explored potential solutions. In this report, we describe the ecosystem of astronomical data, its existing flaws, and the many gaps, duplication, inconsistencies, barriers to access, drags on productivity, missed opportunities, and risks to the long-term integrity of essential data sets. We also highlight the successes and failures in a set of deep dives into several different illustrative components of the ecosystem, included as an appendix., Comment: 59 pages; please send comments and/or questions to foadi@googlegroups.com

We present the Rubin Observatory system for data storage/retrieval and pipelined code execution. The layer for data storage and retrieval is named the Butler. It consists of a relational database, known as the registry, to keep track of metadata and relations, and a system to manage where the data is located, named the datastore. Together these systems create an abstraction layer that science algorithms can be written against. This abstraction layer manages the complexities of the large data volumes expected and allows algorithms to be written independently, yet be tied together automatically into a coherent processing pipeline. This system consists of tools which execute these pipelines by transforming them into execution graphs which contain concrete data stored in the Butler. The pipeline infrastructure is designed to be scalable in nature, allowing execution on environments ranging from a laptop all the way up to multi-facility data centers. This presentation will focus on the data management aspects as well as an overview on the creation of pipelines and the corresponding execution graphs., Comment: 4 pages, submitted to Astronomical Data Analysis Software and Systems XXXII, October 2022

Data processing pipelines need to be executed at scales ranging from small runs up through large production data release runs resulting in millions of data products. As part of the Rubin Observatory's pipeline execution system, BPS is the abstraction layer that provides an interface to different Workflow Management Systems (WMS) such as HTCondor and PanDA. During the submission process, the pipeline execution system interacts with the Data Butler to produce a science-oriented execution graph from algorithmic tasks. BPS converts this execution graph to a workflow graph and then uses a WMS-specific plugin to submit and manage the workflow. Here we will discuss the architectural design of this interface and report briefly on the recent production of the Data Preview 0.2 release and how the system is used by pipeline developers., Comment: 4 pages, submitted to Astronomical Data Analysis Software and Systems XXXII, October 2022

Starting from a description of the Rubin Observatory Data Management System Architecture, and drawing on our experience with and involvement in a range of other projects including Gaia, SDSS, UKIRT, and JCMT, we derive a series of generic design patterns and lessons learned., Comment: 10 pages ADASS XXXII submission

The Rubin Observatory's Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a laptop using a local file system. The Butler and pipeline system are now in daily use during Rubin construction and early operations., Comment: 14 pages, 3 figures, submitted to Proc SPIE 12189, "Software and Cyberinfrastructure for Astronomy VII", Montreal, CA, July 2022

The Astropy Project supports and fosters the development of open-source and openly-developed Python packages that provide commonly needed functionality to the astronomical community. A key element of the Astropy Project is the core package $\texttt{astropy}$, which serves as the foundation for more specialized projects and packages. In this article, we summarize key features in the core package as of the recent major release, version 5.0, and provide major updates for the Project. We then discuss supporting a broader ecosystem of interoperable packages, including connections with several astronomical observatories and missions. We also revisit the future outlook of the Astropy Project and the current status of Learn Astropy. We conclude by raising and discussing the current and future challenges facing the Project., Comment: 43 pages, 5 figures. To appear in ApJ. The author list has two parts: the authors that made significant contributions to the writing and/or coordination of the paper, followed by maintainers of and contributors to the Astropy Project. The position in the author list does not correspond to contributions to the Astropy Project as a whole

We describe Rubin Observatory's experience with offering a data access facility (and associated services including our Science Platform) deployed on Google Cloud infrastructure as part of our pre-Operations Data Preview program., Comment: 4 pages 1 figure

The Legacy Survey of Space and Time, operated by the Vera C. Rubin Observatory, is a 10-year astronomical survey due to start operations in 2022 that will image half the sky every three nights. LSST will produce ~20TB of raw data per night which will be calibrated and analyzed in almost real time. Given the volume of LSST data, the traditional subset-download-process paradigm of data reprocessing faces significant challenges. We describe here, the first steps towards a gateway for astronomical science that would enable astronomers to analyze images and catalogs at scale. In this first step we focus on executing the Rubin LSST Science Pipelines, a collection of image and catalog processing algorithms, on Amazon Web Services (AWS). We describe our initial impressions on the performance, scalability and cost of deploying such a system in the cloud., Comment: 5 pages, 6 figures, Gateways 2020 Conference see: https://osf.io/2rqfb/

The International Virtual Observatory Alliance (IVOA) held its bi-annual Interoperability Meetings in May 2019, and in October 2019 following the ADASS XXIX conference. We provide a brief report on the status of the IVOA and the activities of the Interoperability Meetings., Comment: 4 pages, submitted to proceedings of ADASS XXIV to be published in ASP Conf. Series

Many astronomy data centres still work on filesystems. Industry has moved on; current practice in computing infrastructure is to achieve Big Data scalability using object stores rather than POSIX file systems. This presents us with opportunities for portability and reuse of software underlying processing and archive systems but it also causes problems for legacy implementations in current data centers., Comment: Submitted as decadal APC paper 2019. arXiv admin note: text overlap with arXiv:1905.05116

The International Virtual Observatory Alliance (IVOA) held its bi-annual Interoperability Meeting over two and half days prior to the ADASS 2018 conference. We provide a brief report on the status of the IVOA and the activities of the Interoperability Meeting held in College Park., Comment: 6 pages, to appear in the proceedings of Astronomical Data Analysis Software and Systems XXVIII published by ASP

Writing generic data processing pipelines requires that the algorithmic code does not ever have to know about data formats of files, or the locations of those files. At LSST we have a software system known as "the Data Butler," that abstracts these details from the software developer. Scientists can specify the dataset they want in terms they understand, such as filter, observation identifier, date of observation, and instrument name, and the Butler translates that to one or more files which are read and returned to them as a single Python object. Conversely, once they have created a new dataset they can give it back to the Butler, with a label describing its new status, and the Butler can write it in whatever format it has been configured to use. All configuration is in YAML and supports standard defaults whilst allowing overrides., Comment: 4 pages, 1 figure, submitted to proceedings of ADASS XXVIII to be published in ASP Conf. Series

The Large Synoptic Survey Telescope (LSST) is an ambitious astronomical survey with a similarly ambitious Data Management component. Data Management for LSST includes processing on both nightly and yearly cadences to generate transient alerts, deep catalogs of the static sky, and forced photometry light-curves for billions of objects at hundreds of epochs, spanning at least a decade. The algorithms running in these pipelines are individually sophisticated and interact in subtle ways. This paper provides an overview of those pipelines, focusing more on those interactions than the details of any individual algorithm., Comment: Submitted to proceedings for ADASS XXVIII

The JCMT Gould Belt Survey was one of the first Legacy Surveys with the James Clerk Maxwell Telescope in Hawaii, mapping 47 square degrees of nearby (< 500 pc) molecular clouds in both dust continuum emission at 850 $\mu$m and 450 $\mu$m, as well as a more-limited area in lines of various CO isotopologues. While molecular clouds and the material that forms stars have structures on many size scales, their larger-scale structures are difficult to observe reliably in the submillimetre regime using ground-based facilities. In this paper, we quantify the extent to which three subsequent data-reduction methods employed by the JCMT GBS accurately recover emission structures of various size scales, in particular, dense cores which are the focus of many GBS science goals. With our current best data-reduction procedure, we expect to recover 100% of structures with Gaussian sigma sizes of $\le$30" and intensity peaks of at least five times the local noise for isolated peaks of emission. The measured sizes and peak fluxes of these compact structures are reliable (within 15% of the input values), but source recovery and reliability both decrease significantly for larger emission structures and for fainter peaks. Additional factors such as source crowding have not been tested in our analysis. The most recent JCMT GBS data release includes pointing corrections, and we demonstrate that these tend to decrease the sizes and increase the peak intensities of compact sources in our dataset, mostly at a low level (several percent), but occasionally with notable improvement., Comment: Accepted for publication in ApJS

Since its inception in the early 2000, the Virtual Observatory (VO), developed as a collaboration of many national and international projects, has become a major factor in the discovery and dissemination of astronomical information worldwide. The IVOA has been coordinating all these efforts worldwide to ensure a common VO framework that enables transparent access to and interoperability of astronomy resources (data and software) around the world. The VO is not a magic solution to all astronomy data management challenges but it does bring useful solutions in many areas borne out by the fact that VO interfaces are broadly found in astronomy major data centres and projects worldwide. Astronomy data centres have been building VO services on top of their existing data services to increase interoperability with other VO-compliant data resources to take advantage of the continuous and increasing development of VO applications. VO applications have made multi-instrument and multi-wavelength science, a difficult and fruitful part of astronomy, somewhat easier. More recently, several major new astronomy projects have been directly adopting VO standards to build their data management infrastructure, giving birth to VO built-in archives. Embracing the VO framework from the beginning brings the double gain of not needing to reinvent the wheel and ensuring from the start interoperability with other astronomy VO resources. Some of the IVOA standards are also starting to be used by neighbour disciplines like planetary sciences. There is still quite a lot to be done on the VO, in particular tackling the upcoming big data challenge and how to find interoperable solutions to the new data analysis paradigm of bringing and running the software close to the data., Comment: 4 pages, no figures, ADASS 2015 conference proceedings

The unprecedented volume and rate of transient events that will be discovered by the Large Synoptic Survey Telescope (LSST) demands that the astronomical community update its followup paradigm. Alert-brokers -- automated software system to sift through, characterize, annotate and prioritize events for followup -- will be critical tools for managing alert streams in the LSST era. The Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES) is one such broker. In this work, we develop a machine learning pipeline to characterize and classify variable and transient sources only using the available multiband optical photometry. We describe three illustrative stages of the pipeline, serving the three goals of early, intermediate and retrospective classification of alerts. The first takes the form of variable vs transient categorization, the second, a multi-class typing of the combined variable and transient dataset, and the third, a purity-driven subtyping of a transient class. While several similar algorithms have proven themselves in simulations, we validate their performance on real observations for the first time. We quantitatively evaluate our pipeline on sparse, unevenly sampled, heteroskedastic data from various existing observational campaigns, and demonstrate very competitive classification performance. We describe our progress towards adapting the pipeline developed in this work into a real-time broker working on live alert streams from time-domain surveys., Comment: 33 pages, 14 figures, submitted to ApJS

The LSST software systems make extensive use of Python, with almost all of it initially being developed solely in Python 2. Since LSST will be commissioned when Python 2 is end-of-lifed it is critical that we have all our code support Python 3 before commissioning begins. Over the past year we have made significant progress in migrating the bulk of the code from the Data Management system onto Python 3. This paper presents our migration methodology, and the current status of the port, with our eventual aim to be running completely on Python 3 by early 2018. We also discuss recent modernizations to our Python codebase., Comment: 4 pages, presented at the Astronomical Data Analysis Software and Systems (ADASS) XXVII conference, Santiago, Chile, October 2017

Debris discs are evidence of the ongoing destructive collisions between planetesimals, and their presence around stars also suggests that planets exist in these systems. In this paper, we present submillimetre images of the thermal emission from debris discs that formed the SCUBA-2 Observations of Nearby Stars (SONS) survey, one of seven legacy surveys undertaken on the James Clerk Maxwell telescope between 2012 and 2015. The overall results of the survey are presented in the form of 850 microns (and 450 microns, where possible) images and fluxes for the observed fields. Excess thermal emission, over that expected from the stellar photosphere, is detected around 49 stars out of the 100 observed fields. The discs are characterised in terms of their flux density, size (radial distribution of the dust) and derived dust properties from their spectral energy distributions. The results show discs over a range of sizes, typically 1-10 times the diameter of the Edgeworth-Kuiper Belt in our Solar System. The mass of a disc, for particles up to a few millimetres in size, is uniquely obtainable with submillimetre observations and this quantity is presented as a function of the host stars' age, showing a tentative decline in mass with age. Having doubled the number of imaged discs at submillimetre wavelengths from ground-based, single dish telescope observations, one of the key legacy products from the SONS survey is to provide a comprehensive target list to observe at high angular resolution using submillimetre/millimetre interferometers (e.g., ALMA, SMA)., Comment: 61 pages, 51 figures

The SCUBA-2 Ambitious Sky Survey (SASSy) is composed of shallow 850-$\umu$m imaging using the Sub-millimetre Common-User Bolometer Array 2 (SCUBA-2) on the James Clerk Maxwell Telescope. Here we describe the extraction of a catalogue of beam-sized sources from a roughly $120\,{\rm deg}^2$ region of the Galactic plane mapped uniformly (to an rms level of about 40\,mJy), covering longitude 120\degr\,$<$\,\textit{l}\,$<$\,140\degr\ and latitude $\abs{\textit{b}}$\,$<$\,2.9\degr. We used a matched-filtering approach to increase the signal-to-noise (S/N) ratio in these noisy maps and tested the efficiency of our extraction procedure through estimates of the false discovery rate, as well as by adding artificial sources to the real images. The primary catalogue contains a total of 189 sources at 850\,$\umu$m, down to a S/N threshold of approximately 4.6. Additionally, we list 136 sources detected down to ${\rm S/N}=4.3$, but recognise that as we go lower in S/N, the reliability of the catalogue rapidly diminishes. We perform follow-up observations of some of our lower significance sources through small targeted SCUBA-2 images, and list 265 sources detected in these maps down to ${\rm S/N}=5$. This illustrates the real power of SASSy: inspecting the shallow maps for regions of 850-$\umu$m emission and then using deeper targeted images to efficiently find fainter sources. We also perform a comparison of the SASSy sources with the Planck Catalogue of Compact Sources and the \textit{IRAS} Point Source Catalogue, to determine which sources discovered in this field might be new, and hence potentially cold regions at an early stage of star formation.

The Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES) is a joint effort of NOAO and the Department of Computer Science at the University of Arizona to build prototype software to process alerts from time-domain surveys, especially LSST, to identify those alerts that must be followed up immediately. Value is added by annotating incoming alerts with existing information from previous surveys and compilations across the electromagnetic spectrum and from the history of past alerts. Comparison against a knowledge repository of properties and features of known or predicted kinds of variable phenomena is used for categorization. The architecture and algorithms being employed are described.

The LSST data management science pipelines software consists of more than 100,000 lines of Python 2 code. LSST operations will begin after support for Python 2 has been dropped by the Python community in 2020, and we must therefore plan to migrate the codebase to Python 3. During the transition period we must also support our community of active Python 2 users and this complicates the porting significantly. We have decided to use the Python future package as the basis for our port to enable support for Python 2 and Python 3 simultaneously, whilst developing with a mindset more suited to Python 3. In this paper we report on the current status of the port and the difficulties that have been encountered., Comment: 4 pages, presented at Astronomical Data Analysis Software and Systems XXVI conference, Trieste, Italy, October 2016

We present observations of the Cepheus Flare obtained as part of the James Clerk Maxwell Telescope (JCMT) Gould Belt Legacy Survey (GBLS) with the SCUBA-2 instrument. We produce a catalogue of sources found by SCUBA-2, and separate these into starless cores and protostars. We determine masses and densities for each of our sources, using source temperatures determined by the Herschel Gould Belt Survey. We compare the properties of starless cores in four different molecular clouds: L1147/58, L1172/74, L1251 and L1228. We find that the core mass functions for each region typically show shallower-than-Salpeter behaviour. We find that L1147/58 and L1228 have a high ratio of starless cores to Class II protostars, while L1251 and L1174 have a low ratio, consistent with the latter regions being more active sites of current star formation, while the former are forming stars less actively. We determine that, if modelled as thermally-supported Bonnor-Ebert spheres, most of our cores have stable configurations accessible to them. We estimate the external pressures on our cores using archival $^{13}$CO velocity dispersion measurements and find that our cores are typically pressure-confined, rather than gravitationally bound. We perform a virial analysis on our cores, and find that they typically cannot be supported against collapse by internal thermal energy alone, due primarily to the measured external pressures. This suggests that the dominant mode of internal support in starless cores in the Cepheus Flare is either non-thermal motions or internal magnetic fields., Comment: Accepted for publication by MNRAS, 29 pages, 16 figures, 8 tables

In view of increased interest in object-oriented systems for describing coordinate information, we present a description of the data model used by the Starlink AST library. AST provides a comprehensive range of facilities for attaching world co-ordinate systems to astronomical data, and for retrieving and interpreting that information in a variety of formats, including FITS-WCS. AST is a mature system that has been in use for more than 17 years, and may consequently be useful as a means of informing development of similar systems in the future.

The Large Synoptic Survey Telescope (LSST) is a large-aperture, wide-field, ground-based survey system that will image the sky in six optical bands from 320 to 1050 nm, uniformly covering approximately $18,000$deg$^2$ of the sky over 800 times. The LSST is currently under construction on Cerro Pach\'on in Chile, and expected to enter operations in 2022. Once operational, the LSST will explore a wide range of astrophysical questions, from discovering "killer" asteroids to examining the nature of Dark Energy. The LSST will generate on average 15 TB of data per night, and will require a comprehensive Data Management system to reduce the raw data to scientifically useful catalogs and images with minimum human intervention. These reductions will result in a real-time alert stream, and eleven data releases over the 10-year duration of LSST operations. To enable this processing, the LSST project is developing a new, general-purpose, high-performance, scalable, well documented, open source data processing software stack for O/IR surveys. Prototypes of this stack are already capable of processing data from existing cameras (e.g., SDSS, DECam, MegaCam), and form the basis of the Hyper-Suprime Cam (HSC) Survey data reduction pipeline., Comment: 10 pages, 3 figures, to appear in the Proceedings of ADASS XXV

The Large Synoptic Survey Telescope (LSST) is an 8-m optical ground-based telescope being constructed on Cerro Pachon in Chile. LSST will survey half the sky every few nights in six optical bands. The data will be transferred to NCSA and within 60 seconds they will be reduced using difference imaging techniques and detected transients will be announced to the community in the VOEvent format. Annual data releases will be made from all the data during the 10-year mission, with unprecedented depth of coadds and time resolution of catalogs for such a large region of sky. In this paper we present the current status of the data processing software, and describe how to obtain it., Comment: 4 pages, poster paper presented at the Astronomical Data Analysis Software and Systems XXV Conference held in October 2015 in Sydney, Australia

With the advent of modern multi-detector heterodyne instruments that can result in observations generating thousands of spectra per minute it is no longer feasible to reduce these data as individual spectra. We describe the automated data reduction procedure used to generate baselined data cubes from heterodyne data obtained at the James Clerk Maxwell Telescope. The system can automatically detect baseline regions in spectra and automatically determine regridding parameters, all without input from a user. Additionally it can detect and remove spectra suffering from transient interference effects or anomalous baselines. The pipeline is written as a set of recipes using the ORAC-DR pipeline environment with the algorithmic code using Starlink software packages and infrastructure. The algorithms presented here can be applied to other heterodyne array instruments and have been applied to data from historical JCMT heterodyne instrumentation., Comment: 18 pages, 13 figures, submitted to Monthly Notices of the Royal Astronomical Society

The General Single-Dish Data format (GSDD) was developed in the mid-1980s as a data model to support centimeter, millimeter and submillimeter instrumentation at NRAO, JCMT, the University of Arizona and IRAM. We provide an overview of the GSDD requirements and associated data model, discuss the implementation of the resultant file formats, describe its usage in the observatories and provide a retrospective on the format., Comment: 16 pages, 2 figures, accepted for publication in the Astronomy & Computing special issue on file formats

The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard is, however, showing its age. Developed in the late 1970s the FITS authors made a number of implementation choices for the format that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not appreciate the challenges which we would be facing today in astronomical computing. Difficulties we now face include, but are not limited to, having to address the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets and the need to capture significantly more complex metadata and data relationships. There are members of the community today who find some (or all) of these limitations unworkable, and have decided to move ahead with storing data in other formats. This reaction should be taken as a wakeup call to the FITS community to make changes in the FITS standard, or to see its usage fall. In this paper we detail some selected important problems which exist within the FITS standard today. It is not our intention to prescribe specific remedies to these issues; rather, we hope to call attention of the FITS and greater astronomical computing communities to these issues in the hopes that it will spur action to address them., Comment: Astronomical Data Analysis Software and Systems XXIII. Proceedings of a meeting held 29 September - 3 October 2013 at Waikoloa Beach Marriott, Hawaii, USA. Edited by N. Manset and P. Forshay ASP conference series, vol. 485, 2014, p.351

The Starlink Hierarchical Data System has been a very successful niche astronomy file format and library for over 30 years. Development of the library was frozen ten years ago when funding for Starlink was stopped and almost no-one remains who understands the implementation details. To ensure the long-term sustainability of the Starlink application software and to make the extensible N-Dimensional Data Format accessible to a broader range of users, we propose to re-implement the HDS library application interface as a layer on top of the Hierarchical Data Format version 5. We present an overview of the new implementation of version 5 of the HDS file format and describe differences between the expectations of the HDS and HDF5 library interfaces. We finish by comparing the old and new HDS implementations by looking at a comparison of file sizes and by comparing performance benchmarks., Comment: Accepted for publication in the Astronomy & Computing special issue on file formats

The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not anticipate the challenges which we are facing today in astronomical computing. Difficulties we now face include, but are not limited to, addressing the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets, and capturing significantly more complex metadata and data relationships. There are members of the community today who find some or all of these limitations unworkable, and have decided to move ahead with storing data in other formats. If this fragmentation continues, we risk abandoning the advantages of broad interoperability, and ready archivability, that the FITS format provides for astronomy. In this paper we detail some selected important problems which exist within the FITS standard today. These problems may provide insight into deeper underlying issues which reside in the format and we provide a discussion of some lessons learned. It is not our intention here to prescribe specific remedies to these issues; rather, it is to call attention of the FITS and greater astronomical computing communities to these problems in the hope that it will spur action to address them.

We present, as a case study, a description of the partnership between an observatory (JCMT) and a data centre (CADC) that led to the development of the JCMT Science Archive (JSA). The JSA is a successful example of a service designed to use Virtual Observatory (VO) technologies from the start. We describe the motivation, process and lessons learned from this approach., Comment: Accepted for publication in the second Astronomy & Computing Special Issue on the Virtual Observatory; 10 pages, 5 figures

The future of astronomy is inextricably entwined with the care and feeding of astronomical data products. Community standards such as FITS and NDF have been instrumental in the success of numerous astronomy projects. Their very success challenges us to entertain pragmatic strategies to adapt and evolve the standards to meet the aggressive data-handling requirements of facilities now being designed and built. We discuss characteristics that have made standards successful in the past, as well as desirable features for the future, and an open discussion follows., Comment: Birds of a Feather discussion at Astronomical Data Analysis Software and Systems Conference, October 6, 2014 in Calgary, Alberta

CCAT will be a large sub-millimeter telescope to be built near the ALMA site in northern Chile. The telescope must support a varied set of instrumentation including large format KID cameras, a large heterodyne array and a KID-based direct detection multi-object spectrometer. We are developing a new raw data model based on HDF5 that can cope with the expected data rates of order Gbit/s and is flexible enough to hold raw data from all planned instruments., Comment: Submitted to Proceedings of ADASS XXIV

CCAT is a large submillimetre telescope to be built near the ALMA site in northern Chile. A large-format KID camera, with up to 48,000 detectors at a single waveband sampled at about 1 kHz, will have a data rate about 50 times larger than SCUBA-2, the largest existing submillimetre camera. Creating a map from this volume of data will be a challenge, both in terms of memory and processing time required. We investigate how to extend SMURF, the iterative map-maker used for reducing SCUBA-2 observations, to a distributed-node parallel system, and estimate how the processing time scales with the number of nodes in the system., Comment: Submitted to Proceedings of ADASS XXIV

ORAC-DR is a general purpose data reduction pipeline system designed to be instrument and observatory agnostic. The pipeline works with instruments as varied as infrared integral field units, imaging arrays and spectrographs, and sub-millimeter heterodyne arrays & continuum cameras. This paper describes the architecture of the pipeline system and the implementation of the core infrastructure. We finish by discussing the lessons learned since the initial deployment of the pipeline system in the late 1990s., Comment: 11 pages, 1 figure, accepted for publication in Astronomy and Computing

The extensible N-Dimensional Data Format (NDF) was designed and developed in the late 1980s to provide a data model suitable for use in a variety of astronomy data processing applications supported by the UK Starlink Project. Starlink applications were used extensively, primarily in the UK astronomical community, and form the basis of a number of advanced data reduction pipelines today. This paper provides an overview of the historical drivers for the development of NDF and the lessons learned from using a defined hierarchical data model for many years in data reduction software, data pipelines and in data acquisition systems., Comment: 19 pages, 7 figures, submitted to the Astronomy & Computing special issue on astronomy data formats

The Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES) is a joint project of the National Optical Astronomy Observatory and the Department of Computer Science at the University of Arizona. The goal is to build the software infrastructure necessary to process and filter alerts produced by time-domain surveys, with the ultimate source of such alerts being the Large Synoptic Survey Telescope (LSST). The ANTARES broker will add value to alerts by annotating them with information from external sources such as previous surveys from across the electromagnetic spectrum. In addition, the temporal history of annotated alerts will provide further annotation for analysis. These alerts will go through a cascade of filters to select interesting candidates. For the prototype, `interesting' is defined as the rarest or most unusual alert, but future systems will accommodate multiple filtering goals. The system is designed to be flexible, allowing users to access the stream at multiple points throughout the process, and to insert custom filters where necessary. We describe the basic architecture of ANTARES and the principles that will guide development and implementation., Comment: 8 pages, 2 figures, submitted to Observatory Operations: Strategies, Processes and Systems V, (Proc. SPIE 9149), June 2014, Montreal

SPLAT-VO is a powerful graphical tool for displaying, comparing, modifying and analyzing astronomical spectra, as well as searching and retrieving spectra from services around the world using Virtual Observatory (VO) protocols and services. The development of SPLAT-VO started in 1999, as part of the Starlink StarJava initiative, sometime before that of the VO, so initial support for the VO was necessarily added once VO standards and services became available. Further developments were supported by the Joint Astronomy Centre, Hawaii until 2009. Since end of 2011 development of SPLAT-VO has been continued by the German Astrophysical Virtual Observatory, and the Astronomical Institute of the Academy of Sciences of the Czech Republic. From this time several new features have been added, including support for the latest VO protocols, along with new visualization and spectra storing capabilities. This paper presents the history of SPLAT-VO, it's capabilities, recent additions and future plans, as well as a discussion on the motivations and lessons learned up to now., Comment: 15 pages, 6 figures, accepted for publication in Astronomy & Computing

CCAT will be a 25m diameter sub-millimeter telescope capable of operating in the 0.2 to 2.1mm wavelength range. It will be located at an altitude of 5600m on Cerro Chajnantor in northern Chile near the ALMA site. The anticipated first generation instruments include large format (60,000 pixel) kinetic inductance detector (KID) cameras, a large format heterodyne array and a direct detection multi-object spectrometer. The paper describes the architecture of the CCAT software and the development strategy., Comment: 17 pages, 6 figures, to appear in Software and Cyberinfrastructure for Astronomy III, Chiozzi & Radziwill (eds), Proc. SPIE 9152, paper ID 9152-109

Current ground-based submillimeter instruments (e.g. SCUBA-2, SHARC-2 and LABOCA) have hundreds to thousands of detectors, sampled at tens to hundreds of hertz, generating up to hundreds of gigabytes per night. Since noise is correlated between detectors and in time, due to atmospheric signals and temperature oscillations, naive map-making is not applicable. In addition, the size of the data sets makes direct likelihood based inversion techniques intractable. As a result, the data reduction approach for most current submm cameras is to adopt iterative methods in order to separate noise from sky signal, and hence effectively produce astronomical images. We investigate how today's map-makers scale to the next generation of instruments, which will have tens of thousands of detectors sampled at thousands of hertz, leading to data sets of challenging size. We propose strategies for reducing such large data sets., Comment: 23rd Annual Astronomical Data Analysis Software and Systems (ADASS) Conference; to be published in ASP Conference Proceedings

CCAT is a sub-millimeter telescope to be built on Cerro Chajnantor in Chile near the ALMA site. The remote location means that all observing will be done by remote observers with the future goal of fully autonomous observing using a dynamic scheduler. The fully autonomous observing mode provides a natural means for accepting transient alert notifications for immediate follow up., Comment: 6 pages, 2 figures, to appear in proceedings of Hot-Wiring the Transient Universe 3

CCAT will be a 25-meter telescope for sub millimeter astronomy located at 5600m altitude on Cerro Chajnantor in northern Chile. CCAT will combine high sensitivity, a wide field of view, and a broad wavelength range (0.35 to 2.1mm) to provide an unprecedented capability for deep, large-area multicolor submillimeter surveys. It is planned to have a suite of instruments including large format KID cameras, a large heterodyne array and a KID-based direct detection multi-object spectrometer. The remote location drives a desire for fully autonomous observing coupled with data reduction pipelines and fast feedback to principal investigators., Comment: 4 pages, 1 figure, to be published in the Astronomical Data Analysis Software and Systems XXXIII conference proceedings (ASP Conf Ser)

We mapped five massive star forming regions with the SCUBA-2 camera on the James Clerk Maxwell Telescope (JCMT). Temperature and column density maps are obtained from the SCUBA-2 450 and 850 $\mu$m images. Most of the dense clumps we find have central temperatures below 20 K with some as cold as 8 K, suggesting that they have no internal heating due to the presence of embedded protostars. This is surprising, because at the high densities inferred from these images and at these low temperatures such clumps should be unstable, collapsing to form stars and generating internal heating. The column densities at the clump centres exceed 10$^{23}$ cm$^{-2}$, and the derived peak visual extinction values are from 25-500 mag for $\beta$ = 1.5-2.5, indicating highly opaque centres. The observed cloud gas masses range from $\sim$ 10 to 10$^{3}$ M$_{\odot}$. The outer regions of the clumps follow an $r^{-2.36\pm0.35}$ density distribution and this power-law structure is observed outside of typically 10$^{4}$ AU. All these findings suggest that these clumps are high-mass starless clumps and most likely contain high-mass starless cores., Comment: Accepted for publication in AJ