Search

Survey-based measurements of the spectral energy distributions (SEDs) of galaxies have flux density estimates on badly misaligned grids in rest-frame wavelength. The shift to rest frame wavelength also causes estimated SEDs to have differing support. For many galaxies, there are sizeable wavelength regions with missing data. Finally, dim galaxies dominate typical samples and have noisy SED measurements, many near the limiting signal-to-noise level of the survey. These limitations of SED measurements shifted to the rest frame complicate downstream analysis tasks, particularly tasks requiring computation of functionals (e.g., weighted integrals) of the SEDs, such as synthetic photometry, quantifying SED similarity, and using SED measurements for photometric redshift estimation. We describe a hierarchical Bayesian framework, drawing on tools from functional data analysis, that models SEDs as a random superposition of smooth continuum basis functions (B-splines) and line features, comprising a finite-rank, nonstationary Gaussian process, measured with additive Gaussian noise. We apply this *Splines 'n Lines* (SnL) model to a collection of 678,239 galaxy SED measurements comprising the Main Galaxy Sample from the Sloan Digital Sky Survey, Data Release 17, demonstrating capability to provide continuous estimated SEDs that reliably denoise, interpolate, and extrapolate, with quantified uncertainty, including the ability to predict line features where there is missing data by leveraging correlations between line features and the entire continuum., Comment: 20 pages, 7 figures

We present a computationally efficient expectation-maximization framework for multi-frame image deconvolution and super-resolution. Our method is well adapted for processing large scale imaging data from modern astronomical surveys. Our Tensorflow implementation is flexible, benefits from advanced algorithmic solutions, and allows users to seamlessly leverage Graphical Processing Unit (GPU) acceleration, thus making it viable for use in modern astronomical software pipelines. The testbed for our method is a set of $4$K by $4$K Hyper Suprime-Cam exposures, which are closest in terms of quality to imaging data from the upcoming Rubin Observatory. The preliminary results are extremely promising: our method produces a high-fidelity non-parametric reconstruction of the night sky, from which we recover unprecedented details such as the shape of the spiral arms of galaxies, while also managing to deconvolve stars perfectly into essentially single pixels.

Recovering sharper images from blurred observations, referred to as deconvolution, is an ill-posed problem where classical approaches often produce unsatisfactory results. In ground-based astronomy, combining multiple exposures to achieve images with higher signal-to-noise ratios is complicated by the variation of point-spread functions across exposures due to atmospheric effects. We develop an unsupervised multi-frame method for denoising, deblurring, and coadding images inspired by deep generative priors. We use a carefully chosen convolutional neural network architecture that combines information from multiple observations, regularizes the joint likelihood over these observations, and allows us to impose desired constraints, such as non-negativity of pixel values in the sharp, restored image. With an eye towards the Rubin Observatory, we analyze 4K by 4K Hyper Suprime-Cam exposures and obtain preliminary results which yield promising restored images and extracted source lists.

Building on previous Bayesian approaches, we introduce a novel formulation of probabilistic cross-identification, where detections are directly associated to (hypothesized) astronomical objects in a globally optimal way. We show that this new method scales better for processing multiple catalogs than enumerating all possible candidates, especially in the limit of crowded fields, which is the most challenging observational regime for new-generation astronomy experiments such as the Rubin Observatory Legacy Survey of Space and Time (LSST). Here we study simulated catalogs where the ground-truth is known and report on the statistical and computational performance of the method. The paper is accompanied by a public software tool to perform globally optimal catalog matching based on directional data.

We describe a computational framework for hierarchical Bayesian inference with simple (typically single-plate) parametric graphical models that uses graphics processing units (GPUs) to accelerate computations, enabling deployment on very large datasets. Its C++ implementation, CUDAHM (CUDA for Hierarchical Models) exploits conditional independence between instances of a plate, facilitating massively parallel exploration of the replication parameter space using the single instruction, multiple data architecture of GPUs. It provides support for constructing Metropolis-within-Gibbs samplers that iterate between GPU-accelerated robust adaptive Metropolis sampling of plate-level parameters conditional on upper-level parameters, and Metropolis-Hastings sampling of upper-level parameters on the host processor conditional on the GPU results. CUDAHM is motivated by demographic problems in astronomy, where density estimation and linear and nonlinear regression problems must be addressed for populations of thousands to millions of objects whose features are measured with possibly complex uncertainties. We describe a thinned latent point process framework for modeling such demographic data. We demonstrate accurate GPU-accelerated parametric conditional density deconvolution for simulated populations of up to 300,000 objects in ~1 hour using a single NVIDIA Tesla K40c GPU. Supplementary material provides details about the CUDAHM API and the demonstration problem., Comment: 28 pages, 7 figures, 2 appendices

We introduce a new method to estimate the probability that an extragalactic transient source is associated with a candidate host galaxy. This approach relies solely on simple observables: sky coordinates and their uncertainties, galaxy fluxes and angular sizes. The formalism invokes Bayes' rule to calculate the posterior probability P(O_i|x) from the galaxy prior P(O), observables x, and an assumed model for the true distribution of transients in/around their host galaxies. Using simulated transients placed in the well-studied COSMOS field, we consider several agnostic and physically motivated priors and offset distributions to explore the method sensitivity. We then apply the methodology to the set of 13~fast radio bursts (FRBs) localized with an uncertainty of several arcseconds. Our methodology finds nine of these are securely associated to a single host galaxy, P(O_i|x)>0.95. We examine the observed and intrinsic properties of these secure FRB hosts, recovering similar distributions as previous works. Furthermore, we find a strong correlation between the apparent magnitude of the securely identified host galaxies and the estimated cosmic dispersion measures of the corresponding FRBs, which results from the Macquart relation. Future work with FRBs will leverage this relation and other measures from the secure hosts as priors for future associations. The methodology is generic to transient type, localization error, and image quality. We encourage its application to other transients where host galaxy associations are critical to the science, e.g. gravitational wave events, gamma-ray bursts, and supernovae. We have encoded the technique in Python on GitHub: https://github.com/FRBs/astropath., Comment: In press, ApJ; comments still welcome; Visit https://github.com/FRBs/astropath to use and build PATH

We discuss the history and lessons learned from a series of deployments of environmental sensors measuring soil parameters and CO2 fluxes over the last fifteen years, in an outdoor environment. We present the hardware and software architecture of our current Gen-3 system, and then discuss how we are simplifying the user facing part of the software, to make it easier and friendlier for the environmental scientist to be in full control of the system. Finally, we describe the current effort to build a large-scale Gen-4 sensing platform consisting of hundreds of nodes to track the environmental parameters for urban green spaces in Baltimore, Maryland., Comment: 12 pages, 16 figures, Sensornets 2021 Conference

Cross-matching catalogues from radio surveys to catalogues of sources at other wavelengths is extremely hard, because radio sources are often extended, often consist of several spatially separated components, and often no radio component is coincident with the optical/infrared host galaxy. Traditionally, the cross-matching is done by eye, but this does not scale to the millions of radio sources expected from the next generation of radio surveys. We present an innovative automated procedure, using Bayesian hypothesis testing, that models trial radio-source morphologies with putative positions of the host galaxy. This new algorithm differs from an earlier version by allowing more complex radio source morphologies, and performing a simultaneous fit over a large field. We show that this technique performs well in an unsupervised mode., Comment: 9 pages, 7 figures

The spectroscopic features of white dwarfs are formed in the thin upper layer of their stellar photosphere. These features carry information about the white dwarf's surface temperature, surface gravity, and chemical composition (hereafter 'labels'). Existing methods to determine these labels rely on complex ab-initio theoretical models which are not always publicly available. Here we present two techniques to determine atmospheric labels from white dwarf spectra: a generative fitting pipeline that interpolates theoretical spectra with artificial neural networks, and a random forest regression model using parameters derived from absorption line features. We test and compare our methods using a large catalog of white dwarfs from the Sloan Digital Sky Survey (SDSS), achieving the same accuracy and negligible bias compared to previous studies. We package our techniques into an open-source Python module 'wdtools' that provides a computationally inexpensive way to determine stellar labels from white dwarf spectra observed from any facility. We will actively develop and update our tool as more theoretical models become publicly available. We discuss applications of our tool in its present form to identify interesting outlier white dwarf systems including those with magnetic fields, helium-rich atmospheres, and double-degenerate binaries., Comment: 11 pages, 7 figures. Accepted for publication in MNRAS; updated references

Astrometric calibration of images with a small field of view is often inferior to the internal accuracy of the source detections due to the small number of accessible guide stars. One important experiment with such challenges is the Hubble Space Telescope (HST). A possible solution is to cross-calibrate overlapping fields instead of just relying on standard stars. Following the approach of \citet{2012ApJ...761..188B}, we use infinitesimal 3D rotations for fine-tuning the calibration but devise a better objective that is robust to a large number of false candidates in the initial set of associations. Using Bayesian statistics, we accommodate bad data by explicitly modeling the quality, which yields a formalism essentially identical to an $M$-estimation in robust statistics. Our results on simulated and real catalogs show great potentials for improving the HST calibration, and those with similar challenges.

Refraction by the atmosphere causes the positions of sources to depend on the airmass through which an observation was taken. This shift is dependent on the underlying spectral energy of the source and the filter or bandpass through which it is observed. Wavelength-dependent refraction within a single passband is often referred to as differential chromatic refraction (DCR). With a new generation of astronomical surveys undertaking repeated observations of the same part of the sky over a range of different airmasses and parallactic angles, DCR should be a detectable and measurable astrometric signal. In this paper we introduce a novel procedure that takes this astrometric signal and uses it to infer the underlying spectral energy distribution of a source; we solve for multiple latent images at specific wavelengths via a generalized deconvolution procedure built on robust statistics. We demonstrate the utility of such an approach for estimating a partially deconvolved image, at higher spectral resolution than the input images, for surveys such as the Large Synoptic Survey Telescope (LSST).

We present a blind multiframe image-deconvolution method based on robust statistics. The usual shortcomings of iterative optimization of the likelihood function are alleviated by minimizing the M-scale of the residuals, which achieves more uniform convergence across the image. We focus on the deconvolution of astronomical images, which are among the most challenging due to their huge dynamic ranges and the frequent presence of large noise-dominated regions in the images. We show that high-quality image reconstruction is possible even in super-resolution and without the use of traditional regularization terms. Using a robust \r{ho}-function is straightforward to implement in a streaming setting and, hence our method is applicable to the large volumes of astronomy images. The power of our method is demonstrated on observations from the Sloan Digital Sky Survey (Stripe 82) and we briefly discuss the feasibility of a pipeline based on Graphical Processing Units for the next generation of telescope surveys.

Cosmological $N$-body simulations play a vital role in studying models for the evolution of the Universe. To compare to observations and make a scientific inference, statistic analysis on large simulation datasets, e.g., finding halos, obtaining multi-point correlation functions, is crucial. However, traditional in-memory methods for these tasks do not scale to the datasets that are forbiddingly large in modern simulations. Our prior paper proposes memory-efficient streaming algorithms that can find the largest halos in a simulation with up to $10^9$ particles on a small server or desktop. However, this approach fails when directly scaling to larger datasets. This paper presents a robust streaming tool that leverages state-of-the-art techniques on GPU boosting, sampling, and parallel I/O, to significantly improve performance and scalability. Our rigorous analysis of the sketch parameters improves the previous results from finding the centers of the $10^3$ largest halos to $\sim 10^4-10^5$, and reveals the trade-offs between memory, running time and number of halos. Our experiments show that our tool can scale to datasets with up to $\sim 10^{12}$ particles while using less than an hour of running time on a single GPU Nvidia GTX 1080., Comment: preprint

Matching astronomical catalogs in crowded regions of the sky is challenging both statistically and computationally due to the many possible alternative associations. Budav\'ari and Basu (2016) modeled the two-catalog situation as an Assignment Problem and used the famous Hungarian algorithm to solve it. Here we treat cross-identification of multiple catalogs by introducing a different approach based on integer linear programming. We first test this new method on problems with two catalogs and compare with the previous results. We then test the efficacy of the new approach on problems with three catalogs. The performance and scalability of the new approach is discussed in the context of large surveys., Comment: 9 pages, 8 figure; Accepted for publication in The Astrophysical Journal

Probabilistic cross-identification has been successfully applied to a number of problems in astronomy from matching simple point sources to associating stars with unknown proper motions and even radio observations with realistic morphology. Here we study the Bayes factor for clustered objects and focus in particular on galaxies to assess the effect of typical angular correlations. Numerical calculations provide the modified relationship, which (as expected) suppresses the evidence for the associations at the shortest separations where the 2-point auto-correlation function is large. Ultimately this means that the matching probability drops at somewhat shorter scales than in previous models., Comment: Accepted for publication in Astronomy and Computing, 6 pages, 3 figures

Observational astronomy in the time-domain era faces several new challenges. One of them is the efficient use of observations obtained at multiple epochs. The work presented here addresses faint object detection with multi-epoch data, and describes an incremental strategy for separating real objects from artifacts in ongoing surveys, in situations where the single-epoch data are summaries of the full image data, such as single-epoch catalogs of flux and direction estimates for candidate sources. The basic idea is to produce low-threshold single-epoch catalogs, and use a probabilistic approach to accumulate catalog information across epochs; this is in contrast to more conventional strategies based on co-added or stacked image data across all epochs. We adopt a Bayesian approach, addressing object detection by calculating the marginal likelihoods for hypotheses asserting there is no object, or one object, in a small image patch containing at most one cataloged source at each epoch. The object-present hypothesis interprets the sources in a patch at different epochs as arising from a genuine object; the no-object (noise) hypothesis interprets candidate sources as spurious, arising from noise peaks. We study the detection probability for constant-flux objects in a simplified Gaussian noise setting, comparing results based on single exposures and stacked exposures to results based on a series of single-epoch catalog summaries. Computing the detection probability based on catalog data amounts to generalized cross-matching: it is the product of a factor accounting for matching of the estimated fluxes of candidate sources, and a factor accounting for matching of their estimated directions. We find that probabilistic fusion of multi-epoch catalog information can detect sources with only modest sacrifice in sensitivity and selectivity compared to stacking., Comment: 11 pages, 11 figures

We present a flexible template-based photometric redshift estimation framework, implemented in C#, that can be seamlessly integrated into a SQL database (or DB) server and executed on-demand in SQL. The DB integration eliminates the need to move large photometric datasets outside a database for redshift estimation, and utilizes the computational capabilities of DB hardware. The code is able to perform both maximum likelihood and Bayesian estimation, and can handle inputs of variable photometric filter sets and corresponding broad-band magnitudes. It is possible to take into account the full covariance matrix between filters, and filter zero points can be empirically calibrated using measurements with given redshifts. The list of spectral templates and the prior can be specified flexibly, and the expensive synthetic magnitude computations are done via lazy evaluation, coupled with a caching of results. Parallel execution is fully supported. For large upcoming photometric surveys such as the LSST, the ability to perform in-place photo-z calculation would be a significant advantage. Also, the efficient handling of variable filter sets is a necessity for heterogeneous databases, for example the Hubble Source Catalog, and for cross-match services such as SkyQuery. We illustrate the performance of our code on two reference photo-z estimation testing datasets, and provide an analysis of execution time and scalability with respect to different configurations. The code is available for download at https://github.com/beckrob/Photo-z-SQL., Comment: 14 pages, 5 figures. Minor revision accepted by Astronomy & Computing on 2017 March 11

Modern astronomy increasingly relies upon systematic surveys, whose dedicated telescopes continuously observe the sky across varied wavelength ranges of the electromagnetic spectrum; some surveys also observe non-electromagnetic "messengers," such as high-energy particles or gravitational waves. Stars and galaxies look different through the eyes of different instruments, and their independent measurements have to be carefully combined to provide a complete, sound picture of the multicolor and eventful universe. The association of an object's independent detections is, however, a difficult problem scientifically, computationally, and statistically, raising varied challenges across diverse astronomical applications. The fundamental problem is finding records in survey databases with directions that match to within the direction uncertainties. Such astronomical versions of the record linkage problem are known by various terms in astronomy: cross-matching, cross-identification, and directional, positional, or spatio-temporal coincidence assessment. Astronomers have developed several statistical approaches for such problems, largely independently of related developments in other disciplines. Here we review emerging approaches that compute (Bayesian) probabilities for the hypotheses of interest: possible associations, or demographic properties of a cosmic population that depend on identifying associations. Many cross-identification tasks can be formulated within a hierarchical Bayesian partition model framework, with components that explicitly account for astrophysical effects (e.g., source brightness vs. wavelength, source motion, or source extent), selection effects, and measurement error. We survey recent developments, and highlight important open areas for future research., Comment: 21 pages, 9 figures

We apply a novel spectral graph technique, that of locally-biased semi-supervised eigenvectors, to study the diversity of galaxies. This technique permits us to characterize empirically the natural variations in observed spectra data, and we illustrate how this approach can be used in an exploratory manner to highlight both large-scale global as well as small-scale local structure in Sloan Digital Sky Survey (SDSS) data. We use this method in a way that simultaneously takes into account the measurements of spectral lines as well as the continuum shape. Unlike Principal Component Analysis, this method does not assume that the Euclidean distance between galaxy spectra is a good global measure of similarity between all spectra, but instead it only assumes that local difference information between similar spectra is reliable. Moreover, unlike other nonlinear dimensionality methods, this method can be used to characterize very finely both small-scale local as well as large-scale global properties of realistic noisy data. The power of the method is demonstrated on the SDSS Main Galaxy Sample by illustrating that the derived embeddings of spectra carry an unprecedented amount of information. By using a straightforward global or unsupervised variant, we observe that the main features correlate strongly with star formation rate and that they clearly separate active galactic nuclei. Computed parameters of the method can be used to describe line strengths and their interdependencies. By using a locally-biased or semi-supervised variant, we are able to focus on typical variations around specific objects of astronomical interest. We present several examples illustrating that this approach can enable new discoveries in the data as well as a detailed understanding of very fine local structure that would otherwise be overwhelmed by large-scale noise and global trends in the data., Comment: 34 pages. A modified version of this paper has been accepted to The Astrophysical Journal

One of the outstanding challenges of cross-identification is multiplicity: detections in crowded regions of the sky are often linked to more than one candidate associations of similar likelihoods. We map the resulting maximum likelihood partitioning to the fundamental assignment problem of discrete mathematics and efficiently solve the two-way catalog-level matching in the realm of combinatorial optimization using the so-called Hungarian algorithm. We introduce the method, demonstrate its performance in a mock universe where the true associations are known, and discuss the applicability of the new procedure to large surveys., Comment: 6 pages, 4 figures, accepted for publication in the Astronomical Journal

Data from the Herschel Space Observatory is freely available to the public but no uniformly processed catalogue of the observations has been published so far. To date, the Herschel Science Archive does not contain the exact sky coverage (footprint) of individual observations and supports search for measurements based on bounding circles only. Drawing on previous experience in implementing footprint databases, we built the Herschel Footprint Database and Web Services for the Herschel Space Observatory to provide efficient search capabilities for typical astronomical queries. The database was designed with the following main goals in mind: (a) provide a unified data model for meta-data of all instruments and observational modes, (b) quickly find observations covering a selected object and its neighbourhood, (c) quickly find every observation in a larger area of the sky, (d) allow for finding solar system objects crossing observation fields. As a first step, we developed a unified data model of observations of all three Herschel instruments for all pointing and instrument modes. Then, using telescope pointing information and observational meta-data, we compiled a database of footprints. As opposed to methods using pixellation of the sphere, we represent sky coverage in an exact geometric form allowing for precise area calculations. For easier handling of Herschel observation footprints with rather complex shapes, two algorithms were implemented to reduce the outline. Furthermore, a new visualisation tool to plot footprints with various spherical projections was developed. Indexing of the footprints using Hierarchical Triangular Mesh makes it possible to quickly find observations based on sky coverage, time and meta-data. The database is accessible via a web site (http://herschel.vo.elte.hu) and also as a set of REST web service functions., Comment: Accepted for publication in Experimental Astronomy

We propose a new method of constraining the redshifts of individual extragalactic sources based on celestial coordinates and their ensemble statistics. Techniques from integer linear programming are utilized to optimize simultaneously for the angular two-point cross- and autocorrelation functions. Our novel formalism introduced here not only transforms the otherwise hopelessly expensive, brute-force combinatorial search into a linear system with integer constraints but also is readily implementable in off-the-shelf solvers. We adopt Gurobi, a commercial optimization solver, and use Python to build the cost function dynamically. The preliminary results on simulated data show potential for future applications to sky surveys by complementing and enhancing photometric redshift estimators. Our approach is the first application of integer linear programming to astronomical analysis., Comment: 10 pages with 3 figures, accepted for publication in The Astronomical Journal on 08/04/16

We present the methodology and data behind the photometric redshift database of the Sloan Digital Sky Survey Data Release 12 (SDSS DR12). We adopt a hybrid technique, empirically estimating the redshift via local regression on a spectroscopic training set, then fitting a spectrum template to obtain K-corrections and absolute magnitudes. The SDSS spectroscopic catalog was augmented with data from other, publicly available spectroscopic surveys to mitigate target selection effects. The training set is comprised of $1,976,978$ galaxies, and extends up to redshift $z\approx 0.8$, with a useful coverage of up to $z\approx 0.6$. We provide photometric redshifts and realistic error estimates for the $208,474,076$ galaxies of the SDSS primary photometric catalog. We achieve an average bias of $\overline{\Delta z_{\mathrm{norm}}} = 5.84 \times 10^{-5}$, a standard deviation of $\sigma \left(\Delta z_{\mathrm{norm}}\right)=0.0205$, and a $3\sigma$ outlier rate of $P_o=4.11\%$ when cross-validating on our training set. The published redshift error estimates and photometric error classes enable the selection of galaxies with high quality photometric redshifts. We also provide a supplementary error map that allows additional, sophisticated filtering of the data., Comment: 12 pages, 7 figures. Original submitted to MNRAS on 2016 March 03, revision submitted on 2016 April 21

The Hubble Source Catalog is designed to help optimize science from the Hubble Space Telescope by combining the tens of thousands of visit-based source lists in the Hubble Legacy Archive into a single master catalog. Version 1 of the Hubble Source Catalog includes WFPC2, ACS/WFC, WFC3/UVIS, and WFC3/IR photometric data generated using SExtractor software to produce the individual source lists. The catalog includes roughly 80 million detections of 30 million objects involving 112 different detector/filter combinations, and about 160 thousand HST exposures. Source lists from Data Release 8 of the Hubble Legacy Archive are matched using an algorithm developed by Budavari & Lubow (2012). The mean photometric accuracy for the catalog as a whole is better than 0.10 mag, with relative accuracy as good as 0.02 mag in certain circumstances (e.g., bright isolated stars). The relative astrometric residuals are typically within 10 mas, with a value for the mode (i.e., most common value) of 2.3 mas. The absolute astrometric accuracy is better than $\sim$0.1 arcsec for most sources, but can be much larger for a fraction of fields that could not be matched to the PanSTARRS, SDSS, or 2MASS reference systems. In this paper we describe the database design with emphasis on those aspects that enable the users to fully exploit the catalog while avoiding common misunderstandings and potential pitfalls. We provide usage examples to illustrate some of the science capabilities and data quality characteristics, and briefly discuss plans for future improvements to the Hubble Source Catalog., Comment: 25 pages, 30 figures, 2 tables, 4 appendices; AJ accepted

We apply clustering-based redshift inference to all extended sources from the Sloan Digital Sky Survey photometric catalogue, down to magnitude r = 22. We map the relationships between colours and redshift, without assumption of the sources' spectral energy distributions (SED). We identify and locate star-forming, quiescent galaxies, and AGN, as well as colour changes due to spectral features, such as the 4000 \AA{} break, redshifting through specific filters. Our mapping is globally in good agreement with colour-redshift tracks computed with SED templates, but reveals informative differences, such as the need for a lower fraction of M-type stars in certain templates. We compare our clustering-redshift estimates to photometric redshifts and find these two independent estimators to be in good agreement at each limiting magnitude considered. Finally, we present the global clustering-redshift distribution of all Sloan extended sources, showing objects up to z ~ 0.8. While the overall shape agrees with that inferred from photometric redshifts, the clustering redshift technique results in a smoother distribution, with no indication of structure in redshift space suggested by the photometric redshift estimates (likely artifacts imprinted by their spectroscopic training set). We also infer a higher fraction of high redshift objects. The mapping between the four observed colours and redshift can be used to estimate the redshift probability distribution function of individual galaxies. This work is an initial step towards producing a general mapping between redshift and all available observables in the photometric space, including brightness, size, concentration, and ellipticity., Comment: 12 pages, 9 figures, accepted to MNRAS

Crossmatching catalogs at different wavelengths is a difficult problem in astronomy, especially when the objects are not point-like. At radio wavelengths an object can have several components corresponding, for example, to a core and lobes. {Considering not all radio detections correspond to visible or infrared sources, matching these catalogs can be challenging.} Traditionally this is done by eye for better quality, which does not scale to the large data volumes expected from the next-generation of radio telescopes. We present a novel automated procedure, using Bayesian hypothesis testing, to achieve reliable associations by explicit modelling of a particular class of radio-source morphology. {The new algorithm not only assesses the likelihood of an association between data at two different wavelengths, but also tries to assess whether different radio sources are physically associated, are double-lobed radio galaxies, or just distinct nearby objects.} Application to the SWIRE and ATLAS CDF-S catalogs shows that this method performs well without human intervention., Comment: 8 pages, 7 figures

We present a case study about the spatial indexing and regional classification of billions of geographic coordinates from geo-tagged social network data using Hierarchical Triangular Mesh (HTM) implemented for Microsoft SQL Server. Due to the lack of certain features of the HTM library, we use it in conjunction with the GIS functions of SQL Server to significantly increase the efficiency of pre-filtering of spatial filter and join queries. For example, we implemented a new algorithm to compute the HTM tessellation of complex geographic regions and precomputed the intersections of HTM triangles and geographic regions for faster false-positive filtering. With full control over the index structure, HTM-based pre-filtering of simple containment searches outperforms SQL Server spatial indices by a factor of ten and HTM-based spatial joins run about a hundred times faster., Comment: appears in Proceedings of the 26th International Conference on Scientific and Statistical Database Management (2014)

We present photometric redshifts and associated probability distributions for all detected sources in the Extended Chandra Deep Field South (ECDFS). The work makes use of the most up-to-date data from the Cosmic Assembly Near-IR Deep Legacy Survey (CANDELS) and the Taiwan ECDFS Near-Infrared Survey (TENIS) in addition to other data. We also revisit multi-wavelength counterparts for published X-ray sources from the 4Ms-CDFS and 250ks-ECDFS surveys, finding reliable counterparts for 1207 out of 1259 sources ($\sim 96\%$). Data used for photometric redshifts include intermediate-band photometry deblended using the TFIT method, which is used for the first time in this work. Photometric redshifts for X-ray source counterparts are based on a new library of AGN/galaxy hybrid templates appropriate for the faint X-ray population in the CDFS. Photometric redshift accuracy for normal galaxies is 0.010 and for X-ray sources is 0.014, and outlier fractions are $4\%$ and $5.4\%$ respectively. The results within the CANDELS coverage area are even better as demonstrated both by spectroscopic comparison and by galaxy-pair statistics. Intermediate-band photometry, even if shallow, is valuable when combined with deep broad-band photometry. For best accuracy, templates must include emission lines., Comment: The paper has been accepted by ApJ. The materials we provide are available under [Surveys] > [CDFS] through the portal http://www.mpe.mpg.de/XraySurveys

Not only source catalogs are extracted from astronomy observations. Their sky coverage is always carefully recorded and used in statistical analyses, such as correlation and luminosity function studies. Here we present a novel method for catalog matching, which inherently builds on the coverage information for better performance and completeness. A modified version of the Zones Algorithm is introduced for matching partially overlapping observations, where irrelevant parts of the data are excluded up front for efficiency. Our design enables searches to focus on specific areas on the sky to further speed up the process. Another important advantage of the new method over traditional techniques is its ability to quickly detect dropouts, i.e., the missing components that are in the observed regions of the celestial sphere but did not reach the detection limit in some observations. These often provide invaluable insight into the spectral energy distribution of the matched sources but rarely available in traditional associations., Comment: 14 pages, 6 figures

The fraction of galaxies with red colours depends sensitively on environment, and on the way in which environment is measured. To distinguish competing theories for the quenching of star formation, a robust and complete description of environment is required, to be applied to a large sample of galaxies. The environment of galaxies can be described using the density field of neighbours on multiple scales - the multiscale density field. We are using the Millennium simulation and a simple HOD prescription which describes the multiscale density field of Sloan Digital Sky Survey DR7 galaxies to investigate the dependence of the fraction of red galaxies on the environment. Using a volume limited sample where we have sufficient galaxies in narrow density bins, we have more dynamic range in halo mass and density for satellite galaxies than for central galaxies. Therefore we model the red fraction of central galaxies as a constant while we use a functional form to describe the red fraction of satellites as a function of halo mass which allows us to distinguish a sharp from a gradual transition. While it is clear that the data can only be explained by a gradual transition, an analysis of the multiscale density field on different scales suggests that colour segregation within the haloes is needed to explain the results. We also rule out a sharp transition for central galaxies, within the halo mass range sampled., Comment: 24 pages, 21 figures, accepted for publication by MNRAS

Understanding the diversity in spectra is the key to determining the physical parameters of galaxies. The optical spectra of galaxies are highly convoluted with continuum and lines which are potentially sensitive to different physical parameters. Defining the wavelength regions of interest is therefore an important question. In this work, we identify informative wavelength regions in a single-burst stellar populations model by using the CUR Matrix Decomposition. Simulating the Lick/IDS spectrograph configuration, we recover the widely used Dn(4000), Hbeta, and HdeltaA to be most informative. Simulating the SDSS spectrograph configuration with a wavelength range 3450-8350 Angstrom and a model-limited spectral resolution of 3 Angstrom, the most informative regions are: first region-the 4000 Angstrom break and the Hdelta line; second region-the Fe-like indices; third region-the Hbeta line; fourth region-the G band and the Hgamma line. A Principal Component Analysis on the first region shows that the first eigenspectrum tells primarily the stellar age, the second eigenspectrum is related to the age-metallicity degeneracy, and the third eigenspectrum shows an anti-correlation between the strengths of the Balmer and the Ca K and H absorptions. The regions can be used to determine the stellar age and metallicity in early-type galaxies which have solar abundance ratios, no dust, and a single-burst star formation history. The region identification method can be applied to any set of spectra of the user's interest, so that we eliminate the need for a common, fixed-resolution index system. We discuss future directions in extending the current analysis to late-type galaxies., Comment: 36 Pages, 13 Figures, 4 Tables. AJ Accepted

Many fields of science rely on relational database management systems to analyze, publish and share data. Since RDBMS are originally designed for, and their development directions are primarily driven by, business use cases they often lack features very important for scientific applications. Horizontal scalability is probably the most important missing feature which makes it challenging to adapt traditional relational database systems to the ever growing data sizes. Due to the limited support of array data types and metadata management, successful application of RDBMS in science usually requires the development of custom extensions. While some of these extensions are specific to the field of science, the majority of them could easily be generalized and reused in other disciplines. With the Graywulf project we intend to target several goals. We are building a generic platform that offers reusable components for efficient storage, transformation, statistical analysis and presentation of scientific data stored in Microsoft SQL Server. Graywulf also addresses the distributed computational issues arising from current RDBMS technologies. The current version supports load balancing of simple queries and parallel execution of partitioned queries over a set of mirrored databases. Uniform user access to the data is provided through a web based query interface and a data surface for software clients. Queries are formulated in a slightly modified syntax of SQL that offers a transparent view of the distributed data. The software library consists of several components that can be reused to develop complex scientific data warehouses: a system registry, administration tools to manage entire database server clusters, a sophisticated workflow execution framework, and a SQL parser library., Comment: SSDBM 2013 proceedings

We present a data-driven method to infer the redshift distribution of an arbitrary dataset based on spatial cross-correlation with a reference population and we apply it to various datasets across the electromagnetic spectrum to show its potential and limitations. Our approach advocates the use of clustering measurements on all available scales, in contrast to previous works focusing only on linear scales. We also show how its accuracy can be enhanced by optimally sampling a dataset within its photometric space rather than applying the estimator globally. We show that the ultimate goal of this technique is to characterize the mapping between the space of photometric observables and redshift space as this characterization then allows us to infer the clustering-redshift p.d.f. of a single galaxy. We apply this technique to estimate the redshift distributions of luminous red galaxies and emission line galaxies from the SDSS, infrared sources from WISE and radio sources from FIRST. We show that consistent redshift distributions are found using both quasars and absorber systems as reference populations. This technique brings valuable information on the third dimension of astronomical datasets. It is widely applicable to a large range of extra-galactic surveys., Comment: 10 pages, 5 figures. Improved description of the formalism

We present observations of a luminous galaxy at redshift z=6.573 --- the end of the reioinization epoch --- which has been spectroscopically confirmed twice. The first spectroscopic confirmation comes from slitless HST ACS grism spectra from the PEARS survey (Probing Evolution And Reionization Spectroscopically), which show a dramatic continuum break in the spectrum at restframe 1216 A wavelength. The second confirmation is done with Keck + DEIMOS. The continuum is not clearly detected with ground-based spectra, but high wavelength resolution enables the Lyman alpha emission line profile to be determined. We compare the line profile to composite line profiles at redshift z=4.5. The Lyman alpha line profile shows no signature of a damping wing attenuation, confirming that the intergalactic gas is ionized at redshift z=6.57. Spectra of Lyman breaks at yet higher redshifts will be possible using comparably deep observations with IR-sensitive grisms, even at redshifts where Lyman alpha is too attenuated by the neutral IGM to be detectable using traditional spectroscopy from the ground., Comment: 19 pages, four figures. Resubmitted to The Astrophysical Journal after revisions to address the referee's report

We introduce \emph{Astronomy and Computing}, a new journal for the growing population of people working in the domain where astronomy overlaps with computer science and information technology. The journal aims to provide a new communication channel within that community, which is not well served by current journals, and to help secure recognition of its true importance within modern astronomy. In this inaugural editorial, we describe the rationale for creating the journal, outline its scope and ambitions, and seek input from the community in defining in detail how the journal should work towards its high-level goals., Comment: 5 pages, no figures; editorial for first edition of journal

In the lead-up to the Square Kilometre Array (SKA) project, several next-generation radio telescopes and upgrades are already being built around the world. These include APERTIF (The Netherlands), ASKAP (Australia), eMERLIN (UK), VLA (USA), e-EVN (based in Europe), LOFAR (The Netherlands), Meerkat (South Africa), and the Murchison Widefield Array (MWA). Each of these new instruments has different strengths, and coordination of surveys between them can help maximise the science from each of them. A radio continuum survey is being planned on each of them with the primary science objective of understanding the formation and evolution of galaxies over cosmic time, and the cosmological parameters and large-scale structures which drive it. In pursuit of this objective, the different teams are developing a variety of new techniques, and refining existing ones. Here we describe these projects, their science goals, and the technical challenges which are being addressed to maximise the science return., Comment: Accepted by PASA, 22 October 2012

Scientific endeavors such as large astronomical surveys generate databases on the terabyte scale. These, usually multidimensional databases must be visualized and mined in order to find interesting objects or to extract meaningful and qualitatively new relationships. Many statistical algorithms required for these tasks run reasonably fast when operating on small sets of in-memory data, but take noticeable performance hits when operating on large databases that do not fit into memory. We utilize new software technologies to develop and evaluate fast multidimensional indexing schemes that inherently follow the underlying, highly non-uniform distribution of the data: they are layered uniform grid indices, hierarchical binary space partitioning, and sampled flat Voronoi tessellation of the data. Our working database is the 5-dimensional magnitude space of the Sloan Digital Sky Survey with more than 270 million data points, where we show that these techniques can dramatically speed up data mining operations such as finding similar objects by example, classifying objects or comparing extensive simulation sets with observations. We are also developing tools to interact with the multidimensional database and visualize the data at multiple resolutions in an adaptive manner., Comment: 12 pages, 16 figures; CIDR 2007

Multi-wavelength astronomical studies require cross-identification of detections of the same celestial objects in multiple catalogs based on spherical coordinates and other properties. Because of the large data volumes and spherical geometry, the symmetric N-way association of astronomical detections is a computationally intensive problem, even when sophisticated indexing schemes are used to exclude obviously false candidates. Legacy astronomical catalogs already contain detections of more than a hundred million objects while the ongoing and future surveys will produce catalogs of billions of objects with multiple detections of each at different times. The varying statistical error of position measurements, moving and extended objects, and other physical properties make it necessary to perform the cross-identification using a mathematically correct, proper Bayesian probabilistic algorithm, capable of including various priors. One time, pair-wise cross-identification of these large catalogs is not sufficient for many astronomical scenarios. Consequently, a novel system is necessary that can cross-identify multiple catalogs on-demand, efficiently and reliably. In this paper, we present our solution based on a cluster of commodity servers and ordinary relational databases. The cross-identification problems are formulated in a language based on SQL, but extended with special clauses. These special queries are partitioned spatially by coordinate ranges and compiled into a complex workflow of ordinary SQL queries. Workflows are then executed in a parallel framework using a cluster of servers hosting identical mirrors of the same data sets.

Object cross-identification in multiple observations is often complicated by the uncertainties in their astrometric calibration. Due to the lack of standard reference objects, an image with a small field of view can have significantly larger errors in its absolute positioning than the relative precision of the detected sources within. We present a new general solution for the relative astrometry that quickly refines the World Coordinate System of overlapping fields. The efficiency is obtained through the use of infinitesimal 3-D rotations on the celestial sphere, which do not involve trigonometric functions. They also enable an analytic solution to an important step in making the astrometric corrections. In cases with many overlapping images, the correct identification of detections that match together across different images is difficult to determine. We describe a new greedy Bayesian approach for selecting the best object matches across a large number of overlapping images. The methods are developed and demonstrated on the Hubble Legacy Archive, one of the most challenging data sets today. We describe a novel catalog compiled from many Hubble Space Telescope observations, where the detections are combined into a searchable collection of matches that link the individual detections. The matches provide descriptions of astronomical objects involving multiple wavelengths and epochs. High relative positional accuracy of objects is achieved across the Hubble images, often sub-pixel precision in the order of just a few milli-arcseconds. The result is a reliable set of high-quality associations that are publicly available online., Comment: 9 pages, 9 figures, accepted for publication in the Astrophysical Journal

We present a data model describing the structure of spectrophotometric datasets with spectral and temporal coordinates and associated metadata. This data model may be used to represent spectra, time series data, segments of SED (Spectral Energy Distributions) and other spectral or temporal associations., Comment: http://www.ivoa.net

The Simple Spectral Access (SSA) Protocol (SSAP) defines a uniform interface to remotely discover and access one dimensional spectra. SSA is a member of an integrated family of data access interfaces altogether comprising the Data Access Layer (DAL) of the IVOA. SSA is based on a more general data model capable of describing most tabular spectrophotometric data, including time series and spectral energy distributions (SEDs) as well as 1-D spectra; however the scope of the SSA interface as specified in this document is limited to simple 1-D spectra, including simple aggregations of 1-D spectra. The form of the SSA interface is simple: clients first query the global resource registry to find services of interest and then issue a data discovery query to selected services to determine what relevant data is available from each service; the candidate datasets available are described uniformly in a VOTable format document which is returned in response to the query. Finally, the client may retrieve selected datasets for analysis. Spectrum datasets returned by an SSA spectrum service may be either precomputed, archival datasets, or they may be virtual data which is computed on the fly to respond to a client request. Spectrum datasets may conform to a standard data model defined by SSA, or may be native spectra with custom project-defined content. Spectra may be returned in any of a number of standard data formats. Spectral data is generally stored externally to the VO in a format specific to each spectral data collection; currently there is no standard way to represent astronomical spectra, and virtually every project does it differently. Hence spectra may be actively mediated to the standard SSA-defined data model at access time by the service, so that client analysis programs do not have to be familiar with the idiosyncratic details of each data collection to be accessed.

In this work we present an atlas of composite spectra of galaxies based on the data of the Sloan Digital Sky Survey Data Release 7 (SDSS DR7). Galaxies are classified by colour, nuclear activity and star-formation activity to calculate average spectra of high signal-to-noise ratio and resolution (S/N = 132 - 4760 at {Dlambda = 1 A), using an algorithm that is robust against outliers. Besides composite spectra, we also compute the first five principal components of the distributions in each galaxy class to characterize the nature of variations of individual spectra around the averages. The continua of the composite spectra are fitted with BC03 stellar population synthesis models to extend the wavelength coverage beyond the coverage of the SDSS spectrographs. Common derived parameters of the composites are also calculated: integrated colours in the most popular filter systems, line strength measurements, and continuum absorption indices (including Lick indices). These derived parameters are compared with the distributions of parameters of individual galaxies and it is shown on many examples that the composites of the atlas cover much of the parameter space spanned by SDSS galaxies. By co-adding thousands of spectra, a total integration time of several months can be reached, which results in extremely low noise composites. The variations in redshift not only allow for extending the spectral coverage bluewards to the original wavelength limit of the SDSS spectrographs, but also make higher spectral resolution achievable. The composite spectrum atlas is available online at http://www.vo.elte.hu/compositeatlas.

We present a statistical study of the correlations and dimensionality of emission lines carried out on a sample of over 40,000 Sloan Digital Sky Survey (SDSS) galaxies. Using principal component analysis, we found that the equivalent widths of the 11 strongest lines can be well represented using three parameters. We also explore correlations of the emission pattern with the eigenspace representation of the continuum spectrum. The observed relations are used to provide an empirical prescription for expectation values and variances of emission-line strengths as a function of spectral shape. We show that this estimation of emission lines has a sufficient accuracy to make it suitable for photometric applications. The method has already proved useful in SDSS photometric redshift estimation., Comment: 38 pages, 20 figures

This paper outlines certain scenarios from the fields of astrophysics and fluid dynamics simulations which require high performance data warehouses that support array data type. A common feature of all these use cases is that subsetting and preprocessing the data on the server side (as far as possible inside the database server process) is necessary to avoid the client-server overhead and to minimize IO utilization. Analyzing and summarizing the requirements of the various fields help software engineers to come up with a comprehensive design of an array extension to relational database systems that covers a wide range of scientific applications. We also present a working implementation of an array data type for Microsoft SQL Server 2008 to support large-scale scientific applications. We introduce the design of the array type, results from a performance evaluation, and discuss the lessons learned from this implementation. The library can be downloaded from our website at http://voservices.net/sqlarray/

We discuss a novel approach to identifying cosmic events in separate and independent observations. In our focus are the true events, such as supernova explosions, that happen once, hence, whose measurements are not repeatable. Their classification and analysis have to make the best use of all the available data. Bayesian hypothesis testing is used to associate streams of events in space and time. Probabilities are assigned to the matches by studying their rates of occurrence. A case study of Type Ia supernovae illustrates how to use lightcurves in the cross-identification process. Constraints from realistic lightcurves happen to be well-approximated by Gaussians in time, which makes the matching process very efficient. Model-dependent associations are computationally more demanding but can further boost our confidence., Comment: 5 pages, 2 figures, accepted to ApJ

The inclination of galaxies induces both reddening and extinction to their observed spectral energy distribution, which in turn impact the derived properties of the galaxies. Here we report a significant dependence of the error in photometric redshift (photo-z) on the inclination of disk galaxies from the Sloan Digital Sky Survey. The bias in the photo-z based on the template-fitting approach increases from -0.015 in face-on to 0.021 in edge-on galaxies. A Principal Component Analysis on the full sample of photometry reveals the inclination of the galaxies to be represented by the 2nd mode. The corresponding eigenspectrum resembles an extinction curve. The isolation of the inclination effect in a low-order mode demonstrates the significant reddening induced on the observed colors, leading to the over-estimated photo-z in galaxies of high inclinations. We present approaches to correct the photo-z and the other properties of the disk galaxies against the inclination effect., Comment: Accepted for publication in ApJ

Although relative errors can readily be calculated, the absolute astrometric accuracy of the source positions in the Chandra Source Catalog (CSC), Version 1.0, is a priori unknown. However, the cross-match with stellar objects from the Sloan Digital Sky Survey (SDSS) offers the opportunity to compare the apparent separations of the cross-matched pairs with the formally calculated errors. The analysis of these data allowed us to derive a value of 0.16" for the residual absolute astrometric error in CSC positions. This error will be added to the published position errors in the CSC from now on, starting with CSC, Version 1.1., Comment: 18 pages, 6 figures, accepted for publication in ApJSuppl

We show that redshift-space distortions of galaxy correlations have a strong effect on correlation functions with distinct, localized features, like the signature of the baryon acoustic oscillations (BAO). Near the line of sight, the features become sharper as a result of redshift-space distortions. We demonstrate this effect by measuring the correlation function in Gaussian simulations and the Millennium Simulation. We also analyze the SDSS DR7 main-galaxy sample (MGS), splitting the sample into slices 2.5 degrees on the sky in various rotations. Measuring 2D correlation functions in each slice, we do see a sharp bump along the line of sight. Using Mexican-hat wavelets, we localize it to (110 +/- 10) Mpc/h. Averaging only along the line of sight, we estimate its significance at a particular wavelet scale and location at 2.2 sigma. In a flat angular weighting in the (pi,r_p) coordinate system, the noise level is suppressed, pushing the bump's significance to 4 sigma. We estimate that there is about a 0.2% chance of getting such a signal anywhere in the vicinity of the BAO scale from a power spectrum lacking a BAO feature. However, these estimates of the significances make some use of idealized Gaussian simulations, and thus are likely a bit optimistic., Comment: 17 pages, 27 figures. Minor changes to match final version accepted to ApJ

We use measurements of the HI content, stellar mass and star formation rates in ~190 massive galaxies with stellar masses greater than 10^10 Msun, obtained from the Galex Arecibo SDSS Survey (GASS) described in Paper I (Catinella et al. 2010) to explore the global scaling relations associated with the bin-averaged ratio of the star formation rate over the HI mass, which we call the HI-based star formation efficiency (SFE). Unlike the mean specific star formation rate, which decreases with stellar mass and stellar mass surface density, the star formation efficiency remains relatively constant across the sample with a value close to SFE = 10^-9.5 yr^-1 (or an equivalent gas consumption timescale of ~3 Gyr). Specifically, we find little variation in SFE with stellar mass, stellar mass surface density, NUV-r color and concentration. We interpret these results as an indication that external processes or feedback mechanisms that control the gas supply are important for regulating star formation in massive galaxies. An investigation into the detailed distribution of SFEs reveals that approximately 5% of the sample shows high efficiencies with SFE > 10^-9 yr^-1, and we suggest that this is very likely due to a deficiency of cold gas rather than an excess star formation rate. Conversely, we also find a similar fraction of galaxies that appear to be gas-rich for their given specific star-formation rate, although these galaxies show both a higher than average gas fraction and lower than average specific star formation rate. Both of these populations are plausible candidates for "transition" galaxies, showing potential for a change (either decrease or increase) in their specific star formation rate in the near future. We also find that 36+/-5% of the total HI mass density and 47+/-5% of the total SFR density is found in galaxies with stellar mass greater than 10^10 Msun. [abridged], Comment: 18 pages, 11 figures. Accepted for publication in MNRAS. GASS publications and released data can be found at http://www.mpa-garching.mpg.de/GASS/index.php