Search

A number of synoptic sky surveys are underway or being planned. Typically they are done with small telescopes and relatively short exposure times. A search for transient or variable sources involves comparison with deeper baseline images, ideally obtained through the same telescope and camera. With that in mind we have stacked images from the 0.68~m Schmidt telescope on Mt. Bigelow taken over ten years as part of the Catalina Sky Survey. In order to generate deep reference images for the Catalina Real-time Transient Survey, close to 0.8 million images over 8000 fields and covering over 27000~sq.~deg. have gone into the deep stack that goes up to 3 magnitudes deeper than individual images. CRTS system does not use a filter in imaging, hence there is no standard passband in which the optical magnitude is measured. We estimate depth by comparing these wide-band unfiltered co-added images with images in the $g$-band and find that the image depth ranges from 22.0--24.2 across the sky, with a 200-image stack attaining an equivalent AB magnitude sensitivity of 22.8. We compared various state-of-the-art software packages for co-adding astronomical images and have used SWarp for the stacking. We describe here the details of the process adopted. This methodology may be useful in other panoramic imaging applications, and to other surveys as well. The stacked images are available through a server at Inter-University Centre for Astronomy and Astrophysics (IUCAA)., Comment: 16 pages, 15 figures

A systematic study on the long-term periodicities of known Galactic cataclysmic variables (CVs) was conducted. Among 1580 known CVs, 344 sources were matched and extracted from the Palomar Transient Factory (PTF) data repository. The PTF light curves were combined with the Catalina Real-Time Transient Survey (CRTS) light curves and analyzed. Ten targets were found to exhibit long-term periodic variability, which is not frequently observed in the CV systems. These long-term variations are possibly caused by various mechanisms, such as the precession of the accretion disk, hierarchical triple star system, magnetic field change of the companion star, and other possible mechanisms. We discuss the possible mechanisms in this study. If the long-term period is less than several tens of days, the disk precession period scenario is favored. However, the hierarchical triple star system or the variations in magnetic field strengths are most likely the predominant mechanisms for longer periods., Comment: 33 pages, 9 figures (manuscript form), Accepted for publication in PASP

CRTS J084133.15+200525.8 is an optically bright quasar at z=2.345 that has shown extreme spectral variability over the past decade. Photometrically, the source had a visual magnitude of V~17.3 between 2002 and 2008. Then, over the following five years, the source slowly brightened by approximately one magnitude, to V~16.2. Only ~1 in 10,000 quasars show such extreme variability, as quantified by the extreme parameters derived for this quasar assuming a damped random walk model. A combination of archival and newly acquired spectra reveal the source to be an iron low-ionization broad absorption line (FeLoBAL) quasar with extreme changes in its absorption spectrum. Some absorption features completely disappear over the 9 years of optical spectra, while other features remain essentially unchanged. We report the first definitive redshift for this source, based on the detection of broad H-alpha in a Keck/MOSFIRE spectrum. Absorption systems separated by several 1000 km/s in velocity show coordinated weakening in the depths of their troughs as the continuum flux increases. We interpret the broad absorption line variability to be due to changes in photoionization, rather than due to motion of material along our line of sight. This source highlights one sort of rare transition object that astronomy will now be finding through dedicated time-domain surveys., Comment: 6 pages, 4 figures; accepted for publication in ApJ

The nature of scientific and technological data collection is evolving rapidly: data volumes and rates grow exponentially, with increasing complexity and information content, and there has been a transition from static data sets to data streams that must be analyzed in real time. Interesting or anomalous phenomena must be quickly characterized and followed up with additional measurements via optimal deployment of limited assets. Modern astronomy presents a variety of such phenomena in the form of transient events in digital synoptic sky surveys, including cosmic explosions (supernovae, gamma ray bursts), relativistic phenomena (black hole formation, jets), potentially hazardous asteroids, etc. We have been developing a set of machine learning tools to detect, classify and plan a response to transient events for astronomy applications, using the Catalina Real-time Transient Survey (CRTS) as a scientific and methodological testbed. The ability to respond rapidly to the potentially most interesting events is a key bottleneck that limits the scientific returns from the current and anticipated synoptic sky surveys. Similar challenge arise in other contexts, from environmental monitoring using sensor networks to autonomous spacecraft systems. Given the exponential growth of data rates, and the time-critical response, we need a fully automated and robust approach. We describe the results obtained to date, and the possible future developments., Comment: 14 pages, an invited paper for a special issue of Future Generation Computer Systems, Elsevier Publ. (2015). This is an expanded version of a paper arXiv:1407.3502 presented at the IEEE e-Science 2014 conf., with some new content

The exploitation of present and future synoptic (multi-band and multi-epoch) surveys requires an extensive use of automatic methods for data processing and data interpretation. In this work, using data extracted from the Catalina Real Time Transient Survey (CRTS), we investigate the classification performance of some well tested methods: Random Forest, MLPQNA (Multi Layer Perceptron with Quasi Newton Algorithm) and K-Nearest Neighbors, paying special attention to the feature selection phase. In order to do so, several classification experiments were performed. Namely: identification of cataclysmic variables, separation between galactic and extra-galactic objects and identification of supernovae., Comment: Accepted by MNRAS, 11 figures, 18 pages

Hierarchical assembly models predict a population of supermassive black hole (SMBH) binaries. These are not resolvable by direct imaging but may be detectable via periodic variability (or nanohertz frequency gravitational waves). Following our detection of a 5.2 year periodic signal in the quasar PG 1302-102 (Graham et al. 2015), we present a novel analysis of the optical variability of 243,500 known spectroscopically confirmed quasars using data from the Catalina Real-time Transient Survey (CRTS) to look for close (< 0.1 pc) SMBH systems. Looking for a strong Keplerian periodic signal with at least 1.5 cycles over a baseline of nine years, we find a sample of 111 candidate objects. This is in conservative agreement with theoretical predictions from models of binary SMBH populations. Simulated data sets, assuming stochastic variability, also produce no equivalent candidates implying a low likelihood of spurious detections. The periodicity seen is likely attributable to either jet precession, warped accretion disks or periodic accretion associated with a close SMBH binary system. We also consider how other SMBH binary candidates in the literature appear in CRTS data and show that none of these are equivalent to the identified objects. Finally, the distribution of objects found is consistent with that expected from a gravitational wave-driven population. This implies that circumbinary gas is present at small orbital radii and is being perturbed by the black holes. None of the sources is expected to merge within at least the next century. This study opens a new unique window to study a population of close SMBH binaries that must exist according to our current understanding of galaxy and SMBH evolution., Comment: 29 pages, 10 figures, accepted for publication in MNRAS - this version contains extended table and figure

Quasars have long been known to be variable sources at all wavelengths. Their optical variability is stochastic, can be due to a variety of physical mechanisms, and is well-described statistically in terms of a damped random walk model. The recent availability of large collections of astronomical time series of flux measurements (light curves) offers new data sets for a systematic exploration of quasar variability. Here we report on the detection of a strong, smooth periodic signal in the optical variability of the quasar PG 1302-102 with a mean observed period of 1,884 $\pm$ 88 days. It was identified in a search for periodic variability in a data set of light curves for 247,000 known, spectroscopically confirmed quasars with a temporal baseline of $\sim9$ years. While the interpretation of this phenomenon is still uncertain, the most plausible mechanisms involve a binary system of two supermassive black holes with a subparsec separation. Such systems are an expected consequence of galaxy mergers and can provide important constraints on models of galaxy formation and evolution., Comment: 19 pages, 6 figures. Published online by Nature on 7 January 2015

We use seven year's worth of observations from the Catalina Sky Survey and the Siding Spring Survey covering most of the northern and southern hemisphere at galactic latitudes higher than 20 degrees to search for serendipitously imaged moving objects in the outer solar system. These slowly moving objects would appear as stationary transients in these fast cadence asteroids surveys, so we develop methods to discover objects in the outer solar system using individual observations spaced by months, rather than spaced by hours, as is typically done. While we independently discover 8 known bright objects in the outer solar system, the faintest having $V=19.8\pm0.1$, no new objects are discovered. We find that the survey is nearly 100% efficient at detecting objects beyond 25 AU for $V\lesssim 19.1$ ($V\lesssim18.6$ in the southern hemisphere) and that the probability that there is one or more remaining outer solar system object of this brightness left to be discovered in the unsurveyed regions of the galactic plane is approximately 32%.

Effective data visualization is a key part of the discovery process in the era of big data. It is the bridge between the quantitative content of the data and human intuition, and thus an essential component of the scientific path from data into knowledge and understanding. Visualization is also essential in the data mining process, directing the choice of the applicable algorithms, and in helping to identify and remove bad data from the analysis. However, a high complexity or a high dimensionality of modern data sets represents a critical obstacle. How do we visualize interesting structures and patterns that may exist in hyper-dimensional data spaces? A better understanding of how we can perceive and interact with multi dimensional information poses some deep questions in the field of cognition technology and human computer interaction. To this effect, we are exploring the use of immersive virtual reality platforms for scientific data visualization, both as software and inexpensive commodity hardware. These potentially powerful and innovative tools for multi dimensional data visualization can also provide an easy and natural path to a collaborative data visualization and exploration, where scientists can interact with their data and their colleagues in the same visual space. Immersion provides benefits beyond the traditional desktop visualization tools: it leads to a demonstrably better perception of a datascape geometry, more intuitive data understanding, and a better retention of the perceived relationships in the data., Comment: 6 pages, refereed proceedings of 2014 IEEE International Conference on Big Data, page 609, ISBN 978-1-4799-5665-4

We review some aspects of the current state of data-intensive astronomy, its methods, and some outstanding data analysis challenges. Astronomy is at the forefront of "big data" science, with exponentially growing data volumes and data rates, and an ever-increasing complexity, now entering the Petascale regime. Telescopes and observatories from both ground and space, covering a full range of wavelengths, feed the data via processing pipelines into dedicated archives, where they can be accessed for scientific analysis. Most of the large archives are connected through the Virtual Observatory framework, that provides interoperability standards and services, and effectively constitutes a global data grid of astronomy. Making discoveries in this overabundance of data requires applications of novel, machine learning tools. We describe some of the recent examples of such applications., Comment: Keynote talk in the proceedings of ESA-ESRIN Conference: Big Data from Space 2014, Frascati, Italy, November 12-14, 2014, 8 pages, 2 figures

The nature of scientific and technological data collection is evolving rapidly: data volumes and rates grow exponentially, with increasing complexity and information content, and there has been a transition from static data sets to data streams that must be analyzed in real time. Interesting or anomalous phenomena must be quickly characterized and followed up with additional measurements via optimal deployment of limited assets. Modern astronomy presents a variety of such phenomena in the form of transient events in digital synoptic sky surveys, including cosmic explosions (supernovae, gamma ray bursts), relativistic phenomena (black hole formation, jets), potentially hazardous asteroids, etc. We have been developing a set of machine learning tools to detect, classify and plan a response to transient events for astronomy applications, using the Catalina Real-time Transient Survey (CRTS) as a scientific and methodological testbed. The ability to respond rapidly to the potentially most interesting events is a key bottleneck that limits the scientific returns from the current and anticipated synoptic sky surveys. Similar challenge arise in other contexts, from environmental monitoring using sensor networks to autonomous spacecraft systems. Given the exponential growth of data rates, and the time-critical response, we need a fully automated and robust approach. We describe the results obtained to date, and the possible future developments., Comment: 8 pages, IEEE conference format, to appear in the refereed proceedings of the IEEE e-Science 2014 conf., eds. C. Medeiros et al., IEEE, in press (2014). arXiv admin note: substantial text overlap with arXiv:1209.1681, arXiv:1110.4655

We investigate the properties of 367 ultra-short period binary candidates selected from 31,000 sources recently identified from Catalina Surveys data. Based on light curve morphology, along with WISE, SDSS and GALEX multi-colour photometry, we identify two distinct groups of binaries with periods below the 0.22 day contact binary minimum. In contrast to most recent work, we spectroscopically confirm the existence of M-dwarf+M-dwarf contact binary systems. By measuring the radial velocity variations for five of the shortest-period systems, we find examples of rare cool-white dwarf+M-dwarf binaries. Only a few such systems are currently known. Unlike warmer white dwarf systems, their UV flux and their optical colours and spectra are dominated by the M-dwarf companion. We contrast our discoveries with previous photometrically-selected ultra-short period contact binary candidates, and highlight the ongoing need for confirmation using spectra and associated radial velocity measurements. Overall, our analysis increases the number of ultra-short period contact binary candidates by more than an order of magnitude., Comment: 12 pages, 12 figures, accepted ApJ

Astronomy is undergoing through a methodological revolution triggered by an unprecedented wealth of complex and accurate data. The new panchromatic, synoptic sky surveys require advanced tools for discovering patterns and trends hidden behind data which are both complex and of high dimensionality. We present DAMEWARE (DAta Mining & Exploration Web Application REsource): a general purpose, web-based, distributed data mining environment developed for the exploration of large datasets, and finely tuned for astronomical applications. By means of graphical user interfaces, it allows the user to perform classification, regression or clustering tasks with machine learning methods. Salient features of DAMEWARE include its capability to work on large datasets with minimal human intervention, and to deal with a wide variety of real problems such as the classification of globular clusters in the galaxy NGC1399, the evaluation of photometric redshifts and, finally, the identification of candidate Active Galactic Nuclei in multiband photometric surveys. In all these applications, DAMEWARE allowed to achieve better results than those attained with more traditional methods. With the aim of providing potential users with all needed information, in this paper we briefly describe the technological background of DAMEWARE, give a short introduction to some relevant aspects of data mining, followed by a summary of some science cases and, finally, we provide a detailed description of a template use case., Comment: To appear in PASP (accepted for pubblication)

We present ~47,000 periodic variables found during the analysis of 5.4 million variable star candidates within a 20,000 square degree region covered by the Catalina Surveys Data Release-1 (CSDR1). Combining these variables with type-ab RR Lyrae from our previous work, we produce an on-line catalog containing periods, amplitudes, and classifications for ~61,000 periodic variables. By cross-matching these variables with those from prior surveys, we find that > 90% of the ~8,000 known periodic variables in the survey region are recovered. For these sources we find excellent agreement between our catalog and prior values of luminosity, period and amplitude, as well as classification. We investigate the rate of confusion between objects classified as contact binaries and type-c RR Lyrae (RRc's) based on periods, colours, amplitudes, metalicities, radial velocities and surface gravities. We find that no more than few percent of these variables in these classes are misidentified. By deriving distances for this clean sample of ~5,500 RRc's, we trace the path of the Sagittarius tidal streams within the Galactic halo. Selecting 146 outer-halo RRc's with SDSS radial velocities, we confirm the presence of a coherent halo structure that is inconsistent with current N-body simulations of the Sagittarius tidal stream. We also find numerous long-period variables that are very likely associated within the Sagittarius tidal streams system. Based on the examination of 31,000 contact binary light curves we find evidence for two subgroups exhibiting irregular lightcurves. One subgroup presents significant variations in mean brightness that are likely due to chromospheric activity. The other subgroup shows stable modulations over more than a thousand days and thereby provides evidence that the O'Connell effect is not due to stellar spots., Comment: Accepted ApJS, 43 pages, 9 tables, 44 figures (some at reduced resolution)

We present 855 cataclysmic variable candidates detected by the Catalina Real-time Transient Survey (CRTS) of which at least 137 have been spectroscopically confirmed and 705 are new discoveries. The sources were identified from the analysis of five years of data, and come from an area covering three quarters of the sky. We study the amplitude distribution of the dwarf novae CVs discovered by CRTS during outburst, and find that in quiescence they are typically two magnitudes fainter compared to the spectroscopic CV sample identified by SDSS. However, almost all CRTS CVs in the SDSS footprint have ugriz photometry. We analyse the spatial distribution of the CVs and find evidence that many of the systems lie at scale heights beyond those expected for a Galactic thin disc population. We compare the outburst rates of newly discovered CRTS CVs with the previously known CV population, and find no evidence for a difference between them. However, we find that significant evidence for a systematic difference in orbital period distribution. We discuss the CVs found below the orbital period minimum and argue that many more are yet to be identified among the full CRTS CV sample. We cross-match the CVs with archival X-ray catalogs and find that most of the systems are dwarf novae rather than magnetic CVs., Comment: 15 pages, 17 figures, accepted MNRAS

We compare quasar selection techniques based on their optical variability using data from the Catalina Real-time Transient Survey (CRTS). We introduce a new technique based on Slepian wavelet variance (SWV) that shows comparable or better performance to structure functions and damped random walk models but with fewer assumptions. Combining these methods with WISE mid-IR colors produces a highly efficient quasar selection technique which we have validated spectroscopically. The SWV technique also identifies characteristic timescales in a time series and we find a characteristic rest frame timescale of ~54 days, confirmed in the light curves of ~18000 quasars from CRTS, SDSS and MACHO data, and anticorrelated with absolute magnitude. This indicates a transition between a damped random walk and $P(f) \propto f^{-1/3}$ behaviours and is the first strong indication that a damped random walk model may be too simplistic to describe optical quasar variability., Comment: 18 pages, 17 figures, accepted for publication in MNRAS

The amount of collected data in many scientific fields is increasing, all of them requiring a common task: extract knowledge from massive, multi parametric data sets, as rapidly and efficiently possible. This is especially true in astronomy where synoptic sky surveys are enabling new research frontiers in the time domain astronomy and posing several new object classification challenges in multi dimensional spaces; given the high number of parameters available for each object, feature selection is quickly becoming a crucial task in analyzing astronomical data sets. Using data sets extracted from the ongoing Catalina Real-Time Transient Surveys (CRTS) and the Kepler Mission we illustrate a variety of feature selection strategies used to identify the subsets that give the most information and the results achieved applying these techniques to three major astronomical problems., Comment: 7 pages, to appear in refereed proceedings of Scalable Machine Learning: Theory and Applications, IEEE BigData 2013

This paper presents a comparison of popular period finding algorithms applied to the light curves of variable stars from the Catalina Real-time Transient Survey (CRTS), MACHO and ASAS data sets. We analyze the accuracy of the methods against magnitude, sampling rates, quoted period, quality measures (signal-to-noise and number of observations), variability, and object classes. We find that measure of dispersion-based techniques - analysis-of-variance with harmonics and conditional entropy - consistently give the best results but there are clear dependencies on object class and light curve quality. Period aliasing and identifying a period harmonic also remain significant issues. We consider the performance of the algorithms and show that a new conditional entropy-based algorithm is the most optimal in terms of completeness and speed. We also consider a simple ensemble approach and find that it performs no better than individual algorithms., Comment: 24 pages, 21 figures, accepted for publication in Monthly Notices of Royal Astronomical Society

This paper presents a new period finding method based on conditional entropy that is both efficient and accurate. We demonstrate its applicability on simulated and real data. We find that it has comparable performance to other information-based techniques with simulated data but is superior with real data, both for finding periods and just identifying periodic behaviour. In particular, it is robust against common aliasing issues found with other period-finding algorithms., Comment: 8 pages, 7 figures, accepted for publication in Monthly Notices of Royal Astronomical Society; revised version (corrected reference to MACHO)

High-volume feature-rich data sets are becoming the bread-and-butter of 21st century astronomy but present significant challenges to scientific discovery. In particular, identifying scientifically significant relationships between sets of parameters is non-trivial. Similar problems in biological and geosciences have led to the development of systems which can explore large parameter spaces and identify potentially interesting sets of associations. In this paper, we describe the application of automated discovery systems of relationships to astronomical data sets, focussing on an evolutionary programming technique and an information-theory technique. We demonstrate their use with classical astronomical relationships - the Hertzsprung-Russell diagram and the fundamental plane of elliptical galaxies. We also show how they work with the issue of binary classification which is relevant to the next generation of large synoptic sky surveys, such as LSST. We find that comparable results to more familiar techniques, such as decision trees, are achievable. Finally, we consider the reality of the relationships discovered and how this can be used for feature selection and extraction., Comment: 16 pages, 9 figures, accepted for publication in MNRAS

We describe the work of the Meta-Institute for Computational Astrophysics (MICA), the first professional scientific organization based in virtual worlds. MICA was an experiment in the use of this technology for science and scholarship, lasting from the early 2008 to June 2012, mainly using the Second Life and OpenSimulator as platforms. We describe its goals and activities, and our future plans. We conducted scientific collaboration meetings, professional seminars, a workshop, classroom instruction, public lectures, informal discussions and gatherings, and experiments in immersive, interactive visualization of high-dimensional scientific data. Perhaps the most successful of these was our program of popular science lectures, illustrating yet again the great potential of immersive VR as an educational and outreach platform. While the members of our research groups and some collaborators found the use of immersive VR as a professional telepresence tool to be very effective, we did not convince a broader astrophysics community to adopt it at this time, despite some efforts; we discuss some possible reasons for this non-uptake. On the whole, we conclude that immersive VR has a great potential as a scientific and educational platform, as the technology matures and becomes more broadly available and accepted., Comment: 10 pages, 8 figures; invited paper for the refereed proc. of the SLACTIONS 2012 conference

We present the analysis of 1,207 RR Lyrae found in photometry taken by the Catalina Survey's Mount Lemmon telescope. By combining accurate distances for these stars with measurements for ~14,000 type-AB RR Lyrae from the Catalina Schmid telescope, we reveal an extended association that reaches Galactocentric distances beyond 100 kpc and overlaps the Sagittarius streams system. This result confirms earlier evidence for the existence of an outer halo tidal stream resulting from a disrupted stellar system. By comparing the RR Lyrae source density with that expected based on halo models, we find the detection has ~8 sigma significance. We investigate the distances, radial velocities, metallicities, and period-amplitude distribution of the RR Lyrae. We find that both radial velocities and distances are inconsistent with current models of the Sagittarius stream. We also find tentative evidence for a division in source metallicities for the most distant sources. Following prior analyses, we compare the locations and distances of the RR Lyrae with photometrically selected candidate horizontal branch stars and find supporting evidence that this structure spans at least 60 deg of the sky. We investigate the prospects of an association between the stream and unusual globular cluster NGC 2419., Comment: 20 pages, 17 figures, 4 tables, accepted ApJ

There are many occasions when one does not have complete information in order to classify objects into different classes, and yet it is important to do the best one can since other decisions depend on that. In astronomy, especially time-domain astronomy, this situation is common when a transient is detected and one wishes to determine what it is in order to decide if one must follow it. We propose to use the Difference Boosting Neural Network (DBNN) which can boost differences between feature vectors of different objects in order to differentiate between them. We apply it to the publicly available data of the Catalina Real-Time Transient Survey (CRTS) and present preliminary results. We also describe another use with a stellar spectral library to identify spectra based on a few features. The technique itself is more general and can be applied to a varied class of problems., Comment: 8 pages, 4 tables, 1 figure, in proceedings

We present the analysis of 12227 type-ab RR Lyrae found among the 200 million public lightcurves in the Catalina Surveys Data Release 1 (CSDR1). These stars span the largest volume of the Milky Way ever surveyed with RR Lyrae, covering ~20,000 square degrees of the sky (0 < RA < 360, -22 < Dec < 65 deg) to heliocentric distances of up to 60kpc. Each of the RR Lyrae are observed between 60 and 419 times over a six-year period. Using period finding and Fourier fitting techniques we determine periods and apparent magnitudes for each source. We find that the periods at generally accurate to sigma = 0.002% by comparison with 2842 previously known RR Lyrae and 100 RR Lyrae observed in overlapping survey fields. We photometrically calibrate the light curves using 445 Landolt standard stars and show that the resulting magnitudes are accurate to ~0.05 mags using SDSS data for ~1000 blue horizontal branch stars and 7788 of the RR Lyrae. By combining Catalina photometry with SDSS spectroscopy, we analyze the radial velocity and metallicity distributions for > 1500 of the RR Lyrae. Using the accurate distances derived for the RR Lyrae, we show the paths of the Sagittarius tidal streams crossing the sky at heliocentric distances from 20 to 60 kpc. By selecting samples of Galactic halo RR Lyrae, we compare their velocity, metallicity, and distance with predictions from a recent detailed N-body model of the Sagittarius system. We find that there are some significant differences between the distances and structures predicted and our observations., Comment: 28 pages, 29 figures, accepted ApJ

An automated, rapid classification of transient events detected in the modern synoptic sky surveys is essential for their scientific utility and effective follow-up using scarce resources. This presents some unusual challenges: the data are sparse, heterogeneous and incomplete; evolving in time; and most of the relevant information comes not from the data stream itself, but from a variety of archival data and contextual information (spatial, temporal, and multi-wavelength). We are exploring a variety of novel techniques, mostly Bayesian, to respond to these challenges, using the ongoing CRTS sky survey as a testbed. The current surveys are already overwhelming our ability to effectively follow all of the potentially interesting events, and these challenges will grow by orders of magnitude over the next decade as the more ambitious sky surveys get under way. While we focus on an application in a specific domain (astrophysics), these challenges are more broadly relevant for event or anomaly detection and knowledge discovery in massive data streams., Comment: 8 pages, to appear in refereed proceedings of the IEEE eScience 2012 conference, October 2012, IEEE Press

Astronomy has been at the forefront of the development of the techniques and methodologies of data intensive science for over a decade with large sky surveys and distributed efforts such as the Virtual Observatory. However, it faces a new data deluge with the next generation of synoptic sky surveys which are opening up the time domain for discovery and exploration. This brings both new scientific opportunities and fresh challenges, in terms of data rates from robotic telescopes and exponential complexity in linked data, but also for data mining algorithms used in classification and decision making. In this paper, we describe how an informatics-based approach-part of the so-called "fourth paradigm" of scientific discovery-is emerging to deal with these. We review our experiences with the Palomar-Quest and Catalina Real-Time Transient Sky Surveys; in particular, addressing the issue of the heterogeneity of data associated with transient astronomical events (and other sensor networks) and how to manage and analyze it., Comment: 15 pages, 3 figures, to appear in special issue of Distributed and Parallel Databases on Data Intensive eScience

The time domain has been identified as one of the most important areas of astronomical research for the next decade. The Virtual Observatory is in the vanguard with dedicated tools and services that enable and facilitate the discovery, dissemination and analysis of time domain data. These range in scope from rapid notifications of time-critical astronomical transients to annotating long-term variables with the latest modeling results. In this paper, we will review the prior art in these areas and focus on the capabilities that the VAO is bringing to bear in support of time domain science. In particular, we will focus on the issues involved with the heterogeneous collections of (ancillary) data associated with astronomical transients, and the time series characterization and classification tools required by the next generation of sky surveys, such as LSST and SKA., Comment: Submitted to Proceedings of SPIE Observatory Operations: Strategies, Processes and Systems IV, Amsterdam, 2012 July 2-6

In this paper we present the Clustering-Labels-Score Patterns Spotter (CLaSPS), a new methodology for the determination of correlations among astronomical observables in complex datasets, based on the application of distinct unsupervised clustering techniques. The novelty in CLaSPS is the criterion used for the selection of the optimal clusterings, based on a quantitative measure of the degree of correlation between the cluster memberships and the distribution of a set of observables, the labels, not employed for the clustering. In this paper we discuss the applications of CLaSPS to two simple astronomical datasets, both composed of extragalactic sources with photometric observations at different wavelengths from large area surveys. The first dataset, CSC+, is composed of optical quasars spectroscopically selected in the SDSS data, observed in the X-rays by Chandra and with multi-wavelength observations in the near-infrared, optical and ultraviolet spectral intervals. One of the results of the application of CLaSPS to the CSC+ is the re-identification of a well-known correlation between the alphaOX parameter and the near ultraviolet color, in a subset of CSC+ sources with relatively small values of the near-ultraviolet colors. The other dataset consists of a sample of blazars for which photometric observations in the optical, mid and near infrared are available, complemented for a subset of the sources, by Fermi gamma-ray data. The main results of the application of CLaSPS to such datasets have been the discovery of a strong correlation between the multi-wavelength color distribution of blazars and their optical spectral classification in BL Lacs and Flat Spectrum Radio Quasars and a peculiar pattern followed by blazars in the WISE mid-infrared colors space. This pattern and its physical interpretation have been discussed in details in other papers by one of the authors., Comment: 18 pages, 9 figures, accepted for publication in ApJ

Sky surveys represent a fundamental data basis for astronomy. We use them to map in a systematic way the universe and its constituents, and to discover new types of objects or phenomena. We review the subject, with an emphasis on the wide-field imaging surveys, placing them in a broader scientific and historical context. Surveys are the largest data generators in astronomy, propelled by the advances in information and computation technology, and have transformed the ways in which astronomy is done. We describe the variety and the general properties of surveys, the ways in which they may be quantified and compared, and offer some figures of merit that can be used to compare their scientific discovery potential. Surveys enable a very wide range of science; that is perhaps their key unifying characteristic. As new domains of the observable parameter space open up thanks to the advances in technology, surveys are often the initial step in their exploration. Science can be done with the survey data alone or a combination of different surveys, or with a targeted follow-up of potentially interesting selected sources. Surveys can be used to generate large, statistical samples of objects that can be studied as populations, or as tracers of larger structures. They can be also used to discover or generate samples of rare or unusual objects, and may lead to discoveries of some previously unknown types. We discuss a general framework of parameter spaces that can be used for an assessment and comparison of different surveys, and the strategies for their scientific exploration. As we move into the Petascale regime, an effective processing and scientific exploitation of such large data sets and data streams poses many challenges, some of which may be addressed in the framework of Virtual Observatory and Astroinformatics, with a broader application of data mining and knowledge discovery technologies., Comment: An invited chapter, to appear in Astronomical Techniques, Software, and Data (ed. H. Bond), Vol.2 of Planets, Stars, and Stellar Systems (ser. ed. T. Oswalt), Springer Verlag, in press (2012). 62 pages, incl. 2 tables and 3 figures

Astronomical data are gathered through a very large number of heterogeneous techniques and stored in very diversified and often incompatible data repositories. Moreover in the e-science environment, it is needed to integrate services across distributed, heterogeneous, dynamic "virtual organizations" formed by different resources within a single enterprise and/or external resource sharing and service provider relationships. The DAME/VONeural project, run jointly by the University Federico II, INAF (National Institute of Astrophysics) Astronomical Observatories of Napoli and the California Institute of Technology, aims at creating a single, sustainable, distributed e-infrastructure for data mining and exploration in massive data sets, to be offered to the astronomical (but not only) community as a web application. The framework makes use of distributed computing environments (e.g. S.Co.P.E.) and matches the international IVOA standards and requirements. The integration process is technically challenging due to the need of achieving a specific quality of service when running on top of different native platforms. In these terms, the result of the DAME/VO-Neural project effort will be a service-oriented architecture, obtained by using appropriate standards and incorporating Grid paradigms and restful Web services frameworks where needed, that will have as main target the integration of interdisciplinary distributed systems within and across organizational domains., Comment: 10 pages, Proceedings of the Final Workshop of the Grid Projects of the Italian National Operational Programme 2000-2006 Call 1575; Edited by Cometa Consortium, 2009, ISBN: 978-88-95892-02-3

An automated, rapid classification of transient events detected in the modern synoptic sky surveys is essential for their scientific utility and effective follow-up using scarce resources. This problem will grow by orders of magnitude with the next generation of surveys. We are exploring a variety of novel automated classification techniques, mostly Bayesian, to respond to these challenges, using the ongoing CRTS sky survey as a testbed. We describe briefly some of the methods used., Comment: 3 pages, to appear in Proc. IAU 285, "New Horizons in Transient Astronomy", Oxford, Sept. 2011

The Catalina Real-time Transient Survey (CRTS) currently covers 33,000 deg^2 of the sky in search of transient astrophysical events, with time baselines ranging from 10 minutes to ~7 years. Data provided by the Catalina Sky Survey provides an unequaled baseline against which >4,000 unique optical transient events have been discovered and openly published in real-time. Here we highlight some of the discoveries of CRTS., Comment: To appear in proc. IAU Symp. 285, "New Horizons in Time Domain Astronomy", eds. E. Griffin et al., Cambridge Univ. Press (2012), 3 pages

Synoptic sky surveys are becoming the largest data generators in astronomy, and they are opening a new research frontier, that touches essentially every field of astronomy. Opening of the time domain to a systematic exploration will strengthen our understanding of a number of interesting known phenomena, and may lead to the discoveries of as yet unknown ones. We describe some lessons learned over the past decade, and offer some ideas that may guide strategic considerations in planning and execution of the future synoptic sky surveys., Comment: Invited talk, to appear in proc. IAU SYmp. 285, "New Horizons in Time Domain Astronomy", eds. E. Griffin et al., Cambridge Univ. Press (2012). Latex file, 6 pages, style files included

Exploration of the time domain - variable and transient objects and phenomena - is rapidly becoming a vibrant research frontier, touching on essentially every field of astronomy and astrophysics, from the Solar system to cosmology. Time domain astronomy is being enabled by the advent of the new generation of synoptic sky surveys that cover large areas on the sky repeatedly, and generating massive data streams. Their scientific exploration poses many challenges, driven mainly by the need for a real-time discovery, classification, and follow-up of the interesting events. Here we describe the Catalina Real-Time Transient Survey (CRTS), that discovers and publishes transient events at optical wavelengths in real time, thus benefiting the entire community. We describe some of the scientific results to date, and then focus on the challenges of the automated classification and prioritization of transient events. CRTS represents a scientific and a technological testbed and precursor for the larger surveys in the future, including the Large Synoptic Survey Telescope (LSST) and the Square Kilometer Array (SKA)., Comment: 22 pages, 12 figures, invited review for the Bulletin of Astronomical Society of India

We describe the development of a system for an automated, iterative, real-time classification of transient events discovered in synoptic sky surveys. The system under development incorporates a number of Machine Learning techniques, mostly using Bayesian approaches, due to the sparse nature, heterogeneity, and variable incompleteness of the available data. The classifications are improved iteratively as the new measurements are obtained. One novel feature is the development of an automated follow-up recommendation engine, that suggest those measurements that would be the most advantageous in terms of resolving classification ambiguities and/or characterization of the astrophysically most interesting objects, given a set of available follow-up assets and their cost functions. This illustrates the symbiotic relationship of astronomy and applied computer science through the emerging discipline of AstroInformatics., Comment: Invited paper, 15 pages, to appear in Statistical Analysis and Data Mining (ASA journal), ref. proc. CIDU 2011 conf., eds. A. Srivasatva & N. Chawla, in press (2011)

The exponential growth of astronomical data collected by both ground based and space borne instruments has fostered the growth of Astroinformatics: a new discipline laying at the intersection between astronomy, applied computer science, and information and computation (ICT) technologies. At the very heart of Astroinformatics is a complex set of methodologies usually called Data Mining (DM) or Knowledge Discovery in Data Bases (KDD). In the astronomical domain, DM/KDD are still in a very early usage stage, even though new methods and tools are being continuously deployed in order to cope with the Massive Data Sets (MDS) that can only grow in the future. In this paper, we briefly outline some general problems encountered when applying DM/KDD methods to astrophysical problems, and describe the DAME (DAta Mining & Exploration) web application. While specifically tailored to work on MDS, DAME can be effectively applied also to smaller data sets. As an illustration, we describe two application of DAME to two different problems: the identification of candidate globular clusters in external galaxies, and the classification of active galactic nuclei (AGN). We believe that tools and services of this nature will become increasingly necessary for the data-intensive astronomy (and indeed all sciences) in the 21st century.

Catalina Real-Time Transient Survey (CRTS) is a synoptic sky survey uses data streams from 3 wide-field telescopes in Arizona and Australia, covering the total area of ~30,000 deg2, down to the limiting magnitudes ~ 20 - 21 mag per exposure, with time baselines from 10 min to 6 years (and growing); there are now typically ~ 200 - 300 exposures per pointing, and coadded images reach deeper than 23 mag. The basic goal of CRTS is a systematic exploration and characterization of the faint, variable sky. The survey has detected ~ 3,000 high-amplitude transients to date, including ~ 1,000 supernovae, hundreds of CVs (the majority of them previously uncatalogued), and hundreds of blazars / OVV AGN, highly variable and flare stars, etc. CRTS has a complete open data philosophy: all transients are published immediately electronically, with no proprietary period at all, and all of the data (images, light curves) will be publicly available in the near future, thus benefiting the entire astronomical community. CRTS is a scientific and technological testbed and precursor for the grander synoptic sky surveys to come., Comment: Invited review, 6 pages, to appear in proc. "The First Year of MAXI: Monitoring Variable X-ray Sources", eds. T. Mihara & N. Kawai, Tokyo: JAXA Special Publ. (2011)

Nowadays, many scientific areas share the same need of being able to deal with massive and distributed datasets and to perform on them complex knowledge extraction tasks. This simple consideration is behind the international efforts to build virtual organizations such as, for instance, the Virtual Observatory (VObs). DAME (DAta Mining & Exploration) is an innovative, general purpose, Web-based, VObs compliant, distributed data mining infrastructure specialized in Massive Data Sets exploration with machine learning methods. Initially fine tuned to deal with astronomical data only, DAME has evolved in a general purpose platform which has found applications also in other domains of human endeavor. We present the products and a short outline of a science case, together with a detailed description of main features available in the beta release of the web application now released., Comment: 16 pages, 9 figures, software available at http://voneural.na.infn.it/beta_info.html

A survey that can cover the sky in optical bands over wide fields to faint magnitudes with a fast cadence will enable many of the exciting science opportunities of the next decade. The Large Synoptic Survey Telescope (LSST) will have an effective aperture of 6.7 meters and an imaging camera with field of view of 9.6 deg^2, and will be devoted to a ten-year imaging survey over 20,000 deg^2 south of +15 deg. Each pointing will be imaged 2000 times with fifteen second exposures in six broad bands from 0.35 to 1.1 microns, to a total point-source depth of r~27.5. The LSST Science Book describes the basic parameters of the LSST hardware, software, and observing plans. The book discusses educational and outreach opportunities, then goes on to describe a broad range of science that LSST will revolutionize: mapping the inner and outer Solar System, stellar populations in the Milky Way and nearby galaxies, the structure of the Milky Way disk and halo and other objects in the Local Volume, transient and variable objects both at low and high redshift, and the properties of normal and active galaxies at low and high redshift. It then turns to far-field cosmological topics, exploring properties of supernovae to z~1, strong and weak lensing, the large-scale distribution of galaxies and baryon oscillations, and how these different probes may be combined to constrain cosmological models and the physics of dark energy., Comment: 596 pages. Also available at full resolution at http://www.lsst.org/lsst/scibook

We identify 3,113 highly variable objects in 7,200 square degrees of the Palomar-QUEST Survey, which each varied by more than 0.4 magnitudes simultaneously in two broadband optical filters on timescales from hours to roughly 3.5 years. The primary goal of the selection is to find blazars by their well-known violent optical variability. Because most known blazars have been found in radio and/or X-ray wavelengths, a sample discovered through optical variability may have very different selection effects, elucidating the range of behavior possible in these systems. A set of blazars selected in this unusual manner will improve our understanding of the physics behind this extremely variable and diverse class of AGN. The object positions, variability statistics, and color information are available using the Palomar-QUEST CasJobs server. The time domain is just beginning to be explored over large sky areas; we do not know exactly what a violently variable sample will hold. About 20% of the sample has been classified in the literature; over 70% of those objects are known or likely AGN. The remainder largely consists of a variety of variable stars, including a number of RR Lyrae and cataclysmic variables., Comment: 22 pages (preprint format), 2 figures. Accepted for publication in ApJ. References updated

Digital synoptic sky surveys pose several new object classification challenges. In surveys where real-time detection and classification of transient events is a science driver, there is a need for an effective elimination of instrument-related artifacts which can masquerade as transient sources in the detection pipeline, e.g., unremoved large cosmic rays, saturation trails, reflections, crosstalk artifacts, etc. We have implemented such an Artifact Filter, using a supervised neural network, for the real-time processing pipeline in the Palomar-Quest (PQ) survey. After the training phase, for each object it takes as input a set of measured morphological parameters and returns the probability of it being a real object. Despite the relatively low number of training cases for many kinds of artifacts, the overall artifact classification rate is around 90%, with no genuine transients misclassified during our real-time scans. Another question is how to assign an optimal star-galaxy classification in a multi-pass survey, where seeing and other conditions change between different epochs, potentially producing inconsistent classifications for the same object. We have implemented a star/galaxy multipass classifier that makes use of external and a priori knowledge to find the optimal classification from the individually derived ones. Both these techniques can be applied to other, similar surveys and data sets., Comment: 5 pages, 5 figures. To appear in proceedings of the Class2008 conference (Classification and Discovery in Large Astronomical Surveys, Ringberg Castle, 14-17 October 2008)

Exploration of time domain is now a vibrant area of research in astronomy, driven by the advent of digital synoptic sky surveys. While panoramic surveys can detect variable or transient events, typically some follow-up observations are needed; for short-lived phenomena, a rapid response is essential. Ability to automatically classify and prioritize transient events for follow-up studies becomes critical as the data rates increase. We have been developing such methods using the data streams from the Palomar-Quest survey, the Catalina Sky Survey and others, using the VOEventNet framework. The goal is to automatically classify transient events, using the new measurements, combined with archival data (previous and multi-wavelength measurements), and contextual information (e.g., Galactic or ecliptic latitude, presence of a possible host galaxy nearby, etc.); and to iterate them dynamically as the follow-up data come in (e.g., light curves or colors). We have been investigating Bayesian methodologies for classification, as well as discriminated follow-up to optimize the use of available resources, including Naive Bayesian approach, and the non-parametric Gaussian process regression. We will also be deploying variants of the traditional machine learning techniques such as Neural Nets and Support Vector Machines on datasets of reliably classified transients as they build up., Comment: 7 pages, 3 figures, to appear in proceedings of the Class2008 conference (Classification and Discovery in Large Astronomical Surveys, Ringberg Castle, 14-17 October 2008)

S.Co.P.E. is one of the four projects funded by the Italian Government in order to provide Southern Italy with a distributed computing infrastructure for fundamental science. Beside being aimed at building the infrastructure, S.Co.P.E. is also actively pursuing research in several areas among which astrophysics and observational cosmology. We shortly summarize the most significant results obtained in the first two years of the project and related to the development of middleware and Data Mining tools for the Virtual Observatory.

There is an increasing number of large, digital, synoptic sky surveys, in which repeated observations are obtained over large areas of the sky in multiple epochs. Likewise, there is a growth in the number of (often automated or robotic) follow-up facilities with varied capabilities in terms of instruments, depth, cadence, wavelengths, etc., most of which are geared toward some specific astrophysical phenomenon. As the number of detected transient events grows, an automated, probabilistic classification of the detected variables and transients becomes increasingly important, so that an optimal use can be made of follow-up facilities, without unnecessary duplication of effort. We describe a methodology now under development for a prototype event classification system; it involves Bayesian and Machine Learning classifiers, automated incorporation of feedback from follow-up observations, and discriminated or directed follow-up requests. This type of methodology may be essential for the massive synoptic sky surveys in the future., Comment: Latex, 4 pages, 3 figures, macros included. To appear in refereed proceedings of "Hotwiring the Transient Universe 2007", eds. A. Allan, R. Seaman, and J. Bloom, Astron. Nachr. vol. 329, March, 2008

We describe briefly the Palomar-Quest (PQ) digital synoptic sky survey, including its parameters, data processing, status, and plans. Exploration of the time domain is now the central scientific and technological focus of the survey. To this end, we have developed a real-time pipeline for detection of transient sources. We describe some of the early results, and lessons learned which may be useful for other, similar projects, and time-domain astronomy in general. Finally, we discuss some issues and challenges posed by the real-time analysis and scientific exploitation of massive data streams from modern synoptic sky surveys., Comment: Latex, 3 pages, 2 figures, macros included. To appear in refereed proceedings of "Hotwiring the Transient Universe 2007", eds. A. Allan, R. Seaman, and J. Bloom, Astron. Nachr. vol. 329, March, 2008

We review some of the recent developments and challenges posed by the data analysis in modern digital sky surveys, which are representative of the information-rich astronomy in the context of Virtual Observatory. Illustrative examples include the problems of an automated star-galaxy classification in complex and heterogeneous panoramic imaging data sets, and an automated, iterative, dynamical classification of transient events detected in synoptic sky surveys. These problems offer good opportunities for productive collaborations between astronomers and applied computer scientists and statisticians, and are representative of the kind of challenges now present in all data-intensive fields. We discuss briefly some emergent types of scalable scientific data analysis systems with a broad applicability., Comment: 8 pages, compressed pdf file, figures downgraded in quality in order to match the arXiv size limit

The determination of the primary cosmic ray mass composition from the longitudinal development of atmospheric cascades is still a debated issue. In this work we discuss several data analysis methods and show that if the entire information contained in the longitudinal profile is exploited, reliable results may be obtained. Among the proposed methods FCC ('Fit of the Cascade Curve'), MTA ('Multiparametric Topological Analysis') and NNA ('Neural Net Analysis') with conjugate gradient optimization algorithm give the best accuracy., Comment: 22 pages, 11 figures, accepted by Astroparticle Physics, minor misprints and an extra figure removed

We present a neural network based approach to the determination of photometric redshift. The method was tested on the Sloan Digital Sky Survey Early Data Release (SDSS-EDR) reaching an accuracy comparable and, in some cases, better than SED template fitting techniques. Different neural networks architecture have been tested and the combination of a Multi Layer Perceptron with 1 hidden layer (22 neurons) operated in a Bayesian framework, with a Self Organizing Map used to estimate the accuracy of the results, turned out to be the most effective. In the best experiment, the implemented network reached an accuracy of 0.020 (interquartile error) in the range 0