Search

We define and prove limit results for a class of dominant P\'olya sequences, which are randomly reinforced urn processes with color-specific random weights and unbounded number of possible colors. Under fairly mild assumptions on the expected reinforcement, we show that the predictive and the empirical distributions converge almost surely (a.s.) in total variation to the same random probability measure $\tilde{P}$; moreover, $\tilde{P}(\mathcal{D})=1$ a.s., where $\mathcal{D}$ denotes the set of dominant colors for which the expected reinforcement is maximum. In the general case, the predictive probabilities and the empirical frequencies of any $\delta$-neighborhood of $\mathcal{D}$ converge a.s. to one. That is, although non-dominant colors continue to be regularly observed, their distance to $\mathcal{D}$ converges in probability to zero. We refine the above results with rates of convergence. We further hint potential applications of dominant P\'olya sequences in randomized clinical trials and species sampling, and use our central limit results for Bayesian inference.

The Pitman-Yor process is a random probability distribution, that can be used as a prior distribution in a nonparametric Bayesian analy-sis. The process is of species sampling type and generates discrete distribu-tions, which yield of the order nσ different values (“species”) in a random sample of size n, ifthetypeσ is positive. Thus this type parameter can be set to target true distributions of various levels of discreteness, making the Pitman-Yor process an interesting prior in this case. It was previously shown that the resulting posterior distribution is consistent if and only if the true distribution of the data is discrete. In this paper we derive the dis-tributional limit of the posterior distribution, in the form of a (corrected) Bernstein-von Mises theorem, which previously was known only in the con-tinuous, inconsistent case. It turns out that the Pitman-Yor posterior distribution has good behaviour if the true distribution of the data is discrete with atoms that decrease not too slowly. Credible sets derived from the posterior distribution provide valid frequentist confidence sets in this case. For a general discrete distribution, the posterior distribution, although con-sistent, may contain a bias which does not converge to zero at the√n rate and invalidates posterior inference. We propose a bias correction that solves this problem. We also consider the effect of estimating the type parameter from the data, both by empirical Bayes and full Bayes methods. In a small simulation study we illustrate that without bias correction the coverage of credible sets can be arbitrarily low, also for some discrete distributions.

Consider an (observable) random sample of size n from an infinite population of individuals, each individual being endowed with a finite set of "features" from a collection of features (F-j)(j>1) with unknown probabilities (p(j))(j>1), i.e., p(j) is the probability that an individual displays feature F-j. Under this feature sampling framework, in recent years there has been a growing interest in estimating the sum of the probability masses p(j)'s of features observed with frequency r >= 0 in the sample, here denoted by M-n,M-r. This is the natural feature sampling counterpart of the classical problem of estimating small probabilities in the species sampling framework, where each individual is endowed with only one feature (or "species"). In this paper we study the problem of consistent estimation of the small mass M-n,M-r. We first show that there do not exist universally consistent estimators, in the multiplicative sense, of the missing mass M-n,M-0. Then, we introduce an estimator of M-n,M-r and identify sufficient conditions under which the estimator is consistent. In particular, we propose a nonparametric estimator (M) over cap (n,r) of M-n,M-r which has the same analytic form of the celebrated Good-Turing estimator for small probabilities, with the sole difference that the two estimators have different ranges (supports). Then, we show that (M) over cap (n,r) is strongly consistent, in the multiplicative sense, under the assumption that (p(j))(j >= 1) has regularly varying heavy tails.

ln Africa we find incredibly diverse ecosystems which are home to a huge diversity of species. Unfortunately, as the latest literature suggests, the population sizes of mammals, birds, fish, amphibians and reptiles have declined by 65% in the last 50 years. The main drivers include deforestation, human population growth, urbanization, collection of natural resources and clearing of land for agriculture. With the African population expected to double in the next 50 years, it is expected an exacerbated effect of the drivers that currently contribute to biodiversity decline. The fact that a third of Africa lives under the poverty line together with high leveis of corruption makes it especially important that the sparse existing resources dedicated to biodiversity preservation can be transparently and effectively distributed throughout the most important areas for biodiversity. Besides the chapters of general introduction and discussion, this thesis contains four chapters representing published or submitted papers. ln chapter 1, 1 propose a new metric that allows the ranking of areas based on biodiversity importance. ln chapter 2, 1 showcase the lack of biodiversity sampling in Africa unveiling an important sampling bias and making predictions on sampling time and effort to sample biodiversity in Africa. ln chapter 3, 1 compile important baseline information on amphibians and reptiles from a poorly known area in terms of biodiversity in northern Mozambique and in chapter 4, we show that endemisms are very dependent on which scale or taxonomic treatment we decide to use in our analysis. Overall, the work developed in this thesis is a contribution to the transparency and prioritisation procedures at biodiversity management globally and showcases the current situation of the African biodiversity sampling and how it may translate into the future. Furthermore, it provides an example on how biodiversity baseline information can be acquired and made available as well as highlighting the importance of taxonomy and scale when inferring patterns from spatial analysis. Em África podemos encontrar uma diversidade incrível de ecossistemas que alojam uma enorme diversidade de espécies. Infelizmente, de acordo com a literatura mais recente, as populações de mamíferos, aves, peixes, anfíbios e répteis sofreram um declínio de 65% nos últimos 50 anos. As principais causas incluem a desflorestação, o crescimento populacional, urbanização, a recolha de recursos naturais e o desmatamento para a implementação da agricultura. Com a previsão que a população humana em África duplique nos próximos 50 anos, espera-se um efeito exacerbado das causas que neste momento contribuem para o declínio da biodiversidade. O facto de um terço de África viver abaixo da linha de pobreza e dados os altos níveis de corrupção torna especialmente importante que os parcos recursos existentes destinados à preservação da biodiversidade possam ser transparente e efetivamente distribuídos pelas áreas de biodiversidade mais importantes. Para além dos capítulos da Introdução e Discussão gerais, esta tese contém quatro capítulos, que representam artigos publicados ou submetidos. No capítulo 1, propõe-se uma fórmula que permite hierarquizar áreas de acordo com a sua importância em termos de biodiversidade. No capítulo 2 demonstra-se a falta de inventários de biodiversidade em África, revelando-se ainda uma tendência importante da amostragem no continente. São também feitas previsões em relação ao tempo e esforço necessário para amostrar biodiversidade em África. No capítulo 3, compila-se informação importante em anfíbios e répteis de uma área pouco amostrada em termos de biodiversidade no Norte de Moçambique, e no capítulo 4 mostra-se que os endemismos são muito dependentes da escala ou da taxonomia usada que é aplicada durante as análises. Programa Doutoral em Biologia e Ecologia das Alterações Globais

In the present paper, we address the problem of prediction within the setting of species sampling models. We consider d populations composed of different species with unknown proportions. Our goal is to predict specific features of additional and unobserved samples from the d populations by adopting a Bayesian nonparametric model. We focus on a broad class of hierarchical priors. These were introduced and investigated in [1], where also an algorithm for drawing predictions is devised, however, without any specific numerical illustration. The aim of this paper is twofold: on the one hand, we provide an illustration with an actual implementation of the algorithm of [1] and, on the other hand, we discuss its relevance with respect to complex prediction problems with species sampling data.

The definition and investigation of general classes of nonparametric priors has recently been an active research line in Bayesian statistics. Among the various proposals, the Gibbs-type family, which includes the Dirichlet process as a special case, stands out as the most tractable class of nonparametric priors for exchangeable sequences of observations. This is the consequence of a key simplifying assumption on the learning mechanism, which, however, has justification except that of ensuring mathematical tractability. In this paper, we remove such an assumption and investigate a general class of random probability measures going beyond the Gibbs-type framework. More specifically, we present a nonparametric hierarchical structure based on transformations of completely random measures, which extends the popular hierarchical Dirichlet process. This class of priors preserves a good degree of tractability, given that we are able to determine the fundamental quantities for Bayesian inference. In particular, we derive the induced partition structure and the prediction rules and characterize the posterior distribution. These theoretical results are also crucial to devise both a marginal and a conditional algorithm for posterior inference. An illustration concerning prediction in genomic sequencing is also provided.

We describe the distribution of frequencies ordered by sample values in a random sample of size $n$ from the two parameter GEM$(\alpha,\theta)$ random discrete distribution on the positive integers. These frequencies are a $($size$-\alpha)$-biased random permutation of the sample frequencies in either ranked order, or in the order of appearance of values in the sampling process. This generalizes a well known identity in distribution due to Donnelly and Tavar\'e (1986) for $\alpha = 0$ to the case $0 \le \alpha < 1$. This description extends to sampling from Gibbs$(\alpha)$ frequencies obtained by suitable conditioning of the GEM$(\alpha,\theta)$ model, and yields a value-ordered version of the Chinese Restaurant construction of GEM$(\alpha,\theta)$ and Gibbs$(\alpha)$ frequencies in the more usual size-biased order of their appearance. The proofs are based on a general construction of a finite sample $(X_1,\dots,X_n)$ from any random frequencies in size-biased order from the associated exchangeable random partition $\Pi_\infty$ of $\mathbb{N}$ which they generate., Comment: 23 pages, some corrections to the first version

Motivated by the fundamental problem of modeling the frequency of frequencies (FoF) distribution, this paper introduces the concept of a cluster structure to define a probability function that governs the joint distribution of a random count and its exchangeable random partitions. A cluster structure, naturally arising from a completely random measure mixed Poisson process, allows the probability distribution of the random partitions of a subset of a population to be dependent on the population size, a distinct and motivated feature that makes it more flexible than a partition structure. This allows it to model an entire FoF distribution whose structural properties change as the population size varies. A FoF vector can be simulated by drawing an infinite number of Poisson random variables, or by a stick-breaking construction with a finite random number of steps. A generalized negative binomial process model is proposed to generate a cluster structure, where in the prior the number of clusters is finite and Poisson distributed, and the cluster sizes follow a truncated negative binomial distribution. We propose a simple Gibbs sampling algorithm to extrapolate the FoF vector of a population given the FoF vector of a sample taken without replacement from the population. We illustrate our results and demonstrate the advantages of the proposed models through the analysis of real text, genomic, and survey data., Comment: To appear in the Journal of the American Statistical Association (Theory and Methods). 26 pages + 17 page supplement, 19 figures. arXiv admin note: text overlap with arXiv:1410.3155

Rarefaction curves represent a powerful method for comparing species richness among habitats on an equal-effort basis. Three assumptions are required to correctly perform rarefaction analysis: (i) data collection should be a representative sample of the community under study, (ii) individuals are randomly dispersed, and (iii) species are independently dispersed. However, the community structure is spatially organized, and these criteria cannot be guaranteed. Recently, Chiarucci et al. (2009) proposed a new type of rarefaction, named Spatially Constrained Rarefaction (SCR), which allows to include the autocorrelated structure of the samples in the construction of a rarefaction curve. Here we present an easy-to-use procedure to calculate Spatially Constrained Rarefaction curve in the R environment.

The mixture model devised by Fisher, Corbet and Williams [Journal of Animal Ecology 12 (1943) 42–58] for species sampling and the sequential prediction approach pioneered by Good [Biometrika 40 (1953) 237–264] and Good and Toulmin [Biometrika 43 (1956) 45–63] are both closely related to the Ewens sampling formula. Fisher’s two-parameter joint distribution for the species counts includes the Ewens distribution as the conditional distribution given the sample size. The log-series model, as it is known in the ecological literature, is closely related to a Poisson process model devised by Arratia, Barbour and Tavare [Ann. Appl. Probab. 2 (1992) 519–535]. Oddly, despite its advantages for statistical inference, Fisher does not mention the conditional distribution. Likewise, athough Good (1953) pioneered the sequential prediction approach, neither he nor Toulmin discovered the Ewens process in a form equivalent to the modern-day Chinese restaurant process.

We consider the problem of evaluating the probability of discovering a certain number of new species in a new sample of population units, conditional on the number of species recorded in a basic sample. We use a Bayesian nonparametric approach. The different species proportions are assumed to be random and the observations from the population exchangeable. We provide a Bayesian estimator, under quadratic loss, for the probability of discovering new species which can be compared with well-known frequentist estimators. The results we obtain are illustrated through a numerical example and an application to a genomic dataset concerning the discovery of new genes by sequencing additional single-read sequences of cdna fragments. Copyright 2007, Oxford University Press.

Discrete random probability measures and the exchangeable random partitions they induce are key tools for addressing a variety of estimation and prediction problems in Bayesian inference. Indeed, many popular nonparametric priors, such as the Dirichlet and the Pitman-Yor process priors, select discrete probability distributions almost surely and, therefore, automatically induce exchangeable random partitions. Here we focus on the family of Gibbs-type priors, a recent and elegant generalization of the Dirichlet and the Pitman-Yor process priors. These random probability measures share properties that are appealing both from a theoretical and an applied point of view: (i) they admit an intuitive characterization in terms of their predictive structure justifying their use in terms of a precise assumption on the learning mechanism; (ii) they stand out in terms of mathematical tractability; (iii) they include several interesting special cases besides the Dirichlet and the Pitman-Yor processes. The goal of our paper is to provide a systematic and unified treatment of Gibbs-type priors and highlight their implications for Bayesian nonparametric inference. We will deal with their distributional properties, the resulting estimators, frequentist asymptotic validation and the construction of time-dependent versions. Applications, mainly concerning hierarchical mixture models and species sampling, will serve to convey the main ideas. The intuition inherent to this class of priors and the neat results that can be deduced for it lead one to wonder whether it actually represents the most natural generalization of the Dirichlet process., 5 figures

We introduce a three-parameter random walk with reinforcement, called the $(\theta,\alpha,\beta)$ scheme, which generalizes the linearly edge reinforced random walk to uncountable spaces. The parameter $\beta$ smoothly tunes the $(\theta,\alpha,\beta)$ scheme between this edge reinforced random walk and the classical exchangeable two-parameter Hoppe urn scheme, while the parameters $\alpha$ and $\theta$ modulate how many states are typically visited. Resorting to de Finetti's theorem for Markov chains, we use the $(\theta,\alpha,\beta)$ scheme to define a nonparametric prior for Bayesian analysis of reversible Markov chains. The prior is applied in Bayesian nonparametric inference for species sampling problems with data generated from a reversible Markov chain with an unknown transition kernel. As a real example, we analyze data from molecular dynamics simulations of protein folding., Comment: Published in at http://dx.doi.org/10.1214/13-AOS1102 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

In this paper we analyze the asymptotic behaviour of Gibbs-type priors, that represent a natural generalization of the Dirichlet process. After determining their topological support, we investigate their consistency according to the "what if", or frequentist, approach, that postulates the existence of a "true" distribution P 0 . We provide a full taxonomy of their limiting behaviours: consistency holds essentially always for discreteP 0 , whereas inconsistency may occur for diffuseP 0 . Such findings are further illustrated by means of three special cases admitting closed form expressions and exhibiting a wide range of asymptotic behaviours. For both Gibbs-type priors and discrete nonparametric priors in general, the possible inconsistency should not be interpreted as evidence against their usetout court. It rather represents an indication that they are designed for modeling discrete distributions and evidence against their use in the case of diffuseP 0 .

L’explosion du nombre de séquences permet à la phylogénomique, c’est-à-dire l’étude des liens de parenté entre espèces à partir de grands alignements multi-gènes, de prendre son essor. C’est incontestablement un moyen de pallier aux erreurs stochastiques des phylogénies simple gène, mais de nombreux problèmes demeurent malgré les progrès réalisés dans la modélisation du processus évolutif. Dans cette thèse, nous nous attachons à caractériser certains aspects du mauvais ajustement du modèle aux données, et à étudier leur impact sur l’exactitude de l’inférence. Contrairement à l’hétérotachie, la variation au cours du temps du processus de substitution en acides aminés a reçu peu d’attention jusqu’alors. Non seulement nous montrons que cette hétérogénéité est largement répandue chez les animaux, mais aussi que son existence peut nuire à la qualité de l’inférence phylogénomique. Ainsi en l’absence d’un modèle adéquat, la suppression des colonnes hétérogènes, mal gérées par le modèle, peut faire disparaître un artéfact de reconstruction. Dans un cadre phylogénomique, les techniques de séquençage utilisées impliquent souvent que tous les gènes ne sont pas présents pour toutes les espèces. La controverse sur l’impact de la quantité de cellules vides a récemment été réactualisée, mais la majorité des études sur les données manquantes sont faites sur de petits jeux de séquences simulées. Nous nous sommes donc intéressés à quantifier cet impact dans le cas d’un large alignement de données réelles. Pour un taux raisonnable de données manquantes, il appert que l’incomplétude de l’alignement affecte moins l’exactitude de l’inférence que le choix du modèle. Au contraire, l’ajout d’une séquence incomplète mais qui casse une longue branche peut restaurer, au moins partiellement, une phylogénie erronée. Comme les violations de modèle constituent toujours la limitation majeure dans l’exactitude de l’inférence phylogénétique, l’amélioration de l’échantillonnage des espèces et des gènes reste une alternative utile en l’absence d’un modèle adéquat. Nous avons donc développé un logiciel de sélection de séquences qui construit des jeux de données reproductibles, en se basant sur la quantité de données présentes, la vitesse d’évolution et les biais de composition. Lors de cette étude nous avons montré que l’expertise humaine apporte pour l’instant encore un savoir incontournable. Les différentes analyses réalisées pour cette thèse concluent à l’importance primordiale du modèle évolutif., The explosion of sequence number allows for phylogenomics, the study of species relationships based on large multi-gene alignments, to flourish. Without any doubt, phylogenomics is essentially an efficient way to eliminate the problems of single gene phylogenies due to stochastic errors, but numerous problems remain despite obvious progress realized in modeling evolutionary process. In this PhD-thesis, we are trying to characterize some consequences of a poor model fit and to study their impact on the accuracy of the phylogenetic inference. In contrast to heterotachy, the variation in the amino acid substitution process over time did not attract so far a lot of attention. We demonstrate that this heterogeneity is frequently observed within animals, but also that its existence can interfere with the quality of phylogenomic inference. In absence of an adequate model, the elimination of heterogeneous columns, which are poorly handled by the model, can eliminate an artefactual reconstruction. In a phylogenomic framework, the sequencing strategies often result in a situation where some genes are absent for some species. The issue about the impact of the quantity of empty cells was recently relaunched, but the majority of studies on missing data is performed on small datasets of simulated sequences. Therefore, we were interested on measuring the impact in the case of a large alignment of real data. With a reasonable amount of missing data, it seems that the accuracy of the inference is influenced rather by the choice of the model than the incompleteness of the alignment. For example, the addition of an incomplete sequence that breaks a long branch can at least partially re-establish an artefactual phylogeny. Because, model violations are always representing the major limitation of the accuracy of the phylogenetic inference, the improvement of species and gene sampling remains a useful alternative in the absence of an adequate model. Therefore, we developed a sequence-selection software, which allows the reproducible construction of datasets, based on the quantity of data, their evolutionary speed and their compositional bias. During this study, we did realize that the human expertise still furnishes an indispensable knowledge. The various analyses performed in the course of this PhD thesis agree on the primordial importance of the model of sequence evolution.

A Bayesian nonparametric methodology has been recently proposed in order to deal with the issue of prediction within species sampling problems. Such problems concern the evaluation, conditional on a sample of size n, of the species variety featured by an additional sample of size m. Genomic applications pose the additional challenge of having to deal with large values of both n and m. In such a case the computation of the Bayesian nonparametric estimators is cumbersome and prevents their implementation. In this paper we focus on the two parameter Poisson-Dirichlet model and provide completely explicit expressions for the corresponding estimators, which can be easily evaluated for any sizes of n and m. We also study the asymptotic behaviour of the number of new species conditionally on the observed sample: such an asymptotic result allows, combined with a suitable simulation scheme, to derive asymptotic highest posterior density intervals for the estimates of interest. Finally, we illustrate the implementation of the proposed methodology by the analysis of five Expressed Sequence Tags (EST) datasets.

SummaryA Bayesian non-parametric methodology has been recently proposed to deal with the issue of prediction within species sampling problems. Such problems concern the evaluation, conditional on a sample of size n, of the species variety featured by an additional sample of size m. Genomic applications pose the additional challenge of having to deal with large values of both n and m. In such a case the computation of the Bayesian non-parametric estimators is cumbersome and prevents their implementation. We focus on the two-parameter Poisson–Dirichlet model and provide completely explicit expressions for the corresponding estimators, which can be easily evaluated for any sizes of n and m. We also study the asymptotic behaviour of the number of new species conditionally on the observed sample: such an asymptotic result, combined with a suitable simulation scheme, allows us to derive asymptotic highest posterior density intervals for the estimates of interest. Finally, we illustrate the implementation of the proposed methodology by the analysis of five expressed sequence tags data sets.

This paper collects facts about the number of occupied boxes in the classical balls-in-boxes occupancy scheme with infinitely many positive frequencies: equivalently, about the number of species represented in samples from populations with infinitely many species. We present moments of this random variable, discuss asymptotic relations among them and with related random variables, and draw connections with regular variation, which appears in various manifestations., Comment: Published at http://dx.doi.org/10.1214/07-PS092 in the Probability Surveys (http://www.i-journals.org/ps/) by the Institute of Mathematical Statistics (http://www.imstat.org)