Search

Consider an (observable) random sample of size n from an infinite population of individuals, each individual being endowed with a finite set of "features" from a collection of features (F-j)(j>1) with unknown probabilities (p(j))(j>1), i.e., p(j) is the probability that an individual displays feature F-j. Under this feature sampling framework, in recent years there has been a growing interest in estimating the sum of the probability masses p(j)'s of features observed with frequency r >= 0 in the sample, here denoted by M-n,M-r. This is the natural feature sampling counterpart of the classical problem of estimating small probabilities in the species sampling framework, where each individual is endowed with only one feature (or "species"). In this paper we study the problem of consistent estimation of the small mass M-n,M-r. We first show that there do not exist universally consistent estimators, in the multiplicative sense, of the missing mass M-n,M-0. Then, we introduce an estimator of M-n,M-r and identify sufficient conditions under which the estimator is consistent. In particular, we propose a nonparametric estimator (M) over cap (n,r) of M-n,M-r which has the same analytic form of the celebrated Good-Turing estimator for small probabilities, with the sole difference that the two estimators have different ranges (supports). Then, we show that (M) over cap (n,r) is strongly consistent, in the multiplicative sense, under the assumption that (p(j))(j >= 1) has regularly varying heavy tails.

Etant donne un echantillon de taille n dans une population d’individus appartenant a differents types dont les proportions sont inconnues, comment estimer la probabilite de decouvrir un nouveau type au (n+1)-ieme tirage ? C’est un probleme classique en statistique, souvent appele le probleme de l’estimation de la masse manquante. Des resultats recents on montre : (i) l’impossibilite d’estimer la masse manquante sans imposer des hypotheses sur les proportions des types ; (ii) la convergence de l’estimateur de la masse manquante de Good–Turing sous l’hypothese que la queue des proportions des types decroit vers 0 comme une fonction reguliere de parametre α∈(0,1) ; (iii) la vitesse de convergence n−α/2 pour l’estimateur de Good–Turing pour la classe de probabilites a variation reguliere α∈(0,1). Dans cet article, nous proposons une preuve alternative, et remarquablement plus courte, de l’impossibilite de l’estimation de la masse manquante sans hypothese sur la distribution. Au dela de son interet propre, cette preuve alternative suggere une approche naturelle pour ameliorer et etendre les resultats de vitesse de convergence de l’estimateur de Good–Turing sous l’hypothese de proportions a variation reguliere α∈(0,1). En particulier, nous montrons que la vitesse de convergence n−α/2 est la meilleure que peut atteindre un estimateur, a une fonction a variation bornee pres. De plus, nous montrons qu’une borne inferieure a l’estimation du risque minimax est au moins d’echelle n−α/2, ce qui amene a la conjecture que l’estimateur de Good–Turing est l’estimateur minimax de vitesse optimale sous une hypothese de proportions a variation reguliere.

Feature allocation models generalize species sampling models by allowing every observation to belong to more than one species, now called features. Under the popular Bernoulli product model for feature allocation, given $n$ samples, we study the problem of estimating the missing mass $M_{n}$, namely the expected number hitherto unseen features that would be observed if one additional individual was sampled. This is motivated by numerous applied problems where the sampling procedure is expensive, in terms of time and/or financial resources allocated, and further samples can be only motivated by the possibility of recording new unobserved features. We introduce a simple, robust and theoretically sound nonparametric estimator $\hat{M}_{n}$ of $M_{n}$. $\hat{M}_{n}$ turns out to have the same analytic form of the popular Good-Turing estimator of the missing mass in species sampling models, with the difference that the two estimators have different ranges. We show that $\hat{M}_{n}$ admits a natural interpretation both as a jackknife estimator and as a nonparametric empirical Bayes estimator, we give provable guarantees for the performance of $\hat{M}_{n}$ in terms of minimax rate optimality, and we provide with an interesting connection between $\hat{M}_{n}$ and the Good-Turing estimator for species sampling. Finally, we derive non-asymptotic confidence intervals for $\hat{M}_{n}$, which are easily computable and do not rely on any asymptotic approximation. Our approach is illustrated with synthetic data and SNP data from the ENCODE sequencing genome project.