Nonparametric estimation of the probability of discovering a new species

Abstract

A random sample is taken from a population consisting of an unknown number of distinct species. A quantity of interest is the probability of discovering a new species when an additional draw from the population is made. An estimator of this quantity was introduced by Starr (1979). We prove a conjecture of Starr’s that the estimator is uniformly minimum variance unbiased and give various asymptotic properties of the estimator. A nonparametric maximum likelihood estimator that has similar asymptotic properties is introduced. A Monte Carlo study that suggests guidelines for choosing an estimator under various circumstances is given. To amplify, suppose that if we take a sample of size 1 from a population then the probability of drawing a representative of the ith species is p1. If n draws are made with replacement, the (unconditional) probability that species i will be observed for the first time on the n + 1st draw is given simply as the term piqni(qi= 1 − pi) from a geometric distribution. Consequently, (Equation presented) represents the (unconditional) probability that some new species will be drawn for the first time on the n + 1st draw. No unbiased estimate of θn, can be derived from a sample of size n, but if one additional observation is allowed then such an estimator arises naturally. Denoted by V1, this estimator is simply the number of species observed only once divided by n + 1, the number of draws. This estimator was proposed by Robbins (1968). An analogous unbiased estimator, Vm, was proposed by Starr (1979) for the case in which m additional draws are made. Starr conjectured that Vmis a minimum variance unbiased estimate. We prove Starr’s conjecture, using the theory of U statistics. This theory can also be used to show that Vmis asymptotically normally distributed as m → ∞ and that the rate of convergence is faster if the pi’s are all equal. As an alternative to estimating θiby V1, we consider estimating piby the fraction of times species i is observed in n + 1 draws, say (Equation presented), and then estimating θnby (Equation presented). We also construct a similar estimator when m additional draws are made. Although this nonparametric maximum likelihood estimator is biased for small samples, we show that it has asymptotic properties similar to Vm: it is asymptotically unbiased and has the same asymptotic distribution. Moreover, by way of simulations and special cases, we show that (Equation presented) can dominate Vmin terms of mean squared error.