Ytdyklly

The phenomenon of 'over-dispersion' in a GLM arises whenever we use a model that restricts the variance of the response variable, and the data exhibits greater variance than the model restriction allows. This occurs commonly when modelling count data using a Poisson GLM, and it can be diagnosed by well-known tests. If tests show that there is statistically significant evidence of over-dispersion then we usually generalise the model by using a broader family of distributions that free the variance parameter from the restriction occurring under the original model. In the case of a Poisson GLM it is common to generalise either to a negative-binomial or quasi-Poisson GLM.

This situation is pregnant with an obvious objection. Why start with a Poisson GLM at all? One can start directly with the broader distributional forms, which have a (relatively) free variance parameter, and allow the variance parameter to be fit to the data, ignoring over-dispersion tests completely. In other situations when we are doing data analysis we almost always use distributional forms that allow freedom of at least the first two-moments, so why make an exception here?

My Question: Is there any good reason to start with a distribution that fixes the variance (e.g., the Poisson distribution) and then perform an over-dispersion test? How does this procedure compare with skipping this exercise completely and going straight to the more general models (e.g., negative-binomial, quasi-Poisson, etc.)? In other words, why not always use a distribution with a free variance parameter?

In principle, I actually agree that 99% of the time, it's better to just use the more flexible model. With that said, here are two and a half arguments for why you might not.

(1) Less flexible means more efficient estimates. Given that variance parameters tend to be less stable than mean parameters, your assumption of fixed mean-variance relation may stabilize standard errors more.

(2) Model checking. I've worked with physicists who believe that various measurements can be described by Poisson distributions due to theoretical physics. If we reject the hypothesis that mean = variance, we have evidence against the Poisson distribution hypothesis. As pointed out in a comment by @GordonSmyth, if you have reason to believe that a given measurement should follow a Poisson distribution, if you have evidence of over dispersion, you have evidence that you are missing important factors.

(2.5) Proper distribution. While the negative binomial regression comes from a valid statistical distribution, it's my understanding that the Quasi-Poisson does not. That means you can't really simulate count data if you believe $Var[y] = alpha E[y]$ for $alpha neq 1$. That might be annoying for some use cases. Likewise, you can't use probabilities to test for outliers, etc.

Although this is my own question, I'm also going to post my own two-cents as an answer, so that we add to the number of perspectives on this question. The issue here is whether or not it is sensible to initially fit a one-parameter distribution to data. When you use a one-parameter distribution (such as the Poisson GLM, or a binomial GLM with fixed trial parameter), the variance is not a free parameter, and is instead constrained to be some function of the mean. This means that it is ill-advised to fit a one-parameter distribution to data in any situation where you are not absolutely sure that the variance follows the structure of that distribution.

Fitting one-parameter distributions to data is almost always a bad idea: Data is often messier than proposed models indicate, and even when there are theoretical reasons to believe that a particular one-parameter model may obtain, it is often the case that the data actually come from a mixture of that one-parameter distribution, with a range of parameter values. This is often equivalent to a broader model, such as a two-parameter distribution that allows greater freedom for the variance. As discussed below, this is true for the Poisson GLM in the case of count data.

As stated in the question, in most applications of statistics, it is standard practice to use distributional forms that at least allow the first two moments to vary freely. This ensures that the fitted model allows the data to dictate the inferred mean and variance, rather than having these artificially constrained by the model. Having this second parameter only loses one degree-of-freedom in the model, which is a tiny loss compared to the benefit of allowing the variance to be estimated from the data. One can of course extend this reasoning and add a third parameter to allow fitting of skewness, a fourth to allow fitting of kurtosis, etc. The reason that these higher-order moments are not usually as important is that asymptotic theorems for estimators usually show that they converge to a normal distribution (regardless of the higher-order moments of the underlying data) and in this case estimates of the mean and variance are sufficient to get good estimates of the asymptotic distribution of the parameter estimators.

With some extremely minor exceptions, a Poisson GLM is a bad model: In my experience, fitting a Poisson distribution to count data is almost always a bad idea. For count data it is extremely common for the variance in the data to be 'over-dispersed' relative to the Poisson distribution. Even in situations where theory points to a Poisson distribution, often the best model is a mixture of Poisson distributions, where the variance becomes a free parameter. Indeed, in the case of count data the negative-binomial distribution is a Poisson mixture with a gamma distribution for the rate parameter, so even when there are theoretical reasons to think that the counts arrive according to the process of a Poisson distribution, it is often the case that there is 'over-dispersion' and the negative-binomial distribution fits much better.

The practice of fitting a Poisson GLM to count data and then doing a statistical test to check for 'over-dispersion' is an anachronism, and it is hardly ever a good practice. In other forms of statistical analysis, we do not start with a two-parameter distribution, arbitrarily choose a variance restriction, and then test for this restriction to try to eliminate a parameter from the distribution. By doing things this way, we actually create an awkward hybrid procedure, consisting of an initial hypothesis test used for model selection, and then the actual model (either Poisson, or a broader distribution). It has been shown in many contexts that this kind of practice of creating hybrid models from an initial model selection test leads to bad overall models.

An analogous situation, where a similar hybrid method has been used, is in T-tests of mean difference. It used to be the case that statistics courses would recommend first using Levene's test (or even just some much crappier "rules of thumb") to check for equality of variances between two populations, and then if the data "passed" this test you would use the Student T-test that assumes equal variance, and if the data "failed" the test then you would instead use Welch's T-test. This is actually a really bad procedure (see e.g., here and here). It is much better just to use the latter test, which makes no assumption on the variance, rather than creating an awkward compound test that jams together a preliminary hypothesis test and then uses this to choose the model.

For count data, you will generally get good initial results by fitting a two-parameter model such as a negative-binomial or quasi-Poisson model. (Note that the latter is not a real distribution, but it still gives a reasonable two-parameter model.) If any further generalisation is needed at all, it is usually the addition of zero-inflation, where there are an excessive number of zeroes in the data. Restricting to a Poisson GLM is an artificial and senseless model choice, and this is not made much better by testing for over-dispersion.

Okay, now here are the minor exceptions: The only real exceptions to the above are two situations:

(1) You have extremely strong a priori theoretical reasons for believing that the assumptions for the one parameter distribution are satisfied, and part of the analysis is to test this theoretical model against the data; or

(2) For some other (strange) reason, the purpose of your analysis is to conduct a hypothesis test on the variance of the data, and so you actually want to restrict this variance to this hypothesised restriction, and then test this hypothesis.

These situations are very rare. They tend to arise only when there is strong a priori theoretical knowledge about the data-generating mechanism, and the purpose of the analysis is to test this underlying theory. This may be the case in an extremely limited range of applications where data is generated under tightly controlled conditions (e.g., in physics).