Read other letters about this article

• Wednesday, October 8, 2008 04:19 PM

  In defense of 80% confidence intervals.

  Hi. Statistician here. There's really nothing wrong with using an 80% confidence interval. You aren't required to use 95% or greater for a confidence interval. Keep in mind that the only reason alpha levels of 0.1, 0.05, and 0.01 (giving CI's of 90%, 95%, and 99%) became the "standard" is because for decades one had to refer to tables in textbooks to compute these things, and the authors didn't have the space to include tables for each and every percentage value. So they chose the most intuitive to include in their books. It can be argued that if they had the computing power a century ago that we have now, we would be doing confidence intervals a lot differently today.

  There are lots of different kinds of analysis where not only is 80% confidence accepted, but it is necessary to use a value that low. It all comes down to the acceptable Type I error, or how frequently you will conclude that something interesting is going on when it isn't. If it's okay to be wrong 20% of the time, then an 80% CI is perfectly fine.

  Frankly there's no problem in using a 70% CI, 50% CI (I've seen 'em), or a 63.286529% confidence interval. The vitally important thing is that you choose it BEFORE YOU RUN THE STATS. Actually you should choose it before you start collecting data, and in most cases you go with the standard for your academic subject. So here's the real criticism of this study: they should have used the polling standard of 95%. I haven't read the actual study, but they should have phrased their conclusion, "Although the result are no significant at the 95% level, it is significant at the 80% level." If they only reported the 80% CI, then they are weaseling things a bit by implicitly stating that 80% is good enough.

  But does an 80% confidence interval give us good information? Damn right it does!! Anyone automatically dismissing these results because it isn't at 95% doesn't know what they're talking about. An 80% CI says it's more likely than not (or roughly a 4 in 5 chance) that there's something really happening here. There is important imformation here.

  Another point: What's much more important than confidence level and even the size of the sample is the quality of the sample. The confidence level and sample size are really easy for the layman to latch on to because they are always reported (or should be), and the sampling method details are not. But if it's a nonrepresentative sample, you could poll a million people and use 99.9999% confidence and it's still bullshit. The conclusions are worthless, and in comparison a small sample with 80% confidence is infinitely better.

  —Mr Smith Read Mr Smith's other letters
  Permalink
  Flag
  Click "Submit Flag" if you feel this post is inappropriate. Explain why below if you wish.

  Cancel