tag:blogger.com,1999:blog-4900303239154048192.post7415238639457034108..comments2024-03-06T06:34:42.881-05:00Comments on EconoSpeak: The Great P Value ControversyUnknownnoreply@blogger.comBlogger5125tag:blogger.com,1999:blog-4900303239154048192.post-54861367575067551452016-06-08T16:38:11.925-04:002016-06-08T16:38:11.925-04:00As I've studied statistics through the lens of...As I've studied statistics through the lens of epidemiological data analysis, I've become far more skeptical of the (over) reliance on p-values as the linchpin of statistical significance.<br /><br />Rothman, one of the authors of a well known epi textbook, warns repeatedly against relying too heavily upon p-values as the sole (or most important) measure of significance. You can imagine how important it would be to keep this in mind when conducting a drug trial or investigating correlations between risk factors and specific negative health impacts. A test that fails to achieve a predetermined measure of statistical significance may very well hold some vital "real world" significance which could literally be a matter of life and death. Clearly, holding p-values in esteem above the other obtained statistics and inherent limitations of your model can obscure important data points.<br />Rothman et al encourage epidemiologists to use estimation (confidence intervals, p-value functions, and even push for Bayesian analysis) in their research, and if statistical significance is achieved, well that's fine.<br /><br />The overall message of the text, which should be explicit in all stats classes, is that statistical models should all be subject to healthy skepticism. Statistical analysis is one tool in the kit of scientific inquiry, and each model is more of a tree in the forest, rather than being a forest by itself. That point, I think, is too frequently missed in frequentist model stats books and classes.Anonymoushttps://www.blogger.com/profile/15025727965483438909noreply@blogger.comtag:blogger.com,1999:blog-4900303239154048192.post-64906881886494502692016-05-29T13:49:36.422-04:002016-05-29T13:49:36.422-04:00Well, let's take an example. Suppose you'...Well, let's take an example. Suppose you're a pollster for a politician. The campaign strategy depends on whether the candidate is ahead or behind. So you do a poll and have a sample. Your guy (could be female) scores a little higher. But how sure are you of this result? You surveyed 800 people out of an electorate of millions. Yes, your descriptive stats will tell you what percent of the people you polled are in favor/opposed/don't give a shit and you can even slice and dice your sample into demographics/geography/whatever. But how much credence should you give these numbers? How likely would you be to find your candidate behind if you took another sample? That's what significance testing is for.<br /><br />Of course, published polls always report their confidence intervals (a variation on significance testing) and many of them are garbage. There are lots of other factors to consider besides sampling uncertainty. This is part of the critique. But would you want to ignore sampling uncertainty altogether?<br /><br />Meanwhile, there is an important difference between descriptive stats and statistical tests. The descriptives are dependent on the real world out there and your data collection and measurement methods. Statistical tests depend on both of those <b>plus</b> all the modeling choices you made <b>and</b>, for significance testing, the conditional assumption that the null hypothesis is correct. You're adding a lot of if's, so the interpretation has to be different.Peter Dormanhttps://www.blogger.com/profile/00093399591393648071noreply@blogger.comtag:blogger.com,1999:blog-4900303239154048192.post-22430040336165527082016-05-29T13:24:09.060-04:002016-05-29T13:24:09.060-04:00In practical work and real time, all you may have ...In practical work and real time, all you may have is that one sample, so rather than over-generalize on the basis of some hypothesis test, to me it's more reasonable to limit oneself to description. After all, the results of one hypothesis test on one sample is just description itself, no?MaxSpeakhttps://www.blogger.com/profile/08594964334301228571noreply@blogger.comtag:blogger.com,1999:blog-4900303239154048192.post-87179753350787623162016-05-29T13:06:59.146-04:002016-05-29T13:06:59.146-04:00There are two problems with staying at the level o...There are two problems with staying at the level of descriptive stats, Max. One is that they tell you only about the sample and not how well the sample generalizes to the underlying population. For policy purposes, it's <i>future</i> samples, altered by our policies, that we care about, and the generalizations we want to make extend over time. Another is that there are often important patterns in the data whose existence -- or limitations! -- aren't visible to the naked eye. You've got to model it to figure it out.<br /><br />Null hypothesis statistical testing is just one technique in modeling, and the point is that it is being abused. Some argue it has no place at all; I'm not willing to go there (yet). I think if it's done in the spirit of an aggressive challenge to a claim about how the world works it can add some value. But I've come to the view that robustness and replication are more powerful criteria.Peter Dormanhttps://www.blogger.com/profile/00093399591393648071noreply@blogger.comtag:blogger.com,1999:blog-4900303239154048192.post-15561214040261347442016-05-29T12:34:13.056-04:002016-05-29T12:34:13.056-04:00This is informative. My 'metric prof warned us...This is informative. My 'metric prof warned us about this sort of thing way back when. It's why I've always been more interested in descriptive stats than dubious hypothesis testing. Is that wrong?MaxSpeakhttps://www.blogger.com/profile/08594964334301228571noreply@blogger.com