Lying with Statistics: the Sharpshooter & Post-hoc Fallacies

targetI’m not an expert at archery, but there’s a way I can guarantee I get a bull’s eye every time. It’s simple. I shoot the arrow at a large target, one I can’t possibly miss. Then, I walk up and draw a bull’s eye around the arrow.

We laugh at how ridiculous this is, but this sort of error occurs every day in peer reviewed studies published in prestigious journals. Called the Sharpshooter Fallacy, it’s all too common, so it’s important that we be able to recognize it.

I first realized the problem when I read that researchers were investigating the link between processed meats and cancer. They looked at all the possible combinations of meats and cancers, and found that the correlation between hot dogs and cancer was statistically significant. Therefore, they concluded that eating hot dogs causes cancer. The problem is that with so many variables, at least one correlation is bound to crop up. They shot at the wall of meats, looked for a cluster of hits, and then circled hot dogs.

This isn’t to say that hot dogs do not cause cancer. It’s just that this type of study, with so many variables, doesn’t prove it. At best, it points in a direction where more research is needed.

A similar conclusion about chicken hit the headlines a few weeks ago. Here are some articles about a recent British study connecting chicken to certain cancers:

Conducted by researchers at Oxford University, and published in the Journal of Epidemiology and Community Health, the study concluded: “Poultry intake was positively associated with risk for malignant melanoma, prostate cancer and non-Hodgkin’s lymphoma.”

Unfortunately, none of these articles provided the statistical analysis of the study, so I can’t say for certain, but it looks very similar to the hot dog study—the researchers made a long lists of cancer types, then looked for a statistically significant correlation with chicken eaters. Of course they eventually found something, which they then circled… and the media jumped on it.

The problem here is that to be accurate, you have to take into account the number of variables. You can’t simply apply the same statistical test over and over until something crops up. So perhaps chicken is linked to cancers—or perhaps the statistical analysis was done incorrectly. It happens more often than you would think.

The Sharpshooter Fallacy is often combined with another fallacy, the Post-hoc Fallacy. Stating that correlation doesn’t imply causation, this “is incorrectly asserting that there is a direct correlation between two findings.” There isn’t. Just because two data sets appear linked doesn’t mean they are. Look at the obviously silly examples at Spurious Correlations and see for yourself (while you get some laughs).

Perhaps because it’s less obvious than some other statistical errors, this fallacy appears frequently in the media.

Let’s look again at the chicken studies. Because there is an association between eating chicken and certain cancers, the researchers concluded that chicken causes cancer, and therefore that avoiding chicken will help us stay cancer-free. But there could be another explanation, one they didn’t consider. For example, perhaps people who eat chicken are in a lower income bracket. After all, chicken is typically cheaper than steak, or even many fruits and vegetables, especially if they’re organic. Because they have less money, they therefore live in a less desirable area, one that is more polluted—and it’s the pollution that causes those cancers. Researchers try to control for these other variables, but it would be impossible to catch all of them—the list is endless.

How can you recognize a Post-hoc Fallacy? Look for phrases such as “correlates,” “associated with,” and “linked to.” Here is an example (emphasis added):

Other ultra-processed foods, such as packaged baked goods and snacks, fizzy drinks, sugary cereals, ready meals and reconstituted meat products, have also been linked with an increase risk of certain types of cancer, according to a 2018 study in the British Medical Journal. The study … found that a 10% increase in the proportion of ultra-processed foods in the diet was associated with increases of 12% in the risk of overall cancer and 11% in the risk of breast cancer.

While no one believes that the majority of processed snack foods are all that healthy, there is no distinction made between the types of processed foods. Are “reconstituted meat products” and “sugary cereals” equally problematic? Maybe it’s a lack of fresh vegetables that cause the increase in cancer risk, and not junk food. Who knows? According to the information provided in this article, not these researchers.

It also helps to read the entire article, and not just the headlines. Then, if you can access it, look at the original study. All too often, news articles proclaim a shocking conclusion that isn’t really supported by the research, but is rather aimed at scaring you into subscribing. Gee, what a surprise.

Whatcha think about this?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s