To print: Select File and then Print from your browser's menu
Return to blog entry

Adjust Font Size :
 

How Studies Can Mislead

David Spero

August 18, 2010

Don’t you hate it when a new food or drug study comes out, telling you that all previous studies were wrong? How are you supposed to trust studies, or even make sense of them, when they disagree with each other?

In May, I reported on some food studies that contradicted earlier studies of the same things. In one, a Boston-based research team found that egg consumption did not appear to have an association with the development of Type 2 diabetes. An earlier study by the same lead researcher had claimed that eggs were a sure path to diabetes.

Studies of herbs and drugs can also contradict each other. Studies can draw huge amounts of media attention and can shape diabetes treatment and self-management, without necessarily proving anything.

Why are studies so confusing? Isn’t science the most reliable path to the truth? Not always. A number of things can skew study results one way or another. For one thing, who was being studied? Do those subjects’ results reflect what will happen to you if you do the same thing?

Trials of cinnamon for glucose control differ based on the population studied. In Pakistan, researchers found a big benefit, but in Oklahoma there was no significant effect. Is this a surprise? The people were different; their diets were different; their medicines were probably different, so why should the results be the same?

Drugs work better for some people than others, too. The journal BMJ confirms that African-Americans with high blood pressure usually do better with diuretics (water pills). White Americans tend to do better with beta blockers. It’s not true for everyone; there are larger differences within ethnic groups than between them, but you want to know if the people being studied were people like you. Were people of your age, gender, ethnic background, and medical history included in the study group?

Can Studies Show Causes?
In media stories, studies often show cause. A Harvard group reported that eating white rice contributed to Type 2 diabetes. People who ate brown rice were less likely to develop Type 2. The story said that replacing white rice with brown rice could drastically reduce the risk of diabetes.

But how do they know what caused the difference? Most likely, the brown rice eaters had higher economic and educational levels, and were perhaps more health conscious, because that’s who eats brown rice. The researchers didn’t look at subjects’ socioeconomic status (SES), and SES has a strong association with diabetes.

This happens all the time. We’ll read that “being overweight causes diabetes,” for example. But all we know is that overweight people are more likely to develop diabetes. We don’t know why. There is a correlation between weight and T2D, but we don’t know what causes what.

A study that looks at a large group of people and finds that, for example, those who drink red wine have less heart disease is called a population study. Such a study can show correlation but cannot show causation. An article describing some different kinds of studies is available here.

Are Studies Honest?
Many studies are biased. A study might find that people who stayed on a drug had better results than people who quit. But most likely, the people who quit were the ones who weren’t getting much benefit anyway. This is called “attrition bias.” There may be “selection bias” when people who volunteer or are chosen for a study are different in some way than the general population. There may also be “observation bias” when observers see what they thought they would see or what they wanted to see, instead of what is actually happening.

Some studies are biased because of who is paying for them. According to this Veterans Administration paper, you want to know if the researchers have a reason for bias. Were they pushing some kind of an agenda, like a milk researcher who consults for the Dairy Council? Are the drug researchers paid by the drug company?

Sometimes study abstracts disclose information about such conflicts of interest. You can find abstracts at the government research site, PubMed. Just type in a study name, author, or a keyword and you’ll be able to find them.

The bottom line is that we should be skeptical towards studies, and even more so toward media reports. A drug trial might show that people who took Drug X had reduced A1C levels. The media will say “new breakthrough for people with diabetes!” But the actual study might say that the subjects were Ugandan tribesmen and the A1C reduction was .1%, so you might decide not to ask your doctor about this new drug yet.

Or you might see an article that says, “never eat another egg.” Two years later, the same team might come out and say the reverse. I think it’s best to wait for observations to be repeated by different researchers in different places with different populations, before acting on them. This is especially true with drugs, where dangerous side effects might not become known until the drug has been in use for a few years.

A site that has good analysis of diabetes-related studies and often debunks them is Diabetes Update. Are there other sites or resources that you have found helpful?

Disclaimer of Medical Advice:You understand that the blogs posts and comments to such blog posts (whether posted by us, our agents, bloggers, or by users) do not constitute medical advice or recommendation of any kind and you should not rely on any information contained on such posts or comments to replace consultations with your qualified health care professionals to meet your individual needs. The opinions and other information contained in the blog posts and comments do not reflect the opinions or positions of the Site Proprietor.