Monday, July 16, 2012

Baloney Detection: Part 1

"Many patterns are real, and it's good to know what those patterns are, it's called learning…"
~Michael Shermer
Sometimes I think one of the worst things a person can say to SkepticRD when I am doing nutrition education is "I've been reading this book…." Now, why would I, as a good skeptic, groan inwardly whenever I hear that statement?  Shouldn't I be encouraging free inquiry and people learning?  Of course I should, but I also know that there are a lot of people out there who haven't had a good grounding in learning and learning science, and there is so much information out there about what constitutes good nutrition that even those of us who feel we are relatively intelligent people can feel a little hopeless.
There are some basic questions you can ask yourself when you do read something or hear something to help you try to sort out what is worthwhile information or not.  These questions are taken from Michael Shermer's Video: The Baloney Detection Kit, the full video can be viewed here.  Michael Shermer's questions are also based on the Baloney Detection Kit offered up by Carl Sagan in his book  "The Demon Haunted World."  I plan to cover the first of the five questions today and the remaining five questions tomorrow.
1)  How reliable is the source of the claim?
Whenever someone is making a claim about a particular food or nutrient, there are always errors that are going to "creep in."  But if everything appears to be slanted in one direction, you should start to be suspicious.  The way I relate this to nutritional claims is that if you see a claim about a diet, or nutrient, or a particular food that seems to "cure" or "improve" a lot of different ailments (or actually cause a lot of different ailments!) I would be very suspicious.  One example of this is the use of a gluten free diet being used to "treat" autism.  There are plenty of studies that show for people who have celiac disease or related GI illness's that indicate that a gluten free diet will improve (and in celiac disease, this is the only treatment), but very little actual evidence that removal of gluten itself actually improved the symptoms of children with autism.  (Do some children with autism have terrible diets and see improvement because they are eating more nutritious food?  Quite possibly.  Does the removal of one particular item do so?  Show me the evidence.)
2)  Does the source make similar claims?
One of the banes of SkepticRDs existence is that whenever I get pre-printed "education materials" on how vital a certain nutrient is, I find out that said materials come from a particular organization lobbying for an industry that markets/sells a food product that contains that particular vitamin, mineral, food substitute, etc.  If I go to a conference, most likely I will get education materials sponsored by the Beef Council, the Dairy Council, Novo-Nordisk, etc.  At these same conferences I will see booths sponsored by products like Sweet and Low, Splenda, Egg-lands best, etc.  Now, knowing how much money these conferences cost, I am not surprised that money changed hands between these organizations and whatever seminar I was at.  It also costs a lot of money to do research, so I am not surprised to find research articles in my packet or e-mailed to me that say "sponsored by the National Dairy council." But what's really disappointing is that I don't see more people like me asking pointed questions about these materials at conferences.  So whenever you read about a certain food helping you lose weight, etc, do some digging to find out who sponsored the study.  They might be on to something, but be skeptical.
3) Have the claims been verified by someone else?
If I see information in one news article or on one particular website, I search around to see if I can find any other non-related sources to verify it.  For example, if I see that a study has been done that states chocolate consumption is good for you, and I see that the Hershey corp has sponsored it, I'm going to start digging around for any other universities, etc not sponsored by Hershey that may have done other studies to verify this.
You also want to make sure the different sources are using the same terminology.  For example, there was recently a study published in Europe that stated that an "Atkins type diet" was bad for the heart.  An Atkin's type diet, however, only contains about 10-30% of the calories from carbohydrates, and the diet used in the study contained 50% of the calories from carbohydrates.  You can't verify a claim if you are not using a similar study plan.
On a related note, I would also ask "Am I getting all the information here?"  It is so easy to read the headlines and not read the fine print.  How many of us rejoiced when we heard "chocolate is good for you?"  But when you dig deeper, you find out that it was the so call "dark chocolate" (70% or higher) that was actually used in the study, not the Snicker's bar you were eating.  And then you find out that the amount recommend was 1 oz of chocolate—a quarter of a bar or less.  Yeah, you still haven't found your justification for your thrice daily Snickers.
4) Does this really fit with the way the world works?
You are going to find a lot of nutrition related studies that that use animal models, sometimes rats, sometimes primates.  If a certain substance puts lab rats at risk, there's definitely something there for future study but human physiology may not react the same.  Also, think about the amount of a particular substance that is often given in these particular studies.    Would humans realistically consume the equivalent of hundreds of cans of diet soda every day for the rest of their lives as in the controversial research done on saccharine in the 1970's? (That wasn't a challenge!)  In other words, we may not be able to consume enough of a substance for it to do us any harm or good, at least at the present time.
I think here is where I would also state that one should be very, very careful of research that is done where people get to self report what they were eating over a period of time.  There are a lot of us that simply report what we think the researchers want to hear, or you're filling out your food log at the end of the week and your memory becomes faulty and selective, or you don't really know what "half a cup" of something looks like, so you under-report because you just "guessed."  You can't base dietary recommendations on someone's faulty memory.
5)  Has anyone tried to dispute the claims?
One of the things scientists love to do is disprove claims.  A good scientist will enjoy looking at his/her own claims just to see if they can be disproven and they will name any faults or limitations to their own studies.  They welcome a challenge.  So, if you see advertisements for nutritional supplements or even articles that talk about how "they" don't want you know about this special secret nutritional ingredient, somebody is probably afraid of being challenged and that should raise suspicions.
Make sure that you are looking for refutation that has a scientific backing to it.  It can be easy for even the most rational among us to say "Well, my grandfather ate [insert food here] every day of his life and he lived to be 100!" We don't know if your grandfather was eating the same quality of food (he probably wasn't), we don't know what kind other factors came into play (air quality, stress level, sleep, family support), and he may have had some interesting genetics that may or may not have been passed on to you.
Stay tuned, part 2 should be coming.

No comments:

Post a Comment