The amount of research conducted throughout the world on diet and health produces new findings daily, many of which make their way into print and online media. However, not all research results are equal, and the strength of claims made about different studies is based on many factors. Study design, when the research was done, and how the findings fit into the overall research picture on a particular topic, all affect the strength of the evidence.
Red Flag: Definitive statements that are based on anything other than the results of large clinical studies often suggest bias on the part of the study authors. This is especially true when definitive claims are made on the basis of animal research.
Certain types of studies carry more weight than others. Human studies provide more useful findings than studies in animals or in vitro research, the type that uses cell cultures. And among human studies, clinical—also called intervention—studies carry more weight than epidemiological studies. Epidemiological studies (population studies) show associations but can’t show cause and effect.
Epidemiologic studies are still important because they can examine the impact of diet on disease endpoints like heart attacks or fractures, which can’t typically be measured in clinical studies. They are useful for generating hypotheses that can ultimately be tested by clinical studies.
All other factors being equal, the larger the study the more credible it is. This is because there is always the possibility that the outcome of any study occurred by chance. Larger studies are less likely to report “chance” findings than smaller ones.
Red flag: Claims made on the basis of single studies
Even with well-designed clinical studies, it’s generally not possible to draw definitive conclusions from single studies. Rather, the findings need to be repeated before they can be used as a basis for recommendations and conclusions.
Red flag: Claims based on older studies only
Although older studies can provide valuable information, recent studies are more generally more credible. Older studies often suffer from design weaknesses because study design and quality improve as research techniques and understanding evolve. Researchers also learn from the mistakes and limitations of earlier research designs.
In addition, important findings from older studies usually lead to follow up research. If this research exists and it is not cited, this is evidence of bias. If on the other hand, no follow up research has been conducted, it suggests findings from the older research have proven to be invalid.
Red flag: Exaggerated claims, or claims that sound too good or too bad to be true.
Dramatic claims without qualification about the benefits or harm of a single food or diet are almost always evidence of bias. Objective scientists talk in terms of risk and probability because they recognize the limitations of research and the difficulty of obtaining definitive data. Furthermore, because the results from different studies often demonstrate some inconsistency, statements about effects need to be tempered.
Red flag: Claims based on only the selection of studies that support a particular viewpoint without identifying the strengths and weakness of those studies
Scientists reach conclusions based on an examination of the strengths and weakness of all of the relevant data on a particular topic. This process requires considerable time and effort and is only possible when an expert has the knowledge and training to evaluate research. The most commonly used research database, which is called PubMed, contains more than 19 million studies. Since findings are not always 100 percent consistent, it is possible to find studies to support nearly any claim regarding diet and health. That’s why it’s important to look at all of the studies, not just those that support a particular perspective.
Red Flag: Findings not published in a peer-reviewed journal
Research articles published in scientific journals are evaluated first by experts in the field who look for flaws in the study design and assess whether the findings are accurately reported and reflect the results of the study. Some journals reject as many as 95% of the research articles they receive. Research findings that are not published in well-respected, peer-reviewed journals are generally not considered relevant.
With the vast amount of information on the internet, on television and in magazines, it can be difficult to decipher valid findings from those that are exaggerated or otherwise without factual basis. In evaluating whether claims are true, it’s helpful to keep all of the above points in mind. Look for claims that are based on recent clinical studies and that reflect the findings of a large body of research. The findings should always have been published in a reputable peer-reviewed journal.