In our previous post, we discussed the initial steps of dissecting a scientific paper. Now, let’s dive deeper into some red flags and critical considerations to help you discern robust science from potentially misleading claims.
1. The Selective Spotlight: What’s Missing from the Narrative?
Scientific papers highlight significant findings. But a critical reader asks: What predefined results were not mentioned? Many studies, especially clinical trials, register their protocols outlining specific endpoints before data collection. If these pre-specified outcomes are absent from the results, it raises a critical question: Why? Were they not “significant,” or perhaps contradictory?
Example: A drug trial protocol registers several primary and secondary outcomes, including patient-reported pain and a specific quality-of-life measure. The published paper then focuses solely on a statistically significant reduction in pain, making no mention of the quality-of-life data. If predefined outcomes aren’t reported, investigate why. You can often find registered protocols (e.g., on ClinicalTrials.gov) using a trial registration number in the paper’s methods section.
2. The P-Value Predicament: When “Significance” Means Little
A p-value less than 0.05 is often deemed “statistically significant.” However, an over-reliance on p-values without context can mislead.
Example: A new blood pressure medication is tested, and the study finds a statistically significant reduction in systolic blood pressure with a p-value of 0.03. However, upon closer inspection, the absolute reduction in blood pressure is only 1.5 mmHg. While statistically significant, a 1.5 mmHg reduction might be clinically negligible for most patients and may not justify potential side effects or costs of the new medication.
Critical Takeaway: Don’t let a low p-value blind you to the actual magnitude and clinical relevance of the effect. Always consider the effect size and confidence intervals alongside the p-value.
3. Correlation vs. Causation: Beware the Causal Leap
Just because two things are linked doesn’t mean one causes the other. Authors sometimes imply causation when their study design only shows correlation.
Example: A study finds a strong correlation between shoe size and salary. It would be incorrect to conclude that having larger feet leads to a higher salary. A more plausible explanation is that, historically, certain higher-paying professions have been predominantly held by men, who, on average, have larger shoe sizes than women. The underlying factor isn’t foot size, but gender and historical workplace demographics.
Another Example: A study finds a strong correlation between ice cream sales and drowning incidents. This doesn’t mean eating ice cream causes drowning. The confounding variable here is likely the time of year: both ice cream sales and swimming (and thus drowning incidents) increase during hotter summer months.
Critical Takeaway: Always scrutinize the study design. Randomized controlled trials are the gold standard for demonstrating causation, while observational studies (cohort, case-control, cross-sectional) can only suggest associations or correlations.
4. Graph Grievances: Don’t Let Visuals Deceive You
Graphs can powerfully summarize data, but they can also be manipulated to create a misleading impression of results.
Example: A bar graph shows a dramatic difference between a control group and a treatment group. However, if you look closely at the y-axis, you might find that it doesn’t start at zero, or that the scale is truncated, exaggerating a small difference. A change from 10 to 12 (a 20% increase) might look enormous if the y-axis only ranges from 9 to 12.
Advice:
- Check the axes: Always look at the range and starting point of the axes.
- Examine error bars and confidence intervals: These show variability and precision. Overlapping error bars often suggest non-significant differences.
- Compare visuals with reported statistics: Ensure the graph accurately reflects the numerical data.
5. Definitional Discrepancies: What Exactly Are They Studying?
Scientific terms can be defined differently across studies. This creates confusion and makes comparing findings difficult.
Example: The term “high-risk patient” in a cardiology study could be defined differently by various authors. One study might define it based on the presence of three or more cardiovascular risk factors, while another might use a specific score from a risk assessment tool. If the authors don’t clearly define their terms, or if they deviate from standardized and referenced definitions, it becomes challenging to interpret their findings in context.
Additional Red Flags:
- Small Sample Sizes: While sometimes unavoidable, very small sample sizes can lead to underpowered studies, meaning they might miss true effects or, conversely, report spurious findings that are simply due to chance. Look for a power analysis in the methods section, which indicates if the study had sufficient participants to detect a meaningful effect.
- Unaccounted Confounding Variables: Even in well-designed studies, there can be unmeasured or uncontrolled variables that influence the results. Critical readers should consider potential confounders that the authors may not have addressed and how these might impact the conclusions. For instance, in an observational study about diet and disease, did they account for lifestyle factors like exercise or smoking?
By developing a critical eye and looking for these red flags, you can truly engage with scientific literature and distinguish robust research from less reliable claims. Happy reading!
