How to Spot Good Research: Reading Papers with a Critical Eye (Part 2)
In our previous post, we discussed the initial steps of dissecting a scientific paper. Now, let’s dive deeper into some red flags and critical considerations to help you discern robust science from potentially misleading claims. 1. The Selective Spotlight: What’s Missing from the Narrative? Scientific papers highlight significant findings. But a critical reader asks: What predefined results were not mentioned? Many studies, especially clinical trials, register their protocols outlining specific endpoints before data collection. If these pre-specified outcomes are absent from the results, it raises a critical question: Why? Were they not “significant,” or perhaps contradictory? Example: A drug trial protocol registers several primary and secondary outcomes, including patient-reported pain and a specific quality-of-life measure. The published paper then focuses solely on a statistically significant reduction in pain, making no mention of the quality-of-life data. If predefined outcomes aren’t reported, investigate why. You can often find registered protocols (e.g., on ClinicalTrials.gov) using a trial registration number in the paper’s methods section. 2. The P-Value Predicament: When “Significance” Means Little A p-value less than 0.05 is often deemed “statistically significant.” However, an over-reliance on p-values without context can mislead. Example: A new blood pressure medication is tested, and the study finds a statistically significant reduction in systolic blood pressure with a p-value of 0.03. However, upon closer inspection, the absolute reduction in blood pressure is only 1.5 mmHg. While statistically significant, a 1.5 mmHg reduction might be clinically negligible for most patients and may not justify potential side effects or costs of the new medication. Critical Takeaway: Don’t let a low p-value blind you to the actual magnitude and clinical relevance of the effect. Always consider the effect size and confidence intervals alongside the p-value. 3. Correlation vs. Causation: Beware the Causal Leap Just because two things are linked doesn’t mean one causes the other. Authors sometimes imply causation when their study design only shows correlation. Example: A study finds a strong correlation between shoe size and salary. It would be incorrect to conclude that having larger feet leads to a higher salary. A more plausible explanation is that, historically, certain higher-paying professions have been predominantly held by men, who, on average, have larger shoe sizes than women. The underlying factor isn’t foot size, but gender and historical workplace demographics. Another Example: A study finds a strong correlation between ice cream sales and drowning incidents. This doesn’t mean eating ice cream causes drowning. The confounding variable here is likely the time of year: both ice cream sales and swimming (and thus drowning incidents) increase during hotter summer months. Critical Takeaway: Always scrutinize the study design. Randomized controlled trials are the gold standard for demonstrating causation, while observational studies (cohort, case-control, cross-sectional) can only suggest associations or correlations. 4. Graph Grievances: Don’t Let Visuals Deceive You Graphs can powerfully summarize data, but they can also be manipulated to create a misleading impression of results. Example: A bar graph shows a dramatic difference between a control group and a treatment group. However, if you look closely at the y-axis, you might find that it doesn’t start at zero, or that the scale is truncated, exaggerating a small difference. A change from 10 to 12 (a 20% increase) might look enormous if the y-axis only ranges from 9 to 12. Advice: 5. Definitional Discrepancies: What Exactly Are They Studying? Scientific terms can be defined differently across studies. This creates confusion and makes comparing findings difficult. Example: The term “high-risk patient” in a cardiology study could be defined differently by various authors. One study might define it based on the presence of three or more cardiovascular risk factors, while another might use a specific score from a risk assessment tool. If the authors don’t clearly define their terms, or if they deviate from standardized and referenced definitions, it becomes challenging to interpret their findings in context. Additional Red Flags: By developing a critical eye and looking for these red flags, you can truly engage with scientific literature and distinguish robust research from less reliable claims. Happy reading!