How Can We Tell Whether a Study Is Reliable?

Seven questions anyone can ask when evaluating scientific research

Every day, we encounter news stories that begin with the phrase, “According to a new study.” One study suggests that coffee is beneficial, while another warns of its potential harms. So, which research should we trust?

Scientific studies do not offer absolute and immutable truths. They provide evidence generated through particular methods and under specific conditions, and their conclusions may change as new evidence emerges. Instead of asking, “Is this study correct?”, it is therefore more useful to ask: “How strong is the evidence it provides?”

1. Who conducted and funded the research?

The researchers’ areas of expertise, institutional affiliations, and sources of funding should be examined.

If a study investigating the effects of a drug is funded by the company that manufactures it, this does not automatically mean that its findings are false. However, the relationship must be disclosed clearly. Studies that transparently report their funding sources, potential conflicts of interest, and methods are generally more trustworthy.

2. Where was the study published?

Publication in a peer-reviewed journal means that the study has been evaluated by specialists in the relevant field. However, peer review does not guarantee that a study is free from error.

The findings of manuscripts that have not yet undergone peer review (preprints) should be interpreted with greater caution. Preprints enable the rapid dissemination of scientific information, but their findings should not be presented as settled conclusions.

The important question, therefore, is not simply “Was it published?” but “Where was it published, and what kind of evaluation did it undergo?”

3. Is the sample sufficiently large and representative?

A study’s conclusions depend on who was examined. Drawing conclusions about an entire population from an experiment involving 20 university students may be problematic.

Small samples may fail to detect genuine effects and can exaggerate the magnitude of the effects they do identify. Button et al. (2013) showed that low statistical power can weaken the reproducibility of research findings.

However, the number of participants is not the only consideration. A biased sample of thousands may be more misleading than a smaller but appropriately selected sample.

The following questions should be considered:

• How many people participated in the study?
• How were the participants selected?
• Does the sample represent the population to which the conclusions are being applied?
• Might certain groups have been excluded from the research?

4. Does the study show an association or a causal relationship?

The fact that two variables change together does not prove that one causes the other.

For example, a study may identify an association between social media use and symptoms of depression. This finding alone does not demonstrate that social media causes depression. People experiencing depression may use social media more frequently, or a third factor, such as loneliness, may influence both variables.

Correlation does not imply causation.

Hill (1965) argued that several considerations should be examined when assessing whether an observed association might be causal. These include the temporal sequence of events, the strength of the association, consistency across studies, limitations, and plausible mechanisms.

5. Is statistical significance enough

Statistical significance does not necessarily mean that a finding is large or meaningful in everyday life.

The American Statistical Association has emphasized that scientific conclusions should not be based solely on whether a p-value falls below a particular threshold (Wasserstein & Lazar, 2016).

A p-value should therefore be considered alongside:

• Effect size
• Confidence intervals
• Sample size
• Practical significance

For example, in a study involving thousands of participants, a two-minute difference in sleep duration might be statistically significant. Yet such a difference may have little practical importance in daily life.

6. Have the findings been replicated?

Scientific knowledge is not established through a single study. Surprising claims, in particular, should be tested again by independent research teams.

The Open Science Collaboration (2015) attempted to replicate 100 studies previously published in psychology. Although 97% of the original studies reported statistically significant findings, only 36% of the replication studies produced statistically significant results. The effects observed in the replications were also generally smaller than those reported in the original studies.

This does not mean that every study that failed to replicate was necessarily wrong. It does, however, demonstrate why a single statistically significant finding should not be treated as conclusive evidence. Independent replications, systematic reviews, and meta-analyses should also be considered when evaluating a research claim.

7. Do the researchers acknowledge the study’s limitations?

Reliable research does not conceal its limitations.

Researchers should clearly explain issues such as a narrow sample, weaknesses in the measurement tools, and the populations to which the results may not apply. Acknowledging limitations does not make a study worthless; it demonstrates scientific transparency.

Overly definitive claims should be treated with caution. If the data reveal only an association but the researchers make a strong causal claim, the findings may have been interpreted beyond what the evidence supports.

Ioannidis (2005) argued that small samples, flexible analytical methods, multiple statistical tests, and conflicts of interest can reduce the reliability of published findings. Practices such as preregistration, data sharing, and transparent analytical procedures may help address some of these problems (Munafò et al., 2017).

A Quick Checklist

When you encounter a research study, ask:

1. Who conducted and funded the study?
2. Was it published in a peer-reviewed journal?
3. Is the sample sufficiently large and representative?
4. Do the findings show an association or a causal relationship?
5. Is statistical significance sufficient to support the claim?
6. Have the findings been replicated?
7. Are the study’s limitations clearly acknowledged?

Conclusion

Reliable research is not flawless research. It is research that explains its methods clearly, acknowledges uncertainty, and avoids making claims that go beyond what the evidence supports.

Scientific thinking does not mean rejecting every study. It means developing a level of confidence that is proportionate to the strength of the evidence.

The next time you encounter the phrase “According to a new study,” do not ask only what the researchers concluded. Ask how they reached that conclusion.

References

Button, K. S., et al. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.

Hill, A. B. (1965). The environment and disease: Association or causation? Proceedings of the Royal Society of Medicine, 58(5), 295–300.

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLOS Medicine, 2(8), e124.

Munafò, M. R., et al. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133.