Question: I like to do my research on different products or treatments for my arthritis but I don’t have a science background- how can I tell if what I’m reading is from a trustworthy source or if the study I’m reading is of good quality?
Unless you’ve studied it, most of us are never taught how to evaluate science (e.g., quality of scientific studies or scientific claims) or how to determine the good from the bad. Yet it is something that dictates every area of our lives- we are exposed to claims, “facts”, and statistics every single day online, on TV and on social media. Being able to evaluate the quality of evidence behind a scientific claim is important. Being able to recognise bad science reporting and interpretation, or faults in scientific studies is equally important.
This article cannot possibly cover all aspects of spotting bad science- there is just too much to cover! However, the lessons and hot tips below will certainly cover enough to improve your ability to spot bad science.
Lesson number 1 – Bias and preconceived beliefs
When comes to researching information about a certain product, medication, therapy or treatment, be aware of both your personal and the author/s biases. The two biggest ones are:
- Confirmation bias and belief perseverance.
- Confirmation bias is a bias in which people seek out and recall information that supports their preconceived beliefs
- Belief perseverance is the rejection of information that could disprove beliefs
Preconceived beliefs are often implicitly and unconsciously developed, and may be informed by culture/religion, schooling/education, generational, family, friends, and society. Regardless of where these beliefs came from, it is important to be open to alternative scientific outcomes or options, and allow your beliefs to be challenged and even changed or “upgraded” if they are incorrect or “out of date”.
Furthermore, when reading scientific studies and particularly blogs, articles, and information on other social media and websites, it is important to consider if the author/s are practising confirmation bias or belief perseverance- see hot tip below.
Hot Tip: is this author or study paid or endorsed? Do they have any sort of relationship/conflict of interest with a product/treatment/therapy where they could benefit financially or politically? Do they have polarized or ridged views? Does the paper talk objectively about the results or outcomes, mention both significant and non-significant findings, and/or the study limitations? If not, the paper and/or authors may not be telling the whole truth and are shaping the results to fit in with their preconceived beliefs.
Lesson number 2 – ways to spot bad science
The “A Rough Guide to Spotting Bad Science” infographic outlines and explains 12 ways to spot bad or fake science. You can apply this to scientific articles claims and ads you see on the internet and TV. However, please note that this is not a comprehensive overview, nor is it implied that the presence of one of the points noted automatically means that the research should be disregarded. This is merely intended to provide a rough guide to things to be alert to when either reading science articles or evaluating research.
Lesson number 3 – types of scientific evidence
The “A Rough Guide to Types of Scientific Evidence” infographic outlines and explains different types and strengths of scientific evidence. As you can see, the best articles to read are systematic reviews followed by randomized control trials. Animal and cell studies (in vitro studies), and anecdotal and expert opinions are the least “trustworthy” or weakest form of evidence. Having said that, animal and cell studies are not bad they can be very useful in testing the safety, toxicity and efficacy of a drug, and opinions/anecdotes aren’t necessarily bad either, however, as mentioned earlier, they can be biased or there can be a conflict of interest. Each type serves a purpose, however, it is best practice to formulate opinion on the outcomes of high-quality studies that use human participants.
Hot tip: when researching, try typing into the search engine “systematic review on….” Or “randomized control trial on …..” or some variation including those keywords, and for more recent findings, be sure to change the date range to exclude studies that are more than 10 years old. See below using Google Scholar.
Lesson number 4 – common place pitfalls in reasoning
Fallacies are common errors in reasoning that will undermine the logic of your argument, opinion or choice. Fallacies can be either illegitimate arguments or irrelevant points and are often identified because they lack evidence that supports their claim. See below for some common examples.
- Emotional reasoning fallacy
- Bandwagon fallacy
- Not me fallacy
- Bias blind spot fallacy
Hot tip: it’s important to practice a little introspection and objectivity when committing to a belief, reason for trying a product/therapy/treatment, or forming an opinion. Try not to go with the crowd or popular thinking “just because”- instead, investigate for yourself. Also, try not to let your emotions get the best of you and guide your decision making, and always be open to being wrong and/or changing your views.
References
Infographics: Compoundchem