How do you determine the quality of the evidence?

Author: SCIRE Community Team | Reviewer: Janice Eng | Published: 24 April 2017 | Updated: 23 February 2024

Evidence quality can help us determine the value of research evidence in our treatment decisions. Higher quality evidence is usually weighed more heavily. However, lower quality evidence is still valuable when conclusions are made about a treatment, especially if there is no other research to help us understand it.

The quality of an experimental study is determined by how effectively the researchers reduce biases and errors in the study. Some of the features of high-quality experimental studies include:

Randomization, control groups, blinding, and large numbers of participants help reduce biases and errors during data collection and analysis.⁷

Randomization is when study participants are randomly placed into the experimental group or the control group of a study. This is done to reduce biases in how participants are assigned to the groups within the study. Randomization means that all groups start off the same so they can be compared fairly at the end of the study.

A control group is a group of participants in a study that receives an alternative treatment instead of the treatment being tested. This may be a placebo, a comparison treatment, or simply usual care (the care you would have if you were not in the study). At the end of the study, the control group is compared to the experimental group to see if they are different. Because the two groups only differ by which treatment they received, differences are thought to show the effects of the treatment.

Placebos, sometimes called sham treatments, are treatments that have no actual effects, but the person receiving them does not know whether they work or not. Placebos help to estimate the effects that other factors (besides the treatment being tested) have on the results. If someone is given a placebo (such as a pill that does not have any drug in it), but still gets better for some other reason, this is called the placebo effect.

Blinding is when the type of treatment (experimental or control) that a participant receives is intentionally withheld from that person. The type of treatment may also be withheld from the researchers who are collecting information. This is called a double-blind experiment. Blinding is done to reduce the impact that people’s biases can have on how they report on something.

When a study looks at a large group of participants, the people being tested are more likely to represent the general population and statistical analyses are more likely to be accurate. This allows the results of the study to be applied more accurately to real world situations.

When considering all the studies on a topic, the trends and comparisons between different studies can impact the relevance of the overall evidence.  Some factors to look at in a body of evidence include:

The number of research studies published is important because each new study can validate, verify, or contradict the results of previous studies. If there are many studies on a specific topic with consistent results, the evidence is more likely to be reliable and be applicable to a more general population.

Consistency is whether all the studies on a topic have similar results. When different studies produce opposing results and there is no explanation for the inconsistency, one should be cautious about making decisions using the evidence.