Abstract
Purpose :
Systematic reviews (SRs) are often considered the highest level of evidence, yet the certainty of the evidence which they present can be low and unusable in practice because the estimates are not precise, the primary study populations are not directly relevant to the target population, the risk of bias of primary studies is high, there may be too much heterogeneity in the estimates, or there may be a potential for publication bias. Grading of Recommendations, Assessment, Development and Evaluation (GRADE), is one approach to assessing the certainty of evidence collected in systematic reviews. Our objective is to assess the distribution of certainty of evidence from SRs published by Cochrane Eyes and Vision (CEV) and to identify reasons for which evidence is commonly downgraded or upgraded.
Methods :
So far, we have assessed all CEV systematic reviews that included at least one GRADE Summary of Findings (SoF) table published between January 1, 2008 and February 25, 2020 (N = 275). GRADE was developed, published, and adopted by Cochrane in 2008. We abstracted data from all SRs into a Qualtrics form (single abstraction) and conducted all analyses using Stata 15.
Results :
Of the 275 reviews assessed thus far, 149 (54%) presented GRADE SoF tables, which together included 1854 outcomes. Most outcomes were assessed with a median [IQR] of 190 [66 – 551] participants and 2 [1 – 4] studies (Table). Randomized controlled trials were the most common study type used as evidence for outcomes, but 27% (507) of outcomes did not specify the study type (Table). Only 6% (116) of outcomes were judged at a “high” certainty of evidence whereas 47% (869) were either “low” or “very low” certainty, with “risk of bias” and “imprecision” being the most common reasons for downgrading the evidence (Table). Five hundred and fifty-nine (30%) of the outcomes in SoF tables did not have a GRADE rating due to a lack of studies with evidence.
Conclusions :
Overall, the certainty of evidence within CEV reviews, according to GRADE, is generally low and SoF tables often failed to report the specific study types contributing to the outcome. Our findings highlight the necessity to improve the quality of primary research: specifically, studies should be properly powered and focus on minimizing their risk of bias. Additionally, future reviewers should ensure their SoF tables are informative and consistent in all aspects to support transparent decisions.
This abstract was presented at the 2022 ARVO Annual Meeting, held in Denver, CO, May 1-4, 2022, and virtually.