To create the evidence map, we identified studies evaluating methods for overviews of systematic reviews of interventions. We added the additional criterion that methods studies had to have a stated aim to evaluate methods, since our focus was on evaluation and not just application of a method. For studies located from the purposive search, one author reviewed title, abstracts and full text for their potential inclusion against the eligibility criteria.
- Finding and Evaluating Evidence: Systematic Reviews and Evidence-Based Practice.
- Evidence Based Practice (EBP).
- Subject Guides!
- Buddhism and the West;
- Evidence-Based Practice for Nursing: Finding Evidence.
- Health Evidence™ on Twitter.
We extracted data from primary methods studies, or SRs of methods studies that evaluated the measurement properties of tools for assessing the risk of bias in SRs and one study that developed measures to quantify overlap of primary studies in overviews. We had originally planned to extract quantitative results from the methods evaluations relating to the primary objectives; however, on reflection, we opted not to do this since we felt this lay outside the purpose of the evidence map.
We then made an overall judgement about the risk of bias arising from these concerns low, high, or unclear. We did not assess Domain 4 of ROBIS, since this domain covers synthesis methods that are of limited applicability to the included reviews. The yield, characteristics and description of the studies evaluating methods were described and mapped to the framework of methods. Details of our search results are reported in our first companion paper [ 10 ]. Here, we note the results from the additional purposive search and changes in search results between the papers.
Our main search strategy retrieved unique records through searching databases, methods collections and other sources Fig. Of the 24 included stage II studies, 12 evaluated search filters for SRs reported in paper 1 [ 10 ] , 11 evaluated risk of bias assessment tools for SRs, and one evaluated a synthesis method. Of the 11 studies evaluating risk of bias assessment tools for SRs, four were SRs of methods studies [ 12 , 16 , 17 , 18 ] and seven were primary evaluation studies [ 15 , 17 , 19 , 20 , 21 , 22 , 23 ].
Four of the seven primary evaluations of risk of bias assessment tools [ 20 , 21 , 22 , 23 ] and one SR [ 16 ] were included in the results of the SR by Whiting [ 12 ] and so were not considered individually in this paper. Therefore, of the 24 initially eligible stage II studies, 18 met the inclusion criteria, six of which are included in this second paper Fig.
Helping public health use best evidence in practice since 2005
In each section, we orient readers to the structure of the methods framework, which includes a set of steps and sub-steps which are numbered in the text and tables. However, the terms quality assessment and critical appraisal are common, particularly when referring to the assessment of SR methods, and hence, our analysis includes all relevant literature irrespective of terminology. When determining how to assess the RoB in SRs 1. Other tools are described as being for critical appraisal or quality assessment.
Methods for summarising and presenting RoB assessments mirror those used in a SR of primary studies 1. Authors must also decide on how to assess the RoB of primary studies included within SRs 2.
Two main approaches were identified: to either report the RoB assessments from the included SRs 2. When using the first approach, overview authors may also perform quality checks to verify assessments were done without error and consistently 2. In attempting to report RoB assessments from included SRs, overview authors may encounter missing data e. In addition, discrepancies in RoB assessments may be found when two or more SRs report an assessment of the same primary study but use different RoB tools or report discordant judgements for items or domains using the same tool.
We identified multiple methods for dealing with these scenarios, most are applied at the data extraction stage covered in Paper 1 [ 10 ]. Options varied according to the specific scenario, but included the following: a extracting all assessments, recording discrepancies; b extracting from one SR based on a priori criteria; c extracting data elements from the SR that meets pre-specified decision rules and d retrieving primary studies to extract missing data or reconcile discrepancies [ 10 ]. As a note on terminology, we distinguish between discrepant data—meaning data from the same primary study that differs between what is reported in SRs due to error in data extraction, and discordant results, interpretation and conclusions of the results of SRs—meaning differences in results and conclusions of SRs based on the methodological decisions authors make, or different interpretations or judgments about the results.
An identified step of relevance to all overviews is determining the summary approach 1. This includes determining what data will be extracted and summarised from SRs and primary studies e. When the former approach is chosen 1.
A related issue is that of discordance 6. Some overviews aim to compare results, conclusions and interpretations across a set of SRs that address similar questions. These overviews typically address a focused clinical question e. Identified methods included approaches to examine and record discordance 6. In addition to determining the summary approach of SR results, consideration may also be given to undertaking a new quantitative synthesis of SR results 2.
A range of triggers that may lead to a new quantitative synthesis were identified 2. When undertaking a new meta-analysis in an overview, a decision that is unique to overviews is whether to undertake a first-order meta-analysis of effect estimates from primary studies 2.
If undertaking a second-order meta-analysis, methods may be required for dealing with primary studies contributing data to multiple meta-analyses 5. A second-order subgroup analysis was identified as a potential method for investigating whether characteristics at the level of the meta-analysis e. SR quality modify the magnitude of intervention effect 3. If new meta-analyses are undertaken, decisions regarding the model and estimation method are required 2. Investigation of reporting biases may be done through summarising the reported investigations of reporting biases in the constituent SRs 1.
Overviews also provide an opportunity to identify missing primary studies through non-statistical approaches 4. An additional consideration in overviews is investigation of missing SRs. Identified non-statistical approaches to identify missing SRs included searching SR registries and protocols 4. GRADE is the most widely used method for assessing the certainty of evidence in a systematic review of primary studies.
The methods involve assessing study limitations RoB, imprecision, inconsistency, indirectness, and publication bias to provide an overall rating of the certainty of or confidence in results for each comparison [ 30 ]. In an overview, planning how to assess certainty 1. These include deciding how to account for limitations of the included SRs e. One approach is to assess certainty of the evidence using a method designed for overviews 1. However, GRADE methods or equivalent have not yet been adapted for overviews and guidance on addressing issues is not available.
In the absence of agreed guidance for overviews, another option is to assess the certainty of the evidence using an ad hoc method 1. For example, Pollock incorporated the limitations of included SRs in their GRADE assessment by rating down the certainty of evidence for SRs that did not meet criteria deemed to indicate important sources of bias [ 31 , 32 ]. Other identified approaches use methods developed for SRs of primary studies, without adaptation for overviews. Authors may then use approaches specified in the data extraction step to deal with missing or discrepant assessments see paper 1 [ 10 ].
These approaches include simply noting missing data and discrepant assessments, or reporting assessments of certainty from an SR that meets pre-specified methodological eligibility criteria, for example, the review that addressed the overview question most directly or assessed to be at lowest risk of bias. The final option when using methods developed for SRs of primary studies involves completing the assessment of certainty from scratch 1. This option may apply in circumstances where a an assessment was not reported in included SRs, b new primary studies were retrieved that were not included in the SRs or relevant studies were not integrated into the assessment reported in the SR, c included SRs used different tools to assess certainty e.
In our examination of the literature, methods were often proposed in the context of overcoming common methodological scenarios. Only those methods that provide direct solutions are listed, not those that need to be implemented as a consequence of the chosen solution.
SOWK 4201 Evidence for Practice: Evidence Based Practice Resources
Taking an example, a commonly cited approach for dealing with reviews with overlapping primary studies is to specify eligibility criteria or decision rules to select one SR see Paper 1 [ 10 ]. However, multiple methods exist for addressing overlap at later steps of the overview. During synthesis, for example, authors can i use decision rules to select one or a subset of meta-analyses with overlapping studies 5. Alternatively, overlap may be addressed when assessing certainty of the evidence.
Any of these approaches can be combined with methods to quantify and visually present overlap 5. Five studies, published between and , evaluated tools to assess risk of bias in SRs. Two were SRs [ 12 , 17 ] and three were primary studies not included in either of the SRs [ 15 , 19 , 34 ]. We found one study that evaluated methods for synthesis. Pieper b developed and validated two measures to quantify the degree of overlap in primary studies across multiple SRs [ 35 ]. Two SRs reviewed published tools to assess the risk of bias in SRs [ 12 , 17 ].
The review includes a summary of tool content items and domains measured , tool structure e. Studies included in Whiting [ 12 ] reported methods of development for 17 of 40 tools i. Inter-rater reliability assessments were available from 11 of 13 studies included in Pieper [ 17 ], and for five of the 40 tools most reporting kappa or intraclass correlation coefficient in Whiting [ 12 ].
Six of the studies included in Pieper [ 17 ] assessed construct validity. No tests of validity were reported for any of the tools in Whiting [ 12 ] although exploratory factor analysis was used to develop the content of AMSTAR. In addition, Pieper [ 17 ] reported data on the time to complete the assessment of each tool. In addition, two of the three studies assessed the time to complete assessments [ 19 , 34 ]. In this paper, we present our developed framework of overview methods for the final steps in conducting an overview—assessment of the risk of bias in SRs and primary studies; synthesis, presentation and summary of the findings; and assessment of the certainty of evidence arising from the overview.
The evaluations included psychometric testing of tools to assess the risk of bias in SRs and development of a statistical measure to quantify overlap in primary studies across SRs. Results presented in this paper, in combination with our companion paper [ 10 ], provide a framework—the MOoR framework—of overview methods for all steps in the conduct of an overview. The framework makes explicit the large number of steps and methods that need to be considered when planning an overview and the unique decisions that need to be made as compared with a SR of primary studies.
Here, we focus on issues pertinent to this second companion paper and present some overarching considerations. A key observation from our first paper, and aligned with conclusions of others [ 8 , 9 ], was that there are important gaps in the guidance on the conduct of overviews [ 10 ]. Similar conclusions can be drawn from this paper, wherein guidance covers particular options, but not alternatives, and there is a lack of operational guidance for many methods. Detailed guidance on the applications of these tools has also been published.
The framework extends previous guidance on overviews methods [ 4 , 39 ] through provision of a range of methods and options that might be used for each step. For most methods, we identified a lack of evaluation studies, indicating that there is limited evidence to inform methods decision-making in overviews.
However, not all methods presented necessarily require evaluation. Theoretical considerations or poor face or content validity of a method may determine that it should not be used. Since the interpretation of evidence is highly dependent on limitations of primary studies within an SR, this option has little face validity. A further extension to previous guidance is the linking of methods from our framework to address commonly arising challenges in overviews.
The strengths and limitations described in the first paper in this series [ 10 ] are now briefly described here. The strengths of our research included a noting any deviations to our planned protocol [ 11 ], b using consistent language throughout the framework and an intuitive organising structure to group related methods and c drafting of the framework for each step by two authors independently.
An additional limitation is that new methods and methods evaluations may have been published since our last search August However, we sought to identify methods that were missing from the literature through inference so the structure of the framework is unlikely to change. Given the sparsity of evidence about the performance of methods, any new evaluations will be an important addition to the evidence base but are unlikely to provide definitive evidence. While the development of AMSTAR 2 reflects an important advancement on the previous version of AMSTAR extending to non-randomised studies and changing the response format , the tool will require application and further testing in overviews before its measurement properties can be fully established and compared to existing tools.
Overview methods are evolving, and as methods are developed and evaluated, the evidence map can be further refined and populated. There are two related, but distinct streams of research here. The first stream relates to the development and application of methods. Substantial work is needed to provide detailed guidance for applying methods that have been advocated for use in overviews, in addition to developing new methods where gaps exist. The development of GRADE guidance for overviews is an important example where both methods development and detailed guidance is required.
The second stream of research involves methods evaluation. In our first paper, we suggested three domains against which the performance of overview methods should be evaluated: the validity and reliability of overview findings, the time and resources required to complete the overview, and the utility of the overview for decision-makers. For example, researchers could compare the statistical performance of different metrics to assess the degree of overlap, or different statistical methods to adjust for overlap in meta-analyses, using numerical simulation studies.
A further area of research could include evaluation of different visual presentations of the range of summary results extracted from the constituent SRs. The framework will need to be refined, in response to methods development and evaluation. As mentioned in Paper 1, visual representation of an evidence map of overview methods will be useful when more evidence is available. Furthermore, our framework and evidence map only focused on overviews of intervention reviews.
The framework and evidence map could be extended to include methods for other types of overviews, such as overviews of diagnostic test accuracy reviews or prognostic reviews [ 42 ]. A framework of methods for the final steps in conducting, interpreting and reporting overviews was developed, which in combination with our companion paper, provide a framework of overview methods—the MOoR framework—for all steps in the conduct of an overview.
Evaluations of methods for overviews were identified and mapped to the framework. Further evaluation of methods for overviews will facilitate more informed methods decision-making. Results of this research may be used to identify and prioritise methods research, aid authors in the development of overview protocols and offer a basis for the development of reporting checklists.
Mediating policy-relevant evidence at speed: are systematic reviews of systematic reviews a useful approach? Evid Policy. Retrieval of overviews of systematic reviews in MEDLINE was improved by the development of an objectively derived and validated search strategy. J Clin Epidemiol. Overviews of reviews often have limited rigor: a systematic review. Chapter Overviews of reviews. Cochrane Handbook for Systematic Reviews of Interventions. Hoboken: Wiley; Differences in overlapping meta-analyses of vitamin D supplements and falls. J Clin Endocrinol Metab. Cooper H, Koenka AC. The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship.
Am Psychol. Overviews of systematic reviews: great promise, greater challenge. Syst Rev. Ballard M, Montgomery P.
- Brain and Longevity;
- ?thereals Pride (?thereal Series, Book 4).
Risk of bias in overviews of reviews: a scoping review of methodological guidance and four-item checklist. Res Synth Methods. What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary. Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction.
Evidence map of studies evaluating methods for conducting, interpreting and reporting overviews of systematic reviews of interventions: rationale and design. Chapter 4. Phase 2: review of existing quality assessment tools for systematic reviews. Evaluation of the methodological quality of systematic reviews of health status measurement instruments. The purpose is to enable policy officials and other readers to readily distinguish these programs from the many others that claim to have such evidence.
SPRC is funded by the U. Practicing evidence-based prevention means using the best available research and data throughout the process of planning and implementing your suicide prevention efforts. Included are youth facts, funding information, and tools to help you assess community assets, generate maps of local and federal resources, search for evidence-based youth programs, and keep up-to-date on the latest, youth-related news.
Practice Guidelines American Academy of Child and Adolescent Psychiatry Clinical Updates and Practice Guidelines "Clinical Updates - a new series of documents intended to address three topic areas: the psychiatric assessment and management of special populations of children and adolescents; the psychiatric assessment and management of children and adolescents in specific settings; and the application of specific psychiatric techniques to children and adolescents.
APA Professional Practice Guidelines "APA has approved as policy a variety of professional practice guidelines and related criteria in areas such as multicultural practice, child custody evaluations and treatment of gay, lesbian and bisexual clients. These guidelines address psychological practice with particular populations e.
Searching for Evidence | EBBP
DSM Library. The standard diagnostic tool used by mental health professionals worldwide to promote reliable research, accurate diagnosis, and thus appropriate treatment and patient care. Systematic Review and Other Sources Campbell Collaboration The Campbell Collaboration provides systematic reviews in order to help inform policy and service decisions regarding the effectiveness of certain interventions. The Social Welfare Coordinating Group section features both completed reviews and reviews in progress.
The library of systematic reviews can be searched by title, author, or keyword, or browsed by broad subject categories social and human sciences, education, politics, law and economics , most recent documents, or those accessed most frequently. Cochrane Collaboration Library - Free Summaries Although the full-text Cochrane reviews are not available for free, you can access the summaries of these systematic reviews in health care and health policy. Includes PICO search interface; can limit by evidence type.
Most trials have been critically appraised for their validity and interpretability. In one database, OTseeker provides fast and easy access to information from a wide range of sources to inform occupational therapy. Due to a lack of funding for OTseeker, content on OTseeker from and beyond is not as comprehensive as previously. Therefore, we recommend that users of OTseeker also search for more recent evidence using other freely available search engines and databases such as PEDro, and PubMed.
Related Finding and Evaluating Evidence: Systematic Reviews and Evidence-Based Practice
Copyright 2019 - All Right Reserved