Systematic and systematic-like review toolkit

Your protocol (or plan for conducting your review) should include the rationale, objectives, hypothesis, and planned methods used in searching, screening and analysing identified studies used in the review. The rationale should clearly state what will be included and excluded from the review. The aim is to minimise any bias by having pre-defined eligibility criteria.

Base the protocol on the relevant guidelines for the review that you are conducting. PRISMA-P was developed for reporting and development of protocols for systematic reviews. Their Explanation and Elaboration paper includes examples of what to write in your protocol. York's CRD has also created a document on how to submit a protocol to PROSPERO.

There are several registers of protocols, often associated with the organisation publishing the review. Cochrane and Joanna Briggs Institute both have their own protocol registries, and PROSPERO is a wide-reaching registry covering protocols for Cochrane, non-Cochrane and non-JBI reviews on a range of health, social care, education, justice, and international development topics.

Before beginning your protocol, search within protocol registries such as those listed above, or Open Science Framework or Research Registry, or journals such as Systematic Reviews and BMJ Open. This is a useful step to see if a protocol has already been submitted on your review topic and to find examples of protocols in similar areas of research.    

While a protocol will contain details of the intended search strategy, a protocol should be registered before the search strategy is finalised and run, so that you can show that your intention for the review has remained true and to limit duplication of in progress reviews.  

A protocol should typically address points that define the kind of studies to be included and the kind of data required to ensure the systematic review is focused on the appropriate studies for the topic. Some points to think about are:

• What study types are you looking for? For example, randomised controlled trials, cohort studies, qualitative studies
• What sample size is acceptable in each study (power of the study)? 
• What population are you focusing on? Consider age ranges, gender, disease severity, geography of patients.
• What type of intervention are you focusing on?
• What outcomes are of importance to the review, including how those outcomes are measured?
• What context should you be looking for in a study? A lab, acute care, school, community...
• How will you appraise the studies? What methodology will you use?
• Does the study differentiate between the target population and other groups in the data? How will you handle it if it does not?
• Is the data available to access if the article does not specify the details you need? If not, what will you do?
• What languages are you able to review? Do you have funding to translate articles from languages other than English?
   

Further reading

PLoS Medicine Editors. (2011). Best practice in systematic reviews: the importance of protocols and registration. PLoS medicine, 8(2), e1001009.

The Cochrane handbook of systematic reviews of interventions is a world-renowned resource for information on designing systematic reviews of intervention.  

Many other guidelines have been developed from these extensive guidelines.

General systematic reviews

• The PRISMA Statement includes the well-used Checklist and Flow Diagram.
• Systematic Reviews: CRD's guidance on undertaking reviews in health care. One of the founding institutions that developed systematic review procedure. CRD's guide gives detailed clearly written explanations for different fields in Health.
• National Academies Press (US); 2011. 3, Standards for Finding and Assessing Individual Studies. Provides guidance on searching, screening, data collection, and appraisal of individual studies for a systematic review.

Meta-analyses

• An alternative to PRISMA is the Meta‐analysis Of Observational Studies in Epidemiology (MOOSE) for observational studies. It is a 35‐item checklist. It pays more attention to certain aspects of the search strategy, in particular the inclusion of unpublished and non‐English‐language studies.

Surgical systematic reviews

• Systematic reviews in surgery-recommendations from the Study Center of the German Society of Surgery. Provides recommendations for systematic reviews in surgery with or without meta-analysis, for each step of the process with specific recommendations important to surgical reviews.

Nursing/Allied Health systematic reviews

• Joanna Briggs Institute Manual for Evidence Synthesis a comprehensive guide to conducting JBI systematic and similar reviews

Nutrition systematic reviews

• Academy of Nutrition and Dietetics Evidence Analysis Manual is designed to guide expert workgroup members and evidence analysts to understand and carry out the process of conducting a systematic review.

Occupational therapy

• American Occupational Therapy Association: Guidelines for Systematic reviews. The American Journal of Occupational Therapy (AJOT) provides guidance for authors conducting systematic reviews.

Education/Law/ Sociology systematic reviews

• Campbell Collaboration, Cochrane's sister organisation provides guidelines for systematic reviews in the social sciences: MECIR
• Systematic Reviews in Educational Research: Methodology, Perspectives and Application

Other

• Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy

• COSMIN Guideline for Systematic Reviews of Outcome Measurement Instruments – This was developed for patient reported outcomes (PROMs) but has since been adapted for use with other types of outcome measurements in systematic reviews.

• Prinsen, C.A.C., Mokkink, L.B., Bouter, L.M. et al. COSMIN guideline for systematic reviews of patient-reported outcome measures. Qual Life Res 27, 1147–1157 (2018). https://doi.org/10.1007/s11136-018-1798-3

• HuGENet™ Handbook of systematic reviews – particularly useful for describing population-based data and human genetic variants.

• AHRQ: Methods Guide for Effectiveness and Comparative Effectiveness Reviews - from the US Department of Health and Human Services, guidelines on conducting systematic reviews of existing research on the effectiveness, comparative effectiveness, and harms of different health care interventions.

• Mariano, D. C., Leite, C., Santos, L. H., Rocha, R. E., & de Melo-Minardi, R. C. (2017). A guide to performing systematic literature reviews in bioinformatics. arXiv preprint arXiv:1707.05813.

Integrative reviews may incorporate experimental and non-experimental data, as well as theoretical information.  They differ from systematic reviews in the diversity of the study methodologies included.

Guidelines:

• Whittemore, R. and Knafl, K. (2005), The integrative review: updated methodology. Journal of Advanced Nursing, 52: 546–553. doi:10.1111/j.1365-2648.2005.03621.x
• A step-by-step guide to conducting an Integrative Review (2020), edited by C.E. Toronto & Ruth Remington, Springer Books

Rapid reviews differ from systematic reviews in the shorter timeframe taken and reduced comprehensiveness of the search.

Cochrane has a methods group to inform the conduct of rapid reviews with a bibliography of relevant publications.

A modified approach to systematic review guidelines can be used for rapid reviews, but guidelines are beginning to appear:

• Crawford C, Boyd C, Jain S, Khorsan R and Jonas W (2015), Rapid Evidence Assessment of the Literature (REAL©): streamlining the systematic review process and creating utility for evidence-based health care. BMC Res Notes 8:631 DOI 10.1186/s13104-015-1604-z

• Philip Moons, Eva Goossens, David R. Thompson, Rapid reviews: the pros and cons of an accelerated review process, European Journal of Cardiovascular Nursing, Volume 20, Issue 5, June 2021, Pages 515–519, https://doi.org/10.1093/eurjcn/zvab041

• Rapid Review Guidebook: Steps for conducting a rapid review National Collaborating Centre for Methods and Tools (McMaster University and Public Health Agency Canada) 2017

• Tricco AC, Langlois EV, Straus SE, editors (2017) Rapid reviews to strengthen health policy and systems: a practical guide (World Health Organization). This guide is particularly aimed towards developing rapid reviews to inform health policy. 

Scoping reviews can be used to map an area, or to determine the need for a subsequent systematic review. Scoping reviews tend to have a broader focus than many other types of reviews, however, still require a focused question.

Guidelines:

• Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil, H. Chapter 11: Scoping Reviews (2020 version). In: Aromataris E, Munn Z (Editors). Joanna Briggs Institute Reviewer's Manual, JBI, 2020. 
• Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR):
  • Checklist
  • Statement / Explanatory paper

• Scoping reviews: what they are and how you can do them - Series of Cochrane Training videos presented by Dr. Andrea C. Tricco and Kafayat Oboirien

• Martin, G. P., Jenkins, D. A., Bull, L., Sisk, R., Lin, L., Hulme, W., ... & Group, P. H. A. (2020). Toward a framework for the design, implementation, and reporting of methodology scoping reviews. Journal of Clinical Epidemiology, 127, 191-197.

• Khalil, H., McInerney, P., Pollock, D., Alexander, L., Munn, Z., Tricco, A. C., ... & Peters, M. D. (2021). Practical guide to undertaking scoping reviews for pharmacy clinicians, researchers and policymakers. Journal of clinical pharmacy and therapeutics.

• Colquhoun, H (2016) Current best practices for the conduct of scoping reviews (presentation)

• Arksey H & O'Malley L (2005) Scoping studies: towards a methodological framework, International Journal of Social Research Methodology, 8:1, 19-32, DOI: 10.1080/1364557032000119616

• Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L. Chapter V: Overviews of Reviews. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Available from www.training.cochrane.org/handbook.
   
• Aromataris E, Fernandez R, Godfrey C, Holly C, Khalil H, Tungpunkom P. Chapter 10: Umbrella Reviews. In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020. Available from https://jbi-global-wiki.refined.site/space/MANUAL/4687363.

• Aromataris, Edoardo; Fernandez, Ritin; Godfrey, Christina M.; Holly, Cheryl; Khalil, Hanan; Tungpunkom, Patraporn. Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach, International Journal of Evidence-Based Healthcare: September 2015 - Volume 13 - Issue 3 - p 132-140.

• Noyes, J., Booth, A., Cargo, M., Flemming, K., Garside, R., Hannes, K., ... & Thomas, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 1: introduction. Journal of clinical epidemiology, 97, 35-38.

• Harris, J. L., Booth, A., Cargo, M., Hannes, K., Harden, A., Flemming, K., ... & Noyes, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis. Journal of clinical epidemiology, 97, 39-48.

• Noyes, J., Booth, A., Flemming, K., Garside, R., Harden, A., Lewin, S., ... & Thomas, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. Journal of clinical epidemiology, 97, 49-58.

• Cargo, M., Harris, J., Pantoja, T., Booth, A., Harden, A., Hannes, K., ... & Noyes, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 4: methods for assessing evidence on intervention implementation. Journal of clinical epidemiology, 97, 59-69.

• Harden, A., Thomas, J., Cargo, M., Harris, J., Pantoja, T., Flemming, K., ... & Noyes, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. Journal of clinical epidemiology, 97, 70-78.

• Flemming, K., Booth, A., Hannes, K., Cargo, M., & Noyes, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—Paper 6: Reporting guidelines for qualitative, implementation, and process evaluation evidence syntheses. Journal of Clinical Epidemiology, 97, 79-85.

• Walsh, D. and Downe, S. (2005), Meta-synthesis method for qualitative research: a literature review. Journal of Advanced Nursing, 50: 204–211. doi:10.1111/j.1365-2648.2005.03380.x

• Akl, E.A., Meerpohl, J.J., Elliott, J., Kahale, L.A., Schünemann, H.J., Agoritsas, T., Hilton, J., Perron, C., Akl, E., Hodder, R. and Pestridge, C., 2017. Living systematic reviews: 4. Living guideline recommendations. Journal of clinical epidemiology, 91, pp.47-53.

• Dixon-Woods, M., Bonas, S., Booth, A., Jones, D. R., Miller, T., Sutton, A. J., . . . Young, B. (2006). How can systematic reviews incorporate qualitative research? A critical perspective. Qualitative Research,6(1), 27–44.
• Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology,8, 45–45.

• Lizarondo L, Stern C, Carrier J, Godfrey C, Rieger K, Salmond S, Apostolo J, Kirkpatrick P, Loveday H. Chapter 8: Mixed methods systematic reviews. In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020. Available from https://synthesismanual.jbi.global. https://doi.org/10.46658/JBIMES-20-09
• Pearson, A, White, H, Bath-Hextall, F, Salmond, S, Apostolo, J, & Kirkpatrick, P 2015, 'A mixed-methods approach to systematic reviews', International Journal of Evidence-Based Healthcare, vol. 13, no. 3, p. 121-131. Available from: 10.1097/XEB.0000000000000052
• Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005). Synthesising qualitative and quantitative evidence: A review of possible methods. Journal of Health Services Research &Policy,10(1), 45–53.

• The RAMESES Projects - Includes information on publication, quality, and reporting standards, as well as training materials for realist reviews, meta-narrative reviews, and realist evaluation.

• Rycroft-Malone, J., McCormack, B., Hutchinson, A. M., DeCorby, K., Bucknall, T. K., Kent, B., ... & Wilson, V. (2012). Realist synthesis: illustrating the method for implementation research. Implementation Science, 7(1), 1-10.

• Wong, G., Westhorp, G., Manzano, A. et al. RAMESES II reporting standards for realist evaluations. BMC Med 14, 96 (2016). https://doi.org/10.1186/s12916-016-0643-1

• Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: realist syntheses. BMC medicine, 11, 21. https://doi.org/10.1186/1741-7015-11-21

• Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: realist syntheses. BMC medicine, 11(1), 1-14. https://doi.org/10.1186/1741-7015-11-21

• Social sciences

  • Chapman, K. (2021). Characteristics of systematic reviews in the social sciences. The Journal of Academic Librarianship, 47(5), 102396.
  • Crisp, B. R. (2015). Systematic reviews: A social work perspective. Australian Social Work, 68(3), 284-295.
     

  Further Reading

  Uttley, L., Montgomery, P. The influence of the team in conducting a systematic review. Syst Rev 6, 149 (2017). https://doi.org/10.1186/s13643-017-0548-x