Systematic and systematic-like review toolkit

Review type	Review purpose	Recommended timeframe*	Reviewers required
Meta-analysis	Determines the effectiveness of interventions by applying statistical analysis to the combined results of individual studies.	12-24 months	At least 2-3
Meta-synthesis	An interpretive synthesis of qualitative data which provides new insights into the research	12+ months	2-3
Systematic review	May be useful for clinical decision-making, determining best practice.	9-12+ months	At least 2-3
Integrative review	Can be used to determine best practice. May assist where diverse methodologies are required to answer a question, or where empirical evidence may not exist.	12+ months	At least 2-3
Rapid review	Useful for addressing issues needing quick decisions, or synthesising the evidence as part of a systematic review is not practical, often due to time constraints.	1-6 months	At least 2
Scoping review	Exploratory review. May assist with determining areas for new research or the usefulness of a future systematic review.	2-6 months	At least 2
Traditional literature review	Increase self-knowledge of an area, provide a broad overview or identify gaps in the literature, or as background for research.	1 week - 1 month	At least one
Mixed Methods Systematic Reviews (MMSR)	Will usually have multiple questions or several aspects to research question, to provide a more comprehensive picture of the issue.	Same as a systematic review or longer. 12-18+ months.	2+
Umbrella reviews	Most useful when there are competing interventions to consider.	May take about 3 months.	2-3
Living systematic reviews	To maintain currency of findings by ongoing review of the literature; most appropriate when the research field covered is rapidly evolving and new evidence is emerging.	3-6 months	2-3
Realist reviews	To provide a detailed understanding to policy makers and practitioners of the intervention and how it can be implemented to maximise effectiveness.	1-6 months	At least 1

*timeframe may vary from this guide, depending on the number of eligible articles included in the review and other demands on the reviewers' time

Systematic reviews are comprehensive evidence syntheses that are considered the gold standard review with a focus on high level evidence (often randomised controlled trials, but this depends on the question type).

These reviews seek to find, critique and synthesise all the available evidence for a focused question to determine the effectiveness of interventions, the levels of harm in interventions or to prepare for a meta-analysis. They support decision-making and development of best practice and can make recommendations for further research.

These reviews are exhaustive, transparent, and must be reproducible to enable updating and verification. They are ideally conducted by teams to minimise potential bias.

Synthesising the results of single studies and establishing overall findings for the question of interest from the larger body of evidence enables decision-makers to be more confident in the findings. Examples of significant shifts in best practice following the completion of a systematic review include the practice of giving live-saving corticosteroid injections to pregnant women at risk of giving birth prematurely (see the story of the Cochrane logo) and advice about sleeping positions for babies was contrary to the evidence for many years.

Cochrane is considered one of the most eminent publishers of systematic reviews, along with its sister organisation the Campbell Collaboration, and the Joanna Briggs Institute.
 

Note

Not all systematic reviews include meta-analyses.

 

Example review

• Internet-based interventions for smoking cessation.

 

References

American Occupational Therapy Association 2020, Guidelines for systematic reviews, AOTA Press, retrieved 15 June 2022, <https://research.aota.org/DocumentLibrary/AOTA_AJOT_systematic%20reviews%20instructions.pdf>.

Borah, R, Brown, AW, Capers, PL & Kaiser, KA 2017, 'Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry', BMJ Open, vol. 7, no. 2, pp. e012545-e.

Centre for Reviews and Dissemination 2009, Systematic reviews: CRD’s guidance for undertaking reviews in health care, CRD, retrieved 24 August 2017, <https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf>.

Green, S & Higgins, JPT 2011, Cochrane handbook for systematic reviews of interventions, The Cochrane Collaboration, UK.

Khan, KS 2010, Systematic reviews to support evidence-based medicine, 2nd edn, HodderArnold, London

Maglione, MA, Maher, AR, Ewing, B, Colaiaco, B, Newberry, S, Kandrack, R, Shanman, RM, Sorbero, ME & Hempel, S 2017, 'Efficacy of mindfulness meditation for smoking cessation: A systematic review and meta-analysis', Addictive Behaviors, vol. 69, pp. 27-34.

Integrative reviews seek to aggregate experimental and non-experimental data and/or theoretical information into a combined synthesis to answer specific questions. An integrative review will take as long, if not longer than a systematic review, and searching is systematic, exhaustive and transparent. This type of review is appropriate for questions which require studies with diverse methodologies to answer questions.  

An integrative review might seek to evaluate the strength of the existing scientific evidence, identify gaps in the literature, or bridge gaps between related areas of work. Examples include identifying culturally-specific practices and their influence on self-care in people with cardiovascular disease to design more effective treatment interventions, and an examination of the literature on social and psychological health outcomes associated with participation in team sport.
 

Example review

• Passive interventions in primary healthcare waiting rooms are effective in promoting healthy lifestyle behaviours: an integrative review.

References

Beyea, S & Nicoll, LH 1998, 'Writing an integrative review', AORN journal, vol. 67, no. 4, pp. 877-80.

Cass, SJ, Ball, LE & Leveritt, MD 2016, 'Passive interventions in primary healthcare waiting rooms are effective in promoting healthy lifestyle behaviours: an integrative review', Australian Journal of Primary Health, vol. 22, no. 3, pp. 198-210.

Russell, C. L. (2005). An overview of the integrative research review. Progress in Transplantation, vol. 15, no. 1, pp. 8-13.

Whittemore, R & Knafl, K 2005, 'The integrative review: updated methodology', Journal of Advanced Nursing, vol. 52, no. 5, pp. 546-53.

Rapid reviews (also known as rapid evidence assessment) also seek to synthesise research evidence with minimal bias to assist with best practice, however they are conducted on a shorter timeframe than systematic reviews and are less comprehensive. Searching is systematic and transparent, but less exhaustive than a systematic or integrative review.

What might make a Rapid Review 'rapid'?

Some aspects of the systematic review process are either omitted or shortened to enable the review to be conducted in a shorter timeframe. This could involve one or more of the following:

• Searching a smaller selection of databases
• Omitting grey literature and/or hand searching
• Restricting to a particular study type
• Reviewing reviews
• Limiting the number of outcomes being included
• Quality assessed at study design level, rather than appraising individual studies
• Limiting the amount of analysis and interpretation

The details of steps omitted in the rapid review process will be listed in the article, often in the limitations section, (see ‘The rollercoaster of follow-up care’ after bariatric surgery: a rapid review and qualitative synthesis and Cost-Effectiveness of Lifestyle-Related Interventions for the Primary Prevention of Breast Cancer: A Rapid Review).
 

Example reviews

• A rapid review of consumer health information needs and preferences.
• Quarantine alone or in combination with other public health measures to control COVID‐19: a rapid review

References

Crawford, C, Boyd, C, Jonas, W, Jain, S & Khorsan, R 2015, 'Rapid Evidence Assessment of the Literature (REAL©): Streamlining the systematic review process and creating utility for evidence-based health care', BMC Research Notes, vol. 8, no. 1.

Dobbins, M 2017, Rapid review guidebook: steps for conducting a rapid review, NCCMT, retrieved 11 June 2018, <http://www.nccmt.ca/uploads/media/media/0001/01/a816af720e4d587e13da6bb307df8c907a5dff9a.pdf>.

Hamel, C., Michaud, A., Thuku, M., Skidmore, B., Stevens, A., Nussbaumer-Streit, B., & Garritty, C. (2021). Defining rapid reviews: a systematic scoping review and thematic analysis of definitions and defining characteristics of rapid reviews. Journal of Clinical Epidemiology, 129, 74-85.

Garritty C, Gartlehner G, Kamel C, King VJ, Nussbaumer-Streit B, Stevens A, Hamel C, Affengruber L. Cochrane Rapid Reviews. Interim Guidance from the Cochrane Rapid Reviews Methods Group. March 2020.

Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012). Evidence summaries: the evolution of a rapid review approach. Systematic reviews, 1(1), 1-9.

Ramsey, I, Corsini, N, Peters, MDJ & Eckert, M 2017, 'A rapid review of consumer health information needs and preferences', Patient Education and Counseling.

World Health Organization, Alliance for Health Policy and Systems Research. Rapid reviews to strengthen health policy and systems: a practical guide. 2017

Scoping reviews tend to be broader in scope than systematic reviews but have clearly defined questions. Searching is systematic and transparent. The comprehensiveness of searching is determined by time and scope constraints. May take a longer time complete than a systematic review.

Scoping reviews seek to explore and establish overall themes, or determine the extent of available evidence, particularly in an emerging health area, as well as any gaps in the available research literature. Scoping reviews may be used to scope further research and determine the possible usefulness of a future systematic review. A scoping review can also be undertaken as a stand-alone project, especially where an area is complex or has not been reviewed in depth previously.

The aim is to comprehensively summarize evidence to inform practice, programs and policy.  

Example reviews include a scoping review looking to define a concept is Defining value in health care: a scoping review of the literature, or A Scoping Review of the Health and Social Benefits of Dog Ownership for People Who Have Chronic Pain with a recommendation for a systematic review to be conducted.
 

Example review

• Tobacco use cessation interventions for lesbian, gay, bisexual, transgender and queer youth and young adults: A scoping review.
   

Further reading

Armstrong, R, Hall, B., Doyle, J, Waters, E. ‘Scoping the scope’ of a cochrane review, Journal of Public Health, Volume 33, Issue 1, March 2011, Pages 147–150, https://doi.org/10.1093/pubmed/fdr015

This article details the characteristics of a scoping review.

Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, McInerney P, Godfrey C, Khalil H. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. 2020 Oct;18(10):2119-2126. doi: 10.11124/JBIES-20-00167. PMID: 33038124.
 

References

Arksey, H & O'Malley, L 2005, 'Scoping studies: Towards a methodological framework', International Journal of Social Research Methodology: Theory and Practice, vol. 8, no. 1, pp. 19-32.

Colquhoun, H 2016, Current best practices for the conduct of scoping reviews, Equator Network, retrieved June 11 2018, <http://www.equator-network.org/wp-content/uploads/2016/06/Gerstein-Library-scoping-reviews_May-12.pdf>.

Joanna Briggs Institute 2015, The Joanna Briggs Institute Reviewers’ Manual 2015 Methodology for JBI Scoping Reviews, JBI University of Adelaide, Adelaide, retrieved 17 July 2017, <https://jbi-global-wiki.refined.site/space/MANUAL/4687770/11.3+The+scoping+review+and+summary+of+the+evidence>

Levac, D., Colquhoun, H., & O'Brien, K. K. (2010). Scoping studies: advancing the methodology. Implementation science, 5(1), 1-9.

Munn, Z., Peters, M.D.J., Stern, C. et al. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approaches. BMC Med Res Methodol 18, 143 (2018). https://doi.org/10.1186/s12874-018-0611-x

Noble H, Smith J. Reviewing the literature: choosing a review design. Evidence-Based Nursing 2018;21:39-41.

Pollock, D., Davies, E. L., Peters, M. D., Tricco, A. C., Alexander, L., McInerney, P., ... & Munn, Z. (2021). Undertaking a scoping review: A practical guide for nursing and midwifery students, clinicians, researchers, and academics. Journal of advanced nursing, 77(4), 2102-2113.

Not strictly speaking a review, a meta-analysis is usually developed from or within a systematic review. Meta-analyses are quantitative statistical analyses of studies identified during a systematic review which are similar enough to each other for researchers to be able to pool the data of each of them together for further analysis.  These studies will have had a similar research question and methodology. The meta-analysis process can enhance the objectivity and validity of findings.

Small or inconclusive studies lacking in statistical significance can contribute to the larger picture. Conclusions produced by meta-analysis are statistically stronger than the analysis of any single study, due to factors such as increased numbers of subjects and greater diversity among subjects.
 

Example reviews

• Daily steps and all-cause mortality: a meta-analysis of 15 international cohorts
• Efficacy of mindfulness meditation for smoking cessation: a systematic review and meta-analysis.
   

Further reading

This piece notes that a potential pitfall of a meta-analysis is the potential to be misleading if differences in study designs, biases within studies, differences across studies and reporting biases are not carefully considered.

Deeks JJ, Higgins JPT, Altman DG (editors). Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook.
 

References

Dekkers, Olaf M. “Meta-analysis: Key features, potentials and misunderstandings.” Research and practice in thrombosis and haemostasis vol. 2,4 658-663. 3 Oct. 2018, doi:10.1002/rth2.12153

Shorten, A & Shorten, B 2013, 'What is meta-analysis?', BMJ Journals - Evidence-based Nursing, vol. 16, no. 1, pp. 3-4.

Meta-syntheses are syntheses of qualitative research. Usually done as part of a systematic review. Requires a clearly formulated question, to set boundaries for the scope and depth of the review.  

They serve several purposes: theory building, theory explication and descriptive meta-synthesis. A meta-synthesis involves rigorous analysis and interpretation of the data, but does not seek to find a single answer – findings are integrated or compared across many qualitative studies

Best designed for generating new theories of why an intervention works or not, or comparison with trial outcomes.
 

Example review

• Meta-Synthesis of Qualitative Research on Facilitators and Barriers of Return to Work After Stroke
   

References

Edwards, J., & Kaimal, G. (2016). Using meta-synthesis to support application of qualitative methods findings in practice: A discussion of meta-ethnography, narrative synthesis, and critical interpretive synthesis. The Arts in Psychotherapy, 51, 30-35.

Finfgeld, D 2003, 'Metasynthesis: The state of the art - So far', Qualitative Health Research, vol. 13, no. 7, pp. 893-904.

Schwarz, B, Claros-Salinas, D & Streibelt, M 2017, 'Meta-Synthesis of Qualitative Research on Facilitators and Barriers of Return to Work After Stroke', Journal Of Occupational Rehabilitation.

Walsh, D & Downe, S 2005, 'Meta-synthesis method for qualitative research: a literature review', Journal Of Advanced Nursing, vol. 50, no. 2, pp. 204-11.

Traditional literature reviews are not systematic, but some may take a systematic or an iterative approach to searching, particularly if the review seeks to be comprehensive. The search strategy is not typically reported.

They are designed to increase self-knowledge of an area of specialisation, to provide a broad overview of an area of specialisation, or they may provide background to a research thesis. They are unlikely to be published as stand-alone pieces of work.
 

Example review

• Smoking Cessation for People Living With HIV/AIDS: A Literature Review and Synthesis.
   

Further reading

This editorial piece notes that traditional narrative reviews are better for some topics due to their broader coverage but could benefit from by drawing on the rigour of systematic reviews.

Collins, J. A., & Fauser, B. C. J. M. (2005). Balancing the strengths of systematic and narrative reviews. Human Reproduction Update, 11(2), 103–104. https://doi.org/10.1093/humupd/dmh058

This piece provides guidance on how to produce a literature review using methods to minimise bias.

Green, B. N., Johnson, C. D., & Adams, A. (2006). Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of chiropractic medicine, 5(3), 101-117.

Provides some comparison of traditional literature reviews against scoping reviews.

Munn, Z., Peters, M.D.J., Stern, C. et al. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approaches. BMC Med Res Methodol 18, 143 (2018). https://doi.org/10.1186/s12874-018-0611-x

Mixed methods systematic reviews (MMSR) are best suited to topics that are multidisciplinary or topics with a body of literature that includes quantitative, qualitative, and mixed methods studies.

They have multiple questions or several aspects to their research question to provide a more comprehensive picture of the issue. For example, they may look to determine both the effect of an intervention, and when it is clinically appropriate.  This usually requires multiple searches to address subtopics.

MMSR can be resource intensive with a lengthy planning phase to determine the available evidence and refine the review scope and questions. Further questions or hypotheses may emerge as the reviews progress and therefore the protocols are generally developed over time rather than in a predetermined manner. These reviews are less transparent and not inherently reproducible because of their highly iterative nature. There are currently no universally adopted methods for conducting Mixed Methods Reviews.

Synthesis can integrate quantitative and qualitative evidence or data from primary studies, but usually involves both statistical meta-analysis and qualitative synthesis to produce a narrative understanding.  

An example of how qualitative data is incorporated into a synthesis can be seen in A mixed-methods systematic review of the effects of mindfulness on nurses.
 

Example review

A study about managing pain for people experiencing pressure ulcers found that the words used by individuals in four qualitative studies to describe their pressure ulcer pain were not matched by those listed in existing pain measurements instruments. This has implications for the assessment of pain and any changes in a clinical setting, but also raises questions about data quality and interpretation.
 

References

Pearson, A., White, H., Bath-Hextall, F., Salmond, S., Apostolo, J., & Kirkpatrick, P. (2015). A mixed-methods approach to systematic reviews. International journal of evidence-based healthcare, 13(3), 121-131.

A relatively new addition to review types, umbrella reviews are essentially systematic reviews of existing reviews, also known as a "review of reviews". Umbrella reviews have a broader focus than a systematic review and aim to locate and summarise all the evidence from research syntheses related to a topic or question. This evidence can then be used to inform guidelines and clinical practice by:

• considering different treatment or management options for the same condition
• examining the same intervention and condition, but with different outcomes
• evaluate the same intervention but when applied to a different condition, situation or population.

Each comparison is considered separately and used to summarise or synthesise the findings. Where appropriate meta-analyses are performed.  

Generally, there is no search for primary studies.
 

Example review

• Piovani, D., Danese, S., Peyrin-Biroulet, L., Nikolopoulos, G. K., Lytras, T., & Bonovas, S. (2019). Environmental risk factors for inflammatory bowel diseases: an umbrella review of meta-analyses. Gastroenterology, 157(3), 647-659.
   

References

Aromataris E, Fernandez R, Godfrey C, Holly C, Khalil H, Tungpunkom P. Chapter 10: Umbrella Reviews. In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020. Available from https://jbi-global-wiki.refined.site/space/MANUAL/4687363.

Aromataris, E; Fernandez, R; Godfrey, C; Holly, C; Khalil, H; Tungpunkom, P Summarizing systematic reviews, International Journal of Evidence-Based Healthcare: September 2015 - Volume 13 - Issue 3 - p 132-140 doi: 10.1097/XEB.0000000000000055

Living reviews are an approach to updating reviews, rather than a formal review methodology. Living review processes allow the author team to stay up to date with the newest evidence, which is then integrated into their review. The conclusions and citations are updated as new evidence is incorporated.   

A living review requires predefined and transparent decisions on how often new evidence will be searched for, and, as well as when and how this new evidence will be included in the review. There also needs to be some consideration as to when the review no longer requires regular updating.  

Living reviews are most appropriate for topics that are rapidly developing. Over time, it is expected that the scope of questions for which a living systematic review approach is appropriate will expand.  

Living reviews are enabled by new technologies, as well as crowd sourcing initiatives.
 

Example review

COVID-19 is an example of a rapidly emerging condition, with a great deal of research activity in the area. There are many living reviews on this topic.

• Association of frailty with outcomes in individuals with COVID-19: A living review and meta-analysis 
   

References

Akl, E., Meerpohl, J, Elliott, J., Kahale, L., Schünemann, H., Agoritsas, T., Hilton, J., Perron, C., Akl, E., Hodder, R. and Pestridge, C., 2017. Living systematic reviews: 4. Living guideline recommendations. Journal of clinical epidemiology, 91, pp.47-53.

Elliott, J., Synnot, A., Turner, T., Simmonds, M., Akl, E., McDonald, S., Salanti, G., Meerpohl, J., MacLehose, H., Hilton, J. and Tovey, D., 2017. Living systematic review: 1. Introduction—the why, what, when, and how. Journal of clinical epidemiology, 91, pp.23-30.

Elliott J, Turner T, Clavisi O, Thomas J, Higgins J, Mavergames C, et al. (2014) Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap. PLoS Med 11(2): e1001603.

Another relatively new method for synthesising research is the realist review (also called realist syntheses). These reviews differ from traditional methods of review which focus on measuring and reporting on the effectiveness of an intervention, but often do not provide detail as to why the intervention worked (or not) when applied in different circumstances.  

A realist review has an explanatory focus and will investigate both the processes and impacts of health or social care interventions. There is a recognition of the various factors such as policy timing, organizational culture, resource allocation, staffing levels and skill, and competing priorities that can impact on service delivery, meaning it is unlikely that the same intervention will work in the same way in different contexts.

The realist approach will use both, or either, quantitative or qualitative methods.  

The search stage is iterative and lengthy, with multiple search strategies needed to retrieve materials to answer specific questions or test theories. Practically, there may be more potentially relevant sources of information than can be included. Decisions are needed around when to stop looking for evidence and will be limited by time and funding.  

Pawson et al have proposed a method for conducting realist reviews, however this is often customised by research teams.
 

Example reviews

• Flynn, R., Rotter, T., Hartfield, D. et al. A realist evaluation to identify contexts and mechanisms that enabled and hindered implementation and had an effect on sustainability of a lean intervention in pediatric healthcare. BMC Health Serv Res 19, 912 (2019).
• Kantilal K, Hardeman W, Whiteside H, et al. Realist review protocol for understanding the real-world barriers and enablers to practitioners implementing self-management support to people living with and beyond cancer. BMJ Open 2020;10.
   

References

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005 Jul;10 Suppl 1:21-34.  

Rycroft-Malone, J., McCormack, B., Hutchinson, A.M. et al. Realist synthesis: illustrating the method for implementation research. Implementation Sci 7, 33 (2012).