Critical appraisal
After you’ve completed the screening stage and identified relevant studies for your review, you then critically appraise these studies before drawing conclusions or making recommendations.
Critical appraisal is a systematic process which aims to thoroughly evaluate the quality, reliability, and validity of research studies. By systematically identifying potential biases and limitations, it ensures the evidence you rely on is sound and credible.
Approach to critically appraisal in a review
Critical appraisal in a literature review should take a structured approach:
1. Choose the right tool
2. Evaluate the studies
3. Record judgments
Choosing the right tool
Selecting the right critical appraisal tool is key to ensuring a fair and accurate evaluation of a study. Your choice will depend on the review’s scope, included study designs, and overall goals.
Tip
To help identify study types, use resources such as NICE’s Algorithm for classifying quantitative study designs, the Glossary of study designs, or our Study designs guide.
Tools for critical appraisal
There are numerous critical appraisal tools available, each tailored to specific study designs or intended for more versatile application.
Click on the plus (+) icons below to learn about some critical appraisal tools and the study designs they are useful for appraising.
CASP (Critical Appraisal Skills Programme)
Description
CASP (Critical Appraisal Skills Programme) provides structured checklists with clear prompts and questions to guide the appraisal process, helping users assess the quality and relevance of research in a systematic way.
Useful for
A variety of study designs, including qualitative research, randomised controlled trials (RCTs), and cohort studies.
JBI (Joanna Briggs Institute)
Description
JBI (Joanna Briggs Institute) offers detailed critical appraisal checklists with explanatory guidance, supporting a structured and consistent approach to assessing research quality across a wide range of methodologies.
Useful for
Appraising a variety of study designs, including observational studies, systematic reviews, qualitative research, and text or opinion-based papers.
ROB (Risk of Bias) Tools
Description
ROB2 a structured, domain-based tool developed by Cochrane to help reviewers systematically assess the risk of bias in the conduct and reporting of randomised controlled trials.
Useful for
Appraising randomised controlled trials, particularly to evaluate potential biases related to randomisation, deviations from intended interventions, missing data, outcome measurement, and reporting.
Description
ROBINS-I a comprehensive tool for assessing risk of bias in non-randomised studies of interventions, guiding users through multiple domains to evaluate how study design and execution may affect validity.
Useful for
Appraising non-randomised studies such as cohort and case-control designs, with a focus on identifying confounding, selection bias, and measurement issues.
AMSTAR (A Measurement Tool to Assess Systematic Reviews)
Description
AMSTAR (A Measurement Tool to Assess Systematic Reviews) a standardised checklist that helps users assess the methodological rigour of systematic reviews, including how comprehensively they were planned, conducted, and reported.
Useful for
Evaluating the quality of systematic reviews, particularly in healthcare and clinical research contexts.
AACODS
Description
AACODS a structured framework designed to guide critical appraisal of grey literature by assessing six key elements: Authority, Accuracy, Coverage, Objectivity, Date, and Significance.
Useful for
Evaluating grey literature sources such as reports, policy documents, theses, and conference papers, where traditional peer review is absent.
AGREE II
Description
AGREE II a validated tool that helps users assess the methodological quality and transparency of clinical practice guidelines, focusing on aspects like stakeholder involvement, rigour of development, and clarity of presentation.
Useful for
Appraising clinical practice guidelines to determine their trustworthiness, relevance, and potential for implementation in healthcare settings.
MMAT (McMaster Mixed Methods Appraisal Tool)
Description
MMAT (McMaster Mixed Methods Appraisal Tool) a unified appraisal tool that includes distinct criteria for five different methodological categories, making it suitable for use in reviews that include a mix of study types.
Useful for
A versatile tool suited for evaluating qualitative, quantitative, and mixed-methods research within a single framework.
The Aboriginal and Torres Strait Islander Quality Appraisal Tool (QAT)
Description
The Aboriginal and Torres Strait Islander Quality Appraisal Tool (QAT) specifically designed for Aboriginal and Torres Strait Islander research, this tool evaluates the inclusion of Indigenous knowledge, community engagement, and cultural safety.
Useful for
Assessing whether research meaningfully incorporates Indigenous perspectives, respects cultural protocols, and benefits Aboriginal and Torres Strait Islander communities.
Critically Appraising for Antiracism Tool
Description
Critically Appraising for Antiracism Tool provides structured reflective questions with explanatory guidance to help users identify racial bias in research design, conduct, and reporting.
Useful for
Critically reviewing research for racial bias, particularly in studies involving racialised communities or addressing social and health inequities.
Alert
Before fully appraising your selected studies, it’s a good idea to pilot your chosen tool on a small subset of studies. This helps ensure you understand the tool’s criteria and can apply it consistently, and it allows you to evaluate whether the tool is the right fit for your review.
Evaluate the studies
Once you have chosen and piloted your critical appraisal tool, the next step is to evaluate the studies systematically. This involves assessing each study's quality, identifying potential biases.
Following the tool’s protocol:
- Ensure you adhere to the guidelines provided by your chosen critical appraisal tool.
- Involve multiple appraisers: This can help reduce subjectivity and improve the reliability of your assessments.
- Evaluate methodological quality and Risk of Bias (RoB): Use the criteria outlined in the tool to guide your judgments.
- Weight the evidence: When synthesising findings, consider the relative quality of each study, for example, higher-quality studies with lower risk of bias might have greater influence on your conclusions.
Record judgements
Your findings for each study should be documented in a consistent format. Many critical appraisal tools provide templates or scoring systems to help structure your findings.
Documenting your reasoning ensures that your appraisal is transparent and can be revisited if needed.
Note
Following a structured approach to critical appraisal in your review supports transparent reporting and helps uphold the principles of Evidence-Based Practice (EBP), specifically serving as the Appraise step in the EBP process, ultimately contributing to the continuous improvement of research quality and the decisions drawn from it.