Systematic and systematic-like review toolkit

Your completed search framework from Step 1 will help to determine the key concepts and develop relevant search terms.  

There is flexibility in which elements need to be included in the final search strategy.  Additionally, different elements may be interchanged (for example, intervention or exposure or other element of interest in PICO).

Here is an example from the previous scenario:

Each of these search lines can be developed further by exploring additional keywords (synonyms, acronyms) and subject headings (or controlled vocabularies), an important indexing tool used in databases.

Each element of your search can then be linked together in a final search with the AND operator within the search history.
 

Tip

At least one element is usually left out of the search strategy. Consider what you can search for, and what you can screen for.

Often outcomes are excluded from the search, as these may restrict the scope of the search if included. Outcomes can be determined from within the found studies. Generally, outcomes are not well described in abstracts, and the inclusion of the term 'outcome' will retrieve abstracts with a study outcome section, regardless of whether this includes details of a clinical or therapeutic outcome.

If you're conducting a search to scope and explore broader areas, instead of search frameworks, use between two and four major concepts for your search.

If you're new to a discipline area, preliminary searches (sometimes referred to as scoping searching) can help you to get to know the literature.  This simple searching will help with identifying key authors and developing a 'gold set' of key articles for your topic.

The gold set can help you to identify possible search terms - check the title, abstract and subject headings or keywords.

The gold set can also be used to help test the comprehensiveness of your final search for your review. These are articles which should be picked up by your search, as well as retrieving other relevant papers). The more articles you have in your gold set, the more confidence you will have that you are not going to miss relevant papers due to the chosen terms and parameters of your search.

Your gold set may include:

• key papers recommended by others
• references used in similar reviews
• results of background or preliminary searching
   

Tip

You can also use your gold set for snowball searching - following links to citations and reference lists of an article to find related articles.

Your topic, search framework and preliminary search will provide a basis for identifying your search terms.

It's important to consider a wide variety of terms for a comprehensive search.

You can brainstorm search terms by:

• exploring ‘gold set’ of articles either supplied to you or found through preliminary searching.
• reviewing search strategies from published systematic reviews.
• scanning article database records and full text of item found in preliminary searches for applicable controlled vocabulary and relevant and/or frequently occurring terms.  
• using database tools (e.g., thesauri, index, subject headings) to identify controlled vocabulary terms
• using Text Mining tools to identify recurrent words or phrases

For each concept, consider:

• alternative spellings (i.e.: British English vs. American English, and any common misspellings)
• alternative or outdated names (e.g. handicapped for disability) or terms (e.g.: for procedures or diseases)
• abbreviations or acronyms
• synonyms and antonyms
• plurals
• scientific and common or layperson terminology
• if relevant, non-English language alternatives
• database-specific indexing terms (eg. MeSH in Medline, CINAHL Headings in CINAHL)

As an example, searching for 'cancer' might include the following terms:

cancer OR malignancy/malignant OR tumour/tumor(s) OR (MH "Neoplasms") [MeSH]

Tip

Remember to make use of truncation and wildcard symbols, where available.; e.g.: malignan*, tumour*, tumor*, tumo?r

Text mining is a modern search technique which can help you to identify search terms by using software to analyse chunks of text for recurrent words or phrases.  

Some free online text mining tools include:

• PubReMiner - type in keywords and a table of authors. Journal titles and MeSH terms will be displayed.
• Yale MeSH Analyzer - type in PMIDs (PubMed IDs) and major topic terms will be displayed.
• TerMine is a tool that allows for blocks of text to be mined for keywords.
• Voyant Tools - this is a set of tools that allows you to mine text for keywords or do text analysis. There are word and visual tools. You can paste in a block of text or add some URLs or upload documents or RIS files.

At the top of the results list in Embase is a new tool called Index miner. This will display a list of Index (subject, controlled vocabulary) terms harvested from the results.

Further reading

Paynter R, Bañez LL, Berliner E, et al. EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews. Agency for Healthcare Research and Quality (US), Rockville (MD); 2016. PMID: 27195359.

When planning and revising your search it is important to record the search terms you intend to use as well as those you have discarded. Keep track of all your search terms and search strategies that you have used for each database as this will need to be included as supplemental material to accompany your manuscript.

A documented search plan (a search planner) facilitates this and should be used from the outset, as a plan for your search that is edited during the search planning process, progressively refining the search strategy up to the final stage.

Choose at least two to three databases to search for your review to limit bias. There is a degree of overlap between databases, but no database includes all journal articles.  You will need to select databases which cover much of the relevant literature.

Most reviews in Health begin with a search in Medline and Embase and, depending on the topic, Cochrane Central Register of Controlled Trials (CENTRAL).

You should also consider at least one discipline specific database:

• PsycInfo (produced by the American Psychological Association) is the main one for psychology or any behaviour topics
• CINAHL is the core nursing database and includes publications from the National League for Nursing and the American Nurses Association
• Global Health (produced by the Centre for Agriculture and Bioscience International) is a public health database that also provides information on health promotion, environmental & occupational health, nutrition, food safety, epidemiology and chronic diseases
• SportDiscus (produced by the Sport Information Resource Centre in Canada) covers sport, exercise sciences, physical education, recreation, exercise physiology, sports medicine, kinesiology and physical therapy
• SocIndex covers all areas of sociology including  gender studies, marriage & family, social work, substance abuse & other addictions, violence and many others
• Informit databases include Australian content.
• Web of Science and Scopus are multidisciplinary databases with citation searching functions, and make a good addition to a comprehensive search strategy.
   

Note

Some database providers will allow multiple databases to be searched at once; however, for transparency, as well as to make use of individual database features and functionality, we recommended searching only one database at a time.

Further Readings

• The Cochrane Handbook provides guidelines for selecting databases.

• Cochrane - Reporting biases to learn about different types of reporting biases.

• Bramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H. (2017). Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Systematic reviews, 6(1), 1-12.

• Hunter, K. E., Webster, A. C., Page, M. J., Willson, M., McDonald, S., Berber, S., ... & Seidler, A. L. (2022). Searching clinical trials registers: guide for systematic reviewers. bmj, 377.

Tip

Sign up or register with each database you use for your search (it’s free!) so you can save your search history, create alerts, and re-run a search .

You will need to include terms from the database-controlled vocabulary when developing a comprehensive search. Controlled vocabulary is also sometimes called index terms, subject headings or thesarus. The controlled vocabulary is a consistent list of database specific subject headings used to label records and is used in combination with keywords in a comprehensive search.

Searching with a controlled vocabulary is more powerful because you are using the database's indexing. This can give your search a greater balance of breadth and precision.
 

Note

Not all databases have subject headings available. Also, subject headings vary from one database to the next in how they describe a concept. Examples include MeSH for Medline, EMTREE for Embase and CINAHL Headings for CINAHL. Look for the Thesaurus when using PsycINFO.

Be as specific as you can - avoid using terms that are too generic as these can increase the risk of irrelevant search results. Where available, Scope notes can assist with determining the most specific term to match the context needed for your search.

Information in database records is organised into sections called 'fields', such as the title, author, abstract, and so on.

Generally, the default setting will be to search all fields at the same time (it might appear as Default "Select a field (optional)"). This means your terms can be found anywhere in an article record. However, this can mean many irrelevant results.

Some databases allow searching in specific fields. Limiting your search to specific fields is a more targeted approach and will retrieve fewer results than a keyword, search in any field.
 

Tip

Restricting your keyword search to the title and abstract fields, as well as subject headings, where possible, is recommended. If your search terms appear in these fields, the reference is likely to be more relevant.

Databases and other searchable resources often offer the option to add specific limits to a search, either pre- or post-search. Examples of search limit options include demographic details (e.g., age, sex) and year of publication.  

Some databases offer the option to limit search results by language of publication, as English is the predominant language in scientific publication. Narrowing results in this way may exclude relevant articles, although the absence of a translation (if required) may make this a pragmatic decision.

Search filters, also known as hedges, are tested and (theoretically) transparent ‘canned searches’ that allow for the searcher to reliably narrow their results to a certain topic or study type. The filters have been tested to account for certain levels of specificity (relevance) and sensitivity (breadth). Many filter creators publish papers explaining their process and outcome in developing the filter.

Filters are more sophisticated than database limits and balanced for optimum search sensitivity.

For more detailed information and for sources of filters, see the Filters and complex search examples page.

Reviewing and refining your search to improve the search results is an important part of developing the best search possible. It is recommended that you test your search terms, including both subject headings and keywords/phrases, to determine if they return useful results. Identify any terms that are retrieving irrelevant results – this may include acronyms, terms truncated or truncated too early.

Tip

It is best practice to search one concept at a time and combine concepts using the search history.

 

We recommend reading the PRESS Checklist from the Canadian Agency for Drugs and Technologies in Health (CADTH) which will guide you through a check for typos and errors in syntax and logic, appropriate use of subject headings, and relevant filters.

Consider documenting the reasons for adjustments to the search for the review team's benefit.
 

Translating the search to other databases  

An example of the same search for cancer in two different databases can be seen here. In the first example the search is in Embase:

cancer:ab,ti OR neoplasms:ab,ti OR 'malignant neoplasm'/exp

The same search in Ovid Medline would look like this:  

exp neoplasms/ or (cancer or neoplasms).ti,ab.

Recording or documenting the search strategy in enough detail enables appropriate reporting of the search methodology for publication, and for verification. The full search strategies for each database will need to be included in an appendix or supplementary file of the review.

Options for documentation include saving the search history from the database as a PDF file; creating a spreadsheet to record the details of your searches or typing the search line by line in a word processing document. You can also save your search histories by signing up for an account in each database (where this is an option).

For each database searched, it is important to record:

• the date of the search
• the database
• the platform/database provider (e.g., OVID or EBSCO)  
• time period/years covered by search
• the keywords used and how these were combined in the search
• any filters used
• any database limits applied, such as language, etc., and
• the number of studies identified.

Details of other resources searched, including study registries, citation searching, and other methods also need to be detailed. Deduplication processes and software used is also important to outline.

Let's return to our scenario.  This is an example of a search which has been developed for Medline.  It has been designed for comprehensiveness, including both keywords (including synonyms) and controlled vocabulary terms. The keyword terms have been restricted to the abstract and title fields.

Date: 11 January 2022

Database, platform: MEDLINE Complete via Ebsco

Timeframe: 1950- January 2022

EndNote can be used to collect the results of the final search and the Groups function can be used for inclusion / exclusion criteria. More information about using EndNote is available from the EndNote library guide.

Tip

It can be useful to consult other advanced reviews related to at least one of the concepts of your search, to see how the search strategy has been developed. As the quality of searches can vary, it would be beneficial to examine more than one review.

Saving the search strategies enables the search to be rerun at a later point in time and can be useful when updating the systematic review.

Setting an alert on the search strategy allows you to be kept up to date with newly published articles received via email and the articles may also be added to the review.  

The Saving searches handout provides information on saving searches and setting up alerts in many different databases / platforms.

If you are updating a previously published review, or if the initial searches were conducted some time ago, these will need to be updated to capture any newly indexed content. This means rerunning your final database searches closer to your review being ready for publication.  

To avoid screening articles a second time, one method is to restrict records to those added to the database after the search was last run.  

McGill University has a detailed page on using field codes in several databases to update searches.
 

Note

This option is not available as a search field in all databases.
 

Database features such as limits available, or subject headings may have been modified since the initial search. For example, MeSH is updated annually, with new terms added. Also, some terms are changed, and older records are not re-indexed with new headings. You will need to check for changes and modify search as appropriate.  

An alternative method of identifying new material is to use Endnote to create de-duplicated sets of records for each database. However, this is more complex, and some duplicates may not be detected if the records have been revised.  There is some guidance on this on the McGill Updating the database searches page.  
 

Tip

If screening has not yet occurred, you can simply re-run your searches in your chosen databases.

Further Readings

• Bramer, W., & Bain, P. (2017). Updating search strategies for systematic reviews using EndNote. Journal of the Medical Library Association: JMLA, 105(3), 285.
• Bramer, W. M., Giustini, D., de Jonge, G. B., Holland, L., & Bekhuis, T. (2016). De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association: JMLA, 104(3), 240.  
• Cumpston M, Chandler J. Chapter IV: Updating a review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Available from www.training.cochrane.org/handbook 
• Cochrane Handbook - Maintaining the currency of systematic reviews, and Signals for updating
• Cochrane, Reporting standards specific to updates
• Garner, P., Hopewell, S., Chandler, J., MacLehose, H., Akl, E. A., Beyene, J., ... & Schünemann, H. J. (2016). When and how to update systematic reviews: consensus and checklist. bmj, 354.

Depending on the extent of your review, you may need to consider other sources of research beyond traditional scholarly journals and databases.

Further reading

Boulos, L., Ogilvie, R., & Hayden, J. A. (2021). Search methods for prognostic factor systematic reviews: a methodologic investigation. Journal of the Medical Library Association: JMLA, 109(1), 23.

Grey literature

While publication in a scholarly journal is a traditional method of scholarly communication, work published in other avenues such as policy papers, theses and dissertations, or conference proceedings, professional association guidelines, and pharmaceutical or other corporate reports, may still be relevant to your research question.

These sources are termed grey literature and are produced by organizations, governments and industry, rather than through commercial publishing. Systematic reviews are also produced by government, NGOs, and academic institutions and can be located on public websites.   

The guide Grey literature for Health, offers more information on this topic, including details on finding and evaluating grey literature.

Grey literature is particularly important to consider for systematic reviews, but also if your topic covers public health, health economics or policy, guidelines or standards. The inclusion of grey literature as part of an advanced review can prevent publication bias.

Further reading

• Bonato, S. Searching the grey literature: a handbook for searching reports, working papers, and other unpublished research. Lanham, Maryland; Rowman & Littlefield Publishers (2018).
• Hartling, L., Featherstone, R., Nuspl, M. et al. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol 17, 64 (2017).
   

Handsearching

Handsearching is a search technique which attempts to overcome the limitations of database content and indexing.

According to the Cochrane Handbook, handsearching “involves a manual page-by-page examination of the entire contents of a journal issue or conference proceedings to identify all eligible reports of trials.” Many clinical trials are unpublished, and when included in bibliographic databases they may not contain relevant search terms in the titles or abstracts or be indexed with terms that allow them to be easily identified as trials.  

There are three main strategies to use for hand searching. The first is to review the reference list and citing articles of studies included in the review. Another common method to ensure no relevant articles were missed is to manually browse key journals in your field. Clinical trial registries may also be considered for hand searching.

The guide Grey literature for Health, offers more information on this topic, including details on finding unpublished reports and reviews and finding clinical trials.

Research from the Cochrane Methodology Review Group indicates that combining handsearching with standard electronic searching is necessary for complete identification of relevant reports, however, this is not yet a routine expectation and should be considered in conjunction with available resources, expertise, and the nature of the literature in the field.
 

Contacting authors

In instances of incomplete reporting, it may be appropriate to contact the study’s authors directly to seek additional information. However, the Cochrane handbook warns that this “may lead to overly positive answers”, possibly introducing bias to the review.

To reduce the risk of this bias, the Cochrane handbook states: “… review authors should use open-ended questions when asking trial authors for information about study design and conduct.”

Citation searching

Citation searching is sometimes referred to as citation tracking or snowball searching.   

Forward citation searching involves examining relevant articles to find more recent articles that have cited the original articles of interest.  

Backward citation searching instead looks at the reference lists of relevant articles to find other relevant materials.   

Useful databases to do citation searching are Scopus and Web of Science. This search method can also be used in the preliminary stages to help determine that search results contain these records.

Similarly, examining reference lists of related existing systematic reviews and meta-analyses, as well as other studies identified during the search, including excluded studies, can be invaluable to locate other relevant papers.