Troubleshooting and testing
Systematic searching is an iterative process. After building your primary search, it's important to;
- Review your search results - does the first page look relevant to your question?
- Troubleshoot any problems and refine your search.
- Test your search by re-running and testing the results against your gold set.
A systematic search will be refined but never perfect.
Troubleshooting problems with your search
Once you have created your first line by line search strategy, you will need to refine the search by experimenting with individual keywords, search techniques and subject headings, to review the results they produce. This helps ensure that you are retrieving relevant results and it minimises the likelihood of missing important literature.
Click on the plus (+) icons to explore tips for solving common search problems.
Testing your search
Testing your search strategy ensures that you are effectively capturing all relevant literature. A recommended approach is checking if your search strategy accurately finds your gold set articles. By doing this, you can be confident that your search strategy is effective, comprehensive and precise. Additionally, it provides an opportunity to fine-tune your search if any missed articles are identified. Overall, this process helps you further refine your search for better results.
How do we use the gold set to test the search?
We test searches by comparing whether our gold set appears in our search results. As these articles are relevant and representative of the terminology and vocabulary used in the topic area, we would hope to see these in the search results.
To do this, we:
- Check which database your gold set articles are included in.
- Create a new search concept which lists all gold set articles included in that database.
- Use the NOT Boolean operator to filter out all gold set articles which are not picked up by the search.
The below table is an example of how this would look in a search strategy using Pubmed article ID's (you can also use article titles):
| Search (MEDLINE via EBSCO) | Result Number | Explanation |
|---|---|---|
| S21 S20 NOT S19 | 2 | This search tells the database: 'show me my Gold Set' (S20) NOT in 'the search' (S19). So there are 2 Gold Set articles not in the search. |
| S20 AN (31846046 OR 22503075 OR 9022232 OR 33076230 OR 21479509) | 5 | This line includes all Gold Set articles. In this case, we have used PMID (PubMed Identifiers) to search for 5 Gold Set articles within MEDLINE). You can also use the article titles. |
| S19 S18 AND (Limiters) | 509 | This is the final line of the search with any relevant limiters |
| S18 S8 AND S13 AND S17 | 736 | This is the final line of the search |
Once we know which articles are NOT appearing in the search, we can refer back to the search strategy and gold set analysis table to see why the articles are not being returned in the results, (e.g. Are the articles indexed in the database? Are terms missing?).
When is my search ready to run and translate?
Creating an effective search is iterative and all about balance. Your search is ready when:
- All/most of your Gold Set are returned in your results (your search can be adapted to pick up outliers if needed)
- Your results are mostly relevant, with numbers reasonable for screening
- You have proof-read your search, to ensure there are no spelling or syntax errors
- You have reached your time and resource budget for search construction
While building your search, you will become an expert in the topic's literature and recognise relevant papers as you test, modify, and refine your search. If these papers are absent, you may question why or what has changed in your search strategy to cause this.
Remember, after building, testing and refining your search, you will best positioned to determine when to finalise your search and move on to translating and screening.