Troubleshooting and testing
Systematic searching is an iterative process. After building your primary search, it's important to;
- Review your search results - does the first page look relevant to your question?
- Troubleshoot any problems and refine your search.
- Test your search by re-running and testing the results against your gold set.
A systematic search will be refined but never perfect.
Troubleshooting problems with your search
Once you have created your first line by line search strategy, you will need to refine the search by experimenting with individual keywords, search techniques and subject headings, to review the results they produce. This helps ensure that you are retrieving relevant results and it minimises the likelihood of missing important literature.
Click on the plus (+) icons to explore tips for solving common search problems.
Check for errors
- Check for spelling mistakes, missing terms, and missing subject headings.
- Check your Boolean (AND, OR), truncation (are there words that can be truncated), and phrase searching (have you used the required syntax).
- Check your proximity search, are there brackets around multiple terms? For example, (view* OR watch*) N2 (TV OR television).
- Do you have optimum proximity between terms? For example, adjust the number next to the proximity operator to test whether you could get better results with closer or further proximity between keywords.
Too many results
- Double check that you’ve used your search techniques (*, “”) and operators (AND, OR) correctly.
- Try narrowing your broader terms with phrase searching or proximity operators.
- Ensure that you are searching for keywords in only the title and abstract fields.
- Consider refining a broad concept or adding a new concept, if relevant, to focus your search.
- Consider exploded subject headings, could they be adjusted?
Too few results
- Try searching fewer concepts, to make the search broader in scope.
- Brainstorm more keywords for your concepts – the more options you give the database, the more results it can find.
- Consider how search techniques, like truncation or wildcards, can expand your search.
- Double check that you have used your search techniques (*, “”) and operators (AND, OR) correctly, and that there are no spelling errors.
Finding irrelevant results
- Replace ambiguous terms with more specific terms – phrase searching can be helpful for this.
- Consider whether your truncated words are returning unexpected results. For example, using polic* would retrieve results with the words policy and policies, but it would also find results with police and policing.
- Consider exploded subject headings, could they be adjusted to better match your topic?
- Review relevant articles found through citation searching (see video under Citation Databases) – take note of the subject headings and terminology used in the abstract to find more relevant terms.
- Review irrelevant results – determine which keywords or subject headings are bringing in unrelated results and edit or remove them from your search.
Check your concepts
- Do you need every concept?
If you searched fewer concepts would you still get relevant results?
Could some of your concepts be too obvious or redundant because they're inherent to another concept?
- Can some of your concepts be joined?
You might have two concepts (physical activity AND diet) and you want either one or both of them. Join the two concepts with OR (not AND).
- Can some of your concepts be separated?
Might your concept include two different things - an intervention and population. Split them into two different concepts.
Testing your search
Testing your search strategy ensures that you are effectively capturing all relevant literature. A recommended approach is checking if your search strategy accurately finds your gold set articles. By doing this, you can be confident that your search strategy is effective, comprehensive and precise. Additionally, it provides an opportunity to fine-tune your search if any missed articles are identified. Overall, this process helps you further refine your search for better results.
How do we use the gold set to test the search?
We test searches by comparing whether our gold set appears in our search results. As these articles are relevant and representative of the terminology and vocabulary used in the topic area, we would hope to see these in the search results.
To do this, we:
- Check which database your gold set articles are included in.
- Create a new search concept which lists all gold set articles included in that database.
- Use the NOT Boolean operator to filter out all gold set articles which are not picked up by the search.
The below table is an example of how this would look in a search strategy using Pubmed article ID's (you can also use article titles):
| Search (MEDLINE via EBSCO) | Result Number | Explanation |
|---|---|---|
| S21 S20 NOT S19 | 2 | This search tells the database: 'show me my Gold Set' (S20) NOT in 'the search' (S19). So there are 2 Gold Set articles not in the search. |
| S20 AN (31846046 OR 22503075 OR 9022232 OR 33076230 OR 21479509) | 5 | This line includes all Gold Set articles. In this case, we have used PMID (PubMed Identifiers) to search for 5 Gold Set articles within MEDLINE). You can also use the article titles. |
| S19 S18 AND (Limiters) | 509 | This is the final line of the search with any relevant limiters |
| S18 S8 AND S13 AND S17 | 736 | This is the final line of the search |
Once we know which articles are NOT appearing in the search, we can refer back to the search strategy and gold set analysis table to see why the articles are not being returned in the results, (e.g. Are the articles indexed in the database? Are terms missing?).
Step by step instructions: Testing your search with gold set articles
- Retrieve your primary search strategy in your first database.
- Search for each article title as a phrase on individual search lines. This will check if each article is available in your database.
NOTE: If your search does not find any results in an EBSCOHost database, it may run an additional SmartText Search line. Delete these search lines as they are irrelevant. We will not include article S79 in the search as it is not an article in this database.
- Combine the lines of each of your available gold set articles with OR.
Note the new search line in the image below, this is now your gold set line (S80).
- Search for the gold set article line NOT final search results, e.g. S80 NOT S75. This will show you the gold set articles which do not appear in your search.
What to do if an article is not appearing in the results
In our example, we can see that one of our gold set articles did not appear in our final search results. This article is available in this database, so we now need to review the title and abstract for your keywords to determine why it does not appear.
Looking at the title and abstract, we have identified at least one keyword for our ‘Physical Activity’ concept, and one for our ‘Adolescent’ concept. However, there are no keywords for our ‘Australia’ concept. This is why this article does not appear in our final search results. So this isn't a gold set article it isn't Australian.
When is my search ready to run and translate?
Creating an effective search is iterative and all about balance. Your search is ready when:
- All/most of your Gold Set are returned in your results (your search can be adapted to pick up outliers if needed)
- Your results are mostly relevant, with numbers reasonable for screening
- You have proof-read your search, to ensure there are no spelling or syntax errors
- You have reached your time and resource budget for search construction
While building your search, you will become an expert in the topic's literature and recognise relevant papers as you test, modify, and refine your search. If these papers are absent, you may question why or what has changed in your search strategy to cause this.
Remember, after building, testing and refining your search, you will best positioned to determine when to finalise your search and move on to translating and screening.