In 2018 we completed two very different reviews. The first was a 6 week long (or short!) rapid review of the ‘Nearest Relative’ provisions in the UK Mental Health Act. This review was based on qualitative evidence such as interview and focus group studies. The second review was a year-long systematic review on the effectiveness and cost-effectiveness of interventions to reduce length of hospital stay for older people undergoing planned procedures. This review incorporated a wide range of quantitative evidence including randomised controlled trials (RCTs), non-randomised controlled trials, interrupted time-series studies, and controlled and uncontrolled before and after studies (CBAs and UBAs respectively), as well as cost-effectiveness studies.
Although these two reviews, and the type of evidence included in them, were very different, there was an interesting similarity between the two from the point of view of searching for studies. This was the important contribution that backward and forward citation searching made to the identification of eligible studies for each review. Backward and forward citation searching are two similar ‘supplementary’ search methods for identifying studies. Backward citation searching involves inspecting the reference lists of included and/or related studies for additional includable studies. Forward citation searching involves using a citation index – a type of database that indexes citations of studies – to identify studies that cite a source study, i.e. if you search for a study in a citation index, the citation index will produce a list of all the studies that have cited that study.
The main search method in a systematic review typically consists of searching bibliographic databases, which index and make easily accessible large amounts of studies. Citation searching is used to identify studies that are missed by bibliographic databases, particularly ‘hard-to-find’ study types such as qualitative studies. Unlike with RCTs, there is limited standardisation of how qualitative studies are described and labelled in titles and abstracts, and in database indexing terms. For example, qualitative study abstracts do not always include a methods section, which can mean that key methodological terms are missing. There may also be inconsistencies in how studies are indexed in different databases. For example, CINAHL offers a wide range of qualitative indexing terms, whereas MEDLINE offers comparatively very little (Evans 2002). Because bibliographic databases use keywords and indexing terms to retrieve studies, these issues can make it hard to retrieve a comprehensive set of relevant studies using bibliographic databases alone. Hence citation searching, which works on the assumption that studies that cite or are cited by a source study are likely to have similar content, can be particularly useful.
We encountered some of these issues when searching for studies for our rapid review of the Mental Health Act. We spent as long as we dared (in view of the exceedingly short time-scale for the review) developing the bibliographic database search strategy, testing and refining our search terms to maximise the recall (i.e. retrieval of known relevant studies) and precision (i.e. limiting the retrieval of irrelevant studies) of the search. In total we identified around 1700 articles via the bibliographic database searches and a further 126 articles via supplementary searches including backward and forward citation searching. Following the screening process, 35 articles were included in the review, including 22 identified from bibliographic databases and 13 from supplementary search methods – 8 of which from backward and forward citation searching.
The second review on interventions to reduce length of hospital stay for older people included RCTs. RCTs have probably the most standardised title and abstract format of all study types thanks to initiatives such as the CONSORT statement (Moher 1998). However, it also included a range of observational study types including UBAs and CBAs. For similar reasons to qualitative studies, observational studies can be hard to identify via bibliographic databases. We tried our best, however, to test and refine a suitable set of search terms in order to be able to screen the results efficiently with the time and resources available. This involved running test searches and testing whether the studies retrieved included pre-identified studies of interest. Although attempting to identify observational study types using keyword terms is not always recommended, the size of the search without such terms (>20,000 results in MEDLINE alone) necessitated some form of limit. We were helped enormously by the information specialist at the EPOC Cochrane Centre who shared their unpublished search filter for identifying observational studies, which we adapted for our review.
This review was much bigger than the Mental Health Act rapid review – we had more time to cast a wider net and there was more published evidence. Our bibliographic database searches identified around 8,000 unique records and our supplementary searches, including backward and forward citation searching, identified a further 2000 records. Following the screening process, we identified around 80 relevant papers from the bibliographic database search results, and around 30 and 60 relevant records respectively from backward citation searching (including checking the references of relevant systematic reviews) and forward citation searching.
The way forwards…
The large yield from backward and forward citation searching in both reviews shows the value of these search methods for retrieving studies where a text-based search approach is difficult to achieve. Can we leave it there? Probably not. It is possible to be too reliant on citation searching. As noted in the Cochrane Handbook, citation searching is not an ‘objective’ search method; citations are prone to biases, such as the documented phenomenon of studies with positive results receiving more citations than studies with negative results (Lefebvre et al. 2011). In addition, because citation searching happens at a later stage of the review, it can be problematic to find the time and resources to screen and include a large number of additional includes. As such, although we’re confident that our extensive searches in both reviews identified a wide and potentially comprehensive set of studies (it is almost impossible to know whether a set of studies is comprehensive) we will re-visit and re-think how we identify certain kinds of study for future projects – in particular, observational studies such as CBAs and UBAs which made up a large proportion of the included studies in the review of interventions to reduce hospital length of stay.
Two options are on the table: first, whether the study type filters we used can be improved; and secondly, whether we should use one at all. Problematically, the second option was prohibitive in our review of interventions to reduce length of stay for older people, as the number of records retrieved without a filter would have been in the tens of thousands requiring excessive screening time. However, the first option of improving the study type filter is something we can explore. In particular, we plan to inspect the titles, abstracts and indexing terms of the studies that were identified via backward and forward citation searching to see if there are additional terms we could use in future to improve recall via bibliographic databases.
Watch this space for a future blog post on what we find out…
Evans D. Database searches for qualitative research. Journal of the Medical Library Association. 2002;90(3):290–293.
Lefebvre C, Manheimer E, Glanville J. Searching for studies. In: Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration; 2011.
Moher D. CONSORT: an evolving tool to help improve the quality of reports of randomized controlled trials. Consolidated Standards of Reporting Trials. Journal of the American Medical Association. 1998;279:1489-91.