Some reflections on the use of citation searching in our first two reviews

In 2018 we completed two very different reviews.  The first was a 6 week long (or short!) rapid review of the ‘Nearest Relative’ provisions in the UK Mental Health Act. This review was based on qualitative evidence such as interview and focus group studies. The second review was a year-long systematic review on the effectiveness and cost-effectiveness of interventions to reduce length of hospital stay for older people undergoing planned procedures. This review incorporated a wide range of quantitative evidence including randomised controlled trials (RCTs), non-randomised controlled trials, interrupted time-series studies, and controlled and uncontrolled before and after studies (CBAs and UBAs respectively), as well as cost-effectiveness studies.

Although these two reviews, and the type of evidence included in them, were very different, there was an interesting similarity between the two from the point of view of searching for studies. This was the important contribution that backward and forward citation searching made to the identification of eligible studies for each review. Backward and forward citation searching are two similar ‘supplementary’ search methods for identifying studies. Backward citation searching involves inspecting the reference lists of included and/or related studies for additional includable studies. Forward citation searching involves using a citation index – a type of database that indexes citations of studies – to identify studies that cite a source study, i.e. if you search for a study in a citation index, the citation index will produce a list of all the studies that have cited that study.


The main search method in a systematic review typically consists of searching bibliographic databases, which index and make easily accessible large amounts of studies.  Citation searching is used to identify studies that are missed by bibliographic databases, particularly ‘hard-to-find’ study types such as qualitative studies. Unlike with RCTs, there is limited standardisation of how qualitative studies are described and labelled in titles and abstracts, and in database indexing terms. For example, qualitative study abstracts do not always include a methods section, which can mean that key methodological terms are missing. There may also be inconsistencies in how studies are indexed in different databases. For example, CINAHL offers a wide range of qualitative indexing terms, whereas MEDLINE offers comparatively very little (Evans 2002). Because bibliographic databases use keywords and indexing terms to retrieve studies, these issues can make it hard to retrieve a comprehensive set of relevant studies using bibliographic databases alone. Hence citation searching, which works on the assumption that studies that cite or are cited by a source study are likely to have similar content, can be particularly useful.

We encountered some of these issues when searching for studies for our rapid review of the Mental Health Act. We spent as long as we dared (in view of the exceedingly short time-scale for the review) developing the bibliographic database search strategy, testing and refining our search terms to maximise the recall (i.e. retrieval of known relevant studies) and precision (i.e. limiting the retrieval of irrelevant studies) of the search. In total we identified around 1700 articles via the bibliographic database searches and a further 126 articles via supplementary searches including backward and forward citation searching. Following the screening process, 35 articles were included in the review, including 22 identified from bibliographic databases and 13 from supplementary search methods – 8 of which from backward and forward citation searching.

The second review on interventions to reduce length of hospital stay for older people included RCTs. RCTs have probably the most standardised title and abstract format of all study types thanks to initiatives such as the CONSORT statement (Moher 1998). However, it also included a range of observational study types including UBAs and CBAs. For similar reasons to qualitative studies, observational studies can be hard to identify via bibliographic databases. We tried our best, however, to test and refine a suitable set of search terms in order to be able to screen the results efficiently with the time and resources available. This involved running test searches and testing whether the studies retrieved included pre-identified studies of interest. Although attempting to identify observational study types using keyword terms is not always recommended, the size of the search without such terms (>20,000 results in MEDLINE alone) necessitated some form of limit. We were helped enormously by the information specialist at the EPOC Cochrane Centre who shared their unpublished search filter for identifying observational studies, which we adapted for our review.

This review was much bigger than the Mental Health Act rapid review – we had more time to cast a wider net and there was more published evidence. Our bibliographic database searches identified around 8,000 unique records and our supplementary searches, including backward and forward citation searching, identified a further 2000 records. Following the screening process, we identified around 80 relevant papers from the bibliographic database search results, and around 30 and 60 relevant records respectively from backward citation searching (including checking the references of relevant systematic reviews) and forward citation searching.

The way forwards…

The large yield from backward and forward citation searching in both reviews shows the value of these search methods for retrieving studies where a text-based search approach is difficult to achieve. Can we leave it there? Probably not. It is possible to be too reliant on citation searching. As noted in the Cochrane Handbook, citation searching is not an ‘objective’ search method; citations are prone to biases, such as the documented phenomenon of studies with positive results receiving more citations than studies with negative results (Lefebvre et al. 2011). In addition, because citation searching happens at a later stage of the review, it can be problematic to find the time and resources to screen and include a large number of additional includes. As such, although we’re confident that our extensive searches in both reviews identified a wide and potentially comprehensive set of studies (it is almost impossible to know whether a set of studies is comprehensive) we will re-visit and re-think how we identify certain kinds of study for future projects – in particular, observational studies such as CBAs and UBAs which made up a large proportion of the included studies in the review of interventions to reduce hospital length of stay.


Two options are on the table: first, whether the study type filters we used can be improved; and secondly, whether we should use one at all. Problematically, the second option was prohibitive in our review of interventions to reduce length of stay for older people, as the number of records retrieved without a filter would have been in the tens of thousands requiring excessive screening time. However, the first option of improving the study type filter is something we can explore. In particular, we plan to inspect the titles, abstracts and indexing terms of the studies that were identified via backward and forward citation searching to see if there are additional terms we could use in future to improve recall via bibliographic databases.

Watch this space for a future blog post on what we find out…


Evans D. Database searches for qualitative research. Journal of the Medical Library Association. 2002;90(3):290–293.

Lefebvre C, Manheimer E, Glanville J. Searching for studies. In: Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration; 2011.

Moher D. CONSORT: an evolving tool to help improve the quality of reports of randomized controlled trials. Consolidated Standards of Reporting Trials. Journal of the American Medical Association. 1998;279:1489-91.

Posted in Uncategorized | Leave a comment

So, you want to make a podcast?

Podcasts have recently become a popular means of communicating information, whether it be for entertainment purposes such as comedy shows, or discussing more serious issues such as news items or current affairs. For some in the academic community, podcasts represent an opportunity for researchers climbing out of their ’ivory tower’ and engage with members of the public.

Recently, two members of the Exeter HS&DR Evidence Synthesis Centre have been involved in recording their own podcasts. Below, we talk with Michael Nunns and Liz Shaw about why they thought that a podcast was a good way of sharing their work with other people and how they found the recording process.

  1. So what attracted you to the idea of making a podcast?podcast 2.docx

MN:       I think I saw a discussion on Twitter where someone said it would be good to hear academics talk about their research. So I thought, why not? The idea was to have a no-frills breakdown of some research, described by the author. I just had a quick look on Google for some podcast creation software and picked an app that looked as though it might work!tech

LS:          It just seemed like a really fun and accessible way of introducing people who are not researchers to the work that we do. It also gave me the opportunity to push myself out of my comfort zone by experimenting with some basic recording and editing technology!

2. What was your podcast about?

LS:          The team were asked by the Department of Health and Social Care to carry out a review, looking at all the research which has been carried out on people’s experiences of the Nearest Relative provisions of the Mental Health Act 1983. This piece of work was used to inform the governments Independent Review of the Mental Health Act 1983, which was released on the 6th December 2018. My podcast focused on talking about why our review was important as well as discussing the main findings.

MN:       I recorded a podcast to talk about a recently published paper, which was about interventions to reduce anxiety felt by children and young people when they go through some potentially distressing treatments for cancer.

3. You both used the Anchor app (download via the Play store if you’re interested) to record your podcast. How did you find it?

MN:       It was the ideal for solution for getting started quickly and recording with the minimum of fuss. I think it’s a good bit of software, particularly on IOS, given that you have better functionality and a range of transitions on the app (I may have gone slightly overboard with these!).


LS:          The Anchor app is really straightforward to use on the whole. If I can figure it out, anyone can! It is easy to download and start recording. Once you’ve started, you can easily delete or move segments around until you are happy with the final product. I found it incredibly cringe worthy listening to my own voice for hours at a time! The perfectionist in me found it very difficult to “let go” of any mistakes I made whilst recording, which I think contributed towards the reason why the process took a much longer time for me compared to Michael. I kept having to stop and re-record segments, which was frustrating. However, I think this was a “me” issue. You become numb to listening to your own voice after a while and knowing what to say without sounding really stilted becomes a lot easier after a bit of practise.

4. Any things you didn’t like?

LS:          The app inconveniently stopped working for me the day I was trying to finish it off and refused to save any of the changes I made. I needed to switch to using their website, which thankfully worked. I’m still not sure whether this was user error though…..

I also found that editing recordings using the Anchor app, both on a mobile and on the website, was very clunky. I would have a section which I liked, but had made a mistake in the middle of it which I wanted to cut out, but this was rather fiddly using the Anchor app. Instead, Simon Briscoe (another member of the Exeter ESC team), downloaded the completed podcast into another software programme called Audacity. This looks a bit more complicated to use initially to me, but it was definitely easier to edit the podcast.

SB: I had a keen interest in sound engineering and music technology as a teenager, and I knew it would be easy to correct the mistakes Liz mentioned with the right software. Audacity is a free and open source audio editor, and once you know the basics it’s very easy to use!

MN:       Well, nobody likes the sound of their own voice, but I can’t really blame the software for that!

SB: The problem was mainly long gaps of silence between sections, and the musical interlude between sections was too loud.  In Audacity I was able to easily delete and change the volume of parts of the audio waveform as necessary. Once that was done I re-uploaded the audio to the Anchor webpage.

LS:          I think you were a lot more comfortable with the process than me, Michael. Simon’s tweaks to the podcast were very welcome!


5. Any tips for anyone considering making their own podcast?

LS:          I would say just go for it! Start talking until you get used to the process of recording and listening to yourself back. I found it really helpful to have a script to follow when I was talking about more complicated ideas to make sure I didn’t forget anything or use any language that I didn’t want to. There are still a few words and phrases which have slipped through the net, but I’m not going to tell you what they are! Having said that, I think you need to be careful not to make it sound too stilted…

MN:    Just get on and do it! In this scenario, the focus is not on audio quality or production values – that would become too much of a drag and a deterrent for what I hope will be a lively and ‘rough and ready’ series of episodes, designed to be an easily accessible insight into our research. So the onus really is on us to just crack on and not worrying too much about it!

I’m hardly a podcasting expert here, but I found it useful to note down key points and phrases I would say, and then just talk through it. Imagine you’re telling your friends about your work in the pub, and go from there, and if you need to re-record a segment, just do it and move on when happy.

If you would like to listen to Liz’s podcast which talks about the experiences of service users, carers and professionals of the Nearest Relative provisions of the Mental Health Act, you can listen here.

Alternatively, follow this link to hear Michael talk about his paper on reducing procedural anxiety in young people undergoing cancer treatments.

If you want to find out more about what it is like to record a podcast, or about any of our work you can follow us on Twitter at @ExEvidSC

We would like to take this opportunity to thank our followers and supporters for their interest in the blog during 2018 and wish you all a very Merry Christmas and a Happy New Year.

See you in 2019!

Merry Christmas

Disclaimer: The Exeter HS&DR Evidence Synthesis Team does not endorse any of the podcast creation software mentioned in this blogpost. This blog represents the views are the team members who received no financial recompense.

Posted in Jo, Liz, Michael, podcast, Rob, Simon, Uncategorized | Tagged , , , , | Leave a comment

Experiences of the ‘Nearest Relative’ provisions of the Mental Health Act (1983)


handsIf there are concerns that a person with severe mental health problems may harm themselves or another person, they could be admitted to hospital against their wishes under the Mental Health Act (1983).

When people are assessed or treated against their will in hospital, the Mental Health Act 1983 states that a relative or carer should be appointed as their ‘Nearest Relative’ (or ‘Named Person’ if you live in Scotland).

The Nearest Relative will receive important information and be involved in decisions about the person who is unwell. It is an important but controversial role, with some perceived flaws in the way Nearest Relatives are selected, the powers they have and the sensitive information they receive.

We want to know more…..

We wanted to understand more about the experiences of people involved with the Nearest Relative provisions of the Mental Health Act (1983). We were interested in the views of service users, carers, family members, Nearest Relatives and professionals.

What we did

We carried out a systematic review of the evidence. We aimed to bring together the findings of existing studies to find out what is already known about people’s views on the Nearest Relative provisions.

How we did it

We searched different online libraries and organisation websites for all the research library.docxpapers which have been written on this topic since 1998. We also contacted the authors of included studies and experts in the field and searched the reference lists of the papers we found, looking for any more papers of interest.

We then took out the information we needed from the relevant papers identified by our searches. This information included:

  • What people had said about their experiences of the Nearest Relative provision
  • Who was giving their views
  • How many people were giving their views
  • The country where the study took place

For more information on how we made sense of the information we found, you can read our last blog post or visit our project webpage.

 What we found


In this podcast, Dr Liz Shaw (Systematic Reviewer) talks about the main findings in this review.






Key findings

  • It can be better for service users to be actively involved in choosing their Nearest Relative.
  • However, this choice can have a significant impact on family and carer relationships. Professionals involved in the treatment of people with mental illness are in a position to be able to help service users and the Nearest Relative deal with the pressure and responsibilities they will face
  • Nearest Relative law was seen as being very complex and hard to understand by most people.

The main ideas which were identified can be seen in the image below, with the arrows showing how these ideas are related to one another.

MHA results

What  should happen next?

Much of the research included in this review was carried out before the 2007 update to the Mental Health Act 1983. This means that more up to date research is needed, in particular the following areas:

  • Examination of experiences of the Nearest Relative/Named Person role in light of the most recent legislative changes throughout the UK
  • Looking at the experiences of those who are not married or in a civil partnership
  • Exploring the views of mental health professionals and mental health advocates
  • Looking at the experiences of ethnic minorities.

Our review contributes towards the findings of the Independent Review of the Mental Health Act 1983.

If you would like to find out more about what we found, the full report for this project can be found here:

Contact us

We would really like to hear your views on the work we he have done. If you have any questions or comments, please let us know by commenting on this blog post

Alternatively, you can contact us using:

Twitter:                @ExEvidSC or

Email Liz:   

If you’d like to hear more about other work being carried out within our team, please visit our website.

Up next: Are you interested in making your own podcast? Find out how we did it in our next blog post!

Posted in evidence, Mental health, podcast, systematic review | Tagged , , , , , | Leave a comment

STAND BY, ACTION STATIONS! A two month rapid review

You may have noted our absence from the twitter-sphere and blogging zone during the early part of 2018. This is because, upon our immediate return to work following the Christmas break, we received a call to action. A rapid review, to contribute towards an interim report on the Mental Health Act 1983(MHA), was urgently needed. This meant we needed to temporarily ‘park’ our current review (examining organisational interventions to optimise the length of hospital stay for older people following elective treatment) and swiftly change direction….

“Exactly how ‘rapid’ is ‘rapid’?” we enquired.

“Two months please” was our answer.

This is a just a little bit of a shorter time-frame than we are normally used to – in our previous experience, systematic reviews usually take between 8 and 12 months to complete, with even rapid reviews typically taking around 3 months to complete.

We are proud to say that we managed to produce a rapid systematic review within eight weeks. Below we talk you through the process and reflect on some of the dilemmas and challenges we encountered.

T minus 8 weeks: A bit of light background reading…


The first two weeks were spent quickly getting ourselves up to speed with the MHA, the existing literature and working alongside the people who had commissioned this review to identify and refine our research question. The MHA is an old, extremely comprehensive piece of legislation which has gone through several iterations in its lifetime. It was a bit tricky getting our head around this complex legislation so quickly!

T minus 7 weeks: Preliminary searches and conversations

converesationWe were initially provided with quite a broad remit for this review, which meant there were several potentially useful and valid questions we could have tried to answer.  To help us identify our research question quickly, we needed to establish early contact with the people who wanted to commission the review.

We spoke with a representative from the Department of Health and Social Care, who helped us to understand exactly what they wanted us to investigate. This was a two-way conversation as, based upon our preliminary searches, we were also able to let them know what literature we thought was available in particular topic areas and what we felt would be possible to complete in the time available.

At the end of this process; we agreed that we would focus on the experiences of patients, carers and professionals of the Nearest Relative provisions of the Mental Health Act. We made sure that we had regular telephone calls with the people who commissioned our review to let them know our progress and ensure we both had opportunities to ask each other questions.

T minus 6 weeks: All systems go!

coffee cup stack

Once we had agreed our question, we were able to register our protocol and run our searches. One of the difficulties we encountered was how to maintain the methodological rigour of our review, whilst recognising that we needed to work quickly. It was a bit of a balancing act!

In the end we decided to:

  • Write, share and register a full protocol for the review on PROSPERO to maintain transparency
  • Use a full range of databases rather than restricting ourselves to just a small selection.
  • Keep the double screening of title/abstracts/full texts and quality appraisal by two independent reviewers
  • Retain our searching of the grey literature and pursuing articles through author contact. This was because some relevant papers we were aware of through our background reading were not picked up by our electronic searches due to not having an abstract. However we only included papers found through this additional searching if they were found before we completed data extraction.

In order to make sure that our review was as useful as it possibly could be to the people who were going to use it, we worked hard to identify and include only the most relevant evidence.  We therefore limited our searches:

  • To those published from 1998. This considered the impact the publication of The Human Rights Act (1998) had on the implementation of the Nearest Relative provisions
  • To English language publications in consideration of the UK focus of the review
  • To papers regarding people detained under sections 2 or 3 of the Civil pathway
  • To studies using questionnaire and interviews to gather data, which we hoped would provide us with rich qualitative data to synthesis

T minus 5 weeks: Still too many papers….

Despite our efforts, following full text screening, we still had too many eligible papers than it was possible to incorporate into a meaningful synthesis within the two weeks we had available. To overcome this; we conducted a form of purposive sampling to select the papers which had the highest quantity of data most relevant to our research objectives.

The papers with the most relevant data were prioritised for further synthesis, whilst the remaining studies were described in a table. This allowed us to focus our analysis on studies which contained the richest data, whilst ensuring we retained a transparent approach within the systematic review.

T minus 4 weeks: How do you carry out a qualitative synthesis in two weeks?!

post it notes

In a typical qualitative synthesis; we would usually spend more than the 8 weeks we had available for this whole review to complete the synthesis of results. This simply wasn’t possible. We had a strict two weeks allocated for this stage of the review!

Drawing upon inspiration from other systematic reviews conducted within a short time frame, we decided to use a method known as ‘framework synthesis’. This is a useful technique for making sense of qualitative evidence when time is short. Typically; a framework (like a grid in Microsoft Excel) with different categories will be developed, based upon concepts that people already know are important in the subject they are interested in. These concepts or categories could be drawn from bringing together background literature or using a model that has already been created.

rubix cubeProblem: We weren’t confident that our prior knowledge of the background literature would enable us to create a useful framework and we hadn’t found any existing models that we could use.

Solution: We decided to use our research objectives to form the basis of our framework. We then looked at the papers which contributed the most data across all our research objectives. We used the themes from these papers to revise our framework. Finally; we looked at the data within each category and conducted a brief inductive thematic synthesis to in order to create the final themes and subthemes.

T minus 2 weeks: Writing, reviewing and PPI

Our final two weeks of the review were spent finalising the synthesis, alongside writing and reviewing the final report. During this time, we also asked a carer with experience of the Nearest Relative provisions of the MHA to provide feedback on our synthesis whilst it was being conducted. It was challenging to identify people with relevant experiences who would be willing to discuss their experiences to inform the review within the time we had available. We were very fortunate that someone with relevant experience wanted to provide their input. Our discussions with them helped shape our synthesis and ensure that it reflected the experiences of a carer of the Nearest Relative provisions of the Mental Health Act.

Overall thoughts:

  • Negotiation with the funder regarding the scope of the review was essential to ensure that the review met their needs
  • Adherence to a strict timetable was absolutely necessary
  • Clear communication was vital; including delegation of tasks and responsibilities which allowed each member of the team to play to their strengths
  • Whilst it may be challenging to incorporate PPI involvement into rapid reviews; it is useful to draw upon existing resources and provides invaluable insight into the subject of the review
  • It is possible to maintain rigorous standards that underpin high quality systematic reviews, even within a short timeframe.

If you have any thoughts or questions, please do let us know! The final published report can also be read here.

Standby for our next blog post; where we will introduce you to the findings of this review via podcast!

Posted in Jo, Liz, Mental health, Michael, Rob, Simon, systematic review, Uncategorized | Tagged , , , , , | Leave a comment

We’re Not Lost, We’re Exploring…

Well here we are again, albeit a little sooner than anticipated, with a new project! Our previous review topic (effectiveness of different social work models) has been put to one side pending the publication of more primary research and we’re already sinking our teeth into something new.

The “something new” is organisational interventions aiming to reduce the length of hospital stay (LoS) for older adults undergoing elective surgery.  The focus on “elective”, or planned, surgery arose due to interest in supporting hospitals to undertake a proactive approach to managing known factors which may increase the duration of a patient’s stay in hospital.

Our starting point was to identify and understand the factors which may increase LoS and available hospital-based interventions used to manage these. We spoke to a local consultant geriatrician, who helped us to think about the typical journey an older person through the hospital system. We also discussed how the responsibility for ensuring that this potentially vulnerable patient group has an appropriate length of stay does not just lie with the hospital itself.  These early conversations helped us focus our search for a relevant research question.

In At the Deep End…


Diving deep into the existing literature

The next step was to look in the published literature to help us understand the different factors which may influence an older person’s LoS. We had a splash course in getting up to speed with some of the existing organisational interventions used to reduce length of hospital stay, such as Enhanced Recovery After Surgery programmes (ERP/ERAS) and Comprehensive Geriatric Assessments (CGA). As none of the team hold medical degrees, getting to grips with the names of some medical procedures provided an interesting challenge!

Our preliminary scoping searches revealed that this is an area which has generated a lot of interest, including several existing systematic reviews (e.g. Bagnall et al (2014), Conroy et al (2011)). This is perhaps unsurprising given the increased demand experienced by many health services by an older population. If the literature base for social work was too sparse to justify conducting a systematic review, here we appeared to have the opposite problem; identifying a clinically useful question with an associated body of research, which has not already been reviewed. We did not want to conjure up a research question based upon an identified gap in the literature, which was academically interesting but of no use to clinicians/service managers on the ground. Likewise, we needed to make sure that some of the clinical issues we had identified had not already been subject to a systematic review.

A Guiding Light: The Role of Expert Advice


Stakeholders: a guiding light

Discussions continued with our clinical colleagues – we enlisted the help of a physiotherapist and occupational therapist in addition to our friendly consultant geriatrician. This provided us with some knowledge of the organisational systems in NHS hospitals which aim to ensure patients receive the care appropriate to their needs with minimal potential harms which in turn aided us in focusing our research question. We will be continuing these conversations and expanding them to include patients, other clinicians and service managers as the review progresses to ensure that it remains relevant to those it is intended to benefit.

Circling the Question Generation Cycle


Circling our Question

With lots of background knowledge and reading (e.g. Miani et al, 2014; Paton et al, 2014) under our belts we began the “Idea for a question – explore existing research – back to question” cycle. Each iteration through the cycle resulted in the team becoming more immersed in the research. This was a strange period of uncertainty and doubt. Often it felt as though we were taking two steps forward, followed by one-step back, as our preliminary scoping resulted in us returning to older ideas that had been proposed earlier in the process. Some of our early question ideas were as follows:

  1. A systematic review of qualitative research to identify and explore the influencing factors necessary for successful implementation of an ERAS/ERP intervention.
  2. A conceptual mapping review to better define/produce a typology of interventions aiming to reduce length of stay.
  3. A systematic review identifying key components of an ERP intervention for different patient populations (people with dementia etc.).

Spreadsheets and mind-mapping were useful tools to help anchor our thoughts and keep track of progress. Whilst initially frustrating and a little overwhelming, over time our familiarity with the existing literature grew and we were able to refine our ideas.

And Our Question Is….

hand with shell

Whilst all the above questions may represent important topics to address through systematic review, we kept returning to the broad concept of wanting to know whether existing organisational interventions were effective for older adults within a hospital setting. We decided that our focus should continue to be broader than elective surgery and also encompass other elective treatments, such as chemotherapy.

Hence, it was decided that our research questions should be:

  1. What is the effectiveness of hospital-based interventions to reduce length of inpatient stay in hospitals for older adults following planned admission?
  2. What is the cost-effectiveness of hospital-based interventions to reduce length of inpatient stay in hospitals for older adults following planned admission.

Just Keep Swimming

Of course that wasn’t the end of the story, discussions continued to further refine and define the components of the question – some of the things we’ve been grappling with are:

  • Who is an “older” patient? What does being “old” mean, exactly? Is it your number of years on this planet, or is it the change in your health and the support you need as you age? This question has been approached in many different ways….
  • What is an elective treatment? Do we include people who were admitted to an Intensive Care Unit after undergoing an elective treatment? It’s hardly a treatment pathway someone decides they’d like, but there is potential to develop a planned care package in advance.
  • What is an organisational intervention? What is the definition of an organisational intervention? We want to know what it is that is being done, how it is being implemented and who it carrying it out. Some of the studies exploring the effectiveness of certain types of intervention (e.g. CGA) don’t tell us much about what the intervention consists of or how it is carried out.
  • How can we usefully define “length of stay”? Is it just the number of days in hospital after the first admission or should we also be interested in readmission data? How is that defined clinically and in research studies?
  • How can we judge whether interventions intended to reduce length of stay? Should LOS be the primary intended outcome in our included studies?  Should we include only those that stated this as their aim, or also those that said they aimed to ‘improve recovery’ or ‘accelerate rehabilitation’?  Or should we include evaluations of all eligible hospital-based interventions where LOS is an outcome measure?

We have made decisions about most of these uncertainties, based on stakeholder input, pilot screening and discussion among the team, but still have some more thinking to do. Stay tuned whilst we dig into the nitty gritty of defining our inclusion and exclusion criteria and (hopefully!) turn this grain of sand into a pretty pearl.

Posted in Chris, Hospital, Jo, Length of Stay, Liz, methods, Michael, Older Adults, Question Generation, Rob, Simon, systematic review | 2 Comments

Social Work Practice in the UK: a ‘pre-scoping’ exercise

At the Exeter HS&DR Evidence Synthesis Centre, we have just come to the end of what could be described as a ‘pre-scoping’ exercise on the topic of the effectiveness of different models of social work practice in the UK. I call it ‘pre-scoping’ because we haven’t performed a systematic scoping review, but we were tasked with getting on board with the state of play, in anticipation of the outcome of a James Lind Alliance Priority Setting Partnership exercise, to be completed in 2018.

The core review team of Michael, Liz and Simon set about the process of learning what they could about social work in the UK, feeding back this information, and reporting to HS&DR with a broad overview and some suggested review questions. As a team we had almost no prior knowledge of social work, and started from scratch. This is often the case with systematic reviewers who may be ‘guns for hire’ as methods specialists, often coming together with topic experts to help make sense of it all. As this was a new topic area for us, starting in a new team, on our first project, we thought we would blog about our experiences.



We called this a ‘pre-scoping’ exercise

We will usually receive a ‘topic brief’ when beginning new projects. This sets out the rationale for research, some key publications to read and so on. On this occasion, we had only a broad question and a name. That name was Lyn Romeo – Chief Social Worker for Adults in England, at the Department of Health. After a period of digging for publications and relevant policy, we had the chance to speak with Lyn for some more direction, but prior to that it was a case of hitting the internet for some of the basics … like what is social work? For a start, and what on earth is a ‘model’ of social work practice?

At first, we wondered quite what we were getting ourselves into. Our initial database searches revealed very little primary evidence, while searching for ‘social work practice’ and the like on Google Scholar returned, rather alarmingly, a lot of books. Yes, there is an abundance of theory about how to do social work, but thus far the randomised controlled trials of said practice were eluding us.

We decided to ‘go grey’ and party with Scholar’s older, cooler sibling, Google. Inevitably, this revealed an array of web pages, blogs, commentaries and internet resources, which started to shed some light on the topic and, of particular value, some of the big issues. One really useful resource was Lyn Romeo’s blog (duh!). This was a great way to get up to speed with hot topics in social work. Not only this, but in her role she has published annual reports on the status and progress of social work in the UK. Akin to this was the British Association of Social Workers website, which is home to the largest professional association for social workers in the UK, and a rich resource for learning about the role.


Going to the grey literature was a particularly helpful starting point

Other key websites for anyone looking include (Social Care Institute for Excellence), (The Institute for Research and Innovation in Social Services) and  (the NIHR School for Social Care Research). We even found the odd research paper – for example a directly relevant review of adult social work by Jo Moriarty and Jill Manthorpe from King’s College published in 2016. It was clear from reading this review, and our own digging, that we weren’t about to tap into a body of high quality evidence for the effectiveness of social work practice – despite seeing calls for evidence from various bodies and individuals for the last 20 years.

While we weren’t uncovering tons of evidence, we were putting together a rogue’s gallery of stories about social work, with some quite startling details emerging. Here are some of the eye-openers:

  • Being a social worker is tough! Most people probably appreciate that on some level, but did you realise the average career length of a social worker in Britain is around seven to eight years, compared with 12 years for nurses and 25 years for doctors? The main reason for this seems to be the high-levels of burnout caused by a combination of stress, time-pressures and limited resources. A 2015 Research in Practice Strategic Briefing gives a lot of the gory details about workload, recruitment and retention – it’s not a pretty picture
  • Social workers have unique skill sets and often undertake roles that cross a variety of disciplines. The now defunct College of Social Work describes the varied role of social workers, which can require expertise in legal issues, social skills, delivering physical and mental health therapy and advice, and often making highly important and impactful decisions in pressurised situations. The crucial and delicate nature of the role and the guiding principles of social work practice are elegantly illustrated in this quote:

“Social workers play a pivotal and often leading role in safeguarding people’s rights, building relationships to support and empower children, adults and families to make important choices about the direction of their lives” (The College of Social Work, 2015, p3)

Social workers need skills that make them personable but assertive, empathetic but decisive, and able to work under great pressure. An ethnographic exploration of the ‘Front Door System’ highlights the sheer workload faced by social workers receiving a steady stream of referrals about which they must make the sort of decisions that could have severe consequences if incorrect.

  • Government policy around social work seems to be ever-evolving. The Care Act 2014 emphasises a shift towards a person-centred or strengths-based approach, although we couldn’t find research evidence about whether such an approach improves outcomes. Education and regulation of social workers is also in a state of flux. Social work is a protected profession, currently regulated by the HCPC (Health Care Professions Council), which regulates a range of other professional bodies in the UK. However in April, The Children and Social Work Act was given Royal Assent, and one of the key implications of this Act is that regulation of social worker education will be the responsibility of a new body called Social Work England as of September 2018.

This is another change in a landscape which has been in something of a hiatus since The College of Social Work– a body that aimed to raise professional standards and present a worker-led voice for reform – was deemed not financially viable and was closed down in 2015 after just four years. The HCPC worked with the College of Social Work between 2012 and 2015 to raise standards in social work education and improve the dual accreditation/endorsement process. Since its closure in 2015, the responsibility for social worker endorsement has fallen to educators and employers, with uncertainty about the next steps.

Through our relatively informal synthesis of a range of reports, commentaries, blogs and research articles, we were struck by how challenging the role of social worker can be and disappointed by reports that social workers can struggle to fulfil their vital role within a difficult culture, which at times may lack organisational clarity and support. There seems to be a need to streamline processes, provide evidence to support difficult decisions, and allow time for reflection on practice. In one of her blog posts, Lyn Romeo highlights the need for a culture shift within social care practice, from the risk-averse system which promotes defensive practice to one which encourages learning from mistakes and supported practice.


A strengths-based approach aims to work closely with the individual

As reviewers, we struggled to (and anticipate further struggle) get to grips with such a broad area, and in particular the challenge of defining and identifying models of practice, such as identifying when a ‘strengths-based approach’ is being used. The Moriarty and Manthorpe report highlights such challenges, and additional considerations such as the measurement of ‘effectiveness’ of social work, or that many evaluations of health and social care interventions poorly describe the professional background of individuals providing or coordinated the care. However, this is our field of expertise, and we’re keen to take on this challenge (and tell you about it).

While evidence synthesis is needed to help inform policy makers and social workers alike, our ‘pre-scoping’ exercise reaffirmed the need for more and better primary research to take place first, particularly in the UK. The next period of research prioritisation, through the James Lind Alliance, is an important step in the right direction in an area crying out for better evidence, and we will be poised and ready to fulfil our role when the time is right!


Posted in methods, Michael, social work, systematic review | 7 Comments

From new posts to blog posts – Welcome!

Welcome to the blog of the team of the book of the film based on the true-life adventures of the Exeter HS&DR Evidence Synthesis Centre!  We are Rob, Jo, Chris, Michael, Liz, Simon and Sue, and over the next three years we will be producing rapid, responsive and relevant systematic reviews of evidence about the organisation and delivery of health care and other services.  We are based in the ESMI research group at the University of Exeter Medical School.

The juicy bit in the middle …

We are really excited to be embarking on this programme of work.  This is partly to do with the variety and challenge of doing service-relevant work, but it’s also because we are passionate about what can be achieved through health services research. We are looking forward to synthesising evidence about the middle ground between individual treatments and the design and funding of the health system at the national level. This is the juicy bit!

2017-05-19 20.17.13

Such work involves identifying and making sense of the evidence looking at how different models of care or service commissioning can influence the lives of the patients. It could encompass subjects such as; evaluating different appointment systems, how person-centred care works, and what pathways of referral are best.

So why blog?

First and foremost, we hope that blogging will enable us to reach out to a wide range of stakeholders to whom our work will be relevant.  These stakeholders will be the potential users of our evidence synthesis findings, including health service managers, care professionals, commissioners and health care leaders and policy-makers.

But, we also want to engage with researchers and others who grapple with the inevitable challenges of producing useful insights and conclusions from diverse forms of evidence, on complex topics.

So the second aim of our blog will be as a kind of ‘reflective research diary’, for all of us in the team to capture and share what we learn through delivering this programme of work.  Rather than well-formed conclusions or clear suggestions for improvement, these ‘shared learnings’ will more often be recurrent ambiguities, intriguing differences, irrepressible excitements, practical frustrations, unexpected realisations, and undiscovered connections.  The kinds of thing that hardly warrant a full journal article but seem important and interesting enough to capture in passing.

At this point – being both new to the programme of work, and (for most of us) new to blogging – we don’t really know what we will actually blog about.  The range of likely topics is already hugely diverse and exciting (our first review topic, for example **watch this space** is not even within the realm of healthcare).  So some of our blog posts will inevitably share what we are learning about service or policy areas, or areas of management or implementation science, that are new to us.

We also have a strong shared interest in some methodological issues – for example, we want to explore the social process of systematic reviews.  Within team-based review processes, particularly of complex topics, much of the effort of synthesis and sense-making does not happen through a prescribed and linear processing of information.  Useful reviews of evidence about complex health service interventions require iterative cycles of data gathering, reading, reflection, dialogue, explanation-building/theorising, and numerous small judgements about the relevance and rigour of individual sources and pieces of evidence.  We believe these processes are an inevitable part of conducting high quality, useful evidence syntheses on complex topics – yet they are often poorly described (or even acknowledged) in most published systematic reviews.  Through our blog we hope to generate discussion on this and other methodological realities and fascinations that we face – and hopefully suggest and test some solutions that are useful to others too.

Please bookmark us.  Follow us on Twitter for notification of new blog posts: @ExEvidSC


Whose views? (disclaimer)

We should underline that the views and opinions in this blog are our own (specifically, the named member of the team who signs any post to the blog), and do not necessarily reflect those of the University of Exeter, the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health.


Posted in evidence, methods, Rob, systematic review, team | Leave a comment