…or answers to some of the questions we are asked most often in our Search and Review clinics.

We’ve been running fortnightly Search & Review clinics for more than 4 years, providing advice and guidance on systematic review and search methods to over 350 researchers, health care professionals, students and patients within the South West peninsula.

Over the next week we’ll be sharing some of the most common questions we hear at the clinics and offering our learned advice in the hope that it is helpful to others embarking on a systematic review.

Question Twelve:  Do I need to do a systematic review? I’m thinking a literature review might be better…

Short answer = Why?

Read on for more…

I’m sorry we can’t help you with this one but we can wish you a happy and healthy Christmas and much fun and excitement in 2017.


Question Eleven:  Do I have to register my protocol – what if someone steals my idea?

Short answer = Yes.

Blog answer = It is good practice to register your protocol for the reasons stated in answer to Q1 – transparency, rigour, avoiding bias etc. but registration also allows others to know what you are doing so that they don’t repeat your efforts and contribute to research waste.

Further reading:

Lancet REWARD campaign

Question Ten:  Do I need to search Pubmed and Medline?

Short answer = It depends…

Read on for more…

MEDLINE is a subset of the PubMed database, so in theory you can find all of the MEDLINE content on PubMed. MEDLINE is streamlined through a journal selection process so it does not contain a lot of the broader content of PubMed, covering areas such as astrophysics, plate tectonics and chemistry, that might add considerably to your recall should you search PubMed rather than MEDLINE. In addition, MEDLINE can be more efficiently searched than PubMed via platforms such as OvidSp, and also benefits from the National Library of Medicine’s controlled vocabulary (MeSH).

It is worth considering, however, that some “ahead of print” citations appear on PubMed before they are finally published in a MEDLINE indexed journal, so you might want to run your search on PubMed and restrict it to very recent non-MEDLINE citations only.

Further reading:

Fact Sheet:  MEDLINE, PubMed and PMC (PubMed Central):  How are they different?


Question Nine:  How do I find grey literature?

Short answer = There is no short answer.

Read on for more…

Some databases are good sources of grey literature, for example HMIC (Health Management Information Consortium) is excellent for UK health related grey literature, and SPP (Social Policy and Practice) is equally good for social care grey literature. Some conventional databases e.g. CINAHL and PsycINFO also hold types of grey literature. There are also databases that index specific types of grey literature, e.g. Web of Science’s Conference Proceedings Citation Index and ProQuest Dissertations and Theses.

It helps to consider your topic, and the kind of grey literature you might expect to find. Are there likely to be unpublished reports or studies addressing your question? What organisations would produce them? Typical institutions generating grey literature might be hospitals, professional associations, government organisations or private companies, for example. You need to list these in your protocol as part of your overall search strategy.

As well as considering the sources of grey literature, you also need to think about how you are going to retrieve this information. Are you going to contact these organisations? Or search their online repositories? It is not enough to simply say you are going to carry out web-searching. What search engine will you use? What are you going to type in to it? Will you screen all the results? What if there are 100,000 of them?

So…the answer to this question of ‘how do I find grey literature’ is that it depends partly on the type of grey literature you want to find, and partly on who generates it. If you know the answers to these questions, then you are halfway there.

Further reading:

Briscoe S. Web searching for systematic reviews: a case study of reporting standards in the UK Health Technology Assessment programme. BMC Res Notes. 2015 Apr 16;8:153. doi:  10.1186/s13104-015-1079-y.

Stansfield C, Dickson K, Bangpan M. Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? Syst Rev. 2016 Nov 15;5(1):191.


Question Eight:  How am I going to manage all these references?

Short answer – Don’t worry…

Read on for more…

Managing your references and recording the flow of studies through your review is really important.  You will need this information to populate your PRISMA flow diagram.

There are plenty of reference management software options available.  Some like Endnote desktop have useful additional functionality such as SMART groups and CUSTOM fields. It is also possible to carry out your double screening using Endnote desktop.  Take a look at our screencasts for some step by step guidance on how to do this.

If you don’t have access to Endnote desktop – there are other reference management software options that are freely available.  This table compares some of them in terms of features that may be helpful in a systematic review.

Endnote desktop Endnote web Mendeley Zotero
Import refs from databases Yes Yes Yes Yes
De-duplicate refs Yes Yes – auto Yes – auto Yes – auto
Create groups Yes yes Yes Yes
Create SMART groups Yes No No No
Create sub-groups Yes No Yes Yes
Numbers of references visible Yes Yes No No
Edit fields Yes No No No
Create multiple libraries Yes No No No
Cite while you write Yes Yes Yes Yes
Manually input references Yes Yes Yes Yes
Accessible off line Yes No Yes Yes
Searchable Yes Yes Yes Yes



Question Seven:   Do I have to double screen all the search results?

Short answer = It depends…

Ideally, yes if you are to avoid missing potentially relevant studies, two reviewers should independently screen results at both the title/abstract and the full text stage.

Sometimes, when there are insufficient resources available to dedicate to this process, review authors decide to double screen a proportion of the total, perhaps 10%.  Before you do this it is important to consider what you will do if there is ‘significant’ disagreement between reviewers.  How will you decide if the level of disagreement is something to worry about?  What will you do if it is?  The process for dealing with this should be outlined in your protocol.

Double screening a batch of papers e.g. 100 to test and refine understanding of inclusion and exclusion criteria between review team members is a useful step in guiding decisions on the necessary extent of double screening.

Further reading:

Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat Med. 2002 Jun 15;21(11):1635-40.

Shemilt I, Khan N, Park S, Thomas J. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Syst Rev. 2016 Aug 17;5(1):140. 

Question Six:  Which critical appraisal tool should I use?

Short answer = It depends

Read on for more…

There are many critical appraisal tools available.  None are suitable for all systematic reviews.  Your choice will depend on the type of evidence that you plan to include in the review.  An understanding of why you need to critically appraise the included evidence and what you are looking for is a good place to start – the CASP website provides a good overview.

The CRD Handbook of guidance for undertaking reviews of healthcare describes many of the available tools for critical appraisal.  The Cochrane Risk of Bias tool for randomised controlled trials, ROBINS-I for non-randomised studies of interventions and the EPHPP for public health studies are all popular choices.   A new tool for the critical appraisal of cross sectional studies (AXIS) was published last month.  If you decide to critically appraise included studies in a review of qualitative research you could use the criteria described in this paper by Wallace and colleagues.

Further reading:

Wallace A, Croucher K, Quilgars D, Baldwin S. Meeting the challenge: developing systematic reviewing in social policy. Policy & Politics. 2004;32(4):455-70.

Downes MJ, Brennan ML, Williams HC, Dean RS. Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open. 2016 Dec 8;6(12):e011458. doi: 10.1136/bmjopen-2016-011458.

Question Five:  I’m not sure what my research question is, should I do a scoping review, a systematic review or a meta-analysis?

Short answer = You need to match the method to the research question and the available evidence.

Read on for more…

First of all a meta-analysis is a type of synthesis not a type of review.  Your systematic review may or may not involve a meta-analysis depending on the type of evidence available.  Not all systematic reviews will involve a meta-analysis and vice versa.

Different review methods are appropriate for different review questions. The method you use should be suited to the research question and the available evidence.

A scoping review is best suited to questions like ‘What is known about….?’ If your question is ‘What are the effects of X on Y symptoms in people with Z condition? a systematic review of effectiveness studies might be the best method.  Your scoping review will still use systematic, transparent methods.

What you will need to do is scope the literature and talk about your question with people who understand the issue.  This scoping process is really important to make sure that you get the question right and also to know whether your question has already been addressed – but it isn’t a ‘scoping review’.  Many people undervalue this step – getting it right will help to make sure that the findings of your review are relevant, useful and used.

Further reading:

Kastner M, Antony J, Soobiah C, Straus SE, Tricco AC. Conceptual recommendations for selecting the most appropriate knowledge synthesis method to answer research questions related to complex evidence. J Clin Epidemiol. 2016 May;73:43-9.

Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Research Synthesis Methods. 2014;5(4):371-385. doi:10.1002/jrsm.1123.

Question Four: I know all the studies I want to include in my review, do I need to do a search?

Short answer = Yes.

Read on for more…

Searching, like the other elements of a systematic review process, needs to be systematic, transparent and reproducible. This means you need

  • a search log detailing the databases you searched, how you accessed them, the date you searched them and the number of results you found,
  • to document the search strategies for all of the databases you search and,
  • to explain and document how you identified and selected the studies you include in the review – were they double screened by two independent reviewers?

Perhaps you do know all the relevant literature but you still need to show the steps you took to find it.

And you never know, maybe during the process you will discover some you didn’t know about…

Question Three:  I’ve finished my review – could you check my search strategy?

Short answer = We can, but…

Read on for more…

We hope that your search strategy will have been informed by extensive scoping searches, discussion with topic experts and a protocol so that you knew what you were looking for when you ran the searches.

If you are using a cut-off date for the searches you need to explain why (searching for the last two years because there would be too many studies otherwise is not a good reason).  Similarly if you decide to restrict to English only papers why did you do this?

We also hope that you have considered a range of databases, supplementary search methods, locating grey literature and clinical trials databases.

Hopefully your search will be effective, but in the future, it is worth considering that information specialists provide the best value at the start of the project!

If you have don’t have access to an information specialist, there are plenty of good resources on how to search effectively e.g. The Cochrane Handbook of Systematic Reviews (searching chapter).

Further reading:

Petticrew M. Systematic reviews from astronomy to zoology: myths and misconceptions. BMJ. 2001 Jan 13;322(7278):98-101

Question Two:  Do I need to follow the protocol?

Short answer = Yes.

Read on for more…

See answer to  Question One below.

Further reading:

Silagy CA, Middleton P, Hopewell S. Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA. 2002 Jun 5;287(21):2831-4. 

Kirkham JJ, Altman DG, Williamson PR. Bias due to changes in specified outcomes during the systematic review process. PLoS One. 2010 Mar 22;5(3):e9810. 

Question One:  Do I need a protocol?

Short answer = Yes.

Read on for more…

A protocol is an essential step in any systematic review and should be prepared, discussed with co-authors and agreed before work starts on the review.  The protocol should be reported according to the PRISMA-P guidelines and either registered (e.g. with PROSPERO or CEE, published in a peer-reviewed journal or made available on a departmental website.  Having a protocol (that is subsequently adhered to) means that decisions made throughout the process (e.g. whether to include or exclude papers, how to approach the synthesis or whether sub-group analyses will be undertaken) are pre-defined, rigorous and transparent and made without prior knowledge of individual study findings.  Examples of systematic review protocols can be found on PROSPERO and also in the journals Systematic Reviews and BMJ Open.

Further reading:

Planning a Systematic Review:  Think protocols