As I edge ever closer to completing another milestone in my PhD, I thought I would write a post about the parts that have preceeded this. I admit to being no expert in the field of writing systematic reviews (SRs), but thought I might share my experience of what the process of completing and writing one and then getting it published was like for me.
Though, I should mention here that this is the personal experience of writing a systematic review as part of PhD thesis and therefore other researchers that have been involved in producing SRs may have a totally different experience from the one which I am about to account. Writing a PhD is largely an individual endeavour. While you have the support and guidance of a supervision team behind you, you are quite alone in writing and producing the final work. I’m about to be involved in a SR which has had a larger team involved in producing it and I expect this will produce a wholly different experience from the one which I write about here.
Anyway, I digress.
Unfortunately I can’t delve too much into the details of the results and conclusions of my SR, as it has been published in a journal, and there are all issues around copyright etc. But, here is the link to the article and journal if you wish to find out more!
Eur Heart J Acute Cardiovasc Care. 2015 Dec 4. pii: 2048872615620893. [Epub ahead of print]
What is a systematic review?
Most PhDs begin with a review of all of the current literature surrounding your field of interest. This serves a number of purposes; it gives you a good grounding on your topic and ensure you have a good knowledge; it lets you know what has already been researched in order to avoid duplication; it highlights gaps in the body of evidence.
A systematic review goes further than a narrative review or a summary of the existing literature. Literature reviews, narrative reviews and commentaries are useful for background reading and providing summaries on a particular topic, however they differ from a systematic review. A systematic review is based on a pre-determined protocol (see PRISMA-P). This should reduce bias by making the review replicable, so that it can be known whether the authors have selected specific papers to include in order to support their own personal beliefs and also provides a systematic way of appraising the quality of the papers and level of bias within them.
The Cochrane Collaboration defines a systematic review as a review of a clearly formulated question that uses systematic and explcit methods to identify, select and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review.
I can not better that definition, so let’s stick with that for the time being.
It’s also good practice to register your systematic review on a database, such as PROSPERO. This helps avoid unnecessary duplication and allows comparison of reported review methods with what was planned in the protocol. You can even publish your protocol in a journal, so that people can see whether you applied the methods you said you were going to.
A good systematic review should:
- Identify all relevant published and unpublished evidence
- Select studies or reports for inclusion
- Assess the quality of each study or report
- Synthesise the findings from individual studies or reports in an unbiased way
- Interpret the findings and present a balanced and impartial summary of the findings with due consideration of any flaws in the evidence
In medicine and healthcare it’s very difficult to stay on top of the plethora of evidence that is produced, let alone take the time to critically appraise it to find out whether the findings are reliable or not. On the front line there is not much time to seek out all of the individual studies on a particular topic you need information on to help you make a clinical decision. And even if there was the time, they would no doubt produce conflicting results. Systematic reviews are hoped to bring all of that information together to provide the best available evidence, or to highlight an area where research is lacking. There are also useful resources like BestBETS, which provide rapid evidence-based answers to real-life clinical questions, using a systematic approach to reviewing the literature.
However, just as with other levels of evidence, systematic reviews can vary in their quality and should be critically appraised (critical appraisal tool for SR). In order to address suboptimal reporting of systematic reviews and meta-analyses, the PRISMA statement (Preferred Reporting Items for Systematic reviews and Meta-Analysis) was produced. This is a 27-item checklist and flow diagram to help authors improve the reporting of systematic reviews and meta-analysis (primarily focussing on randomised trials, but can be used for other types of studies). A lot of journals will only publish a systematic review if it evidences that it has followed the PRISMA statement.
So, when I had decided that my interest lay in the use of prehospital 12-lead ECG in acute stroke patients and had come up with my research question, I needed to know whether this had already been researched; what the findings were; what are the gaps in our current knowledge?; are the existing pieces of research of good quality?; are they generalizable to my current setting?; and what does the cumulative knowledge of all of the papers put together and synthesised tell me about my field of interest?.
The first stage was to decide on my search strategy. This means choosing the words you are going to put into the search you enter into the electronic databases. Your friendly librarian can help you with things like knowing which electronic databases are best to search, using things like Boolean operators (using AND, OR, NOT, WITHIN etc), ‘exploding’ terms etc. This sounds easy, but in my experience, it took some work. I found that my search strategies were resulting in 400,000+ results from the databases, which is just unfeasible for one PhD student to trawl through by themself.
So I would go back and tweak it.
I would try adding a word, taking a word away, trying different words. I would then get zero results. It was infuriating, I spent the good part of a week trying different combinations of words and terms to try make sure it would include everything I needed, but was a manageable amount for me to search through. The search strategy was either too specific, so that no studies were being included, or too inclusive that absolute irrelelvant rubbish was being included. In the end I came up with a search strategy that resulted in approximately 70,000 results and I took this as hopefully including everything that was relevant to my field of interest and relatively manageable (and I say that liberally).
The next job was to screen the titles and abstracts of the 70,000 articles pulled from my search for relevance. If you haven’t done a SR before, imagine making a search on google and then scrolling through the search results for 7 hours a day for 2 weeks. By the end of each day it felt like my eyes were bleeding. But alas, at the end of the 2 weeks I had screened all of the articles for relevance and whether they fit my inclusion/exclusion criteria.
A systematic review should also be a complete and exhaustive search of all of the available evidence, this means using more than just a MEDLINE search. I searched multiple electronic databases, emailed experts in the field, searched individual key journals, sought out conferences proceedings and any unpublished or ‘grey’ literature. Ideally, non-English studies should also be included.
At the end of this arduous task, do you know how many articles were included for me to appraise? None. Zero. Two weeks of bleeding eyes for nothing. No articles were found that looked at the use of prehospital 12-lead ECG in acute stroke patients. I wanted to cry. One of the inclusion criteria was that they had to be based in the prehospital setting. I took my findings to my supervisors and we decided to open the inclusion criteria to include hospital-based studies. Although it felt like I had done all of that work for nothing, it actually turned out to be a good thing, and the fact there were no studies based in the prehospital study was an important finding of the systematic review itself. This already showed us there was a gap in the evidence for future research (i.e my PhD) to study. Praise the Lord.
Appraise and synthesise
Now that the SR included hospital-based studies I had 54 articles to appraise and synthesise the results. None of the studies were randomised controlled trials (RCTs), so using the Centre of Evidence Based Medicine (CEBM) CASP appraisal tools for observational studies I created a customised tool for data extraction and to critically appraise them. Again, to critically appraise 54 studies is a large, time consuming task, but it helped improve my critical appraisal skills hugely. It was during this time I realised how important critical appraisal skills are when reading research. Before this I was under the impression that if a piece of research had been published in a peer-reviewed journal then it must be good quality. I quickly found this not to be the case. It’s also good to have a second, independent reviewer go through the studies to assess for quality and bias.
The 54 studies were split in to 4 themes and I went to work in trying to synthesise the findings and write them up in to a substantial chapter for my thesis. This synthesis could include a meta-analysis (a statistical technique for combining the findings from independent studies) if all the data was homogenous, however in my review this was not the case. I therefore produced what could be called a narrative summary of the findings and wrote about what impact these findings had for my particular context.
Completing a SR on your own as a novice researcher is not a quick or easy undertaking. It took time and patience. It also takes a lot of skill, skill which I did not possess at the start of the process, but with guidance, support and a lot of practice, managed to develop slowly but surely. From start to finish, with a few interruptions here and there, it took me around 2 years to produce. Bear in mind that I worked on it part-time, while also spending half of my time in clinical practice. Whether you think this was quick or overly long does not really worry me. With my SR completed, I then looked at getting the information out and available for people to use. I presented it at a few conferences and started thinking about getting it published in a journal, which brings us nicely to part 2…