a systematic literature review strategy

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Publications
Account settings
Advanced Search
Journal List

Systematically Reviewing the Literature: Building the Evidence for Health Care Quality

Suzanne austin boren , phd, david moxley , mlis.

Author information
Copyright and License information

Contact: [email protected]

Corresponding author.

There are important research and non-research reasons to systematically review the literature. This article describes a step-by-step process to systematically review the literature along with links to key resources. An example of a graduate program using systematic literature reviews to link research and quality improvement practices is also provided.

Introduction

Systematic reviews that summarize the available information on a topic are an important part of evidence-based health care. There are both research and non-research reasons for undertaking a literature review. It is important to systematically review the literature when one would like to justify the need for a study, to update personal knowledge and practice, to evaluate current practices, to develop and update guidelines for practice, and to develop work related policies. 1 A systematic review draws upon the best health services research principles and methods to address: What is the state of the evidence on the selected topic? The systematic process enables others to reproduce the methods and to make a rational determination of whether to accept the results of the review. An abundance of articles on systematic reviews exist focusing on different aspects of systematic reviews. 2 – 9 The purpose of this article is to describe a step by step process of systematically reviewing the health care literature and provide links to key resources.

Systematic Review Process: Six Key Steps

Six key steps to systematically review the literature are outlined in Table 1 and discussed here.

Systematic Review Steps

1. Formulate the Question and Refine the Topic

When preparing a topic to conduct a systematic review, it is important to ask at the outset, “What exactly am I looking for?” Hopefully it seems like an obvious step, but explicitly writing a one or two sentence statement of the topic before you begin to search is often overlooked. It is important for several reasons; in particular because, although we usually think we know what we are searching for, in truth our mental image of a topic is often quite fuzzy. The act of writing something concise and intelligible to a reader, even if you are the only one who will read it, clarifies your thoughts and can inspire you to ask key questions. In addition, in subsequent steps of the review process, when you begin to develop a strategy for searching the literature, your topic statement is the ready raw material from which you can extract the key concepts and terminology for your strategies. The medical and related health literature is massive, so the more precise and specific your understanding of your information need, the better your results will be when you search.

2. Search, Retrieve, and Select Relevant Articles

The retrieval tools chosen to search the literature should be determined by the purpose of the search. Questions to ask include: For what and by whom will the information be used? A topical expert or a novice? Am I looking for a simple fact? A comprehensive overview on the topic? Exploration of a new topic? A systematic review? For the purpose of a systematic review of journal research in the area of health care, PubMed or Medline is the most appropriate retrieval tool to start with, however other databases may be useful ( Table 2 ). In particular, Google Scholar allows one to search the same set of articles as PubMed/MEDLINE, in addition to some from other disciplines, but it lacks a number of key advanced search features that a skilled searcher can exploit in PubMed/MEDLINE.

Examples of Electronic Bibliographic Databases Specific to Health Care

Note: These databases may be available through university or hospital library systems.

An effective way to search the literature is to break the topic into different “building blocks.” The building blocks approach is the most systematic and works the best in periodical databases such as PubMed/MEDLINE. The “blocks” in a “building blocks” strategy consist of the key concepts in the search topic. For example, let’s say we are interested in researching about mobile phone-based interventions for monitoring of patient status or disease management. We could break the topic into the following concepts or blocks: 1. Mobile phones, 2. patient monitoring, and 3. Disease management. Gather synonyms and related terms to represent each concept and match to available subject headings in databases that offer them. Organize the resulting concepts into individual queries. Run the queries and examine your results to find relevant items and suggest query modifications to improve your results. Revise and re-run your strategy based on your observations. Repeat this process until you are satisfied or further modifications produce no improvements. For example in Medline, these terms would be used in this search and combined as follows: cellular phone AND (ambulatory monitoring OR disease management), where each of the key word phrases is an official subject heading in the MEDLINE vocabulary. Keep detailed notes on the literature search, as it will need to be reported in the methods section of the systematic review paper. Careful noting of search strategies also allows you to revisit a topic in the future and confidently replicate the same results, with the addition of those subsequently published on your topic.

3. Assess Quality

There is no consensus on the best way to assess study quality. Many quality assessment tools include issues such as: appropriateness of study design to the research objective, risk of bias, generalizability, statistical issues, quality of the intervention, and quality of reporting. Reporting guidelines for most literature types are available at the EQUATOR Network website ( http://www.equator-network.org/ ). These guidelines are a useful starting point; however they should not be used for assessing study quality.

4. Extract Data and Information

Extract information from each eligible article into a standardized format to permit the findings to be summarized. This will involve building one or more tables. When making tables each row should represent an article and each column a variable. Not all of the information that is extracted into the tables will end up in the paper. All of the information that is extracted from the eligible articles will help you obtain an overview of the topic, however you will want to reserve the use of tables in the literature review paper for the more complex information. All tables should be introduced and discussed in the narrative of the literature review. An example of an evidence summary table is presented in Table 3 .

Example of an evidence summary table

Notes: BP = blood pressure, HbA1c = Hemoglobin A1c, Hypo = hypoglycemic, I = Internet, NS = not significant, PDA = personal digital assistant, QOL = quality of life, SMBG = self-monitored blood glucose, SMS = short message service, V = voice

5. Analyze and Synthesize Data and information

The findings from individual studies are analyzed and synthesized so that the overall effectiveness of the intervention can be determined. It should also be observed at this time if the effect of an intervention is comparable in different studies, participants, and settings.

6. Write the Systematic Review

The PRISMA 12 and ENTREQ 13 checklists can be useful resources when writing a systematic review. These uniform reporting tools focus on how to write coherent and comprehensive reviews that facilitate readers and reviewers in evaluating the relative strengths and weaknesses. A systematic literature review has the same structure as an original research article:

TITLE : The systematic review title should indicate the content. The title should reflect the research question, however it should be a statement and not a question. The research question and the title should have similar key words.

STRUCTURED ABSTRACT: The structured abstract recaps the background, methods, results and conclusion in usually 250 words or less.

INTRODUCTION: The introduction summarizes the topic or problem and specifies the practical significance for the systematic review. The first paragraph or two of the paper should capture the attention of the reader. It might be dramatic, statistical, or descriptive, but above all, it should be interesting and very relevant to the research question. The topic or problem is linked with earlier research through previous attempts to solve the problem. Gaps in the literature regarding research and practice should also be noted. The final sentence of the introduction should clearly state the purpose of the systematic review.

METHODS: The methods provide a specification of the study protocol with enough information so that others can reproduce the results. It is important to include information on the:

Eligibility criteria for studies: Who are the patients or subjects? What are the study characteristics, interventions, and outcomes? Were there language restrictions?

Literature search: What databases were searched? Which key search terms were used? Which years were searched?

Study selection: What was the study selection method? Was the title screened first, followed by the abstract, and finally the full text of the article?

Data extraction: What data and information will be extracted from the articles?

Data analysis: What are the statistical methods for handling any quantitative data?

RESULTS: The results should also be well-organized. One way to approach the results is to include information on the:

Search results: What are the numbers of articles identified, excluded, and ultimately eligible?

Study characteristics: What are the type and number of subjects? What are the methodological features of the studies?

Study quality score: What is the overall quality of included studies? Does the quality of the included studies affect the outcome of the results?

Results of the study: What are the overall results and outcomes? Could the literature be divided into themes or categories?

DISCUSSION: The discussion begins with a nonnumeric summary of the results. Next, gaps in the literature as well as limitations of the included articles are discussed with respect to the impact that they have on the reliability of the results. The final paragraph provides conclusions as well as implications for future research and current practice. For example, questions for future research on this topic are revealed, as well as whether or not practice should change as a result of the review.

REFERENCES: A complete bibliographical list of all journal articles, reports, books, and other media referred to in the systematic review should be included at the end of the paper. Referencing software can facilitate the compilation of citations and is useful in terms of ensuring the reference list is accurate and complete.

The following resources may be helpful when writing a systematic review:

CEBM: Centre for Evidence-based Medicine. Dedicated to the practice, teaching and dissemination of high quality evidence based medicine to improve health care Available at: http://www.cebm.net/ .

CITING MEDICINE: The National Library of Medicine Style Guide for Authors, Editors, and Publishers. This resource provides guidance in compiling, revising, formatting, and setting reference standards. Available at http://www.ncbi.nlm.nih.gov/books/NBK7265/ .

EQUATOR NETWORK: Enhancing the QUAlity and Transparency Of health Research. The EQUATOR Network promotes the transparent and accurate reporting of research studies. Available at: http://www.equator-network.org/ .

ICMJE RECOMMENDATIONS: International Committee of Medical Journal Editors Recommendations for the Conduct, Reporting, Editing and Publication of Scholarly Work in Medical Journals. The ICJME recommendations are followed by a large number of journals. Available at: http://www.icmje.org/about-icmje/faqs/icmje-recommendations/ .

PRISMA STATEMENT: Preferred Reporting Items for Systematic Reviews and Meta-Analyses. Authors can utilize the PRISMA Statement checklist to improve the reporting of systematic reviews and meta-analyses. Available at: http://prisma-statement.org .

THE COCHRANE COLLABORATION: A reliable source for making evidence generated through research useful for informing decisions about health. Available at: http://www.cochrane.org/ .

Examples of Systematic Reviews To Link Research and Quality Improvement

Over the past 17 years more than 300 learners, including physicians, nurses, and health administrators have completed a course as part of a Master of Health Administration or a Master of Science in Health Informatics degree at the University of Missouri. An objective of the course is to educate health informatics and health administration professionals about how to utilize a systematic, scientific, and evidence-based approach to literature searching, appraisal, and synthesis. Learners in the course conduct a systematic review of the literature on a health care topic of their choosing that could suggest quality improvement in their organization. Students select topics that make sense in terms of their core educational competencies and are related to their work. The categories of topics include public health, leadership, information management, health information technology, electronic medical records, telehealth, patient/clinician safety, treatment/screening evaluation cost/finance, human resources, planning and marketing, supply chain, education/training, policies and regulations, access, and satisfaction. Some learners have published their systematic literature reviews 14 – 15 . Qualitative comments from the students indicate that the course is well received and the skills learned in the course are applicable to a variety of health care settings.

Undertaking a literature review includes identification of a topic of interest, searching and retrieving the appropriate literature, assessing quality, extracting data and information, analyzing and synthesizing the findings, and writing a report. A structured step-by-step approach facilitates the development of a complete and informed literature review.

Suzanne Austin Boren, PhD, MHA, (above) is Associate Professor and Director of Academic Programs, and David Moxley, MLIS, is Clinical Instructor and Associate Director of Executive Programs. Both are in the Department of Health Management and Informatics at the University of Missouri School of Medicine.

graphic file with name ms112_p0058f1.jpg

None reported.

1. Polit DF, Beck CT. Nursing Research: principles and methods. 9th edition. Lippincott, Williams & Wilkins; Philadelphia: 2011. [ Google Scholar ]
2. Bruce J, Mollison J. Reviewing the literature: adopting a systematic approach. Journal of Family Planning and Reproductive Health Care. 2004;30(1) doi: 10.1783/147118904322701901. [ DOI ] [ PubMed ] [ Google Scholar ]
3. Cronin P, Ryan F, Coughlin M. Undertaking a literature review: A step-by-step approach. British Journal of Nursing. 2008;17(1):38–43. doi: 10.12968/bjon.2008.17.1.28059. [ DOI ] [ PubMed ] [ Google Scholar ]
4. Crowther DM. A clinician’s guide to systematic reviews. Nutr Clin Pract. 2013;28:459–462. doi: 10.1177/0884533613490742. [ DOI ] [ PubMed ] [ Google Scholar ]
5. Hasse SC. Systematic reviews and meta-analysis. Plast Reconstr Surg. 2011;127:955–966. doi: 10.1097/PRS.0b013e318200afa9. [ DOI ] [ PubMed ] [ Google Scholar ]
6. Mandrekar JN, Mandreker SJ. Systematic reviews and meta analysis of published studies: An over view and best practices. J Thorac Oncol. 2011;6(8):1301–1303. doi: 10.1097/JTO.0b013e31822461b0. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
7. Ng KH, Peh WC. Writing a systematic review. Singapore Med J. 2010 May;51(5):362–6. [ PubMed ] [ Google Scholar ]
8. Price B. Guidance on conducting a literature search reviewing mixed literature. Nursing Standard. 2009;23(24):43–49. doi: 10.7748/ns2009.02.23.24.43.c6829. [ DOI ] [ PubMed ] [ Google Scholar ]
9. Engberg S. Systematic reviews and meta-analysis: Studies of studies. J Wound Ostomy Continence Nurs. 2008;35(3):258–265. doi: 10.1097/01.WON.0000319122.76112.23. [ DOI ] [ PubMed ] [ Google Scholar ]
10. Benhamou PY, Melki V, Boizel R, et al. One-year efficacy and safety of Web-based follow-up using cellular phone in type 1 diabetic patients under insulin pump therapy: the PumpNet study. Diabetes & Metabolism. 2007;33(3):220–6. doi: 10.1016/j.diabet.2007.01.002. [ DOI ] [ PubMed ] [ Google Scholar ]
11. Marquez Contreras E, de la Figuera von Wichmann M, Gil Guillen V, Ylla-Catala A, Figueras M, Balana M, Naval J. Effective news of an inter vention to provide information to patients with hypertension as short text messages and reminder sent to their mobile phone [Spanish] Aten Primaria. 2004;34(8):399–405. doi: 10.1016/S0212-6567(04)78922-2. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
12. Moher D, Liberati A, Tetzlaff J, Altman DG The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Ann Intern Med. 2009;151(4):264–269. W64. doi: 10.7326/0003-4819-151-4-200908180-00135. [ DOI ] [ PubMed ] [ Google Scholar ]
13. Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12(1):181. doi: 10.1186/1471-2288-12-181. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
14. Hart MD. Informatics competency and development within the US nursing population workforce. CIN: Computers, Informatics, Nursing. 2008;26(6):320–329. doi: 10.1097/01.NCN.0000336462.94939.4c. [ DOI ] [ PubMed ] [ Google Scholar ]
15. Bryan C, Boren SA. The use and effectiveness of electronic clinical decision support tools in the ambulatory, primary care setting: A systematic review of the literature. Informatics in Primary Care. 2008 Jun;16(2):79–91. doi: 10.14236/jhi.v16i2.679. [ DOI ] [ PubMed ] [ Google Scholar ]
PDF (510.6 KB)
Collections