Mastering Meta-Analysis

A systematic review is a comprehensive process that involves gathering all relevant studies on a specific topic and research design, followed by a thorough evaluation and analysis of their findings (Ahn & Kang, 2018). They have been conducted since 1753 when James Lind published a paper that aimed to provide a concise and impartial summary of evidence on scurvy. However, it wasn’t until after the 1970s that systematic reviews began to receive greater attention, highlighting the importance of synthesizing research findings (Clarke & Chalmers, 2018).

A systematic review follows a predefined protocol, including clear inclusion criteria and search strategies (Paul & Leibovici, 2014). Systematic reviews can be divided into: (i) qualitative: if they provide a synthesis of research studies; or (ii) quantitative: if they involve the processing of a dataset gathered from previous publications (Philibert et al., 2012). Generally, quantitative systematic reviews are referred to as “meta‐analyses,” when a statistical method is applied to a dataset derived from a systematic review. While all meta‐analyses should be based on systematic reviews, not all systematic reviews include meta‐analyses (Ryś et al., 2009). In other words, systematic reviews provide a comprehensive overview of available evidence on a topic, whereas meta‐analyses offer a quantitative summary of effect sizes (Table 1). Both methods are valuable for supporting clinical decision‐making and developing evidence‐based guidelines, and they require careful interpretation and critical evaluation.

Steps to Conduct a Systematic Review and Meta‐Analysis

Defining the Research Question

A systematic review should always begin with identifying a research question that is clear, interesting, and unique. In fact, having a well‐defined research question sets a systematic review apart from a general literature review while striving to answer a quantitative question turns it into a meta‐analysis. There are many approaches to formulating a research question, for instance—authors can use a FINER (Feasible, Interesting, Novel, Ethical, Relevant) criteria or PICO (Population, Intervention, Comparison, Outcome) or SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) framework to formulate a clear and focused research question. Researchers suggest a combined approach, utilizing either or both the SPIDER and PICO tools, to achieve a comprehensive search, depending on the available time and resources (Tawfik et al., 2019).

Here’s an example of a research question authors may ask:

“What is the effect of organic mulch application (Intervention) compared with inorganic mulch (Comparison) on soil moisture retention (Outcome) in arid regions (Population)?”

Feel ready to formulate a research question? Look up in depth on FINER (https://tinyurl.com/finercriteria) criteria and PICO (https://www.cochranelibrary.com/about‐pico) through the links.

Inclusion/Exclusion Criteria

Once authors have identified a research question that is novel and feasible for study, the next step is to delineate the boundary (scope) of the study. This can be achieved through establishing specific criteria for including or excluding certain kinds of studies in the review to ensure consistency of research and relevance in applicability of findings.

For instance, in the above example, authors can decide to only include peer‐reviewed journal articles that report field studies with a minimum of two years or more, or studies that report the effect on both a soil and crop parameter (or a particular parameter), or studies conducted in a particular type of soil (e.g., sandy) or climatic regions (e.g., temperate).

Title and Abstract Screening

Then, the next step is the tedious act of going through hundreds of titles and abstracts to identify the studies that meet the inclusion/exclusion criteria that the authors devise. There are different software or websites that can help to organize this step, for example HUBMETA (https://hubmeta.com/). After these steps, it is a good idea to formulate a tentative title of the review, develop a protocol, and register the title and protocol with platforms like OSF (https://osf.io/) or PROSPERO (https://www.crd.york.ac.uk/prospero/) etc. to ensure transparency. These platforms allow authors to add titles, author name(s), objectives, inclusion/exclusion criteria, search strategy, data extraction methods, and statistical analysis plan.

Searching Database, Data Acquiring, and Quality Assessment

Then, authors should conduct a comprehensive literature search across multiple databases and sources, document the selection process, and assess the risk of bias in each study to further identify the studies that can be trusted and used. It involves finding relevant literature databases, picking suitable keywords, and employing search filters (Langenfeld & Singh, 2024). The common literature databases are Google Scholar, Scopus, Web of Science, PubMed, and ResearchGate. The screening process should be well documented and reproducible, often guided by tools like PRISMA flow diagrams (https://www.prisma‐statement.org/prisma‐2020‐flow‐diagram). At this stage, if the aim is to conduct a meta‐analysis, authors should make sure that either the data presented in the paper are sufficient, are available online, or can be collected from the authors for the study. For a narrative systematic review, this may not be a requirement.

In the previous example, authors can combine key words such as “biochar,” “soil carbon,” “crop yield,” and “field trials.”

Full‐Text Screening and Information/Data Extraction

This involves a rigorous step of reading full papers once the eligibility is confirmed to extract information or data (for meta‐analysis) required to answer the research question. In each step, it is key to document the search strategy in detail, including databases searched, key words used, and date of the search, especially when working on a collaborative project. For extracting data, it is recommended to use standardized forms to extract data on study characteristics, methodology, outcomes, and quality. When such information is not readily available, authors can be contacted to gather more information.

Study characteristics may include publication metadata (source, date, authors, peer‐review status), study design (randomized, observational, cross‐sectional, split, etc.), study eligibility criteria (PICO), study implementation (intervention type, length of follow up, study population, number of samples, etc.), participant information (average age, sex, demographics, etc.), and more. On the other hand, outcomes information can include outcome measures and effect sizes (mean, median, range, standard deviation, [standardized] treatment difference, correlations), sample sizes (total, per group, stratified by participant characteristics, enrolled, lost to follow up, analyzed), statistical inference (standard errors [SE], confidence intervals, p‐values), and analysis methods (statistical model, statistical test, estimation method, assumptions).

For instance, in the running example, it can be crucial to note down the substrate of organic mulch used, its composition, physical and chemical characteristics, application method etc. (study characteristics) as well as parameters such as changes in soil physical and biological properties and impact of crop yield (outcome of interest information).

Data Synthesis and Analysis

For meta‐analysis, selecting appropriate effect size measures and statistical models (e.g., fixed‐effect or random‐effects models) is key for data synthesis. Depending on the outcome type (continuous, ordinal, dichotomous, counts, etc.), appropriate summary statistics should be selected (Table 2). To conduct a multivariate analysis, it can be a good idea to perform subgroup analyses and meta‐regression to explore heterogeneity and potential moderators of effect. Meta‐analysis can be conducted with any statistical software that allows random effect models. Most meta‐analysis methods employ a variation on a weighted average of the effect estimates from the different studies. Meta‐analysis‐specific options provide commonly reported statistics and graphics R packages (dmetar, forester, meta, metadata, metafor, etc.), STATA (meta, many others from community), RevMan (specifically for Cochrane reviews), SPSS, SAS (PROC, MIXED, PROC NLMIXED, macro, etc.)

It can be crucial to conduct a heterogeneity and sensitivity analysis, which can be accomplished through: assessing heterogeneity (use the I² statistic and Q test to assess variability among study results), subgroup analysis (conduct subgroup analyses to explore potential sources of heterogeneity) and sensitivity analysis (perform sensitivity analyses to examine the robustness of the results). Authors can use tools like the Cochrane Risk of Bias tool for randomized controlled trials or the Newcastle‐Ottawa Scale for observational studies. Using tools such as funnel plots (scatter plot of study effect estimates vs. precision [SE, sample size]), authors can identify and address outliers that may disproportionately influence the results.

Reporting and Writing a Manuscript

After gathering all the necessary information and summarizing the key findings, the next step is finalizing the story and presenting/interpreting the results following guidelines like PRISMA, ensuring transparency and reproducibility in the reporting of the systematic review and meta‐analysis (Table 3). It is also key to identify and address potential biases (e.g., publication bias, small study bias) and outliers in the data and report accordingly. It is a good idea to reference some exemplary systematic reviews/meta‐analyses including those published in high‐impact journals such as the Cochrane Database of Systematic Reviews (Deeks et al., 2023), which adhere to rigorous standards and provide valuable insights into specific research questions.

Caution When Making Inferences Based on Systematic Review

Although meta‐analyses and systematic reviews are effective methods for combining the data from research, they have several drawbacks. When interpreting a systematic review’s conclusions, keep the following important cautions in mind:

• Publication bias: Studies with positive or significant results are more likely to be published, which can skew the conclusions of a systematic review.
• Heterogeneity: Differences in study populations, outcomes, and methodologies can lead to diverse findings that are challenging to combine or compare directly.
• Data dependence: Systematic reviews rely on the availability and accuracy of data in primary studies. Incomplete reporting, missing data, or errors can limit the reliability of the meta‐analysis.
• Time lag: The gap between primary research and systematic review completion can exclude recent studies, especially in rapidly evolving fields.
• Overinterpretation: Summarizing complex data into a single effect estimate can lead to overinterpretation of results.
• Non‐significant findings: Interpret non‐significant findings with caution, as they may still provide valuable insights.

Conclusion

Systematic reviews and meta‐analyses are powerful tools for synthesizing research evidence, offering increased precision and power to evaluate interventions and relationships across studies. Adhering to rigorous methodologies and transparent reporting standards ensures the reliability and validity of their findings, making them essential for evidence‐based practice and policymaking. Despite their limitations, when conducted properly, they provide invaluable insights and guide informed decision‐making in various fields of research.

References

Ahn, E., & Kang, H. (2018). Introduction to systematic review and meta‐analysis. Korean Journal of Anesthesiology, 71(2), 103–112. https://doi.org/10.4097/kjae.2018.71.2.103 

Clarke, M., & Chalmers, I. (2018). Reflections on the history of systematic reviews. BMJ Evidence‐Based Medicine, 23(4), 121–122. https://doi.org/10.1136/bmjebm‐2018‐110968 

Deeks, J.J., Higgins, J.P.T., Altman, D.G. (2023). Analysing data and undertaking meta‐analyses. In J.P.T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M.J. Page, V.A. Welch (Eds.), Cochrane handbook for systematic reviews of interventions (6.4th ed.) Cochrane. https://www.training.cochrane.org/handbook 

Langenfeld, N. J., & Singh, L. (2024). Writing Advice for Graduate Students to Become Published Scientists. CSA News, 69(2), 38–42. https://doi.org/10.1002/csan.21224 

Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., … & McKenzie, J. E. (2021). PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ, 372. https://doi.org/10.1136/bmj.n160 

Paul, M., & Leibovici, L. (2014). Systematic review or meta‐analysis? Their place in the evidence hierarchy. Clinical Microbiology and Infection, 20(2), 97–100. https://doi.org/10.1111/1469‐0691.12489 

Philibert, A., Loyce, C., & Makowski, D. (2012). Assessment of the quality of meta‐analysis in agronomy. Agriculture, Ecosystems & Environment, 148, 72–82. https://doi.org/10.1016/j.agee.2011.12.003 

Ryś, P., Władysiuk, M., Skrzekowska‐Baran, I., & Małecki, M. T. (2009). Review articles, systematic reviews and meta‑analyses: Which can be trusted? Polish Archives of Internal Medicine, 119(3), 148–156. https://doi.org/10.20452/pamw.634 

Tawfik, G. M., Dila, K. A. S., Mohamed, M. Y. F., Tam, D. N. H., Kien, N. D., Ahmed, A. M., & Huy, N. T. (2019). A step‐by‐step guide for conducting a systematic review and meta‐analysis with simulation data. Tropical Medicine and Health, 47(1), 46. https://doi.org/10.1186/s41182‐019‐0165‐6