Multivariate meta-analysis

Multivariate meta-analysis: advantages and limitations

In the aggregation of outcomes of interest across related studies, there are often multivariate effect estimates rather than univariate. Examples usually include bivariate outcomes such as specificities and sensitivities in diagnosis tests, systolic and diastolic measures of blood pressure, and the treatment effects on overall survival and disease-free survival in cancer research. In these situations, multivariate meta-analysis methods (also denoted as multiple endpoint meta-analysis [1]) would allow the joint synthesis of effect estimates, accounting for within-study and between-study correlations of the outcomes [2]. The applications of multivariate meta-analysis—as an alternative to a separate univariate meta-analysis of each outcome independently—have been growing; however, due to the lack of understanding of the correlation structure of study effects (particularly within-study correlations) as well as difficulties of the use of multivariate methods, many researchers still prefer to conduct univariate analysis [3]. Research already shows that if separate univariate meta-analysis is conducted, where the strong and often incorrect assumption of independency among the effects is made, the synthesized estimates are biased and the variance of the summary effect size is overestimated [4].

Overall, multivariate meta-analysis methods can help [5]: 1) obtain estimates for all effects in a single modeling framework, 2) describe the relationship between the multiple effects, and 3) provide estimates with better statistical properties than univariate meta-analysis—due to utilizing the correlations between the effects. The utilization of the correlations results in ‘borrowing of strength’ [6]—the most commonly referred to advantage of multivariate methods—which applies to the between study variance estimates as well as pooled estimates [7], and helps reduce the impact of outcome reporting bias [8]. Furthermore, recent refined methods in multivariate meta-analysis can also help pooling the study results more precisely than univariate approaches when the number of studies is small—which is typical in many meta-analyses [9].    

Despite these advantages, multivariate meta-analysis methods come at the price of making more assumptions—in comparison to univariate meta-analysis. They are often based on multivariate normality assumption which is hard to verify. Moreover, an implicit assumption in the multivariate case is that the effects have a linear relationship among studies. Multivariate meta-analysis methods are also often difficult to estimate, potentially due to difficulty in the estimation of the between-study correlation [5]—data structure complexities, e.g., categorical data, also adds to this difficulty. Another notable limitation is that when there are more than two outcomes, e.g., X, Y, and Z, it is required that the same multivariate analysis is performed across studies. For instance, if studies report the regression of Y on X controlling for Z, those studies that control a fourth variable, e.g., W, or both Z and W could not be included [10]. Not only these limitations constrain researchers to make assumptions which do not result in better inference, but they also limit the applications of multivariate methods.

My current research on aggregation of statistical findings from prior studies shares the advantages of multivariate meta-analysis methods while it is not bounded to their limitations. More to come in my forthcoming publications… Stay tuned!



1.            Gleser LJ, Olkin I. Stochastically dependent effect sizes. In: Cooper HM, Hedges LV, editors. The Handbook of Research Synthesis. New York: Russell Sage Foundation; 1994. p. 339–55.

2.            Wei YH, Higgins JPT. Estimating within-study covariances in multivariate meta-analysis with multiple outcomes. Statistics in Medicine. 2013;32(7):1191-205. doi: 10.1002/sim.5679. PubMed PMID: WOS:000316210800007.

3.            Mavridis D, Salanti G. A practical introduction to multivariate meta-analysis. Statistical methods in medical research. 2013;22(2):133-58. doi: 10.1177/0962280211432219. PubMed PMID: WOS:000317938800003.

4.            Riley RD. Multivariate meta-analysis: the effect of ignoring within-study correlation. Journal of the Royal Statistical Society: Series A (Statistics in Society). 2009;172(4):789-811. doi: 10.1111/j.1467-985X.2008.00593.x.

5.            Jackson D, Riley R, White IR. Multivariate meta-analysis: Potential and promise. Statistics in Medicine. 2011;30(20):2481-98. doi: 10.1002/sim.4172.

6.            Jackson D, White IR, Price M, Copas J, Riley RD. Borrowing of strength and study weights in multivariate and network meta-analysis. Statistical methods in medical research. 2015. Epub 2015/11/08. doi: 10.1177/0962280215611702. PubMed PMID: 26546254.

7.            Jackson D, White IR, Riley RD. Quantifying the impact of between-study heterogeneity in multivariate meta-analyses. Stat Med. 2012;31(29):3805-20. Epub 2012/07/06. doi: 10.1002/sim.5453. PubMed PMID: 22763950; PubMed Central PMCID: PMCPMC3546377.

8.            Kirkham JJ, Riley RD, Williamson PR. A multivariate meta-analysis approach for reducing the impact of outcome reporting bias in systematic reviews. Statistics in Medicine. 2012;31(20):2179-95. doi: 10.1002/sim.5356. PubMed PMID: WOS:000307648900003.

9.            Jackson D, Riley RD. A refined method for multivariate meta-analysis and meta-regression. Statistics in Medicine. 2014;33(4):541-54. doi: 10.1002/sim.5957. PubMed PMID: WOS:000330802300001.

10.         Card NA, Casper DM. Meta-Analysis and Quantitative Research Synthesis. In: Little TD, editor. The Oxford Handbook of Quantitative Methods in Psychology: Vol 2: Statistical Analysis. New York, NY: Oxford University Press; 2013.