Meta-analysis - Some Considerations
Some considerations when planning the statistical approach to an IPD meta-analysis of time-to-event data
We recently undertook a meta-analysis of time-to-adverse-event data across 12 randomized controlled trials in chronic obstructive pulmonary disease and share a few lessons learned when planning and implementing this type of analyses. Meta-analysis of individual patient data (IPD) offers a number of advantages over the traditional meta-analysis of aggregate data - in particular the ability to reliably estimate covariate and interaction effects. IPD can be modelled in a ‘one-stage" analysis (fitting a single model to all trials simultaneously), or the trials can be modelled separately and summary statistics analysed using standard meta-analytic techniques, known as a ‘two-stage" approach. One-stage models are appealing when IPD are available, but two-stage approaches should not be discounted as they are simpler to implement (particularly if assuming random effects) and any loss of power appears to be small (Stewart et al. 2012, Fisher et al. 2011).
In our analysis we aimed to establish three things:
1)Treatment effect: was there evidence of a treatment effect?
2)Covariate effects: were some subgroups at greater risk of experiencing the adverse event than others?
3)Interaction effects: did any apparent treatment effect differ across subgroups?
Finally, we wanted to investigate any heterogeneity in our meta-analysis results.
Conclusion
Overall, there are many things to consider when planning an IPD meta-analysis. Although a one‑stage analysis might seem preferable, two-stage approaches should not be discounted. A combination of both approaches may be required depending on the purpose and complexity of the various analyses required. Finally, special consideration should be given to possible biases in the results of an IPD analysis due to the exclusion of studies for which IPD may not be available. Further details of considerations for our analysis are provided below.
Treatment effect
To obtain an overall treatment effect, a one-stage model using Cox regression model stratified by study is a recommended starting place (Simmonds et al. 2005, Stewart et al. 2012). However, it may be worth considering an exponential model (assuming the hazard functions are suitably constant) to simultaneously obtain an estimate of the absolute risk rate, whilst appropriately taking into account censoring and follow-up times. We obtained treatment effects using both one-stage and two-stage approaches (with the latter as a sensitivity analysis) and found negligible differences, which is line with the findings of Stewart et al. (2012).
It should be carefully considered whether treatment effects adjusted for other covariates are required. One issue is that some studies may only include patients with a particular covariate category – for example, in our case we wanted to adjust for categories of smoking status at baseline, but some studies only included patients who were current smokers. In such cases, the statistician should be aware that not all strata will contribute to all effect estimates. A second issue is whether all covariates required for adjustment are collected in all studies. A sophisticated two-stage solution is provided by the Fibrinogen Studies Collaboration (2009) and could be considered. Alternatively, a two-stage approach using estimates obtained from each study, adjusted as best can be achieved given the data available, may be a pragmatic if not ideal solution.
Covariate effects
One benefit of modelling covariate effects is that they can put any apparent treatment effect in context. The ability to model covariate effects appropriately is a key advantage of meta-analysing IPD over standard aggregate data meta-analyses. In an aggregate data meta-analysis we are usually restricted to meta-regression i.e. modelling the relationship between treatment effects and study‑level covariates, which is subject to aggregation bias. Put simply, aggregation bias is where relationships between aggregated values may not reflect associations at a patient level (see for example Stewart et al. 2012). With IPD we can model the within-study effect of covariates using patient-level data. Meta-regression of aggregate data has been shown to be considerably underpowered and has poor agreement with estimates obtained using patient-level covariates (Lambert et al. 2002). However, modelling IPD data does not necessarily negate biases in covariate effects. A stratified Cox model assuming a single, constant effect across strata may be too simplistic, and covariate-by-stratum interactions might be required (see Fisher et al. 2011). One straightforward way of obtaining a summary estimate of the covariate effect would be to model each stratum individually and combine the estimates using a two-stage approach.
Interaction effects
Interaction effects are subject to all of the same issues outlined for covariate effects. One additional issue, particularly if analysing a rare event, could be lack of events in each treatment-by-covariate category. Without events in each category, the stratum may be effectively ignored in the analysis. If the covariate contains many categories and not all categories are within each study then, again, some strata will not contribute to the analysis. For a two-stage approach we're required to meta‑analyse each estimate which makes up an interaction effect. For example for an interaction between a 3-level factor (subgroups A, B or C) and a two-level factor (treatment), the interaction is made up of two estimates: the treatment effects for groups A versus B, and A versus C.
Heterogeneity
Heterogeneity of the treatment effect in a stratified Cox model can be formally assessed by fitting a model with a treatment-by-stratum interaction and comparing the two models using a likelihood ratio test (Tudur-Smith 2005). Typically in aggregate data meta-analysis we report an I-squared statistic and formally test for evidence of heterogeneity using a Q-test (Higgins et al. 2003). To obtain an I-squared statistic, one option is to implement a two-stage analysis, possibly as part of a sensitivity analysis. In our analyses we found the likelihood ratio tests to be very slightly more powerful than the Q-test. As mentioned previously, differences in estimates from the one-stage and two-stage methods were negligible in our case.
One particular issue we had when investigating heterogeneity was whether the treatment effect was related to the duration of the study. We approached this analysis using piecewise constant Cox models (still stratifying by study) with pieces corresponding to the 3 or 4 common study durations. This requires careful interpretation as with studies of varying duration we have studies dropping-out over time and so the pieces then represent a subset of studies over time. As a further sensitivity analysis we assessed the constancy of treatment effect in only the longest studies, using piecewise Cox models and then treatment-by-time interactions, but were left with few studies and low power to detect non-constant treatment effects. Study drop-out also impacts a Kaplan-Meier curve of all studies combined, and if this is required we recommend displaying the drop-out of studies over time as well as declining patient frequencies in a risk table.
References
Fibrinogen Studies Collaboration (2009) Systematically missing confounders in individual participant data meta-analysis of observational cohort studies. Statist. Med. 2009; 28:1218–1237
Fisher DJ, et al. (2011) A critical review of methods for the assessment of patient-level interactions in individual participant data meta-analysis of randomized trials, and guidance for practitioners. Journal of Clinical Epidemiology 64 (2011) 949e967
Higgins JPT, et al. (2003) Measuring inconsistency in meta-analyses. BMJ 2003;327:557–60
Simmonds MC, et al. (2005) Meta-analysis of individual patient data from randomized trials: a review of methods used in practice. Clinical Trials 2005; 2: 209–217
Stewart GB, et al. (2012) Statistical Analysis of Individual Participant Data Meta-Analyses: A Comparison of Methods and Recommendations for Practice. PLoS ONE 7(10): e46042. doi:10.1371/journal.pone.0046042
Tudur-Smith et al. (2005) Investigating heterogeneity in an individual patient data meta-analysis of time to event outcomes. Statist. Med. 2005; 24:1307–1319