Integrated summary of effectiveness (ISE) - statistical challenges, insights & opportunities

Any single clinical study can present numerous challenges and the same can certainly be true for an Integrated Summary of Effectiveness (ISE). An ISE takes several clinical studies and pulls (or pools) them together to present a coherent and hopefully compelling story of a drug or treatment’s effectiveness. Importantly, an ISE is also considered important for explaining the differences in results between the individual studies. Why might one study have been more or less effective than another – is it because of the differences in doses between studies or perhaps the subset of the target population that was included? From a statistical perspective, an ISE might be able to answer interesting questions that not only provide a better understanding of the treatment’s efficacy overall, but the greater power of pooled analyses may offer additional insight into the nature of the treatment’s effectiveness in demographic and other subpopulations.

In an ideal world, combining studies for an ISE might be straightforward, perhaps even with the same study structure or data metrics collected throughout all the included studies. However, in reality, each clinical study that is included in an ISE may have been designed with different objectives in mind. This can result in several factors differing between studies that can make it difficult to integrate all of the data into one summary. The study designs may be set up in different ways, for example it is possible that data from both crossover and parallel-arm designs are available. Studies may have different entry requirements based on disease severity, prior treatment, or other demographic considerations. Treatment periods and assessment schedules for the studies will likely differ making direct comparisons at different visits difficult. Different treatment doses may also be used on different studies. In a recent ISE study that PHASTAR undertook, all of these differences were present, in addition to these differences, the metrics collected, baseline definitions and times at which endpoint metrics were collected also differed. In this case the data were not pooled and instead the statistical analysis methods were carefully considered to ensure the best comparison of the data could be made. When it is possible to pool data, meta-analytic methods may be appropriate.

As already discussed, within an ISE the studies are likely to have different goals in mind, one may be interested in the increase of symptoms after withdrawal of treatment, whereas others may be interested in the decrease of symptoms during treatment. By setting criteria so that each subject either meets or does not meet the predetermined metric it is possible to demonstrate for each study whether certain goals were met, even if these goals were diametrically opposed. A set of outputs showing the results from this example scenario could show both symptom improvement with the treatment and the persistence of this improvement after removal of treatment.

Another issue that can be resolved with the appropriate use of statistical analysis is when the endpoints of a study are recorded at different timepoints. If some studies within the ISE only record the appropriate metric at baseline and an end of study visit, while others record the metric weekly, the same statistical method may not be appropriate for both. A simple pooled analysis using a judiciously selected common timepoint across studies may be informative. Such analyses can enable a clearer comparison of studies within the ISE, painting a more complete picture of the results. It should be noted however that such analyses should be considered as exploratory only, since they are not typically pre-planned.

While an ISE project can be challenging, as data are rarely as straightforward as desired, the ISE can provide new insights into treatments that truly demonstrate the treatment’s efficacy in a way that an individual trial cannot hope to do. That is one of the reasons why the FDA requires ISE studies as part of a New Drug Application (NDA) [1]. A wider diversity of data, design and endpoints requires extra consideration to ensure that appropriate methods are being used not only for the individual study data but to answer the questions of the ISE as a whole. Although challenging to construct, the ISE is the best source of effectiveness information that can be used to identify strengths
and weaknesses of a treatment and inform appropriate decisions about drug approval for the benefit of patients.

[1] Guidance for Industry: Integrated Summary of Effectiveness, 2015.