The COVID impact on clinical trials requires new approaches for statisticians

COVID-19 and its subsequent variants have provided challenges in many ways, not the least of which are in the conduct and management of clinical trials. Global quarantines and disruptions in investigational product supply, patient recruitment and sustainability have been major concerns in an era when it’s more important than ever to proceed with drug development. But how can we ensure that clinical trials can continue with quality data?  We review how some of these challenges can be overcome now and, in the future, to make sure that quality data can be collected and analyzed.  

Now into our second year of facing these challenges, clinical trial sponsors, managers and regulators will have to even more closely collaborate and share learnings that will be critical for the long-term, particularly in the use of different methods and assessments for the collection of end points. Additional data, such as reasons for missed/delayed visits/assessments, will also need to be collected in the clinical database, as it is important to capture all COVID-19-related data in current trials.   

The range of disruptions of COVID-19 to clinical trials have been so diverse that no single solution will be appropriate for all trials. Although there are many common issues, each trial must be assessed on a case-by-case basis.   

This article will examine some best practices to assure that the trial can continue during the ongoing pandemic – including current vaccine trials – so that the development of treatments and medicines isn’t interrupted. 

 Assessing the Risks 

A risk assessment is the first step in determining the impact to the trial and if the trial can still meet its planned objectives while protecting patient safety and statistical integrity. In blinded trials, the risk assessment should be performed on blinded data. This is an ongoing process throughout the trial, allowing contingency measures/mitigation to be reviewed on a regular basis.  

Should contingency measures be necessary, any changes made to the trial in the protocol and the statistical analysis plan (SAP) must be reported and the impact of COVID-19 will need to be addressed in the clinical study report (CSR). If changes are necessary, it’s best to bring regulatory agencies into the discussion as early as possible.  

Depending on the type of trial, its design, and the stage at which it has progressed, the main questions for the assessment should include (but are not limited to): 

  • Are design modifications necessary? 
  • Are trial participants affected by the pandemic and if so, how? 
  • Has the trial population changed? 
  • Does additional data need to be collected? 
  • Are there any treatment or study discontinuations and if so, why? 
  • Is there missing data and if so, why? 
  • Are the initial assumptions and objectives of the trial still viable? 
  • Are alternative methods of data collection required and if so, how will this impact the trial?  
  • Do the statistical analyses need to be modified?  
  • Are sensitivity analyses required?  

Contingency Measures and Mitigation 

Stopping a trial as a possible contingency measure could do more harm than good, particularly if participants are benefitting from treatment, or if there are no other treatment options. Consideration should also be given to availability of treatment supply and distribution, or the collection of lab samples. The key goal throughout is to minimize the impact of any of these measures on the outcome of the trial.  

If interim analyses were planned to be conducted during the trial, required and acceptable quality data must be available to conduct the interim analysis. Otherwise, it may be necessary to determine the risks of proceeding with the interim analysis with a large amount of missing data or simply delaying or not proceeding with the planned interim analysis. If data is sub-par, discussion should be had regarding increasing the sample size or extending the trial follow-up period. 

Considering Estimands 

Prior to the pandemic, for most trials, the primary interest has been simply Drug A versus Drug B. However, there could be secondary interest in Drug A versus Drug B in the presence of individual COVID-19 infections. Therefore, it is necessary to think about the impact on the treatment effects, which leads to consideration of estimands.  

The estimands framework is based on linking the trial objectives to the treatment effects of interest (which is specified through the estimand), which helps inform the trial design, the data to be collected, and the planned methods of estimation. 

The estimands framework includes five main areas for consideration:  

  • The treatment condition of interest should remain the same as originally intended, even if there are operational and logistical issues with study drug supply/delivery. Any pandemic-related issues with adherence to study drug and concomitant medications should be considered as intercurrent events. 
  • Population should remain as originally intended to avoid deviation from the objective of the trial. However, considerations may occur that address the potential effect of COVID-19 vaccines if they become available at different times in a multi-region study. 
  • If there are alternative data methods employed, they should be assessed for their impact on endpoint variability to make sure the endpoint is not compromised.  
  • Population-level summaries between treatments should remain as originally intended. There may be rare cases where a population-level summary may need to be changed. For example, looking at COVID-19 exposed participants versus non-exposed participants or participants who have received a COVID-19 vaccine versus those who have not. 
  • Only pandemic-related intercurrent events should be considered and categorized based on study treatment adherence or the ability to assess target outcome. Non-pandemic-related intercurrent events should continue to be handled as originally planned. 

Intercurrent Events  

Only pandemic-related intercurrent events should be considered and categorized according to their impact on study treatment adherence such as discontinuation of study treatment or death (not an expected outcome). These events then should be further categorized according to pandemic-related factors. Non-pandemic-related intercurrent events should continue to be handled as originally planned. Strategies employed include: 

  • Treatment policy strategy: under this strategy, whether an intercurrent event has occurred or not is considered irrelevant in defining the treatment effect and the data is still analyzed regardless. Under this strategy, the trial conclusions would not be generalizable post-pandemic and thus would not be an appropriate method for most pandemic-related intercurrent events for the majority of trials. 
  • Composite variable strategy: under this strategy, the occurrence of an intercurrent event is incorporated into the definition of the endpoint. This is also unlikely to be an appropriate method for dealing with most pandemic-related intercurrent events. 
  • Principal stratification strategy: for this strategy, the aim is to estimate the treatment effect within the stratum of participants for whom the intercurrent event did not occur. Stratifying on a pandemic-related event is unlikely to be appropriate since this limits the trial results and conclusions to a specific sub-population of participants applicable during the COVID-19 pandemic only, rather than a future post-pandemic perspective. Furthermore, it is not possible to define the stratum of participants in advance since it is impossible to know exactly which participants will have the intercurrent event. 
  • While-on-treatment strategy: this strategy considers measurements of the endpoint up until the time of the intercurrent event. If the endpoint is measured at multiple time points during the trial, this strategy may still be relevant if it was originally planned for non-pandemic-related intercurrent events. 
  • Hypothetical strategy: Under this strategy, the interest is in estimating the treatment effect in the hypothetical situation where the intercurrent event did not occur. This strategy is the most logical and the natural choice of method for handling most pandemic-related intercurrent events. 

These strategies highlight two key points. First, that it is critical to collect additional data associated with pandemic-related factors in order to assess intercurrent events as pandemic- or non-pandemic related and to select appropriate strategies for handling them. Secondly, trials will handle an intercurrent event of death due to COVID-19 in different ways depending on the type of disease and the endpoint of the trial. 

Missing Data Caused by COVID-19 

It is generally accepted that there will be an increased amount of missing data in most clinical trials due to the COVID-19 pandemic, thus it should be anticipated. Missing data and premature termination are not new to statisticians and well-established methods exist on how to deal with these issues, but the disruptions caused by the pandemic are expected to be so diverse that no single solution will be appropriate for all trials. 

As per regulatory guidance, all missed visits and assessments need to be captured and summarized in the tables, figures, and listings (TFLs) and CSR. Assessment of the reasons for the missing data, the impact of missed visits on a case-by-case basis and for each endpoint in the trial needs to be included.  

The extent of missing data and its impact on the trial needs to be assessed as part of the risk assessment process. It is likely that the mechanism of missingness caused by the pandemic will be random (either missing completely at random or missing at random) but it will still be necessary to understand the source of the missing data. An imbalance of missingness between treatment arms can be of concern and may require an IDMC (International Data Monitoring Committee) to investigate the imbalance. Since trial integrity is of primary concern, any data monitoring should be on baseline and safety data rather than efficacy data in order to avoid inflating the Type I error. 

Blinded reviews of the data should be carried out to further explore the missing data and to determine appropriate methods of handling it in the statistical analyses. If there is a significant amount of missing data, it may be necessary to consider replacing participants and re-estimating the sample size. Methods for handling missing data should be proposed and fully documented in the SAP along with proposed plans for sensitivity and supplementary analyses. Any changes that are implemented need to be applied to all treatment arms equally and any amendments to the SAP need to be completed prior to any database locks (interim or final). Strategies for dealing with the missing data should be considered with the estimands strategy. 

Once the trial has been unblinded (and for open-label studies), summaries should be produced to determine patterns in the missing data. These summaries should be overall as well as by treatment group. It is expected that for most pandemic-related missing data, there would be no differences between the treatment groups. 

With more missing data, the assumptions used can have greater leverage over the results, so it is important to pay extra special attention to method assumptions, thus it is necessary to do a range of sensitivity analyses and supportive analyses, including methods to evaluate the missing data mechanism, the impact of alternative endpoint measurements, and the impact of intercurrent events.  

In Summary 

It is clear that COVID-19 and its lingering variants will continue to have an impact on clinical trial methodologies and will require creative strategies to ensure the efficacy of trial data and patient safety. There is no exhaustive list of the possible effects on clinical trials and there are no fixed rules on how each effect should be handled. It will vary greatly from one trial to another.  

However, in terms of the statistical impact, standard processes are well-equipped to deal with almost anything. The important point is to carefully assess the effect of the COVID-19 pandemic on each aspect of the trial and then to decide on the appropriate methods of handling the issues, while maintaining the quality and integrity of both the clinical trial and the trial data, with the aim of ensuring that the main aim of the trial can still be met.