Why we’re asking the wrong questions when it comes to COVID-19 hospitalisations

On 22nd October, PHASTAR’s Vice President for Statistical Research and Consultancy, Jennifer Rogers, appeared on BBC Radio 5 Live (Click Here for Radio 5 Live Interview which starts at 12:50) to discuss the latest COVID-19 statistics and one of the questions that she was asked considered the expected split of vaccinated versus unvaccinated people in hospital throughout the potential “winter wave”. Here Jen looks at hospitalisation data for vaccinated versus unvaccinated people in more detail and argues that the wrong questions are being asked.

On the 19th July, Sir Patrick Vallance announced that 60% of people in hospital from COVID-19 in the UK were not from double vaccinated people (after a brief bit of confusion when he mistakenly said in the press conference that 60% of people in hospital were double vaccinated). This means that 40% of people in hospital from COVID-19 were double vaccinated. Following this, we have seen the percentage of people in hospital being vaccinated versus unvaccinated be repeatedly reported as a useful statistic.

This 40% figure of hospitalised people being vaccinated seems to be at odds with the narrative we have heard consistently that vaccines break the link between cases and hospitalisations. If COVID-19 vaccines break the link between cases and hospitalisation, why have we, on occasion, seen approaching half of all COVID-19 hospitalisations being from people who are double vaccinated?

If we look at this data on its own, however, it is misleading and uninformative. It turns out that 40% of COVID-19 hospitalisations being from vaccinated individuals could actually be a good thing, as it means we’re doing a good job of vaccination, and here I explain why.

Let’s consider a population of 1 million and assume that 40% are fully vaccinated. First, we need to consider how many people we would expect to become infected with COVID-19, both fully vaccinated and unvaccinated. We can use estimated prevalence statistics to calculate how many would be expected to contract COVID-19 using the latest data from the Office for National Statistics Infection Survey.1 Prevalence has varied (due to lockdown measures, etc.) and ultimately it doesn’t really matter what number we choose, but the UK prevalence at the time of the first vaccine being administered was around 1%, so let’s take this as our chosen prevalence value for those who are unvaccinated.

Next, we need an estimate of how much vaccines affect an individual’s chances of catching COVID-19. Again, it doesn’t really matter what vaccine efficacy is for the purposes of this illustration, but let’s make it something vaguely accurate and science based. A publication in the Lancet that examined the impact of the Pfizer-BioNTech vaccine on COVID-19 cases, hospitalisation, and deaths in Israel found that the adjusted estimate of vaccine effectiveness at 7 days or longer after the second dose was 95·3% (95% CI 94·9–95·7) against COVID-19 infection.2 What does this mean? It means that for every 100 unvaccinated individuals who have COVID-19, we would only expect to see 4.7 COVID-19 cases (so around 5) if the same 100 individuals had been vaccinated.

What does this mean for our 1 million people? Figure 1 shows us how many people we would expect to be vaccinated and how many COVID-19 cases we would expect if vaccine take-up was 40%.

This means that out of a total of 6,188 COVID-19 cases, we would expect 188, or around 3% of them to be from vaccinated people, and 6,000, or around 97% to be from unvaccinated people.

And what about hospitalisation? The same Lancet publication stated that vaccine effectiveness against COVID-19-related hospitalisation was 97·2% (95% CI 96·8–97·5). This means that for every 100 unvaccinated people hospitalised for COVID-19, we would expect to see only 2.8 (approximately 3) vaccinated people hospitalised for COVID-19. Some of this protection is due to the fact that there are fewer COVID-19 cases among vaccinated individuals, but it turns out that COVID-19 vaccines are protective against COVID-19 infection and also protective against COVID-19 hospitalisation conditional on having COVID-19. If we consider 2.8/4.7 = 59.6%, this tells us that for every 100 unvaccinated people in hospital with COVID-19, if we took the same 100 people with COVID-19 but they were vaccinated, we expect only 40.4 of them to be hospitalised.

A previous PHASTAR blog examined the relationship between COVID-19 cases and hospitalisation and found that at the peak of the 2020/2021 winter wave, over 10% of COVID-19 cases resulted in hospitalisation.3 Again, it doesn’t really matter what number we choose here, but let’s say that 10% of unvaccinated COVID-19 cases result in hospitalisation.

Figure 2 shows us how many of our 1 million people we would expect to be hospitalised if we had 40% vaccination. We would expect a total of 611.2 hospitalisations with 11.2, or around 2% of them coming from vaccinated individuals, and 600, or around 98% from vaccinated individuals.

According to the official UK government COVID-19 daily update, almost 80% of the UK population have had two doses of the COVID-19 vaccine.4 So let’s consider how these figures might change if we instead had 80% vaccination.

Considering COVID-19 cases first. Figure 3 shows us that if we had 80% vaccination, we would expect a total of 2,376 COVID-19 cases from our 1 million individuals. Of these, we would expect 376, or around 16% of them to be from vaccinated individuals, and we would expect 2,000, or around 84% of them to be from unvaccinated individuals.

For COVID-19 hospitalisations, Figure 4 tells us that for our 1 million individuals we would expect to see a total of 222.4 hospitalisations. We would expect 22.4, or around 10% of these to be from vaccinated individuals, and 200, or 90% of these to be from unvaccinated individuals.

Let’s review those numbers again. When we had 40% vaccination, we expected 3% of cases and 2% of hospitalisations to be from vaccinated individuals. When we considered 80% vaccination, we expected 16% of cases and 10% of hospitalisations to be from vaccinated individuals. So, as we increased the proportion of the population that are vaccinated, we also increased the proportion of cases and hospitalisations coming from vaccinated individuals. You can see why it might be easy for people to then conclude that vaccines don’t seem to be working.

But statements like this miss out a crucial piece of information. So crucial, that I’m not sure an underline, bold, and italic quite do it justice! Statements like this miss out the overall number of COVID-19 cases and hospitalisations for the different vaccination levels. When we considered 40% vaccination, we expected to see a total of 6,188 COVID-19 cases and 611.1 hospitalisations from our 1 million individuals. Compare this to 80% vaccination, where we expected to see 2,376 COVID-19 cases and 222.4 hospitalisations from our 1 million people. So, whilst we saw the proportion of vaccinated individuals increase, the overall numbers decrease by almost two-thirds with 3,812 fewer expected COVID-19 cases and 388.7 fewer expected hospitalisations.

Statistics is full of these kinds of counterintuitive scenarios – take the Monty Hall problem and Simpson’s Paradox as prime examples. When faced with these somewhat confusing problems, I find it helpful to think of the extremes. We know that vaccines aren’t 100% effective, so even if the entire population was vaccinated, we still would expect to see some COVID-19 cases and some COVID-19 hospitalisations. But if everyone is vaccinated, then all COVID-19 cases and all COVID-19 hospitalisations would be in vaccinated people.

As the proportion of the population fully vaccinated against COVID-19 increases, we would expect to see the proportion of COVID-19 cases and hospitalisations coming from vaccinated individuals also increasing. But, crucially, you would expect to see far fewer overall COVID-19 cases and hospitalisations than if you hadn’t vaccinated anyone.

Really, all these statements about the proportion vaccinated versus unvaccinated do is give us an insight into what proportion of the population might be vaccinated. Like some extravagant proxy for vaccine take up. They don’t really tell us much, on face value, about the performance of vaccines.

1.https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/datasets/coronaviruscovid19infectionsurveydata - downloaded 12 November 2021

  1. EJ Haas, FJ Angulo, JM McLaughlin, E Anis, SR Singer, F Khan, N Brooks, M Smaja, G Mircus, K Pan, J Southern, DL Swerdlow, L Jodar, Y Levy, S Alroy-Preis. Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data. The Lancet, 2021. 397(10287):1819-1829.
  2. https://phastar.com/blog/276-covid-19-what-does-the-latest-surge-in-numbers-mean-and-what-is-the-impact-of-vaccinations
  3. https://coronavirus.data.gov.uk/ - 12 November 2021