The MaxCombo Test severely inflates the type 1 error rate

Log-rank test

We’ll use the excellent {nph} package by Ristl et al. Firstly, the standard log-rank test

library(nph)

# Data set 1
logrank.test(
  time  = dat$time,
  event = dat$event,
  group = dat$arm,
  rho   = c(0),
  gamma = c(0)
)

## Call:
## logrank.test(time = dat$time, event = dat$event, group = dat$arm, 
##     rho = c(0), gamma = c(0))
## 
##     N Observed Expected (O-E)^2/E (O-E)^2/V
## 1 162       84     72.9      1.68      3.19
## 2 166       71     82.1      1.49      3.19
## 
##  Chisq= 3.2  on 1 degrees of freedom, p= 0.07
##  rho   =  0 gamma =  0

So, for Data Set 1, the observed - expected number of events on arm 1 is 84 - 72.9, with Z-statistic sqrt(3.19)=1.786, one-sided p-value 1 - pnorm(1.786)=0.037, and two-sided p-value 0.07.

Now, for Data Set 2…

# Data set 2
logrank.test(
  time  = dat_new$time,
  event = dat_new$event,
  group = dat_new$arm,
  rho   = c(0),
  gamma = c(0)
)

## Call:
## logrank.test(time = dat_new$time, event = dat_new$event, group = dat_new$arm, 
##     rho = c(0), gamma = c(0))
## 
##     N Observed Expected (O-E)^2/E (O-E)^2/V
## 1 162       84       72      2.00      3.78
## 2 167       70       82      1.75      3.78
## 
##  Chisq= 3.8  on 1 degrees of freedom, p= 0.05
##  rho   =  0 gamma =  0

the observed - expected number of events is 84 - 72, with Z-statistic sqrt(3.78)=1.944, one-sided p-value 1 - pnorm(1.944)=0.026, and two-sided p-value 0.05.

This makes complete sense. The results in Data Set 1 are not better than in Data Set 2.

MaxCombo test

The version of the MaxCombo test recommended by Mukhopadhyay et al is the maximum of the standard log-rank test and the Fleming-Harrington-(0, 0.5) test. Let’s try it out…

# Data set 1
logrank.maxtest(
  time  = dat$time,
  event = dat$event,
  group = dat$arm,
  rho   = c(0, 0),
  gamma = c(0, 0.5)
)

## Call:
## logrank.maxtest(time = dat$time, event = dat$event, group = dat$arm, 
##     rho = c(0, 0), gamma = c(0, 0.5))
## 
##  Two sided p-value = 0.0289 (Bonferroni corrected: 0.0418)
## 
##  Individual weighted log-rank tests:
##   Test    z      p
## 1    1 1.79 0.0741
## 2    2 2.31 0.0209

For Data Set 1 the two-sided p-value is 0.0289. This is a multiplicity corrected p-value (since we are taking the maximum of two statistics), but we can see that it is driven by the large z-value z = 2.31 from the second statistic – the Fleming-Harrington-(0,0.5) statistic.

When we try this on Data Set 2…

# Data set 2
logrank.maxtest(
  time  = dat_new$time,
  event = dat_new$event,
  group = dat_new$arm,
  rho   = c(0, 0),
  gamma = c(0, 0.5)
)

## Call:
## logrank.maxtest(time = dat_new$time, event = dat_new$event, group = dat_new$arm, 
##     rho = c(0, 0), gamma = c(0, 0.5))
## 
##  Two sided p-value = 0.0457 (Bonferroni corrected: 0.0669)
## 
##  Individual weighted log-rank tests:
##   Test    z      p
## 1    1 1.94 0.0519
## 2    2 2.13 0.0335

… the two-sided p-value is 0.0457 driven by a Fleming-Harrington-(0,0.5) z-value of 2.13.

Hang on a second! We’ve agreed that the results in Data Set 1 are not ‘better’ than the results in Data Set 2. Yet the p-value is lower, 0.03 compared with 0.05.

Remember the context. Mukhopadhyay et al are using this as a case study for when the log-rank statistic is not statistically significant but the MaxCombo test is. However, we now see that this happens (partly) by the test statistic “rewarding” an early detrimental effect on survival.

What’s going on?

The Fleming-Harrington-(0,0.5) test gives extremely low weights to early events.

Remember that the weighted log-rank statistic is

Let’s plot the weights for the original data set, for which I’ll use {nphRCT}

library(nphRCT)

## At-risk table
at_risk <- nphRCT::find_at_risk(Surv(time, event) ~ arm, 
                                data = dat,
                                include_cens = FALSE)

## F-H-(0,0.5) weights
weights_fh <- nphRCT::find_weights(Surv(time, event) ~ arm, 
                                   data = dat,
                                   method = "fh",
                                   rho = 0,
                                   gamma = 0.5)


plot(at_risk$t_j, weights_fh, ylim = c(0,1),
     xlab = "Event time",
     ylab = "Weight",
     main = "Fleming-Harrington (0,0.5)")

# Standard log-rank test 
nphRCT::find_scores(Surv(time, event) ~ arm, 
             data = dat,
             method = "lr") %>% 
  plot() +
  ggtitle("Standard log-rank test")

# Fleming-Harrington (0,0.5)
nphRCT::find_scores(Surv(time, event) ~ arm, 
             data = dat,
             method = "fh",
             rho = 0,
             gamma = 0.5) %>% 
  plot() +
  ggtitle("Fleming-Harrington (0,0.5)")

Modestly-weighted tests

One possible approach that we favour is a modestly-weighted test. We won’t go into the full details again here, but essentially we can give later events higher weight than earlier events (notice that the first weight is 1 rather than 0)…

weights_mw_s <- nphRCT::find_weights(Surv(time, event) ~ arm, 
                                     data = dat,
                                     method = "mw",
                                     s_star = 0.5)


plot(at_risk$t_j, weights_mw_s, ylim = c(0,2),
     xlab = "Event time",
     ylab = "Weight",
     main = "Modestly weighted, s*=0.5")

while still ensuring that the scores are not increasing for early follow-up times…

# Modestly-weighted (s*=0.5)
nphRCT::find_scores(Surv(time, event) ~ arm, 
             data = dat,
             method = "mw",
             s_star = 0.5) %>% 
  plot() +
  ggtitle("Modestly-weighted (s*=0.5)")

References

• Mukhopadhyay, Pralay, Jiabu Ye, Keaven M. Anderson, Satrajit Roychoudhury, Eric H. Rubin, Susan Halabi, and Richard J. Chappell. “Log-Rank Test vs MaxCombo and Difference in Restricted Mean Survival Time Tests for Comparing Survival Under Nonproportional Hazards in Immuno-oncology Trials: A Systematic Review and Meta-analysis.” JAMA oncology (2022).

• Magirr, Dominic, and Carl‐Fredrik Burman. “Modestly weighted logrank tests.” Statistics in medicine 38, no. 20 (2019): 3782-3790.

• Magirr, Dominic, and Carl-Fredrik Burman. “The Strong Null Hypothesis and the MaxCombo Test: Comment on “Robust Design and Analysis of Clinical Trials with Nonproportional Hazards: A Straw Man Guidance form a Cross-Pharma Working Group.”.” Statistics in Biopharmaceutical Research (2021): 1-2.

• Ristl, Robin, Nicolás M. Ballarini, Heiko Götte, Armin Schüler, Martin Posch, and Franz König. “Delayed treatment effects, treatment switching and heterogeneous patient populations: How to design and analyze RCTs in oncology.” Pharmaceutical statistics 20, no. 1 (2021): 129-145.