- Journal List
- HHS Author Manuscripts
- PMC4886856

As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsem*nt of, or agreement with, the contents by NLM or the National Institutes of Health.

Learn more: PMC Disclaimer | PMC Copyright Notice

Lifetime Data Anal. Author manuscript; available in PMC 2017 Apr 1.

*Published in final edited form as:*

Lifetime Data Anal. 2016 Apr; 22(2): 263–279.

Published online 2015 May 29. doi:10.1007/s10985-015-9330-3

PMCID: PMC4886856

NIHMSID: NIHMS783569

PMID: 26022558

Glenn Heller^{1,}^{*} and Qianxing Mo^{2}

Author information Copyright and License information PMC Disclaimer

The publisher's final edited version of this article is available at Lifetime Data Anal

## Abstract

A clinical risk classification system is an important component of a treatment decision algorithm. A measure used to assess the strength of a risk classification system is discrimination, and when the outcome is survival time, the most commonly applied global measure of discrimination is the concordance probability. The concordance probability represents the pairwise probability of lower patient risk given longer survival time. The c-index and the concordance probability estimate have been used to estimate the concordance probability when patient-specific risk scores are continuous. In the current paper, the concordance probability estimate and an inverse probability censoring weighted c-index are modified to account for discrete risk scores. Simulations are generated to assess the finite sample properties of the concordance probability estimate and the weighted c-index. An application of these measures of discriminatory power to a metastatic prostate cancer risk classification system is examined.

**Keywords: **C-index, Concordance probability estimate, Discrimination, Inverse probability censoring weight, Risk classification

## 1. Introduction

Treatment decisions are based on prognosis derived from a set of molecular, pathological, and histological patient factors. A treatment decision algorithm is often determined from a reduced set of important prognostic markers that are categorized to create a risk classification system. The quality of the treatment decision algorithm is tied to the strength of the risk classification.

Discrimination is an important metric in assessing the strength of a risk classification system in survival analysis. When the endpoint is survival time, good discrimination implies that the ordinality of the risk groups predicts the ranking of survival times. If a risk group classification can be constructed to confidently delineate long-term survivors from short-term survivors, then it will provide utility in clinical decision making.

The most widely used metric for the global assessment of discrimination is the concordance probability. Its popularity stems from its equivalence to the area under the receiver operating curve when the outcome is binary, and its relation to Kendall’s tau and the Goodman-Kruskal gamma for continuous outcomes (Sprent 1989). For survival outcomes, the concordance probability is defined for two hypothetical subjects as Pr[*R*_{2} > *R*_{1}|*T*_{1} > *T*_{2}], where *R* represents the risk score and *T* is survival time. The patient-specific risk scores represent the discrete risk groups and may be determined through a regression model via the linear combination **$\widehat{\beta}$**^{T}* X*, with

*as a dummy variable. If there are*

**X***p*risk groups, the

*k*component of the dummy variable

^{th}*takes the value 1 if the patient is in group*

**X***k*(for

*k*= 1, …,

*p*− 1) and all other components of

*are zero. In this paper it is assumed that the proportional hazards model*

**X***h*(

*t*|

*) =*

**X***h*

_{0}(

*t*) exp(

**β**

^{T}

*) was used to determine patient risk. The application of this model requires that the adequacy of the proportional hazards assumptions have been empirically tested and that the model assumptions cannot be rejected. Two common methods used to test the proportional hazards assumption are developed in Lin et al. (1993) and Grambsch and Therneau (1994).*

**X**The outline of this paper is as follows. In Section 2, two previously developed estimates of the concordance probability are presented for continuous risk scores. The estimates, the weighted c-index and the concordance probability estimate (CPE), have derived asymptotic distributions with continuous risk scores. In Section 3, these estimates are adapted for discrete risk scores. The asymptotic distributions of the CPE and the weighted c-index with discrete risk scores are presented. An analytic expression for the asymptotic variance of the CPE is derived. For the weighted c-index, a resampling based estimate of the asymptotic variance is utilized. Simulations are presented in Section 4 to examine the finite sample properties of the estimates. The development of a risk group classification system in prostate cancer is used illustrate the methods in Section 5 and further discussion points are provided in Section 6.

## 2. Concordance probability with continuous risk scores

It is assumed that that a correctly specified proportional hazards model was used to develop the risk score and that larger values of **β**^{T}* X* represent a greater risk of death. The concordance probability, derived from this regression model, is

*𝒞* =Pr(*β*^{T}*X*_{1}>*β*^{T}*X*_{2}|*T*_{2}>*T*_{1}).

An early estimate of the concordance probability, adapted for survival analysis, is the c-index (Harrell et al. 1982)

$$\frac{{\sum}_{i}{\sum}_{j}I({y}_{i}<{y}_{j})I({\delta}_{i}=1)I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}>{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})}{{\sum}_{i}{\sum}_{j}I({y}_{i}<{y}_{j})I({\delta}_{i}=1)},$$

where *y* is the minimum of the survival time *t* and censoring time *u*, δ = *I*(*t* ≤ *u*), and **$\widehat{\beta}$**^{T}* X* is the estimated risk score from the regression model. Although this has been the predominant discrimination statistic used in survival analysis, it has been observed that the statistic is biased and its distribution is a function of the censoring distribution (Gönen and Heller 2005, Uno et al. 2011). Indeed, the limiting value of Harrell’s c-index is Pr[

**β**

^{T}

**X**_{1}>

**β**

^{T}

**X**_{2}|

*T*

_{1}<

*T*

_{2},

*T*

_{1}<

*U*

_{1}∧

*U*

_{2}]. Thus, the c-index has the undesirable property that it is influenced by the rate of patient accrual and the length of the study.

To address this limitation, Uno et al. (2011) introduced inverse probability censoring weights to the c-index. The inverse probability censoring weights are incorporated to produce a consistent estimate of the concordance probability when the support of the survival distribution is less than the support of the censoring distribution. The weighted c-index is computed as

$$\frac{{\sum}_{i}{\sum}_{j}I({y}_{i}<{y}_{j})I({\delta}_{i}=1)I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}>{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j}){\{\u011c({y}_{i})\}}^{-2}}{{\sum}_{i}{\sum}_{j}I({y}_{i}<{y}_{j})I({\delta}_{i}=1){\{\u011c({y}_{i})\}}^{-2}},$$

where *Ĝ*(*y*) is the Kaplan-Meier estimated survival function for the censoring time random variable. For this weighted c-index, it is assumed that the censoring random variable is independent of the risk covariates. An extension developed by modeling the censoring distribution as a function of the covariates was proposed by Gerds et al. (2013). In general, measures based on inverse probability censoring weights require care as they may be sensitive to late failure times.

If {*T _{i}, X_{i}*} are continuous, independent identically distributed random variables, an application of Bayes theorem shows that the concordance probability is also equal to Pr[

*T*

_{2}>

*T*

_{1}|

**β**

^{T}

**X**_{1}>

**β**

^{T}

**X**_{2}]. This form of the concordance probability is used to derive an alternative estimate (Gönen and Heller 2005), based on the proportional hazards specification

*m*(*T*) =*β*^{T}** X** +

*ε*,

where ε is a standard extreme value random variable and *m* is a monotone function (Kalbfleisch 1978). It follows that

$$\text{Pr}({T}_{2}>{T}_{1}|{\mathit{\beta}}^{T}{\mathit{X}}_{1}>{\mathit{\beta}}^{T}{\mathit{X}}_{2})=\phantom{\rule{0ex}{0ex}}\frac{\underset{{\mathit{\beta}}^{T}{\mathit{X}}_{1}>{\mathit{\beta}}^{T}{\mathit{X}}_{2}}{\int \int}{[1+\text{exp}\{{\mathit{\beta}}^{T}({\mathit{X}}_{2}-{\mathit{X}}_{1})\}]}^{-1}dF({\mathit{\beta}}^{T}{\mathit{X}}_{1})dF({\mathit{\beta}}^{T}{\mathit{X}}_{2})}{\underset{{\mathit{\beta}}^{T}{\mathit{X}}_{1}>{\mathit{\beta}}^{T}{\mathit{X}}_{2}}{\int \int}dF({\mathit{\beta}}^{T}{\mathit{X}}_{1})dF({\mathit{\beta}}^{T}{\mathit{X}}_{2})}$$

where *F* is the distribution function of the risk score **β**^{T}* X*. The concordance probability estimate (CPE) is attained by substituting the partial likelihood estimate for

**β**and the empirical distribution function for

*F*

$$\frac{2}{n(n-1)}\underset{i<j}{\sum \sum}\{\frac{I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{ji}<0)}{1+\text{exp}({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{ji})}+\frac{I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{ij}<0)}{1+\text{exp}({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{ij})}\},$$

where * X_{ji}* =

*−*

**X**_{j}*.*

**X**_{i}## 3. Concordance probability with discrete risk scores

The concordance probability is adapted for use with discrete risk scores. In general, the creation of risk groups induces a large number of ties in the risk scores. For *p* groups, with *n _{k}* subjects in group

*k*, the proportion of subject pairs in the same group is

$$\frac{{\sum}_{k=1}^{p}{n}_{k}({n}_{k}-1)}{({\sum}_{k=1}^{p}{n}_{k})[({\sum}_{k=1}^{p}{n}_{k})-1]}.$$

Thus for example, a 200 patient study with 50 patients in each of 4 risk groups, nets approximately 25% of the patient pair comparisons from the same group.

### 3.1 Concordance parameters

There are two paths to modify the concordance probability 𝒞 for discrete risk scores. The first is to acknowledge the ambiguity resulting from the inclusion of ties in the risk scores

*𝒞*_{I} =Pr(*β*^{T}*X*_{1}>*β*^{T}*X*_{2}|*T*_{2}>*T*_{1}) +[0.5×Pr(*β*^{T}*X*_{1} =*β*^{T}*X*_{2}|*T*_{2}>*T*_{1})],

and the second is to exclude patient pairs with the same risk score

*𝒞*_{E} =Pr(*β*^{T}*X*_{1}>*β*^{T}*X*_{2}|*T*_{2}>*T*_{1},*β*^{T}*X*_{1}≠*β*^{T}*X*_{2}).

The concordance measures 𝒞_{I} and 𝒞_{E} are the survival analogs of Kendall’s tau-a and Kendall’s tau-b, respectively. Yan and Greene (2008) demonstrate that 𝒞_{I} is a convex combination of 𝒞_{E} and 0.5

*𝒞*_{I} =[*𝒞*_{E}×{1−Pr(*β*^{T}*X*_{1} =*β*^{T}*X*_{2})}] +[0.5×Pr(*β*^{T}*X*_{1} =*β*^{T}*X*_{2})].

The parameter 𝒞_{E} is a common concordance measure which ranges from 0.5 to 1.0. In contrast, the inclusion of ties attenuates 𝒞_{I} toward 0.5, with its upper bound equal to 1−0.5 × Pr(**β**^{T}**X**_{1} = **β**^{T}**X**_{2}). Thus, the use of 𝒞_{I} penalizes the concordance measure, with fewer risk groups incurring a greater penalty.

It is assumed that **$\widehat{\beta}$**^{T}**X**_{1} ≠ **$\widehat{\beta}$**^{T}**X**_{2} unless **X**_{1} = **X**_{2} (patient pairs within the same risk group). Under this assumption, conditional on * X*, the asymptotic distribution of the estimate of 𝒞

_{I}follows immediately from the asymptotic distribution of the estimated 𝒞

_{E}. As a result, the focus in this paper is on the statistical properties of concordance estimates based on 𝒞

_{E}.

### 3.2 Concordance parameter estimates

The weighted c-index adapted for discrete risk scores is

$${C}_{n,E}(\hat{\mathit{\beta}})=\frac{{\sum}_{i}{\sum}_{j}I({y}_{i}<{y}_{j})I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}>{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})I({\delta}_{i}=1){\{\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i})\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})\}}^{-1}}{{\sum}_{i}{\sum}_{j}I({y}_{i}<{y}_{j})I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}\ne {\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})I({\delta}_{i}=1){\{\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i})\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})\}}^{-1}}.$$

This estimate explicitly excludes ties in the risk scores and has been modified to account for possibly unequal follow-up between risk groups, using *Ĝ*(*y*|**$\widehat{\beta}$**^{T}* X*) as the within group Kaplan-Meier survival function for the censoring time random variable. The weighted c-index is a consistent estimate of 𝒞

_{E}when the right support of the survival distribution is less than the right support of the censoring distribution. If the support of the survival distribution exceeds the support of the censoring distribution, then the estimate is stabilized by truncating the follow-up time at τ so that

*G*(τ|

**β**

^{T}

*) > 0. For the weighted c-index, this is incorporated into the statistic as*

**X**$${C}_{n,E}(\hat{\mathit{\beta}};\mathit{\tau})=\phantom{\rule{0ex}{0ex}}\frac{{\sum}_{i}{\sum}_{j}I({y}_{i}<{y}_{j})I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}>{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})I({\delta}_{i}=1)I({y}_{i}<{\tau}_{i}){\{\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i})\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})\}}^{-1}}{{\sum}_{i}{\sum}_{j}I({y}_{i}<{y}_{j})I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}\ne {\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})I({\delta}_{i}=1)I({y}_{i}<{\tau}_{i}){\{\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i})\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})\}}^{-1}}$$

where τ_{i} represents the truncation time within the *i ^{th}* subject’s risk group.

The asymptotic normality of the weighted c-index for continuous risk scores was demonstrated in Uno et al. (2011). For discrete risk scores, the asymptotic normality is derived in the appendix. This extension follows directly from the work in Cheng et al. (1995) and Fine et al. (1998). Estimation of the asymptotic variance, however, requires density estimation for the within group censoring random variable. To avoid kernel smoothing and bandwidth selection, a stratified by risk group bootstrap standard error estimate of the weighted c-index in the simulations. An alternative resampling approach was used in Uno et al (2011).

The CPE with discrete risk scores is

$${K}_{n,E}(\hat{\mathit{\beta}})=\frac{{\sum}_{i}{\sum}_{j}I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}>{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j}){[1+\text{exp}({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{ji})]}^{-1}}{{\sum}_{i}{\sum}_{j}I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}>{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})}.$$

The CPE is a consistent estimate of 𝒞_{E} when the proportional hazards specification is correct. The consistency and the asymptotic distribution of the CPE, including its asymptotic variance, are derived in the appendix.

#### Theorem

Under the standard conditions for the proportional hazards model, *n*^{1/2}[*K _{n,E}*(

**$\widehat{\beta}$**) − 𝒞

_{E}] is asymptotically normal with mean 0.

The CPE is a model based measure of concordance and therefore relies on the correctness of the proportional hazards model. The advantage of using the CPE is that it does not require inverse probability censoring weights, which may be sensitive to late failures. Use of the CPE provides additional insight into the discriminatory power of a risk classification system derived with *p* risk groups by noting that the CPE is a weighted average of pairwise probabilities, with each element representing the probability the lower risk patient lives longer

$${K}_{n,E}(\hat{\mathit{\beta}})=\frac{{\sum}_{k=1}^{p}{\sum}_{l=1}^{p}I[{\hat{\beta}}_{k}>{\hat{\beta}}_{l}]{n}_{k}{n}_{l}{[1+\text{exp}\{{\hat{\beta}}_{l}-{\hat{\beta}}_{k}\}]}^{-1}}{{\sum}_{k=1}^{p}{\sum}_{l=1}^{p}I[{\hat{\beta}}_{k}>{\hat{\beta}}_{l}]{n}_{k}{n}_{l}}.$$

Thus, in addition to computing the overall CPE, we can extract the individual probability [1+exp{$\widehat{\beta}$_{l} − $\widehat{\beta}$_{k}}]^{−1}, that a patient in risk group *k* lives longer than a patient in risk group *l*.

## 4. Simulations

Simulation experiments were conducted to examine the accuracy of the CPE and weighted c-index for a four group risk classification system. The data were generated from Weibull and lognormal regression models. For the Weibull model the survival times were produced from

*T*_{i} =exp[−0.5*x*_{1i}−0.25*x*_{2i}−0.10*x*_{3i}]×*ε*_{i}

where * X* = (

*x*

_{1},

*x*

_{2},

*x*

_{3})

^{T}are dummy variables. The errors {ε

_{i}} were independent identically distributed Weibull random variables with scale parameter 1 and shape parameters {1.85, 4.1, 7.3, 13.5}, which were chosen to produce concordance probabilities equal to {0.6, 0.7, 0.8, 0.9}. The Weibull regression model satisfies the proportional hazards assumption. For the lognormal model, the log survival times were created using the same covariates and regression parameters. The log of the error random variables {log ε

_{i}} were independent identically distributed normal random variables with mean 0 and scale parameters {0.62, 0.28, 0.16, 0.09}, which were again chosen to produce concordance probabilities equal to {0.6, 0.7, 0.8, 0.9}. The lognormal simulations were constructed to examine the robustness of the CPE and the weighted c-index statistics when the regression coefficients were computed using the proportional hazards model but the data do not satisfy the proportional hazards assumptions.

For each group, independent uniform censoring times (0, *M _{k}*), (

*k*= 1, 2, 3, 4), were generated to determine the overall proportion censored. The sample size for each simulation was 200 and the proportion of patients in the four groups was {0.1, 0.2, 0.3, 0.4}. One thousand simulations were run in each setting to compute the weighted c-index excluding ties (

*C*), and the CPE excluding ties (

_{n,E}*K*). The results of the simulations are summarized in Tables 1–3.

_{n,E}### Table 1

Proportional hazards data

CP | Prop cen | C_{n,E}avg | K_{n,E}avg | C_{n,E}sim se | K_{n,E}sim se | se(C)_{n,E}avg | se(K)_{n,E}avg | cov(C)_{n,E} | cov(K)_{n,E} | RE |
---|---|---|---|---|---|---|---|---|---|---|

0.6 | 0.00 | 0.602 | 0.605 | 0.028 | 0.025 | 0.028 | 0.027 | 0.953 | 0.958 | 1.212 |

0.6 | 0.25 | 0.606 | 0.605 | 0.031 | 0.029 | 0.030 | 0.032 | 0.938 | 0.956 | 1.151 |

0.6 | 0.50 | 0.607 | 0.609 | 0.037 | 0.036 | 0.037 | 0.041 | 0.927 | 0.958 | 1.030 |

0.6 | 0.75 | 0.618 | 0.624 | 0.054 | 0.046 | 0.056 | 0.058 | 0.929 | 0.941 | 1.204 |

0.7 | 0.00 | 0.700 | 0.702 | 0.026 | 0.022 | 0.025 | 0.024 | 0.944 | 0.958 | 1.385 |

0.7 | 0.24 | 0.701 | 0.703 | 0.027 | 0.026 | 0.029 | 0.028 | 0.955 | 0.963 | 1.066 |

0.7 | 0.49 | 0.698 | 0.702 | 0.035 | 0.034 | 0.035 | 0.035 | 0.935 | 0.955 | 1.059 |

0.7 | 0.75 | 0.693 | 0.701 | 0.053 | 0.046 | 0.052 | 0.051 | 0.929 | 0.955 | 1.350 |

0.8 | 0.50 | 0.798 | 0.800 | 0.021 | 0.019 | 0.021 | 0.020 | 0.937 | 0.963 | 1.233 |

0.8 | 0.24 | 0.800 | 0.800 | 0.023 | 0.022 | 0.023 | 0.023 | 0.949 | 0.953 | 1.093 |

0.8 | 0.50 | 0.797 | 0.801 | 0.029 | 0.027 | 0.030 | 0.028 | 0.944 | 0.947 | 1.164 |

0.8 | 0.75 | 0.784 | 0.797 | 0.045 | 0.040 | 0.045 | 0.041 | 0.950 | 0.946 | 1.418 |

0.9 | 0.00 | 0.900 | 0.900 | 0.015 | 0.013 | 0.015 | 0.014 | 0.928 | 0.949 | 1.331 |

0.9 | 0.24 | 0.899 | 0.901 | 0.016 | 0.014 | 0.016 | 0.015 | 0.939 | 0.957 | 1.305 |

0.9 | 0.50 | 0.900 | 0.901 | 0.020 | 0.018 | 0.020 | 0.019 | 0.936 | 0.940 | 1.231 |

0.9 | 0.75 | 0.894 | 0.899 | 0.031 | 0.027 | 0.031 | 0.026 | 0.942 | 0.928 | 1.366 |

Open in a separate window

CP = Concordance Probability; Prop cen = proportion censored

*C _{n,E}* = Weighted c-index excluding ties;

*K*= CPE excluding ties

_{n,E}avg = average; sim se = simulation standard error; cov = coverage

RE = Mean square error efficiency of the CPE relative to the weighted c-index

### Table 3

Log normal survival time data

CP | Prop cen | C_{n,E}avg | K_{n,E}avg | C_{n,E}sim se | K_{n,E}sim se | se(C)_{n,E}avg | se(K)_{n,E}avg | cov(C)_{n,E} | cov(K)_{n,E} |
---|---|---|---|---|---|---|---|---|---|

0.6 | 0.00 | 0.602 | 0.590 | 0.028 | 0.026 | 0.030 | 0.027 | 0.953 | 0.940 |

0.6 | 0.24 | 0.605 | 0.601 | 0.032 | 0.028 | 0.031 | 0.032 | 0.934 | 0.975 |

0.6 | 0.50 | 0.612 | 0.614 | 0.037 | 0.034 | 0.037 | 0.040 | 0.920 | 0.950 |

0.6 | 0.75 | 0.629 | 0.641 | 0.055 | 0.048 | 0.055 | 0.057 | 0.903 | 0.873 |

0.7 | 0.00 | 0.699 | 0.675 | 0.028 | 0.026 | 0.027 | 0.024 | 0.931 | 0.836 |

0.7 | 0.25 | 0.701 | 0.686 | 0.031 | 0.030 | 0.031 | 0.028 | 0.934 | 0.916 |

0.7 | 0.50 | 0.700 | 0.700 | 0.038 | 0.035 | 0.037 | 0.035 | 0.946 | 0.942 |

0.7 | 0.75 | 0.705 | 0.726 | 0.056 | 0.045 | 0.052 | 0.049 | 0.911 | 0.906 |

0.8 | 0.00 | 0.800 | 0.769 | 0.020 | 0.023 | 0.021 | 0.021 | 0.950 | 0.691 |

0.8 | 0.24 | 0.801 | 0.779 | 0.024 | 0.026 | 0.025 | 0.023 | 0.948 | 0.829 |

0.8 | 0.49 | 0.800 | 0.789 | 0.032 | 0.030 | 0.031 | 0.028 | 0.929 | 0.936 |

0.8 | 0.75 | 0.795 | 0.807 | 0.050 | 0.040 | 0.046 | 0.038 | 0.921 | 0.926 |

0.9 | 0.00 | 0.900 | 0.873 | 0.015 | 0.018 | 0.014 | 0.015 | 0.939 | 0.583 |

0.9 | 0.24 | 0.899 | 0.877 | 0.017 | 0.020 | 0.017 | 0.017 | 0.944 | 0.725 |

0.9 | 0.49 | 0.900 | 0.885 | 0.020 | 0.024 | 0.021 | 0.019 | 0.933 | 0.858 |

0.9 | 0.75 | 0.898 | 0.893 | 0.033 | 0.031 | 0.033 | 0.026 | 0.926 | 0.913 |

Open in a separate window

CP = Concordance Probability; Prop cen = proportion censored

*C _{n,E}* = Weighted c-index excluding ties;

*K*= CPE excluding ties

_{n,E}avg = average; sim se = simulation standard error; cov = coverage

When proportional hazards is correct, the results shown in Table 1 demonstrate that the CPE is relatively unbiased, except when the concordance probability is small. The bias that occurs when the regression model is weak is a function of the bias in the Cox regression coefficient estimates in these simulations. For the weighted c-index, in addition to the bias when the CP = 0.60, a bias is present when the censoring proportion is high. The deficiency of the weighted c-index when data are highly censored is likely attributable to the shorter censoring distribution support relative to the survival distribution support; a violation of a key assumption for the weighted c-index stated in Section 2. The weighted c-index with a time truncation, *C _{n,E}*(

**$\widehat{\beta}$, τ**) in Section 3.2, attenuated the weighted c-index and was not helpful in reducing the bias (Table 2). The truncation time was chosen such that within each risk group,

*G*(τ|

**β**

^{T}

*) > 0.10.*

**X**### Table 2

Weighted c-index with time truncation; proportional hazards data

CP | Prop cen | C_{n,E}avg | C(τ)_{n,E}avg | C_{n,E}sim se | C(τ)_{n,E}sim se | se(C)_{n,E}avg | se(C(τ))_{n,E}avg | cov(C)_{n,E} | cov(C(τ))_{n,E} |
---|---|---|---|---|---|---|---|---|---|

0.6 | 0.00 | 0.602 | 0.602 | 0.028 | 0.028 | 0.028 | 0.028 | 0.953 | 0.953 |

0.6 | 0.25 | 0.606 | 0.606 | 0.031 | 0.031 | 0.030 | 0.030 | 0.938 | 0.938 |

0.6 | 0.50 | 0.607 | 0.604 | 0.037 | 0.038 | 0.037 | 0.037 | 0.927 | 0.930 |

0.6 | 0.75 | 0.618 | 0.612 | 0.054 | 0.053 | 0.056 | 0.055 | 0.929 | 0.945 |

0.7 | 0.00 | 0.700 | 0.700 | 0.026 | 0.026 | 0.025 | 0.025 | 0.944 | 0.944 |

0.7 | 0.24 | 0.701 | 0.701 | 0.027 | 0.027 | 0.029 | 0.029 | 0.955 | 0.955 |

0.7 | 0.49 | 0.698 | 0.696 | 0.035 | 0.036 | 0.035 | 0.035 | 0.935 | 0.933 |

0.7 | 0.74 | 0.693 | 0.677 | 0.053 | 0.056 | 0.052 | 0.055 | 0.929 | 0.914 |

0.8 | 0.00 | 0.798 | 0.798 | 0.021 | 0.021 | 0.021 | 0.021 | 0.937 | 0.937 |

0.8 | 0.24 | 0.800 | 0.800 | 0.023 | 0.023 | 0.023 | 0.023 | 0.949 | 0.949 |

0.8 | 0.50 | 0.797 | 0.796 | 0.029 | 0.030 | 0.030 | 0.030 | 0.944 | 0.944 |

0.8 | 0.75 | 0.784 | 0.769 | 0.045 | 0.051 | 0.045 | 0.050 | 0.950 | 0.935 |

0.9 | 0.00 | 0.900 | 0.900 | 0.015 | 0.015 | 0.015 | 0.015 | 0.928 | 0.928 |

0.9 | 0.24 | 0.899 | 0.899 | 0.016 | 0.016 | 0.016 | 0.016 | 0.939 | 0.939 |

0.9 | 0.49 | 0.900 | 0.900 | 0.020 | 0.020 | 0.020 | 0.020 | 0.936 | 0.936 |

0.9 | 0.75 | 0.894 | 0.888 | 0.031 | 0.033 | 0.031 | 0.033 | 0.942 | 0.945 |

Open in a separate window

CP = Concordance Probability; Prop cen = proportion censored

*C _{n,E}* = Weighted c-index excluding ties

*C _{n,E}*(τ) = Time-truncated weighted c-index excluding ties,

*G*(

*y*|

**β**

^{T}

*) > 0.10*

**X**avg = average; sim se = simulation standard error; cov = coverage

The analytic estimated standard error of the CPE derived in this paper is close to its simulation standard error, and the empirical coverage for the concordance probability based the asymptotic 95% confidence interval *K _{n,E}* ± 1.96 × se(

*K*) is good. As expected, the estimated standard error of the CPE increases as the percent censoring increases and the concordance probability (CP) decreases. For the weighted c-index, the stratified bootstrap resampling standard error and coverage estimate are also accurate.

_{n,E}If the proportional hazards assumption is true, a benefit to using the CPE relative to the weighted c-index is efficiency. The simulation standard errors of *K _{n,E}* are uniformly smaller than the simulation standard errors of

*C*. In terms of mean squared error, the relative efficiency of the CPE to the weighted c-index is always greater than 1 and is as high as 1.4.

_{n,E}For nononproportional hazards data, Table 3 indicates that the CPE is not accurate. The CPE is biased and its coverage may be poor. The inadequacy of the CPE in this case is unsurprising, since its kernel is based on a result stemming from extreme value survival times

$$\text{Pr}[{T}_{2}>{T}_{1}|{\mathit{X}}_{1},{\mathit{X}}_{2}]=\frac{1}{1+\text{exp}\{{\mathit{\beta}}^{T}({\mathit{X}}_{2}-{\mathit{X}}_{1})\}},$$

whereas for normally distributed log survival times

$$\text{Pr}[{T}_{2}>{T}_{1}|{\mathit{X}}_{1},{\mathit{X}}_{2}]=1-\mathrm{\Phi}(\frac{{\mathit{\beta}}^{T}{\mathit{X}}_{1}-{\mathit{\beta}}^{T}{\mathit{X}}_{2}}{\sqrt{2}\sigma}),$$

with Φ(·) representing the standard normal distribution function and σ the scale parameter. The results in Table 3 reinforce the use of diagnostics to assess proportionality before applying the CPE to a given set of data. Conversely, the weighted c-index appears to be sufficently robust and provides similar bias and variance accuracy for both error distributions.

## 5. Prostate cancer example

In recent years, considerable progress in the treatment of metastatic prostate cancer has resulted in new treatments using immunotherapy, hormone therapy, and chemotherapy, with each demonstrating a prolongation of life. As a result, patient prognosis, as determined through risk groups, can affect the choice of treatment. Importantly, the incorporation of a risk classification system in the treatment decision process requires an understanding of the strength of this system to discriminate clinical outcome. A risk classification system was developed from 148 newly diagnosed metastatic prostate cancer patients. After screening multiple factors, patient risk was based on a categorization of two important biomarkers in this patient population: circulating tumor cells (CTC) and lactate dehydrogenase (LDH) (Scher et al. 2009). CTC is a blood-based assay that provides information on the accumulation of tumor cells in the peripheral blood. LDH is a marker of cell turnover and is considered an indirect marker of tumor burden. The finding that five or more CTCs at baseline is associated with shorter survival times has been found in multiple metastatic solid tumor populations (Danila et al. 2011). The upper limit of normal for LDH was defined as 250 units per liter as determined by the central laboratory used for this study. As a result of these previously determined normal/abnormal ranges, the risk classification system derived from the two biomarkers was:

Low risk | CTC < 5 |

Intermediate risk | CTC ≥ 5 LDH ≤ 250 |

High risk | CTC ≥ 5 LDH > 250 |

Open in a separate window

The Kaplan-Meier estimates of survival were well separated by risk and are depicted in Figure 1. The median survival times in the three groups were: 23.6 months (low risk), 13.6 months (intermediate risk), and 8.2 months (high risk). A proportional hazards model was generated using this risk classification and a test of the proportional hazards assumption (Grambsch and Therneau 1994) provided insufficient evidence that the proportional hazards assumption was violated (p= 0.293). From the results of the proportional hazards model, the estimated log relative risk of death was 0.860 (se=0.304) and 1.584 (se = 0.280) for the intermediate and high risk groups respectively, compared to the low risk group. This risk classification is clearly prognostic, but p-values alone, which were both <0.005, are insufficient to assess the strength of this classification. In general, p-values are sensitive to the sample size of the data, with a large number of failures producing a downward influence on the p-value. In order to ascertain the strength of this classification system in discriminating patient risk, the concordance probability estimate and weighted c-index were computed.

Figure 1

Kaplan-Meier estimates of survival based on clinical risk.

The CPE and weighted c-index excluding ties were computed to assess the discrimination strength of this risk group classification. The CPE and weighted c-index were equal to 0.753 (se=0.036) and 0.761 (se = 0.060), respectively. These results indicate that the three group staging system provides good discriminatory power and may be used to assist in the treatment decision process.

When patient pairs in the same risk group were included in the discrimination measures, the CPE was equal to 0.659 and the weighted c-index was equal to 0.666, indicating the the derived metastatic prostate cancer risk groups were not sufficiently strong at discriminating survival rates. However, the overall proportion of pairwise comparisons with patients in the same risk group was 0.30, which explains the attenuation of these discrimination measures.

The CPE and weighted c-index provide a measure of the overall discriminatory power of the risk group classification system. In addition to this overall measure of strength, the CPE can also be used to estimate the discriminatory power between individual risk groups. Using the proportional hazards specification

Pr[*T*_{2}>*T*_{1}|*X*_{1},*X*_{2}] =[1+exp{*β*^{T}(*X*_{2}−*X*_{1})}]^{−1},

and substituting **$\widehat{\beta}$**, the probability is 0.83 that a patient in the low risk group will survive longer than a patient in the high risk group. The probability is 0.70 that a patient in the low risk group survives longer than a patient in the intermediate risk group, and approximately the same probability holds for a patient in the intermediate risk group relative to a patient in the high risk group. This illustrates that although the overall CPE is 0.753, the discriminatory strength between individual groups varies. As noted in Section 3, the overall CPE is a weighted average of these individual measures.

The discriminatory power of a risk classification system should be validated before the system is put into practice. An independent cohort of 97 metastatic prostate cancer patients (Danila et al. 2014) was used to validate the estimates of the concordance probability. The CPE and weighted c-index with ties excluded was 0.772 and 0.783, respectively. These results are comparable to the development study results and confirm the good discriminatory power of the CTC+LDH risk classification system in this patient population.

## 6. Discussion

The interpretation of the concordance probability differs depending on whether ties in the risk groups are included or excluded. The inclusion of patient pairs within the same risk group reduces the concordance probability (𝒞_{I}) parameter toward the null value 0.50 and is therefore affected by the distribution of the subjects across risk groups. In contrast, the parameter (𝒞_{E}), based on the exclusion of patient pairs within the same risk group is unaffected by the distribution of patients across groups.

The most commonly applied measure of discrimination is the unweighted c-index. An advantage of the c-index is that it is nonparametric and may be useful if the proportional hazards assumptions do not hold. We previously showed that with continuous risk scores the unweighted c-index had a bias that was a function of the percent censored (Gönen and Heller 2005). We have adapted the weighted c-index (Uno et al. 2011) to the case when patient pairs in the same risk group are excluded in the calculation. Our simulations indicate that the inverse probability censoring weighted c-index excluding ties is robust to the violation of proportional hazards.

Under proportional hazards, the semiparametric CPE excluding ties was more efficient than the nonparametric weighted c-index. Both statistics were relatively unbiased, but at high censoring rates the weighted c-index demonstrated some bias. In general, time truncation is used to reduce bias for an inverse probability censoring weight statistic, but our simulations demonstrated a lack of improvement uisng the time truncated weighted c-index. In contrast, the CPE is a model based measure of discrimination. It does not require inverse probability censoring weights and therefore does not require a stabilizing truncation point. It does, however, assume that the proportional hazards relation is correct.

Clinical risk classification systems are ubiquitous in medicine and their usefulness in clinical decision making is a function of their discriminatory power. Risk classification systems differ in their discrimination capability and a measure to assess the strength of a classification system should be applied. Our findings indicate that if the proportional hazards model is employed to develop the classification system, and diagnostics are used to confirm the proportional assumptions, then the CPE is an efficient measure of discrimination. If an alternative approach to risk classification is employed, then the nonparametric weighted c-index produces an accurate assessment of discrimination.

`R` code to compute the CPE and its standard error, with the option to include or exclude ties in the risk scores is available in the `R` package `CPE`. For the weighted c-index, the `R` package `pec` enables the user the option to include or exclude ties in the risk scores.

## Appendix

### The asymptotic distribution for the CPE excluding ties

It is assumed that the proportional hazards model $h(t|\mathit{X})={h}_{0}(t)\text{exp}({\mathit{\beta}}_{0}^{T}\mathit{X})$ is the correct specification for the data and that the standard conditions for the asymptotic normality of *n*^{1/2}(**$\widehat{\beta}$** − **β**_{0}) apply (Andersen and Gill, 1982).

*Consistency of the CPE excluding ties*: ${K}_{n,E}(\hat{\mathit{\beta}})\stackrel{p}{\to}{\mathcal{C}}_{E}$ where

$${K}_{n,E}(\hat{\mathit{\beta}})=\frac{{\sum}_{i}{\sum}_{j}I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}>{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j}){[1+\text{exp}({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{ji})]}^{-1}}{{\sum}_{i}{\sum}_{j}I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}>{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})}$$

$$\text{and}{\mathcal{C}}_{E}=\text{Pr}[{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}>{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}|{T}_{2}>{T}_{1},{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}\ne {\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}].$$

Using the proportional hazards specification provided in Section 2 and the law of large numbers,

$${K}_{n,E}(\hat{\mathit{\beta}})\stackrel{p}{\to}\text{Pr}[{T}_{2}>{T}_{1}|{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}>{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}].$$

Note that the concordance probability excluding ties may be written as

$${\mathcal{C}}_{E}=\text{Pr}[{T}_{2}>{T}_{1}|{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}>{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}]\times \frac{\text{Pr}[{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}>{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}]}{\text{Pr}[{T}_{2}>{T}_{1},{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}\ne {\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}]}.$$

(A.1)

Now using the independence between subjects, the following relations hold

$$\text{Pr}[{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}>{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}]=\frac{\text{Pr}[{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}\ne {\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}]}{2}$$

(A.2)

$$\frac{1}{2}=\text{Pr}[{T}_{2}>{T}_{1}|{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}\ne {\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}]\text{Pr}[{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}\ne {\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}]\phantom{\rule{0ex}{0ex}}+\text{Pr}[{T}_{2}>{T}_{1}|{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}={\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}]\text{Pr}[{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}={\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}]$$

(A.3)

Substituting (A.2) into (A.3), it follows that

$$\text{Pr}[{T}_{2}>{T}_{1},{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}\ne {\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}]=\text{Pr}[{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{1}>{\mathit{\beta}}_{0}^{\mathrm{T}}{\mathit{X}}_{2}]$$

(A.4)

and substitution of (A.4) into the denominator of (A.1) proves the consistency of the CPE.

*Asymptotic distribution of n*^{1/2} [*K _{n,E}*(

**$\widehat{\beta}$**) − 𝒞

_{E}]

To prove the asymptotic normality of the CPE, it is first shown that if

$$|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ij}|>\epsilon >0,$$

(A.5)

then *n*^{1/2}*K _{n,E}*(

**$\widehat{\beta}$**) is asymptotically equal to

$${n}^{1/2}{\tilde{K}}_{n,E}(\hat{\mathit{\beta}})=\frac{{n}^{-3/2}{\sum}_{i}{\sum}_{j}I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ij}>0){[1+\text{exp}({\hat{\mathit{\beta}}}^{\mathrm{T}}{\mathit{X}}_{ji})]}^{-1}}{{n}^{-2}{\sum}_{i}{\sum}_{j}I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ij}>0)}$$

where within the indicator functions the estimate **$\widehat{\beta}$** is replaced with the true regression coefficient **β**_{0} and * X_{ij}* =

*−*

**X**_{i}*. Note that in the discrete covariate case there are a finite number of covariate values*

**X**_{j}*. As a result, (A.5) is not a strong assumption.*

**X**_{ij}The asymptotic equality is demonstrated by considering the cases * X_{ij}* = 0 and

*≠ 0 separately.*

**X**_{ij}If

= 0, then clearly $I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{ij}>0)=I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ij}>0)$.**X**_{ij}If

≠ 0, then using the consistency of**X**_{ij}**$\widehat{\beta}$**and assumption (A.5), for*n*sufficiently large, |**$\widehat{\beta}$**^{T}| > ν > 0 and $I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{ij}>0)=I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ij}>0)$ for all**X**_{ij}. Therefore, for**X**_{ij}*n*large,$${n}^{1/2}{K}_{n,E}(\hat{\mathit{\beta}})={n}^{1/2}{\tilde{K}}_{n,E}(\hat{\mathit{\beta}}).$$

A Taylor expansion for the asymptotically equivalent CPE produces

$${n}^{1/2}[{\tilde{K}}_{n,E}(\hat{\mathit{\beta}})-{\mathcal{C}}_{E}]={n}^{1/2}[{\tilde{K}}_{n,E}({\mathit{\beta}}_{0})-{\mathcal{C}}_{E}]+{\left(\frac{\partial {\tilde{K}}_{n,E}}{\partial \mathit{\beta}}\right)}^{\mathrm{T}}[{n}^{1/2}(\hat{\mathit{\beta}}-{\mathit{\beta}}_{0})]+{o}_{p}(1).$$

The partial derivative ∂*$\stackrel{\u0303}{K}$ _{n,E}*/∂

**β**is asymptotically constant. Since

*n*

^{1/2}(

**$\widehat{\beta}$**−

**β**

_{0}) has asymptotic mean zero conditional on

*, it is asymptotically independent of*

**X***n*

^{1/2}[

*$\stackrel{\u0303}{K}$*(

_{n,E}**β**

_{0}) − 𝒞

_{E}]. Therefore the asymptotic variance of

*$\stackrel{\u0303}{K}$*(

_{n,E}**$\widehat{\beta}$**) is

$$\text{Var}\left\{{n}^{1/2}\right[{\tilde{K}}_{n,E}({\mathit{\beta}}_{0})-{\mathcal{C}}_{E}\left]\right\}+{\left(\frac{\partial {\tilde{K}}_{n,E}}{\partial \mathit{\beta}}\right)}^{\mathrm{T}}\text{Var}[\hat{\mathit{\beta}}]\left(\frac{\partial {\tilde{K}}_{n,E}}{\partial \mathit{\beta}}\right).$$

The individual components of the asymptotic variance can be estimated as follows. In each case, the substitution of **$\widehat{\beta}$** for **β**_{0} provides a consistent estimate.

The Var(**$\widehat{\beta}$**) is estimated from the inverse of the second derivative of the partial likelihood.

The partial derivative evaluated at **β**_{0} is equal to

$$\frac{\partial {\tilde{K}}_{n,E}(\mathit{\beta})}{\partial \mathit{\beta}}=-\frac{{\sum}_{i}{\sum}_{j}I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ij}>0){\mathit{X}}_{ji}\text{exp}({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ji}){[1+\text{exp}({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ji})]}^{-2}}{{\sum}_{i}{\sum}_{j}I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ij}>0)}.$$

The first term, *n*^{1/2} [*$\stackrel{\u0303}{K}$ _{n,E}*(

**β**

_{0}) − 𝒞

_{E}], may be approximated by the U-statistic

$${\pi}^{-1}{n}^{-3/2}\sum _{i}\sum _{j}I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ij}>0)\{{[1+\text{exp}({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ji})]}^{-1}-{\mathcal{C}}_{E}\}$$

where $\pi ={\text{lim}}_{n\to \infty}{n}^{-2}{\sum}_{i}{\sum}_{j}I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ij}>0)$.

The asymptotic variance of this U-statistic is

$${\pi}^{-2}{n}^{-3}\sum _{i}\sum _{j}\sum _{k\ne j}[{u}_{ij}+{u}_{ji}][{u}_{ik}+{u}_{ki}].$$

where

$${u}_{ij}=I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ij}>0)\{{[1+\text{exp}({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{ji})]}^{-1}-{\mathcal{C}}_{E}\}.$$

Combining these results provides the estimated asymptotic variance of the CPE excluding ties.

*Asymptotic distribution of n*^{1/2} [*C _{n,E}*(

**$\widehat{\beta}$**) − 𝒞

_{E}]

From Section 3.2, this expression may be written as

$$\frac{{n}^{-3/2}{\sum}_{i}{\sum}_{j}{\delta}_{i}I({y}_{i}<{y}_{j})I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}\ne {\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j}){\{\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i})\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})\}}^{-1}[I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}>{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})-{\mathcal{C}}_{E}]}{{n}^{-2}{\sum}_{i}{\sum}_{j}{\delta}_{i}I({y}_{i}<{y}_{j})I({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i}\ne {\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j}){\{\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i})\u011c({y}_{i}|{\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})\}}^{-1}}\phantom{\rule{0ex}{0ex}}={\psi}^{-1}{n}^{-3/2}\sum _{i}\sum _{j}\frac{{\delta}_{i}I({y}_{i}<{y}_{j})I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{i}\ne {\mathit{\beta}}_{0}^{T}{\mathit{X}}_{j})[I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{i}>{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{j})-{\mathcal{C}}_{E}]}{G({y}_{i}|({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{i})G({y}_{i}|({\hat{\mathit{\beta}}}^{T}{\mathit{X}}_{j})}+{o}_{p}(1)$$

where ψ = lim_{n→∞}*n*^{−2} ∑_{i} ∑_{j} δ_{i}*I*(*y _{i}* <

*y*)

_{j}*I*(

**$\widehat{\beta}$**

^{T}

*≠*

**X**_{i}**$\widehat{\beta}$**

^{T}

*)*

**X**_{j}*Ĝ*

^{−1}(

*y*|

_{i}**$\widehat{\beta}$**

^{T}

*)*

**X**_{i}*Ĝ*

^{−1}(

*y*|

_{i}**$\widehat{\beta}$**

^{T}

*).*

**X**_{j}Letting

$${e}_{ij}={\delta}_{i}I({y}_{i}<{y}_{j})I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{i}\ne {\mathit{\beta}}_{0}^{T}{\mathit{X}}_{j})[I({\mathit{\beta}}_{0}^{T}{\mathit{X}}_{i}>{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{j})-{\mathcal{C}}_{E}]$$

and Taylor expanding **$\widehat{\beta}$** around **β**_{0},

$${n}^{1/2}[{C}_{n,E}(\hat{\mathit{\beta}})-{\mathcal{C}}_{E}]={\psi}^{-1}{n}^{-3/2}\sum _{i}\sum _{j}\frac{{e}_{ij}}{G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{i})G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{j})}+\phantom{\rule{0ex}{0ex}}{\psi}^{-1}{n}^{-3/2}{\sum}_{i}{\sum}_{j}\frac{{e}_{ij}}{G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{i})G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{j})}[\frac{G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{i})-\u011c({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{i})}{G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{i})}+\frac{G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{j})-\u011c({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{j})}{G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{j})}]+\phantom{\rule{0ex}{0ex}}{\psi}^{-1}{n}^{-3/2}{\sum}_{i}{\sum}_{j}\left\{{\left[\frac{\partial}{\partial \mathit{\beta}}\frac{{e}_{ij}}{G({y}_{i}|{\mathit{\beta}}^{T}{\mathit{X}}_{i})G({y}_{i}|{\mathit{\beta}}^{T}{\mathit{X}}_{j})}\right]}^{T}\right\}(\hat{\mathit{\beta}}-{\mathit{\beta}}_{0})+{o}_{p}(1)$$

The second term may be rewritten using the martingale representation theorem as

$$2{n}^{-1/2}\sum _{k=1}^{n}{\int}_{t=0}^{\infty}\frac{{q}_{{\mathit{X}}_{k}}(t)}{{\theta}_{{\mathit{X}}_{k}}(t)}d{M}_{{\mathit{X}}_{k}}(t)$$

where

$${q}_{{\mathit{X}}_{k}}(t)=\underset{n\to \infty}{\text{lim}}\sum _{i}\sum _{j}\left\{\frac{{e}_{ij}}{G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{i})G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{j})}\right\}\{I({y}_{j}\ge t)[I({\mathit{X}}_{i}={\mathit{X}}_{k})+I({\mathit{X}}_{j}={\mathit{X}}_{k})]\}$$

$${\theta}_{{\mathit{X}}_{k}}(t)=\underset{n\to \infty}{\text{lim}}\sum _{j}I({y}_{j}\ge t,{\mathit{X}}_{j}={\mathit{X}}_{k})$$

$${M}_{{\mathit{X}}_{k}}(t)=I({y}_{k}\le t,{\delta}_{k}=0)-{\int}_{u=0}^{t}I({y}_{k}\ge u)d{\mathrm{\Lambda}}_{{\mathit{X}}_{k}}(u)$$

and Λ_{Xk} is the cumulative hazard of the censoring random variable belonging to group * X_{k}* (Cheng et al. 1995, Fine et al. 1998).

It follows that *n*^{1/2} [*C _{n,E}*(

**$\widehat{\beta}$**) − 𝒞

_{E}]

$$={\psi}^{-1}{n}^{-3/2}\sum _{i}\sum _{j}\frac{{e}_{ij}}{G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{i})G({y}_{i}|{\mathit{\beta}}_{0}^{T}{\mathit{X}}_{j})}+2{\psi}^{-1}{n}^{-1/2}\sum _{k=1}^{n}{\int}_{t=0}^{\infty}\frac{{q}_{{\mathit{X}}_{k}}(t)}{{\theta}_{{\mathit{X}}_{k}}(t)}d{M}_{{\mathit{X}}_{k}}(t)\phantom{\rule{0ex}{0ex}}+{\psi}^{-1}\left\{{n}^{-2}\sum _{i}\sum _{j}{\left[\frac{\partial}{\partial \mathit{\beta}}\frac{{e}_{ij}}{G({y}_{i}|{\mathit{\beta}}^{T}{\mathit{X}}_{i})G({y}_{i}|{\mathit{\beta}}^{T}{\mathit{X}}_{j})}\right]}^{T}\right\}{n}^{1/2}(\hat{\mathit{\beta}}-{\mathit{\beta}}_{0})+{o}_{p}(1)$$

Each component is asymptotically normal with mean zero. Analytic estimation of the asymptotic variance, however, requires density estimation of the censoring random variable. Alternatively, a stratified bootstrap resampling approach is utilized to estimate the asymptotic variance.

## References

- Andersen PK, Gill RD. Cox’s regression model for counting procsses: A large sample study. Annals of Statistics. 1982;10:1100–1120. [Google Scholar]
- Cheng SC, Wei LJ, Ying Z. Analysis of transformation models with censored data. Biometrika. 1995;82:835–845. [Google Scholar]
- Cox DR. Regression models and life tables (with Discussion) Journal of the Royal Statistical Society, Series B. 1972;34:187–220. [Google Scholar]
- Cox DR. Partial likelihood. Biometrika. 1975;62:269–276. [Google Scholar]
- Danila DC, Fleisher M, Scher HI. Circulating tumor cells as biomarkers in prostate cancer. Clinical Cancer Research. 2011;17:3903–3912. [PMC free article] [PubMed] [Google Scholar]
- Danila DC, Anand A, Schultz N, Heller G, Wan M, Sung CC, Dai C, Khanin R, Fleisher M, Lilja H, Scher HI. Analytic and clinical validation of a prostate cancer-enhanced messenger RNA detection assay in whole blood as a prognostic biomarker for survival. European Urology. 2014;65:1191–1197. [PMC free article] [PubMed] [Google Scholar]
- Fine JP, Ying Z, Wei LJ. On the linear transformation model for censored data. Biometrika. 1998;85:980–986. [Google Scholar]
- Gerds TA. Prediction Error Curves for risk prediction models in survival analysis. R package version 2.4.2. 2014 [Google Scholar]
- Gerds TA, Kattan MW, Schumacher M, Yu C. Estimating a time-dependent concordance index for survival prediction models with covariate dependent censoring. Statistics in Medicine. 2013;32:2173–2184. [PubMed] [Google Scholar]
- Gönen M, Heller G. Concordance probability and discriminatory power in proportional hazards regression. Biometrika. 2005;92:965–970. [Google Scholar]
- Grambsch PM, Therneau TM. Proportional Hazards Tests and Diagnostics Based on Weighted Residuals. Biometrika. 1994;81:515–526. [Google Scholar]
- Harrell FE, Califf RM, Pryor DB, Lee KL, Rosati RA. Evaluating the yield of medical tests. Journal of the American Medical Association. 1982;247:2543–2546. [PubMed] [Google Scholar]
- Kalbfleisch JD. Likelihood methods and nonparametric tests. Journal of the American Statistical Association. 1978;73:167–170. [Google Scholar]
- Lin DY, Wei LJ, Ying Z. Checking the Cox Model with Cumulative Sums of Martingale-Based Residuals. Biometrika. 1993;80:557–572. [Google Scholar]
- Mo Q, Gönen M, Heller G. CPE: Concordance Probability Estimates in Survival Analysis. R package version 1.4.3. 2012 [Google Scholar]
- R Development Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2011. URL http://www.R-project.org/ [Google Scholar]
- Scher HI, de Bono JS, Fleisher M, Pienta KJ, Raghavan D, Heller G. Circulating tumour cells as prognostic markers in progressive, castration-resistant prostate cancer: a reanalysis of IMMC38 trial data. Lancet Oncology. 2009;10:233–239. [PMC free article] [PubMed] [Google Scholar]
- Sprent P. Applied Nonparametric Statistical Methods, Second edition. London: Chapman and Hall; 1989. [Google Scholar]
- Uno H, Cai T, Pencina MJ, D’Agostino RB, Wei LJ. On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in Medicine. 2011;30:1105–1117. [PMC free article] [PubMed] [Google Scholar]
- Yan G, Greene T. Investigating the effects of ties on measures of concordance. Statistics in Medicine. 2008;27:4190–4206. [PubMed] [Google Scholar]