(B-42) Improving outcome predictions for ovarian cancer through use of ordinal loss functions

Introduction:: Accurate and reproducible methods for prognostication are vital for the individualized care of cancer patients. In cancer grading systems there is a natural ordering to the disease severity and survival follow-up times. Intuition would suggest that data would be more similar between adjacent ordinal categories than between distant categories and that loss functions that incorporate this intuition may be superior to conventional loss functions. Several prior studies have demonstrated mixed results with ordinal-aware loss functions with improvements for an ovarian cancer survival task (Misganaw, IEEE Life Sciences Letters, 2015) and no improvements for many other tasks (Janocha, 2016). In an effort to probe the limitations of these findings, we sought to test whether training deep learning models with "smart” loss functions tailored to penalize the severity of the classification error can improve overall model performance compared to standard crossentropy (CE) loss.


Materials and Methods:: We evaluated two loss functions: categorical CE and an ordinal-weighted CE (OWCE). For this study, weights were selected to exponentially increase with the distance from the correct class. We used simulated and real-world ordinal datasets to compare the performance of loss functions. To disentangle the impact of loss functions from spurious properties of datasets, we first simulated five datasets with 10,000 samples each containing four ordinal target classes. Sparsity, number of informative and non-informative features, noise, bias, effective rank, trail strength, and/or the functional relationship between the features and targets were controlled during data generation.

Next, we sought to compare loss functions for the real-world task of risk stratification of ovarian cancer patients. Surveillance, Epidemiology, and End Results (SEER) data was curated from adults diagnosed with ovarian cancer between 2010 and 2011. A total of 7,052 subjects and 23 features, including 6 demographic, 6 clinical, 5 pathologic, 1 biomarker, and 4 treatment features, were included. To generate an ordinal target for the SEER, survival times were partitioned into quartiles.

To evaluate the performance of the different loss functions we trained on 75% of the data and tested on a 25% hold out set. From the training set, we used stratified sampling to select 5-99% of the samples to evaluate the ability of the loss functions to train with limited data. For each dataset, we used 5-fold cross-validation and iterated 10 times resulting in 50 folds per dataset. We report the mean and standard deviation of the folds.


Results, Conclusions, and Discussions:: Results:

For the five synthetic datasets, the models trained with OWCE consistently outperformed the ones trained with CE, with improvements in accuracy ranging from 10% for smaller training sample sizes to 3% for larger sample sizes. In general, as training size increased, CE performance increased relative to OWCE, although the latter still significantly improved accuracy (p-value < 0.05). Similar to the synthetic data, use of the OWCE loss function on the SEER data improved accuracy relative to CE loss, although to a lesser degree, with improvement ranging from 4% in smaller training sizes to 2% in larger training sizes.

Conclusion:

We compared an ordinal loss function to the standard CE loss function on five synthetic datasets and one real world dataset. Across all data sets, OWCE loss outperformed the CE loss function for these ordinal classification tasks. Furthermore, results suggest that ordinal loss functions offer increased value in the context of limited training data, which is a common challenge in many medical machine learning applications. Concordant with intuition, utilizing an ordinal loss function improves classification of data with an underlying ordinal structure relative to using a flat classifier which does not include data structure.


Acknowledgements (Optional): :

References (Optional): : B. Misganaw and M. Vidyasagar, "Exploiting Ordinal Class Structure in Multiclass Classification: Application to Ovarian Cancer," in IEEE Life Sciences Letters, vol. 1, no. 1, pp. 15-18, June 2015, doi: 10.1109/LLS.2015.2451291.

Lei Y, Zhang J, Shan H. Strided Self-Supervised Low-Dose CT Denoising for Lung Nodule Classification. Phenomics. 2021 Oct 26;1(6):257-268. doi: 10.1007/s43657-021-00025-y. PMID: 36939784; PMCID: PMC9590543.