Add P1C7: Start-to-finish ML survival analysis chapter

book/attic/P1C7_mlsa.qmd

@@ -0,0 +1,79 @@
+---
+abstract: |
+  This chapter draws together the foundational concepts of Parts I-IV into a single end-to-end view of a machine learning survival analysis. It restates the standard supervised-learning pipeline and identifies the survival-analysis-specific adaptation required at each step, from understanding the censoring/truncation structure of the data, through stratified splitting, censoring-aware model training, choice of prediction type (survival distribution, relative risk score, or survival time), and survival-specific evaluation. A nine-step worked example on a hypothetical retrospective study of 30-day in-hospital mortality of COVID-19 patients with type 2 diabetes (CHESS dataset) makes the abstract pipeline concrete: it walks through the question of interest, data sourcing, outcome definition, censoring/truncation handling, prediction-type choice, metric selection, model selection, splitting strategy, and final evaluation. The chapter is designed to be read either as a closing summary of Part I or as a starting orientation for practitioners about to apply the rest of the book to their own data.
+---
+
+::: {.content-visible when-format="html"}
+{{< include _macros.tex >}}
+:::
+
+# Machine Learning Survival Analysis {#sec-ml-sa-pipeline}
+
+To help make the concepts in this book less abstract, this chapter outlines what to consider when undertaking a machine learning survival analysis from start to finish.
+@fig-ml-sa-pipeline shows the standard supervised machine learning pipeline (top, teal) and, directly below each step, the survival-analysis-specific adaptations that are required (bottom, orange).
+
+![The standard supervised machine learning pipeline (top, teal) and the survival-analysis adaptations required at each step (bottom, orange). After splitting, the test features (test X) bypass model training and flow directly to prediction; the test labels (test Y) flow to evaluation. If resampling is used, the split--train--predict--evaluate cycle repeats.](Figures/introduction/ml-sa-pipeline.png){#fig-ml-sa-pipeline fig-alt="Two-row diagram. Top row (teal boxes, left to right): Data (features X, outcome t delta) -> Split (train set, test set X Y) -> Fit Model (algorithm on train set) -> Predict (model on test X, output y-hat) -> Evaluate (y-hat vs test Y, performance). Dashed lines show test X bypassing Fit Model to Predict, and test Y bypassing both to Evaluate. A dashed arc labelled 'if resampling' loops from Evaluate back to Split. Bottom row (orange boxes): Understand Setting (censoring type, truncation, competing risks, recurrent events; Part I Ch 4-6) -> Adjust Splitting (stratify by event/censoring status or competing risk type; Part I Ch 4-5) -> Adapted Training (censoring-aware loss/likelihood, truncation-adjusted risk sets; Part III Ch 13-17) -> Prediction Type (survival distribution, relative risk score, survival time, not a point estimate; Part I Ch 6) -> SA Evaluation (test data also has censoring/truncation/competing risks; Part II Ch 8-11). Orange dashed vertical lines connect each top box to its corresponding bottom box."}
+
+The most important first step, before any data splitting or model selection, is to understand what kind of survival data is available.
+The type of censoring, presence of truncation, and whether competing risks or recurrent events are relevant all determine which methods are valid at every subsequent stage (@sec-surv, @sec-eha).
+Data splitting may need to be stratified by event or censoring status to preserve the proportion of censored observations across train and test sets.
+Model training must use algorithms that correctly account for censoring and, where applicable, truncation or competing risks, rather than treating censored observations as missing outcomes (Part III).
+The output of a survival model is not a single point estimate but a survival distribution, a relative risk score, or a survival time (@sec-survtsk).
+Finally, evaluation must use measures designed for survival data, since the test set itself contains censored observations and standard regression or classification metrics are not applicable (Part II).
+
+## A worked example {#sec-ml-sa-worked}
+
+There are many ways to break down a machine learning survival analysis; the following nine-step decomposition is one such possibility:
+
+1. Establish the question of interest.
+2. Gather relevant data.
+3. Describe the outcome.
+4. Identify censoring and truncation in the data and decide on a strategy for handling them.
+5. Describe the prediction type.
+6. Select metrics based on the data and prediction type.
+7. Select models based on the data and metrics.
+8. Split data for model training.
+9. Evaluate the model.
+
+A hypothetical example, adapted from @Dennis2020, runs through these nine steps below.
+Terms and methods used here are defined throughout the rest of this book; the cross-references point to the relevant chapters.
+
+**1. Question of interest.**
+Predict the 30-day survival distribution for a patient with type 2 diabetes admitted to a critical care setting in England, UK for COVID-19.
+
+**2. Data.**
+This is a retrospective study, since the pandemic is in the past, and many datasets carry the relevant information; one example is the COVID-19 Hospitalisation in English Surveillance System (CHESS) dataset.
+To reduce biases that could arise from data collection (such as left-truncation biases), a fixed observation period is set: 1 March 2020 to 27 July 2020.
+
+**3. Outcome.**
+The outcome of interest is 30-day in-hospital all-cause mortality, the risk of dying from any cause within 30 days of being admitted to hospital, which can occur at most once (@sec-surv).
+Other potential outcomes include discharge, loss to follow-up, or surviving to 30 days.
+
+**4. Censoring and truncation.**
+The study focuses on admissions during the observation period and therefore excludes patients diagnosed with COVID-19 before 1 March 2020.
+Immortal time bias and survivorship bias are avoided by defining study entry as hospital admission and using a fixed 30-day observation window, with inclusion or exclusion not dependent on any outcomes, treatment, patient age, or illness duration.
+Loss to follow-up is assumed to be a random, independent right-censoring event.
+Surviving to 30 days is an administrative right-censoring event (@sec-surv); the reason for censoring is the closing of the observation window, not the patient.
+Discharge is a competing risk: once a patient is discharged (@sec-eha), they are no longer at risk of in-hospital mortality.
+
+**5. Prediction type.**
+The goal is survival probability over time, so the survival task is to predict a probability distribution over the 30-day horizon (@sec-survtsk).
+
+**6. Metrics.**
+The task is to evaluate survival distribution predictions while accounting for competing risks.
+Proper scoring rules (@sec-rules) are well-suited for distribution predictions; the right-censored log-loss is one example of a scoring rule that can be extended to evaluate predictions when competing risks are present.
+
+**7. Models.**
+The sample of interest from the CHESS dataset has more than 20{,}000 observations with relatively few features.
+This means that traditional survival models (@sec-models-classical) can be considered alongside more modern alternatives.
+Given the focus on survival distribution predictions, one might start by considering a Fine and Gray model with optional boosting to improve predictions (@sec-boost).
+Cause-specific random survival forests with log-rank splitting are also worth exploring (@sec-ranfor).
+
+**8. Data splitting.**
+Given the size of the dataset, a simple hold-out splitting strategy may suffice (@sec-ml).
+For example, a random subset of 15{,}000 observations could be used to develop models and the remainder used to evaluate predictions.
+Since the rate of censoring is unlikely to be too small or too large (which can be confirmed by exploratory analysis), stratified sampling is not strictly required.
+
+**9. Evaluation.**
+With all the above in place, the candidate models are trained on the same training data and evaluated using the metrics outlined in step 6.
+Once an optimal model has been identified, it can be retrained on the complete dataset for feature analysis or deployment.