Fail Fast: Predicting Outcomes of Medications in IBD

Transcript

1. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Fail Fast: Early Prediction of Outcomes and Optimizing Medications in IBD Peter D.R. Higgins, MD, PhD, MSc (CRDSA) Director, IBD Program Director, Clinical Trials Support Unit University of Michigan @ibddoctor
2. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Disclosures • Research Funding: • NIH • CCF • Broad Foundation • Abbvie • Ascentage Pharma • Siemens • Consulting • Lycera, UCB, Eli Lilly, BMS • Amgen, PRIME Medical Education • Speakers Bureau • GI Health Foundation • Clinical Trial Site PI: • Abbvie • Amgen • AstraZeneca • BCBS • Celgene • Genentech • Gilead • Janssen • MedImmune • Nestle • Pfizer • Salix • Seres • Takeda • UCB
3. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST What is “Fail Fast”? • A concept from agile software development • It is better for your code to fail early and explicitly than to quietly continue to process and crash later • It is easier and cheaper to fix a bug early than it is to fix it later • This works surprisingly well as an analogy for IBD care
4. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Failing Fast in IBD Preclinical Diagnosis First Second Third Complications / Surgery Inflammation Rx Rx Rx Point at Which Inflammation Effectively Treated Accumulating Bowel Damage Steroids and placebo effects can conceal early evidence of medication failure Early evidence of medication failure – Change med Late evidence of medication failure – Surgery
5. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Speed of Rx Failure in IBD • Clinicians can be pretty slow to recognize therapeutic failure • Thiopurines, methotrexate – wait 12-16 weeks • Dose adjust? • Vedolizumab – wait up to 40 weeks? • Go to q 4 weeks? • Ustekinumab • How long? Go to q 4 weeks? • Can spend a lot of time rearranging deck chairs • Long-term consequences from resulting bowel damage
6. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Clinical trials with • Imagine a new biologic for IBD named Supermab • 2 RCTs are conducted 25% 27% 0% 50% 100% Trial 1: Clinical & Endoscopic Remission Placebo Supermab 26% 24% 0% 50% 100% Trial 2: Clinical & Endoscopic Remission Placebo Supermab Would you prescribe this drug for IBD? Should the FDA approve this drug for IBD?
7. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Notes on Proactive TDM in IBD • 3 prospective RCTs of thiopurine metabolite monitoring and adjustment have failed to produce better outcomes • 2 prospective RCTs (TAXIT, TAILORIX) of anti-TNF trough level monitoring and adjustment have failed to produce better outcomes 69% 66% 0% 20% 40% 60% 80% 100% TAXIT Remission Clinical & CRP Proactive TDM 47% 38% 48% 0% 20% 40% 60% 80% 100% TAILORIX Remission Clinical Proactive TDM2 AGA Guideline: “Prospective studies have failed to show clinical benefits of proactive TDM"
8. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Can We Fail Faster? • Can we predict early failure or success with IBD medications? • Thiopurines • Vedolizumab in UC • Vedolizumab in CD • Ustekinumab in CD
9. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Methodology • Collect data on lab values associated with response/failure • High-quality, CLIA-certified data • Collect biologic response outcomes, rigorously documented • Use clinical trial data when available – YODA, CSDR, etc. • Split data into two random subsets • 70% for training a model • 30% for testing whether the model is generalizable • Model-building • Multiple machine learning approaches, mostly random forest • If possible, identify a simple predictor
10. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Identifying Nearly Pure Splits 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Time Spent Digging Holes Time Spent Sitting in Laps Cats Dogs
11. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST 3 Variables = 3 Dimensions Lap Sitting Digging Holes Curling up in boxes
12. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST MICHIGAN IBD • FAIL FAST Thiopurine Machine Learning Will this Patient benefit from this particular Therapy?
13. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST The Problem • Thiopurines have a poor therapeutic index • Metabolism and efficacy are variable • Metabolites are poor predictors of efficacy • Expert physicians claim there are recognizable patterns in labs when patients are doing well on thiopurines • Can we optimize thiopurines with machine learning patterns?
14. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST MICHIGAN IBD • FAIL FAST Can Laboratory Patterns Distinguish Biologic Remission vs. no Remission? Results
15. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Background • Machine learning algorithms are superior to 6-TGN in predicting clinical remission No data on biologic remission. No data on clinical outcomes. Waljee, et al. Clin Gastro Hep. 2010 8:143-50. 6-TGN optimization has failed to improve outcomes in 3 RCTs Reinshagen, et al. Clin Chem 2007 53:1306 Gonzalez-Lama, Y., et al, APT 2011; 34: 544-54 Dassopoulos, et al. APT 2014.39:163-175.
16. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST 3 Hypotheses Lab values Blood counts Differential Chemistry Age Patterns Biologic Remission Objective Inflammation (not BR) Adequate Immune Suppression Inadequate Immune Suppression Lower Rate of Clinical Events (Steroids, Hospitalization, Surgery) Higher Rate of Clinical Events (Steroids, Hospitalization, Surgery)
17. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Methods • IBD patients on stable thiopurine dose ≥ 4 weeks • Random Forest machine learning • Cross-validation on out of bag (OOB) sample • Algorithms were developed for • Adequate Immune Suppression (BR) • Adherence • Shunting GOOD LOW Time GOOD GOOD LOW LOW Not Immune Suppressed Sustained Predicted Immune Suppression GOOD
18. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST 1 Figure 1: Thiopurine Patients with Metabolites and Labs Cases: 10,726 Individuals: 1,909 Not on a Thiopurine Cases: 1,215 Individuals: 601 Evidence of C. difficile Cases: 584 Individuals: 155 ThioMon v2 Dataset Cases: 3,269 Individuals: 1,082 On a Biologic Cases: 4,010 Individuals: 706 Not IBD Cases: 38 Individuals: 16 Duplicate cases Cases: 1,599 Individuals: 558 Missing Values Cases: 31 Individuals: 26 Biologic Remission Cases: 1,843 Individuals: 731 Non-BR Cases: 1,426 Individuals: 699 All normal (30d) ESR, CRP, FCP Endoscopy, Histology CTE, MRE At least 1 abnormal ESR, CRP, FCP Endoscopy, Histology CTE, MRE
19. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST MCV/WBC ratio for everyday use MCV 100 WBC 4 25 MCV 80 WBC 4 20 MCV 90 WBC 6 15 MCV 75 WBC 7.5 10 MCV 72 WBC 9 8 ratio values 13
20. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Does This Externally Validate? • SONIC data • Compared • Azathioprine • Infliximab • Aza + IFX • In Crohn’s disease Sensitivity Waljee, et al. Clinical Gastroenterology and Hepatology 2018;16:449–451
21. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST MICHIGAN IBD • FAIL FAST Are Clinical Outcomes Better with Sustained Immune Suppression Pattern vs. Not Immune Suppressed? Results
22. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST 0.76 2.39 0.26 1.31 0.06 0.25 1.08 3.95 0 1 2 3 4 5 Mean Events Per Year SPIS Steroid NIS Steroid SPIS Hospital NIS Hospital SPIS Surgery NIS Surgery SPIS Total NIS Total Clinical Events Clinical Event Rate with Sustained Predicted Immune Suppression vs. Non-Immune Suppression p < 0.001 N=274 N=235 p=0.065 p < 0.001 p < 0.001 Clinical Events Per Year GOOD GOOD GOOD GOOD BAD BAD BAD BAD
23. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST MICHIGAN IBD • FAIL FAST If patients change from Not Immune Suppressed (BAD) to Sustained Immune Suppressed (GOOD), do Clinical outcomes improve? RESULTS
24. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST -2.4 -1.5 -0.5 -4.4 -6 -5 -4 -3 -2 -1 0 Steroid Prescriptions Hospitalizations Surgeries Total Clinical Events Change in Mean Events Per Year with Thiopurines Effect of Achieving Sustained Predicted Immune Suppression N=32 cases Change in Clinical Events Per Year
25. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Limitations • These results are only as generalizable as the data from the phase 3 clinical trial that led to FDA approval. • The simple MCV/WBC model was developed post hoc, and tested on the full data set, and would benefit from external validation.
26. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST ThioMon implementation Activates orders for CBCPD, COMP, and THIOCALC THIOCALC looks for results from CBCPD, COMP R server uses results to calculate IMMUNE SUPPRESSION SCORE, NON- ADHERENCE, and SHUNTING 3 results trigger branching logic which selects an interpretive paragraph Report back to Soft LIS ThioMon is a ‘SuperOrder’ In Soft LIS User Orders ThioMon In Epic EMR Report back to Epic EMR Data sent as csv to R server (virtual machine) Results saved on server Any failure to run/result – message to path IT team If too much missing data, reports test failure. Single missing values imputed with warning message.
27. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Result Reporting • Provide limited interpretation with branching logic • Some providers will use this test < 10 times per year • Education from a year ago will not stick. Thiopurine Monitoring Test Immunosuppression Score 102.3 (>100) GOOD Result – patient has achieved effective immune suppression with thiopurines. If continued symptoms, consider infection, IBS, or drug failure necessitating a different class of therapy. Thiopurine Monitoring Test Immunosuppression Score 92.7 (>100) Shunting Score 91.1 (>100) Non-adherence Score 94.4 (>100) LOW Result – patient has not achieved effective immune suppression with thiopurines. No evidence of shunting or non- adherence. Consider increasing dose, adding allopurinol, or a different class of therapy.
28. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST The Competition • Metabolite testing from Prometheus Labs/Nestle • 6-TGN and 6-MMP are active and toxic metabolites • Measurable with HPLC, there is a CPT code • Mostly covered by insurance • NOT a good test – 3 prospective RCTs failed • But marketed very well • These are the same people who can sell billions of chocolate bars contaminated with stale rice.
29. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Pathology dollars saved • Previously ordered over 600 metabolite tests per year @ $200 each • Saved > $120,000 per year in external costs • Internal algorithm nearly free (virtual machine) • Happy pathologists and accountants • A research project that actually saved money! Dr. Jeffrey Myers Vice Chair of Clinical Affairs and Quality
30. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST MICHIGAN IBD • FAIL FAST Vedolizumab Machine Learning Will this Patient benefit from this anti-integrin Therapy?
31. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Vedloizumab in Ulcerative Colitis • Vedolizumab is an effective, FDA-approved therapy for UC. • Vedolizumab can be slow to produce complete remission • Only 16.9% in clinical remission at week 6. • Vedolizumab, like other biologic medications, is very expensive. Feagan, BG, et al., NEJM 2013; 369: 699-71 List prices for a single infusion of Vedolizumab: - Illinois: $9,973 - New York: $18,148
32. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Decisions • It can be difficult to decide what to do with the patient with limited early response Keep dosing? Shorten Interval? Change to another Rx? 83.1 % Add a booster Rx? Aim: To predict at an early time point whether a UC patient on vedolizumab will achieve corticosteroid-free endoscopic remission at week 52.
33. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Methods • For entry: active ulcerative colitis with Mayo 6-12 and sigmoidoscopy score of 2-3. • Strict endpoint: • Week 52 Corticosteroid-free endoscopic remission: • No use of systemic steroids nor budesonide at week 52. • Mayo endoscopy score 0-1 at week 52 Feagan, BG, et al. NEJM 2013; 369: 699-710.
34. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Methods • Exclusions: subjects randomized to placebo, OR with missing predictors or outcome data • Randomly divided complete data into 70% model training set and a 30% model testing set • Trained model with 1000 RandomForest trees on data from: • Through week 6 (before 3rd dose of Vedolizumab)
35. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Predictor Variables Tested • All components of complete blood count with differential (0,6) • All components of comprehensive chemistry panel (0,6) • Uric acid, fecal calprotectin (0,6) • Vedolizumab level at weeks (2, 6) • Age, gender, race, height, weight • Immunomodulator at entry, steroids at entry, prior anti-TNF • Assigned vedolizumab interval (4 vs. 8 weeks) • Disease extent, baseline sigmoid activity, disease duration • Slope, mean, min, max, acceleration of all lab values
36. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Dataset 895 subjects 491 subjects in modeling dataset Assigned to Placebo: 275 subjects Missing Predictor Variables: 125 subjects Missing Outcome Data: 4 subjects 342 subjects in training dataset 148 subjects in testing dataset 70% 30%
37. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Results • Variable Importance at Week 6
38. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Results • Week 6 Model • Accuracy on Testing Set
39. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Prediction Success with Week 6 Model UC subjects in testing set N=148 Model Predicts as Likely to Succeed Model Predicts as Likely to Fail 47.3% 52.7% 59% Are Steroid-Free with Healed Mucosa Only 21% Are Steroid-Free with Healed Mucosa
40. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Challenges to Applying these Results • Need labs at multiple time points – complicated logistics • Complex IT Infrastructure required to • Maintain privacy when sending health data • Link labs from multiple time points. • Run random forest models • Is there a simpler way? • Propose a simpler model post hoc • Single model (avoid multiplicity of testing) • Test on full data set Week 6 Fecal Calprotectin Week 6 Vedolizumab Level
41. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Results with Simple Predictor • A week 6 FCP/VDZ ratio of < 12.35 had an AuROC of 0.71 • 95% CI: 0.67 - 0.76 Week 6 Fecal Calprotectin Week 6 Vedolizumab Level UC subjects in complete modeling dataset (N=491) FCP/VDZ < 12.35 (N=225) FCP/VDZ >= 12.35 (N=266) 45.8% 54.2% 52% were Steroid-Free with Healed Mucosa at week 52 Only 21% were Steroid-Free with Healed Mucosa at week 52
42. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Results with Simple Model Week 6 Fecal Calprotectin Week 6 Vedolizumab Level 65.3 40.9 25.8 13.1 0 10 20 30 40 50 60 70 0-3 3-15 15-75 75-Infinity Percentage Achieving Corticosteroid-Free Endoscopic Remission at Week 52 Week 6 FCP/VDZ Ratio FCP 100 VDZ 50 FCP 300 VDZ 30 FCP 600 VDZ 20 FCP 1000 VDZ 10 Week 6 Fecal Calprotectin Week 6 Vedolizumab Level
43. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Conclusions • Random forest models using data through week 6 can accurately classify UC subjects on vedolizumab as likely to succeed or fail in achieving steroid-free week 52 endoscopic remission.
44. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Speculations • For patients classified as likely to fail, the optimal therapeutic approach is not clear, but could include: • Changing to a new medication class • Adding a booster induction medication (anti-TNF, JAK, anti-IL23) • Shortening the interval • Increasing the dose • For the 53% classified as likely to fail, additional RCTs are needed to optimize the use of vedolizumab in this group of patients. Triple combination trial: Vedo + ADA + MTX Co-Induction trial: IFX + Vedo
45. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Can We Do this In Crohn’s Disease? • Vedolizumab is less effective in CD • Only 14% Clinical Remission at Week 52 • Endpoint: steroid-free biologic remission • Can data through week 6 identify two groups of patients: • Those likely to succeed at a high rate • Those likely to fail to achieve steroid-free biologic remission
46. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Methods • Exclusions: subjects randomized to placebo, OR with missing predictors or outcome data, or a normal CRP • Success endpoint: steroid-free, with normal CRP at week 52 • Randomly divided complete data into 70% model training set and a 30% model testing set • Trained model with 1000 RandomForest trees on data: • Through week 6 (before 3rd dose of Vedolizumab)
47. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST AuROC and Variable Importance for CD
48. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Prediction Success with Week 6 Model CD subjects in testing set N=142 Model Predicts as Likely to Succeed Model Predicts as Likely to Fail 37.3% 62.7% 36% Are Steroid-Free with Normal CRP Only 7% Are Steroid-Free with Normal CRP
49. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Can You Use a Simple Predictor? • Not as simple as UC • If Predictor > 186, subject is likely to achieve week 52 CS-free biologic remission • Tested on entire (N=472) dataset, AuROC = 0.75 (95% CI: 0.70-0.81) Week 6 Hgb x Alb x Vedo Level Week 6 CRP x wt (kg) Predictor =
50. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Applications? • Prospective validation is needed • Use of this predictor could support clinical decisions to • Continue medication • Change to an alternative therapy at week 6 • Increase dose or shorten interval • Add a booster co-therapy • A prospective study to compare these options is needed
51. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST MICHIGAN IBD • FAIL FAST Ustekinumab Machine Learning Will this Patient benefit from this anti-IL12/23 Therapy?
52. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST All Patients N= 1409 Final Dataset N= 401 Patients on Ustekinumab N= 741 CRP less than 5 mg/L N= 279 Missing Predictors N= 61 Model Training Data N= 280 Model Testing Data N= 121 Ustekinumab Data • Data from YODA • Phase 3 trials • Drug vs. placebo • Required measurable inflammation • CRP > 5 mg/L at entry • Outcome: • Steroid free remission beyond week 42
53. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Predictor Importance Inflammatory burden >> Drug levels
54. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST AuROC curve with week 8 data
55. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Prediction Success with Week 8 Model CD subjects in testing set N=110 Model Predicts as Likely to Succeed Model Predicts as Likely to Fail 55 (46%) 49% Are Steroid-Free with Normal CRP Only 11% Are Steroid-Free with Normal CRP 65 (54%)
56. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Can You Use a Simple Predictor? • Albumin/CRP ratio at week 6 • Time to decide whether to give next dose at week 8 • If Predictor > 4.92, subject is likely to achieve CS-free biologic remission > week 42 • Tested on entire (N=398) dataset, AuROC = 0.76 (95% CI: 0. 0.71 - 0.82) Week 6 Albumin Week 6 CRP Predictor =
57. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Can You Use A Simple Predictor ? • A week 6 Alb/CRP ratio of > 4.92 had an AuROC of 0.76 • 95% CI: 0.71 - 0.82 Week 6 Albumin Week 6 CRP UC subjects in complete modeling dataset (N=398) Alb/CRP > 4.92 (N=225) Alb/CRP <= 4.92 (N=266) 166 (41.7%) 232 (58.3%) 40% were Steroid-Free with Healed Mucosa at week 52 Only 9% were Steroid-Free with Healed Mucosa at week 52
58. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Applications? • Prospective validation is needed • Use of this predictor could support clinical decisions to • Continue medication • Change to an alternative therapy at week 8 • Increase dose or shorten interval • Add a booster co-therapy • A prospective study to compare these options is needed
59. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST What Have We Learned Along the Way • We can predict early in the course of therapy whether a thiopurine, vedolizumab, or ustekinumab will be effective in the long term using patterns laboratory responses • Drug levels matter a lot for anti-TNFs, somewhat for Vedo, but not much for Uste or Thiopurines • Newer, better drugs are being dosed appropriately, and drug levels are becoming less important for newer biologics (and were never that important for thiopurines)
60. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Inflammatory Burden • You need some inflammation for an anti-inflammatory therapy to work better than placebo 29.6% 40.7% 50.5% 33.3% 61.3% 40.0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Endoscopic Ulcers No Ulcers Clinical Remission At Week 26 in SONIC Aza IFX Aza + IFX
61. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Inflammatory Burden • We are bad at predicting long-term outcomes from lab values before starting a medication – though inflammatory burden helps • Too much inflammation (and intestinal protein leak) make vedolizumab and ustekinumab less likely to work • Less important for small molecules like steroids and JAK inhibitors • Predictors of good outcomes • High Albumin, Hgb, low body weight for Vedo • Predictors of bad outcomes • High CRP and FCP
62. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Thanks To • IBD Research Team • Machine Learning Team Boang Liu Ji Zhu Akbar Waljee Kay Sauder Ryan Stidham
63. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Thanks To • CSDR – Clinical Study Data Request site https://clinicalstudydatarequest.com/ • YODA – Yale Open Data Access project • Takeda and Janssen – for sharing their clinical trial data publicly PETER HIGGINS Peter D.R. Higgins, MD, PhD, MSc
64. MICHIGAN MEDICINE MICHIGAN IBD • FAIL FAST Thank You