The Dark Side of AI: Four Things Worth Building in Diabetes Care

How can AI remove work from diabetes care without creating new risks for patients? Shahid Shah examines four possibilities: better-prepared visits, care between visits, one shared care plan and agreed tasks completed across organizations.

Using research findings and illustrative scenarios, the presentation examines missing information in polished summaries, alerts without an owner, conflicting recommendations and errors that spread across connected systems. It distinguishes AI functions and the evidence each requires, with practical proposals for verification, human oversight, training, bounded authority and recovery when systems fail.

Presented at Diabetes Technology Meeting, October 27, 2026. Timelines are conditional scenarios; proposed roles for ADA and DTS are not adopted standards or commitments.
https://shahidshah.com

Transcript

1. DIABETES TECHNOLOGY MEETING 2026 The Dark Side of AI End 2026 Prepared visits End 2027 Four things worth building in diabetes care, and how each one goes wrong Care between visits End 2028 One shared plan Shahid Shah • shahidshah.com 2029–30 Agreed tasks completed
2. THE CLAIM Most harm will come from the people and process around the model The news talks about rogue models. In a clinic, three ordinary things do the damage. A summary nobody checks The AI note drops a medication change. The clinician trusts it because it reads well. An alert nobody owns The service says it is watching. Nobody is on duty for the problem it just found. A person nobody trained We hand clinicians, patients and caregivers a tool and assume they know how to use it. DO THIS NOW • Before you switch on any AI tool, write three names: who checks its output, who answers its alerts, who teaches it. • If one of the three lines is blank, do not switch it on yet. • Keep the three names on the clinic phone, where the person on call can see them. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 2
3. ONE QUESTION FOR THE ROOM Think of one thing your patients should not have to do Telling their history again at every visit Chasing a refill Getting three different sets of advice Following up on something nobody finished Working out a referral on their own Keep that one in mind. For each future I show, ask whether it removes that job or adds a new one. DO THIS NOW • Ask your next ten patients the same question, at the front desk or in a portal message. • Their answers are your list of what to build first. Start with the one that comes up most. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 3
4. FOUR CONDITIONAL HORIZONS Four futures worth building by 2030. The dates are my guesses End 2026 End 2027 End 2028 2029–2030 Prepared visits Care between visits One shared plan Agreed tasks completed GOOD Less to review. Sources GOOD Gaps found. Follow-up GOOD Glucose, heart and GOOD Done across clinics, you can check. confirmed. kidney goals in one plan. pharmacies and home. DARK A clean summary hides DARK A problem found, and DARK Three tools, each DARK One mistake copied what it left out. nobody assigned to it. chasing its own target. into every system. DO THIS NOW • Find the horizon your clinic is in today and work on that one only. Most clinics are at the first. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 4
5. EVIDENCE WE ALREADY HAVE Automated insulin delivery has a trial behind it. Chatbots have none −0.6 percentage points HbA1c, automated insulin delivery vs usual insulin delivery 319 adults with insulin-treated type 2 diabetes, 13 weeks, both arms on CGM. 95% CI −0.8 to −0.4. One severe hypoglycemia event, in the AID arm. Funded by the device maker. Each kind of tool needs its own evidence Randomized trials. The device, the Insulin-delivery controller dosing, the recovery. Predictive model The target, the patients, the decision it changes. Generative summary Is it true, what did it leave out, how do people use it. Agent with tools What it may do, what it did, how you undo it. DO THIS NOW • Make a one-line list of every AI tool in the clinic: what kind it is, and what trial backs it. • If the trial column says none, treat the tool's output as a draft, and check it before you use it. • Keep the list where staff can see it, and read it out when a new tool arrives. [1] Kudva et al., NEJM 2025 (AID in type 2 diabetes). The functional categories are a comparison, with no legal standing. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 5
6. END OF 2026 • PREPARED VISITS The cleaner the AI summary looks, the less you check it AI visit summary What it left out Reads well. Looks complete. Nothing on the page flags the gap. Glucose: time in range 68%, improving Metformin, semaglutide, basal insulin 24 units Basal insulin was cut to 18 units two weeks ago The CGM data is nine days stale. The sensor failed Last A1c 7.4%. Kidney function stable Plan: continue, follow up in 3 months The patient's own priority: fewer night alarms 3.45% of transcript sentences left out 1.47% of note sentences invented, in 450 simulated note pairs DO THIS NOW • Open the CGM app or pump report on your phone and check the date of the last reading first. • Read the medication list back to the patient and ask what changed since last time. • Time the check with your phone for a week. If checking takes longer than writing, say so. [13] Asgari et al., npj Digital Medicine 2025: 450 simulated note pairs; 286 omissions judged major; sentence rates, not patient events. The visit shown is invented. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 6
7. END OF 2027 • CARE BETWEEN VISITS Finding a problem does nothing unless someone is assigned to fix it 81 of 81 25 16 9 screened the same visit abnormal AI screens reached eye care within 6 months did not ACCESS trial, young people with diabetes, one academic center. The usual-referral arm screened 18 of 82. Same-visit convenience and the algorithm came together. Vision loss was never measured. What breaks • The service says it is watching, and nobody owns the exception. • Patients who stop replying drop off the dashboard. • More messages, and no more completed care. DO THIS NOW • Pick one alert type. Put one person's name and hours in the portal message and the text message. • On the caregiver's phone, save the clinic number as a contact, and test it once while they are with you. • Once a month, pull the list of patients who have not replied and phone them. [3] Wolf et al., Nature Communications 2024 (ACCESS). [2] Mathioudakis et al., JAMA 2025: AI coaching noninferior to human coaching; long-term prevention untested. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 7
8. END OF 2028 • ONE SHARED PLAN Three correct tools can add up to one plan nobody can follow Glucose tool Weight tool tighter targets, more checks new titration, new side effects Each tool hits its own target. Together they give the patient a plan nobody can live with. Digital twin evidence: 19 people, three months the person Ten adults with type 2 diabetes got AI feedback and nine did not, and a person reviewed every recommendation. A model can predict well under today's care and still tell you nothing about whether a change will help. Patients with the least data get the worst models. Personalization can widen the gap. Cardiorenal tool DO THIS NOW add a drug, cut sodium, more labs • Ask to see every app that gives them advice. • Count the plan's daily tasks. Cut before you add. [4] Wang et al., npj Health Systems 2026 (human-in-the-loop predictive digital twin, side-by-side comparison). The scene is made up. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 8
9. 2029–2030 • AGREED TASKS COMPLETED One wrong dose from an agent reaches four systems before anyone checks Clinic EHR agent drafts a refill request Pharmacy fills it, assumes the clinic checked Scheduling books follow-up on the wrong plan Home patient follows the confident message One wrong dose goes through four systems. Each one assumes another one checked. The patient finds out first. What breaks DO THIS NOW • Permissions build up and nobody removes them • An update changes behavior or leaves tasks with no owner • A system that sounds sure can still be wrong • Set the tool to ask before it sends anything. • Read every message it sent, the same day. • Tell patients to call if a message looks wrong. [6] FDA final PCCP guidance, Aug 2025: test planned changes and their system-wide effects (nonbinding). It says nothing about agents acting across organizations. [11] DTSec. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 9
10. GOVERNANCE Let a tool explain before it recommends, and recommend before it acts ACT RECOMMEND M u s t p r o ve EXPLAIN M u s t p r ove M u s t p r o ve Shows where each fact came from. Catches what was left out. Keeps the clinician's plan. Makes sense in the real workflow. Shows how unsure it is. Checked by trained people first. Has permission for that one action. Asks first when it should. Logs it. Can be undone. Doing well at one step earns nothing at the next Neither does the calendar. How much a tool may do on its own depends on the task, the patients, the evidence, and what happens when it goes wrong. IMDRF principle 7: test the people and the AI together, in the real setting, including skill, understanding and over-reliance. DO THIS NOW • For each AI tool in the clinic, write which step it is allowed: explain, recommend or act. • Most tools in a clinic today belong at explain. Say so on the tool's start page. • Move a tool up one step only after you have watched it get the lower step right. [8] IMDRF Good Machine Learning Practice principles, final Jan 2025. The three steps are my suggestion and have no regulatory meaning. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 10
11. HUMAN READINESS Training does not fix a bad interface Clinicians • Check the sources and the dates • Practice correcting a wrong recommendation • Practice working when the system is down Patients • Say the plan back in their own words • Know what is watched, and when a person responds • Know how to reach a person Caregivers • Agree roles and access with the patient • Practice raising the alarm • Practice handing over DO THIS NOW • At the next visit, ask the patient to say the plan back while you listen. It takes two minutes. • Put the clinic help number as a contact on the patient's phone and on the caregiver's phone. • Run one drill this quarter where the system is down and staff work from the paper plan. If training is the only thing keeping a patient safe, the interface is the problem. [12] ADA Standards of Care 2026, recs 7.3a and 7.3b (ongoing CGM and AID education). This framework is my suggestion; no curriculum has been validated. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 11
12. RESEARCH DESIGN A tool can pass its own tests and still not help patients OR 0.77 Count the people who dropped out, or you have measured the wrong group Compare with the best care you already have 95% CI 0.55 to 1.08. No significant difference. Measure outcomes, harm, work and access Treatment failure at 14 days: 102 of 4,693 with generative AI advice vs 94 of 4,654 with usual care. 16 general primary care clinics in Kenya. Include the people who never used it, quit, or went missing This does not mean the tool is useless. It means you only find out whether patients do better by measuring patients. Say who was trained, how, what it cost, and what they remembered Keep the prediction score separate from the patient outcome DO THIS NOW • Before a pilot starts, write down the name of everyone who was offered the tool. • At the end, count who is still using it, and phone five people who stopped. • Ask those five what went wrong. Most pilots skip that call. [5] Agweyu et al., Nature Medicine 2026 (pragmatic cluster-randomized trial). [8] IMDRF GMLP 2025. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 12
13. OPERATING PRACTICE Decide now what happens when it fails Before use At failure After repair • Say what it should do, what it cannot do, and what it costs • Name who responds, and when • Tell people how to reach a person • Find out who was affected, and which version • Keep the evidence, keep care running • Pause or limit the task when you need to • Test the fix, and the things that failed with it • Retrain users when the update changes their job • Explain what changed before you expand use Look at the interface, the workload and the training before you look at the user. DO THIS NOW • Write a one-page failure plan: who gets the call, what gets switched off, how patients are told. • Keep it next to the on-call phone, and practice it once with the people who would use it. [6] FDA final PCCP guidance, Aug 2025, VII.B(4): communicate changes, update user training. [7] Broader lifecycle guidance remains draft, not for implementation. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 13
14. PROPOSALS ONLY. NEITHER SOCIETY HAS AGREED TO THEM Ten jobs ADA and DTS could take on ADA Clinical practice and education • Bring clinicians, educators, patients and caregivers together to define what each role must be able to do • Write review skills and protected learning time into practice guidance • Get the specialties to agree on how to handle competing goals, uncertainty and workload • Set out how to disclose problems and review incidents without blame • Re-check readiness as evidence comes in, and treat no date as a commitment DTS Technical specification and testing • Build on iCoDE to test whether a summary matches its sources and what it left out • Bring developers and human-factors experts together on usability and training • Test understanding, referral completion and escalation with real users • Extend DTSec methods to agent permissions, recovery drills and update training • Standardize how training, retention, usability and recovery get reported DO THIS NOW • Nobody has to wait for either society. Any clinic can start the first two ADA jobs this month. Clinical recommendations, consensus specifications and regulatory authorization answer different questions. None of them alone certifies an AI tool as safe. [9] [10] [11] Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 14
15. ONE DELIVERABLE Give patients one less thing to do 1 Choose 2 Design 3 Show The job to remove, the patients, and the care decision. What you compare against, usually a good care team. Name how it fails, who owns it, who it affects, and what the fallback is. Pay for training and learning time. Test benefit, harm, work and recovery against real care. Test people on real tasks. Repeat as the people and the tools change. DO THIS NOW • This week: pick the job, write the one-page plan, and name who checks, who answers, who teaches. • Test it with ten patients on the phones they already have. Count who still uses it next month. • Use what has proven benefit, and investigate failures in the open. Backup slides and full references follow. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 15
16. BACKUP FOR QUESTIONS For questions What each date depends on, the kinds of AI, how far each study goes, and the references. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 16
17. THE DATES Each date depends on evidence and staff we do not have yet End 2026 Prepared visits Existing tools validated in the local workflow, with training and support in place End 2027 Care between visits Staffed response, patient and caregiver teach-back, accessible support End 2028 One shared plan Outcome evidence for personalized plans, and education about uncertainty 2029–2030 Agreed tasks completed Bounded authority, recovery drills and training after every update Adoption depends on clinical evidence, local capacity, affordability, patient preference and whatever authorization applies. Society guidance can describe the work; it cannot do the work. [9] ADA Standards of Care process • [10] DTS iCoDE 2022 • [11] DTSec Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 17
18. FUNCTIONAL CATEGORIES CAN COEXIST Each kind of AI needs its own kind of evidence Function What it does What needs scrutiny Insulin-delivery control Adjusts delivery within a validated device system Device, dosing and recovery Predictive model Estimates a target or a risk Target, population and the decision Generative AI Creates fluent content; truth not established Truth, omissions and interaction Diagnostic AI Performs a bounded screening task Accuracy and completed care Agent with tools Executes permitted tasks across systems Authority, each action and recovery A comparison of functions with no legal standing. AID may include predictive models, and some control algorithms are not machine learning. Clinical validation, technical conformance and regulatory status answer different questions. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 18
19. THE STUDIES Each study, and how far its result goes Automated insulin delivery Kudva, NEJM 2025 AI prevention coaching Mathioudakis, JAMA 2025 319 insulin-treated adults with T2D, 13 weeks, both arms on CGM. HbA1c −0.6 points (95% CI −0.8 to −0.4). 368 adults with prediabetes and overweight. Combined endpoint met by 31.7% vs 31.9%; margin −15 points, lower bound −8.2. Shows glycemic benefit for the tested system. Says nothing about time savings or AI summaries. Shows noninferiority for one program. Says nothing about long-term prevention or chatbots in general. ACCESS eye screening Predictive digital twin Wolf, Nat Commun 2024 Wang, npj Health Systems 2026 164 youth, ages 8–21. Screening 81 of 81 vs 18 of 82. Follow-up 16 of 25 AI-positive. 19 adults with T2D, 10 AI feedback vs 9 controls, 3 months, human review. Shows better screening completion, with access bundled in. Vision loss prevention was never tested. Early evidence only. Says nothing about autonomous treatment or care across diseases. [1] Kudva et al. • [2] Mathioudakis et al. • [3] Wolf et al. • [4] Wang et al. Methods and denominators in the notes. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 19
20. EVIDENCE AND EXISTING INSTITUTIONAL FOUNDATIONS References 1 of 2 [1] Kudva YC, Raghinaru D, Lum JW, et al. A Randomized Trial of Automated Insulin Delivery in Type 2 Diabetes. N Engl J Med. 2025;392:18011812. doi:10.1056/NEJMoa2415948 [2] Mathioudakis N, Lalani B, Abusamaan MS, et al. An AI-Powered Lifestyle Intervention vs Human Coaching in the Diabetes Prevention Program: A Randomized Clinical Trial. JAMA. 2025;334:2079-2089. doi:10.1001/jama.2025.19563 [3] Wolf RM, Channa R, Liu TYA, et al. Autonomous artificial intelligence increases screening and follow-up for diabetic retinopathy in youth: the ACCESS randomized control trial. Nat Commun. 2024;15:421. doi:10.1038/s41467-023-44676-z [4] Wang J, Faruqui SHA, Alaeddini A, et al. Human-in-the-loop AI predictive digital twin to extend virtual precision diabetes care between visits. npj Health Systems. 2026;3:59. doi:10.1038/s44401-026-00118-8 [5] Agweyu A, Mwaniki P, Menon V, et al. Generative AI-enabled clinical decision support system in primary care: a pragmatic, clusterrandomized trial. Nat Med. 2026;32:3032-3039. doi:10.1038/s41591-026-04503-6 [6] FDA. Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions. Final, 18 August 2025. Nonbinding recommendations. https://www.fda.gov/media/166704/download [7] FDA. Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations. Draft, January 2025, not for implementation. Status checked 26 October 2026. The ADA and DTS tasks in this deck are my proposals. Neither society has endorsed or committed to them. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 20
21. EVIDENCE AND EXISTING INSTITUTIONAL FOUNDATIONS References 2 of 2 [8] IMDRF. Good machine learning practice for medical device development: Guiding principles. IMDRF/AIML WG/N88 FINAL:2025. 27 January 2025. https://www.imdrf.org/ [9] American Diabetes Association. Standards of Care in Diabetes. Official clinical practice recommendation process, evidence review and update policy. Accessed 26 October 2026. https://professional.diabetes.org/standards-of-care [10] Espinoza J, Klonoff D, Vidmar A, et al. 2022 iCoDE Report: CGM-EHR Integration Standards and Recommendations. Diabetes Technology Society. Version 1.0, 7 November 2022. https://www.diabetestechnology.org/icode/ [11] Diabetes Technology Society. DTSec: Standard for Wireless Diabetes Device Security Assurance. Project page and public-release announcement, 23 May 2016. https://www.diabetestechnology.org/dtsec.shtml [12] American Diabetes Association Professional Practice Committee. 7. Diabetes Technology: Standards of Care in Diabetes 2026. Diabetes Care. 2026;49(Suppl 1):S150-S165. doi:10.2337/dc26-S007 [13] Asgari E, Montaña-Brown N, Dubois M, et al. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digital Medicine. 2025;8:274. doi:10.1038/s41746-025-01670-7 The ADA and DTS tasks in this deck are my proposals. Neither society has endorsed or committed to them. Shahid Shah • shahidshah.com • Diabetes Technology Meeting 2026 21