www.netspective.com © 2021 Netspective. All Rights Reserved. 1 Quantitative Regulatory Science (“RegOps”) for FDA Digital Health/Therapeutics Compliance By Shahid N. Shah, Founder, Netspective Communications LLC This Photo by Unknown Author is licensed under CC BY 

www.netspective.com © 2021 Netspective. All Rights Reserved. 2 This deck is available at http://www.speakerdeck.com/shah @ShahidNShah 

www.netspective.com © 2021 Netspective. All Rights Reserved. 3 • Medical Devices, Gov’t Tech & Security Advisor • 15+ years of safety-critical compliance and cybersecurity expertise (in healthcare, government, and other sectors) • 18+ years of healthcare IT and medical devices experience (blog at https://healthcareguy.com and publisher at https://medigy.com) • 20+ years of technology management experience (government, non-profit, commercial) • 30 years of software engineering and multi-discipline complex IT implementations (Gov., defense, health, finance, insurance) Author of two chapters: “Understanding Medical Practice Cybersecurity Risks” and “How to Conduct a Health-Care Environment Electronic Risk Assessment” Who is Shahid? 

www.netspective.com © 2021 Netspective. All Rights Reserved. 4 What’s this talk about? What we will discuss FDA is looking to innovate around how digital therapeutic solutions are regulated (“Software as a Medical Device”, SaMD). Software systems are not “manufactured” so traditional CGMP standards are less applicable. Unique challenges of applying regulatory science to software-intensive medical devices. Key takeaways • Software has a well-defined systems development lifecycle (SDLC). If regulatory science is applied properly to those lifecycle steps, SaMD solutions have a chance of being even safer and more compliant than their software+hardware cousins. • Consider creating an OKR-based compliance strategy at your organization. • Deterministic reproducibility, observability, and transparency assurance through RegOps (regulatory operations) tied to DevOps, DevSecOps, DataOps, and SRE. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 5 https://www.fda.gov/media/145022/download https://www.fda.gov/media/80958/download 

www.netspective.com © 2021 Netspective. All Rights Reserved. 6 Most Popular “AI” in Healthcare Beware of “AI Washing” – it’s all just software https://towardsdatascience.com/ethical-ai-its-implications-for-enterprise-ai-use-cases-and-governance-81602078f5db 

www.netspective.com © 2021 Netspective. All Rights Reserved. 7 Software is often non- deterministic Reproducible reliability difficult to achieve. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 8 Most software is easy to build these days, especially software driven by machine learning. Challenge moves from building to testing. https://insights.sei.cmu.edu/blog/the-challenges-of-testing-in-a-non-deterministic-world/ 

www.netspective.com © 2021 Netspective. All Rights Reserved. 9 Areas of non- determinism that must be addressed https://insights.sei.cmu.edu/blog/the-challenges-of-testing-in-a-non-deterministic-world/ 

www.netspective.com © 2021 Netspective. All Rights Reserved. 10 Lack of AI “explainability” and auditability increase challenges of deterministic reproducibility A model agnostic explainability frameworks, Local Interpretable Model-Agnostic Explanations (LIME), is “a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner by learning an interpretable model locally around the prediction.” https://towardsdatascience.com/ethical-ai-its-implications-for-enterprise-ai-use-cases-and-governance-81602078f5db 

www.netspective.com © 2021 Netspective. All Rights Reserved. 11 NASA Software Safety Guidebook 

www.netspective.com © 2021 Netspective. All Rights Reserved. 12 

www.netspective.com © 2021 Netspective. All Rights Reserved. 13 FHIR-capable systems (EHRs) Non-HL7/FHIR EHRs, RCMs, etc. Government or any Reporting Agencies (CDC, CMS, FDA, et. al.) RegOps Participant 1 (Manufacturer, Health System, HIE, Insurer, etc.) RegOps Participant 2 RegOps Participant N RegOps Participant 3 RegOps Outcomes Observatory HL7-capable systems (EHRs) Source Databases Any Database Multiple source systems within a physician practice, in a hospital department, across a health system, inside a payer, state health departments, or an entire HIE can generate standardized Metrics and Measures through existing or custom RegOps Exporters RegOps Gateways Local Dashboards, Alerts, Reports RegOps Dashboards Participants can join multiple bespoke RegOps business networks over blockchain or DNS-style chains Publishing can be alerts or documents Medical Devices (Class 1, 2, or 3) Safety/Error Reporting Systems RegOps Publishers Specifications providers such as CMS, NQF, NCQA, FDA, and others can create executable specifications that live in open source or commercial exporters (actual software) Given the time-series database kept by RegOps, full claims records and complete EHR exchange is not required. RegOps Reference Architecture Everything we need is available as Open Source Software (OSS). RegOps approach allows regulatory artifacts to be generated as byproducts of the SaMD development effort. Start from the real-world and look inward, not the other way around. Auto-generate compliance artifacts 

www.netspective.com © 2021 Netspective. All Rights Reserved. 14 Trend: Software is eating medical devices1 FDA intends to apply its regulatory oversight to only those software functions that are medical devices and whose functionality could pose a risk to a patient’s safety if the device were to not function as intended. FDA Policy for Device Software Functions and Mobile Medical Applications (September 2019) 1 https://a16z.com/2011/08/20/why-software-is-eating-the-world/ FDA says there are no question that these are regulated: • Software functions that are an extension of one or more medical devices by connecting to such device(s) for purposes of controlling the device(s) or analyzing medical device data. • Software functions (typically, mobile apps) that transform the mobile platform into a regulated medical device by using attachments, display screens, or sensors or by including functionalities similar to those of currently regulated medical devices. • Software functions that become a regulated medical device by performing patient- specific analysis and providing patient-specific diagnosis, or treatment recommendations. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 15 Trend: Software is increasingly working together What happens when software and systems that were never tested together are forced to live and work together in the “wild”? What does “intended use” mean after integration? FDA discretion as to whether these are regulated: • Software functions that supplement professional clinical care by facilitating behavioral change or coaching patients with specific diseases or identifiable health conditions in their daily environment. • Software that provides contextually-relevant information to users by matching patient- specific information (e.g., diagnosis, treatments, allergies, signs, or symptoms) to reference information routinely used in clinical practice (e.g., practice guidelines) to facilitate a user’s assessment of a specific patient. • Software functions that are specifically marketed to help patients communicate with healthcare providers by supplementing or augmenting the data or information by capturing an image for patients to convey to their healthcare providers about potential medical conditions. • Software functions that perform simple calculations routinely used in clinical practice. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 16 Trend: Software is eating medical devices1 FDA intends to apply its regulatory oversight to only those software functions that are medical devices and whose functionality could pose a risk to a patient’s safety if the device were to not function as intended. FDA Policy for Device Software Functions and Mobile Medical Applications (September 2019) 1 https://a16z.com/2011/08/20/why-software-is-eating-the-world/ FDA says many things are unregulated, but many will be at their discretion. https://www.fda.gov/media/80958/download Think of 510(k) and other regulatory compliance as an opportunity, not a cost. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 17 Trend: Compliance “Theatre” We want to sound smart, so we talk about risk-based compliance but we’re no better than airport security theatre 

www.netspective.com © 2021 Netspective. All Rights Reserved. 18 Example: Is the MDS2 Security Theatre? Is this third-party risk-based or just making us feel good? http://www.himss.org/resourcelibrary/MDS2 

www.netspective.com © 2021 Netspective. All Rights Reserved. 19 https://atos.net/en/blog/applying-artificial-intelligence-in-existing-business-processes 

www.netspective.com © 2021 Netspective. All Rights Reserved. 20 Compliance: often binary (yes/no) Safety and security: must be continuous You can be compliant and not safe, safe but not compliant, or both Compliant but unsafe is pretty common Trend: Compliance does not lead to safety Unless generated from real-world evidence, compliance leads to a false sense of security Safe & 

www.netspective.com © 2021 Netspective. All Rights Reserved. 21 There is a safe software development crisis, especially with Machine Learning / AI. Our software development lifecycles, languages, and tools are built for deterministic environments with systems that stand alone, not for fully integrated non-deterministic environments. Safe development lifecycles and modern software supply chain techniques must be implemented. Building regulatory practices directly into the systems development lifecycle could ease the inspection and audit burdens but create more reliable results. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 22 Preparing annual controls catalogs and compliance documentation or passing audits doesn’t mean you’re safe. Not enough organizations differentiate between point in time assessments versus continuous monitoring. Only continuous monitoring of each safety-critical asset, from the bottom-up, ensures safety. This is what I call RegOps – it’s an extension of DevOps, DevSecOps, and SRE. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 23 ▪ concurrency defects: ▪ classic defects, such as deadlock, livelock, starvation, suspension, priority inversion, (data) race conditions, order violations, atomicity violations, and lock and semaphore defects ▪ multicore defects, such as interference between cores due to sharing common resources (e.g., L2/L3 caches, RAM and disc memory, I/O, system bus), improper allocation of software to hardware components, and unacceptable jitter in processing times ▪ virtual machine defects, such as VM escapes and interference between VMs, improper allocation of SW to VMs, and hypervisor defects ▪ performance defects that cause missed deadlines (latency and response time), unacceptable jitter, and incorrect order of processing and events ▪ reliability defects that cause buffer overflow and bugs due to ill-timed automatic garbage collection ▪ robustness defects that cause missing, inadequate, or incorrect error, fault, failure, and environmental tolerance ▪ security defects including vulnerabilities and defects in security controls (e.g., incorrect configurations) ▪ rare alignments of cyclic behaviors that result in intermittent and non-reproducible failures ▪ positive feedback under stress propagated through a system's environment triggering showstoppers or dangerous loss of control ▪ bizarre responses that are internally consistent but externally dangerous https://insights.sei.cmu.edu/blog/the-challenges-of-testing-in-a-non-deterministic-world/ Common defects in non-deterministic software 

www.netspective.com © 2021 Netspective. All Rights Reserved. 24 WHAT IS THE SOLUTION? OKRs, ML, AUTOMATION AUTO SCORING AUTO ASSESSMENTS DISTRIBUTED COMPLIANCE COLLABORATIVE EVIDENCE 

www.netspective.com © 2021 Netspective. All Rights Reserved. 25 Move from an oversight and validation regulatory focus to a continuous assurance strategy grounded in machine verifiable transparency and observability. The risk of a device and the excellence of a company should be related to the transparency/opacity of its measurable data which can easily be compared with other products / companies. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 26 Move from an “intended use” reporting model to one tied to “intended results” model built around the concept of objectives and key results (OKRs). Today we describe what a device does, but we should be describing what it’s designed to accomplish for patients, through objective measures that can be independently verified. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 27 Digitize compliance into structured data and integrate with software engineering THE NEXT CONSULTANT TO GIVE ADVICE ABOUT YET ANOTHER DOCUMENT OR TEMPLATE SHOULD BE FIRED 

www.netspective.com © 2021 Netspective. All Rights Reserved. 28 Use Objectives & Key Results (OKRs) Setting quantifiable goals with objective measures should be the focus INTEL CORPRORATE OBJECTIVE in 1980 Establish the 8086 as the highest performance 16-bit microprocessor family, as measured by: KEY RESULTS (Q2 1980) 1. Develop and publish five benchmarks showing superior 8086 family performance (Applications). 2. Repackage the entire 8086 family of products Marketing). 3. Get the 8MHz part into production (Engineering, Manufacturing). 4. Sample the arithmetic coprocessor no later than June 15 (Engineering). 

www.netspective.com © 2021 Netspective. All Rights Reserved. 29 How OKRs would work in safety critical software Setting measurables goals with objective measures should be the focus Useful Software Safety Objective Automatically notify device operator and cloud service desk when a software exception is thrown. Potential Key Results 1. All configuration checks are automatically performed, with violation findings recorded in Jira. 2. At least five members of IT are trained to understand and act on violation findings by March 15. Useful Cybersecurity Objective Reduce the likelihood of “A production AWS credential was exposed to the public in Q3”. Potential Key Results 1. Commits mentioning AWS_SECRET_KEY show up in the #security slack. 2. The photobackup pipeline will be moved to an AWS role. 3. Complete Security Monkey alerting pipeline towards our detection on-call. 4. Complete a before & after forecast, and CloudTrail hunt. https://medium.com/starting-up-security/how-to-measure-risk-with-a-better-okr-c259bccf359e 

www.netspective.com © 2021 Netspective. All Rights Reserved. 30 Good Safety Critical Professionals use OKRs Tie your safety and compliance goals with corporate or agency goals and make your bosses sign off https://medium.com/starting-up-security/how-to-measure-risk-with-a-better-okr-c259bccf359e https://community.isc2.org/t5/Tech-Talk/OKR-s-for-a-Security-Team/td-p/16048 

www.netspective.com © 2021 Netspective. All Rights Reserved. 31 If you’re not sure, start with FAIR FAIR, short for “Factor Analysis of Information Risk,” is an international standard quantitative model for operational risk. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 32 Asset management –Requirements management –Procurement authority –Intake procedures –Asset ownership/custodian gap –Charging & storage locations Operations –Tracking MD by end users –Disposal procedures Maintenance management –Patch management –Alerts Access management –Unauthorized remote access –Unauthorized local access Loss/theft –Audit logs w/user history Incident management –Incident reporting –Incident response –Forensics capabilities System engineering process –Software vulnerabilities oSecure software development oCommercial Operating Systems oOpen source software –Hardware vulnerabilities oPatching limitations oHardcoding generic credentials oEasily accessible USB ports on portable devices Software maintenance process –Secure coding environment oSeparation of Dev & Ops oCode test & validation oCMDB –Patch distribution process oPatch integrity checks Cloud operations –Access controls oSeparation of Dev & Ops –Supply chain oCode test & validation Insider threats Supply Chain Supply Chain Physical Insider LAN OS Data Are current frameworks & tools useful? Source: Cynergistek Development Risks (Manufacturers) Production Risks (Manufacturers & Customers) How are things supposed to improve when each manufacturer is independent, but customer risks are collaborative? 

www.netspective.com © 2021 Netspective. All Rights Reserved. 33 https://securityintelligence.com/how-stix-taxii-and-cybox-can-help-with-standardizing-threat-information/ http://www.redzonetech.net/podcast/aharon-chernin/ Structured threat sharing is a must Trusted Automated Exchange of Indicator Information (TAXII) Cyber Observable Expression (CybOX) Structured Threat Information Expression (STIX) 

www.netspective.com © 2021 Netspective. All Rights Reserved. 34 Secure the Software Supply Chain • Identify all the stuff in your pipeline; from people, code, dependencies, to infrastructure • Ensure a consistent and quality build process • Protect the product while in storage and transit • Guarantee and validate the final product at delivery against a bill of materials https://m-square.com.au/securing-the-enterprise-software-supply-chain-using-docker/ 

www.netspective.com © 2021 Netspective. All Rights Reserved. 35 Let’s reimagine compliance so it’s focused on continuous assurance of reproducible reliability 

www.netspective.com © 2021 Netspective. All Rights Reserved. 36 Requirements OKRs Structured, machine and human readable, machine computable, and machine assurable requirements. • It’s not a set of requirements we’re after but after a hierarchical set of objectives that the device is designed to achieve. • Need to be able to compare multiple devices using similar definitions (if two devices are looking to accomplish the same thing, we should be able to specify that). 

www.netspective.com © 2021 Netspective. All Rights Reserved. 37 Objectives – what the device will accomplish (not what it does) • Explicit • Qualitative • Address a target user • Align with the manufacturer’s long term mission, KPIs, product vision, and business goals • Evaluated for efficacy during every product release cycle Key Results – how the accomplishment is judged • Specific • Quantitative • Instrumentable • Measurable • No change for disagreement when achieved • Explicitly define the metric by which they will be monitored (e.g., OpenMetrics exposition format) • Accompanied by a monitoring dashboard (e.g., open source JSON definition) 

www.netspective.com © 2021 Netspective. All Rights Reserved. 38 Opaque ➔ Transparent Instrumented Structured, machine and human readable, machine verifiable, model driven development for engineering (AADL, Architecture Analysis & Design Language and contract driven programming). FDA and regulators should be able to test the design of a device by executing its models. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 39 Discrete ➔ Continuous Migrate from SDLC and quality systems-based evaluation to deterministic reproducibility through machine veriable structured definitions and models (machines can verify cybersecurity, requirements, design, architecture, QA, and artifacts). • Measure the outcome of a process, not the process (SDLC) itself • FDA and regulators should be able to run simulators automatically without any custom work 

www.netspective.com © 2021 Netspective. All Rights Reserved. 40 Reportable ➔ Actionable Live, real-world feedback for continuous assurance 

www.netspective.com © 2021 Netspective. All Rights Reserved. 41 Consensus-driven Data-driven Observable Telemetry and Metrics 

www.netspective.com © 2021 Netspective. All Rights Reserved. 42 Every suggestion being made is already being done in production across many, even non-safe-critical, software systems What’s our excuse? THE NEXT CONSULTANT TO GIVE ADVICE ABOUT YET ANOTHER DOCUMENT OR TEMPLATE SHOULD BE FIRED 

www.netspective.com © 2021 Netspective. All Rights Reserved. 43 

www.netspective.com © 2021 Netspective. All Rights Reserved. 44 

www.netspective.com © 2021 Netspective. All Rights Reserved. 45 Eminence-driven Evidence-driven Why do we need to wait for research papers? Ivory tower Real-world 

www.netspective.com © 2021 Netspective. All Rights Reserved. 46 Compliance ➔ Value Provable patient safety Not conjecture BS 

www.netspective.com © 2021 Netspective. All Rights Reserved. 47 “Paper” ➔ Digital Native Microsoft Word ➔ Markdown Microsoft Excel ➔ JSON or XML Manual Reviews ➔ Instrumentation APIs Test Plans ➔ Machine Assurance 

www.netspective.com © 2021 Netspective. All Rights Reserved. 48 Human ➔ Machine Generated Structured, machine and human readable, machine computable, and machine assurable QA metrics (OSS: OpenMetrics, OpenTelemetry style). • Need to specify what’s experimental, what’s confirmation, and what’s observed • FDA and government bodies should be able to receive anonymous telemetry with ability reidentify when necessary 

www.netspective.com © 2021 Netspective. All Rights Reserved. 49 Retrospective reporting Interactive telemetry EVENT DRIVEN SDKs 

www.netspective.com © 2021 Netspective. All Rights Reserved. 50 Institution-framed Patient-framed 

www.netspective.com © 2021 Netspective. All Rights Reserved. 51 Population-based Personalized 

www.netspective.com © 2021 Netspective. All Rights Reserved. 52 Measurement Process Improvement Continuous Assurance 

www.netspective.com © 2021 Netspective. All Rights Reserved. 53 Every suggestion being made is already being done in production across many, even non-safe-critical, software systems What’s our excuse? THE NEXT CONSULTANT TO GIVE ADVICE ABOUT YET ANOTHER DOCUMENT OR TEMPLATE SHOULD BE FIRED 

www.netspective.com © 2021 Netspective. All Rights Reserved. 54 What we learned What we will discuss FDA is looking to innovate around how digital therapeutic solutions are regulated (“Software as a Medical Device”, SaMD). Software systems are not “manufactured” so traditional CGMP standards are less applicable. Unique challenges of applying regulatory science to software-intensive medical devices. Key takeaways • Software has a well-defined systems development lifecycle (SDLC). If regulatory science is applied properly to those lifecycle steps, SaMD solutions have a chance of being even safer and more compliant than their software+hardware cousins. • Consider creating an OKR-based compliance strategy at your organization. • Deterministic reproducibility, observability, and transparency assurance through RegOps (regulatory operations) tied to DevOps, DevSecOps, DataOps, and SRE. 

www.netspective.com © 2021 Netspective. All Rights Reserved. 55 Quantitative Regulatory Science (“RegOps”) for FDA Digital Health/Therapeutics Compliance This Photo by Unknown Author is licensed under CC BY THANK YOU Shahid N. Shah ([email protected]) @ShahidNShah