Building a Enterprise Eventing Platform Bryan Zelle and Neil Buesing 

Centene Introduction Mission Statement: Transforming the health of the community, one person at a time Medicaid: Medicare (Part D): Marketplace: Medicare: Other: Total: 12,700,000 4,000,000 2,000,000 1,000,000 3,700,000 23,400,000 30 States 50 States 21 States 28 States 33 States 50 States Membership Composition: Industry: Largest Medicaid and Medicare Managed Care Provider 0 5 10 15 20 25 Centene United Health Group Humana Anthem CVS Membership (Millions) Largest Managed Care Organizations Medicaid Medicare & Medicare PDP Other Government Marketplace 

$- $10,000 $20,000 $30,000 $40,000 $50,000 $60,000 $70,000 $80,000 $90,000 $100,000 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 Total Revenus (millions) Centene Yearly Revenue Centene Revenue WellCare Revenue Summary of Centene’s Key Challenges in one word… Growth $4.1 Billion Revenue to $96.9 Billion in 10 Years $80.4 Billion in growth in past 5 years $48.6 Billion in growth in past 2½ years Envolve Jan 2015 Wellcare Mar 2019 Fidelis Sep 2017 HealthNet Mar 2016 ? ? Cause of the growth… Mergers & Acquisitions By the numbers: 

Medicare Medicaid International Federal Marketplace Addressable Market Federal Medicare $860 B 40% State Medicaid International Market Federal Services Health Insurance Marketplace $2,000,000,000,000 + Centene Revenue $97,000,000,000 + Centene Revenue 4% Addressable Market 96% Additional Growth Opportunities $710 B 33% $260 B 12% $120B 6% $115 B 5% Centene Growth Outlook Targeted Pipeline ($270 Billion) 

Medicare Medicaid International Federal Marketplace Addressable Market Federal Medicare $860 B 40% State Medicaid International Market Federal Services Health Insurance Marketplace $2,000,000,000,000 + Centene Revenue $97,000,000,000 + Centene Revenue 4% Addressable Market 96% Additional Growth Opportunities $710 B 33% $260 B 12% $120B 6% $115 B 5% Centene Growth Outlook Targeted Pipeline ($270 Billion) Mergers & Acquisitions Data Integration & Data Migration 

Data Integration & Data Migration 1 Shared Database • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure 

Data Integration & Data Migration Shared Database Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File 2 File Transfer (Batch ETL) • Latent Data • Direct Database Load • Consistency Challenges 

Data Integration & Data Migration Export Import Shared Database File Transfer (Batch ETL) • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure • Latent Data • Direct Database Load • Consistency Challenges File API API Function Call Response 3 • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation 

Data Integration & Data Migration Shared Database File Transfer (Batch ETL) Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File • Latent Data • Direct Database Load • Consistency Challenges API API Function Call Response • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation 4 Pub / Sub Messaging (Streaming ETL) Event Message Bus • Loosely Coupled • No Application Refactor • Highly Availability • Highly Scalable • Real-Time Data 

Data Integration & Data Migration Pub / Sub Messaging (Streaming ETL) Event Message Bus • Loosely Coupled • No Application Refactor • Highly Availability • Highly Scalable • Real-Time Data Shared Database File Transfer (Batch ETL) Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File • Latent Data • Direct Database Load • Consistency Challenges API API Function Call Response • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation * What is a Event? Definition: “A significant change in state” • Statement of fact (immutable) • Expects no response (or call to action) • Has a defined “timepoint” Persistence • Stateless: Notification Event • Stateful: Event-Carried State Transfer How can you use events? E1 E2 E3 + • Combine multiple Events E1 E2 + • Absence of an Event E1 E2 • Leverage Single Event 

Pub / Sub Messaging (Streaming ETL) Event Message Bus • Loosely Coupled • No Application Refactor* • Highly Availability • Highly Scalable • Real-Time Data Event Structure Example Event Payload (JSON vis REST) “Metadata” : { “Transaction ID” : “C7084816514A5D260”, “User ID” : “USER1”, “Time Stamp” : “201803051315400000000000”, “Transaction Type” : “UPDATE”, “Source System” : “d8amisou6p.MEMBER_CONTACT” } , “Event Body” : { “Event Type” : “Member-PCP-Change”, “Previous Value” : “Dr. John Smith”, “Updated Value” : “Dr. Bryan Zelle”, “Event Source” : “Inbound-Member-Call”, “Caller Information” : { “Name” : “Jane Doe”, “Inbound Number” : “1-614-847-0982”, “Call Resolution Status” : “5 - Highly Satisfied”, “First Call Resolution” : “Success”, “Internal Representative” : “CN-10238381”, ”Call Duration (Minutes)” : “8:19” } Transaction Metadata Who - Who changed the data ?* What - What data changed ? When - When the data changed ? Where - Where was the data changed ? Why - Why was the data changed ? What Event information are we capturing? Event Body * 

Data Integration & Data Migration Pub / Sub Messaging (Streaming ETL) Event Message Bus • Loosely Coupled • No Application Refactor • Highly Availability • Highly Scalable • Real-Time Data Shared Database File Transfer (Batch ETL) Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File • Latent Data • Direct Database Load • Consistency Challenges API API Function Call Response • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation * How do Events fit with Streaming? Business Driver: Business Enabler: Real Time Data Streaming Integrated Real Time Enterprise 

Event Driven Apps Defining Characteristic: Stream Driven Apps Where is my driver? What is the temperature? How long till driver arrives? How long till room cools? Leverage real-time events to display current state Defining Characteristic: Continually combine multiple streams of real-time events to extract value from state 

Event Driven Apps Defining Characteristic: Stream Driven Apps Where is my driver? What is the temperature? How long till driver arrives? How long till room cools? Leverage real-time events to display current state Defining Characteristic: Continually combine multiple streams of real-time events to extract value from state What is the temperature? How long till driver arrives? Events are KEY Apache Kafka Stores Events: Resilient / Durable Distributed / Highly Available High-Throughput / Low Latency But…. What Isn’t Provided by Kafka? 

Event Driven Apps Defining Characteristic: Stream Driven Apps Where is my driver? What is the temperature? How long till driver arrives? How long till room cools? Leverage real-time events to display current state Defining Characteristic: Continually combine multiple streams of real-time events to extract value from state What is the temperature? How long till driver arrives? But…. What Isn’t Provided by Kafka? Synthetic Events Event Registration (Event Discovery) Sensitive Data Redaction Event Encryption Flexible Schema Validation (JSON not AVRO) Automated Disaster Recovery (Event Rehydration) Searchable Events (Query Event Store) Distributed Tracing of Event (Event Lineage) Event Metrics (Event Dashboards) Consistent Event Structure (Data Governance) 

Pub / Sub Messaging (Streaming ETL) Event Message Bus • Loosely Coupled • No Application Refactor* • Highly Availability • Highly Scalable • Real-Time Data Mediated (Orchestrated) Eventing Mediator Topology Mediator transfers events to assigned event channel (Topic) Centrally Coordinated Event Routing Complete Decoupling of Event Channels Increased Complexity at cost of increased coordination of event execution Advantages: • Consistent / Common Framework • Enforce Data governance • Economy of Scale Advantage • Technology abstraction / decoupling Disadvantages: • External bottleneck (Mediator Owner) • Single Point of Failure • Duplicative data storage 

Generic Event Mediator Common Core Architecture: 1) Event Source 2) Event Intake 3) Event Channel 4) Event Router 5) Event Subscription 6) Event Destination Event Channel Event Router Event Subscription Event Destination Event Intake Event Source Event Mediator 3 2 1 4 5 6 Event Bridge Event Grid Apache Camel Knative Eventing Mule ESB Mediator Alternatives? 

Generic Event Mediator Required Features & Functionality Event Channel Event Router Event Subscription Event Destination Event Intake Event Source Event Mediator 3 2 1 4 5 6 Design Criteria 1) AVRO Event Serialization 2) JSON Validation of Event Body 3) Centralized Event Registry 4) Distributed Tracing of Events 5) Sensitive Data Redaction 6) Turn / Key Self-Service 7) Cloud Agnostic 8) Permanent Event Storage 9) Flexible Ingestion Intake 10) Pre-built Monitoring / Dashboards 11) Synthetic Events Reduced Message Size -> Reduced Storage Cost in Cloud Data Validation -> Clean Data Easily Find Events -> Prevents Event Duplication & Increases Adoption Tracing -> Provides Event Lineage and Auditability Data Restriction -> Protects HIPPA data (including PHI/PII) Automated Configuration -> Reduced manual administrative burden Multi-Cloud Strategy -> No Reliance on Single Cloud Provider Event Persistence -> DR Strategy + Event Sourcing / Hydration Legacy Systems Limitations -> Offer REST, gRPC, SOAP Interfaces & API’s Universal Metrics -> Consistent / Granular Event Visibility Fictitious Event -> Blue/Green Deployments, Prod Smoke Testing, Etc. Business Value 

1) AVRO Event Serialization 2) JSON Validation of Event Body 3) Centralized Event Registry 4) Distributed Tracing of Events 5) Sensitive Data Redaction 6) Turn / Key Self-Service 7) Cloud Agnostic 8) Permanent Event Storage 9) Flexible Ingestion Intake 10) Pre-built Monitoring / Dashboards 11) Synthetic Events Reduced Message Size -> Reduced Storage Cost in Cloud Data Validation -> Clean Data Easily Find Events -> Prevents Event Duplication & Increases Adoption Tracing -> Provides Event Lineage and Auditability Data Restriction -> Protects HIPPA data (including PHI/PII) Automated Configuration -> Reduced manual administrative burden Multi-Cloud Strategy -> No Reliance on Single Cloud Provider Event Persistence -> DR Strategy + Event Sourcing / Hydration Legacy Systems Limitations -> Offer REST, gRPC, SOAP Interfaces & API’s Universal Metrics -> Consistent / Granular Event Visibility Fictitious Event -> Blue/Green Deployments, Prod Smoke Testing, Etc. Generic Event Mediator Event Channel Event Router Event Subscription Event Destination Event Intake Event Source Event Mediator 3 2 1 4 5 6 Required Features & Functionality Design Criteria Business Value Leverage 3rd Party Frameworks or Build Custom? Assessment: Majority of frameworks focused engineering effort on how to get data into framework as easily as possible • Higher Data Ingest = Increased Revenue (SaaS) • Too many gaps with current features* • Limited flexibility because of so many customers Decision: Build Centralized Eventing Framework for Enterprise use across all Centene Domains * * * * * * 

CentEvent Architecture Docker Container Kubernetes Intake Application Axway Gateway Serialize Deserializer Confluent Schema Registry Caffeine Cache Authorization Tokens Event Types Routing Rules Routing HASH Firehose Topic Docker Container Kubernetes Router Application Consumer Topics Client Portal Event Discovery UI Admin UI Docker Container Kubernetes Admin API Mongo Charts Tracing Monitoring 

Field Level Data Redaction Use Case Example: Data Science Predication Models • Restricting Member PHI / PII (SSN, Medicaid / Medicare ID, etc) Encounter Processing • Restricting Bank/ Account Payment Information Business Requirement: Enforce Least Privilege Access to HIPPA / PHI / PII Data Design: I. Capture Event Metadata – What fields are sensitive? II. Capture Team / App level permissions – Who can see what? III. Duplicate Event -> Modify Event - Redact Sensitive Fields with ***REDACTED*** 

Discard Synthetic Events Use Case Example: Pass-Through Example - Service that validates Payment Account Number Discard Example - Service that processes Claim Payment Business Requirement: Integrated End-to-End Testing without Central Coordination • Synthetic Monitoring / Continuous Unit Testing • Leverage non-prod traffic for Blue / Green Deployments • Inline Production Troubleshooting (in real-time) Design: I. Mandate “Synthetic” Event Property II. Establish Micro-Service Pattern • Pass-Through Event • Discard Event Simulate Process Flow by Generating Synthetic Events Event Channel Pass Through Perform Action Discard Action = Synthetic Event 

Event Tagging Use Case Example: Health Plan Claim Processing - Tag Claim Events by which State its associated with Process Orchestration - Tag Events with which Step in the Process is next Business Requirement: Be able to route events based on “Event Tags” • Filter Kibana dashboard metrics by Tags • Aggregate SLA’s by Tags • Filter Jaeger Distributed Traces by Tags Design: I. Tags are Optional (0 to N) II. Tags are attached to event by Producer, to be leveraged by Consumer III. Can be used in any combination or order (flexible) • Same Event -> Different Tags • Different Events -> Same Tags Same Event (Different Tags) Different Events (Same Tags) 

Demo Build a Kafka Topic Request Process leveraging all Asynchronous Events Business Requirement: Additional Requirements: • Only process creation of “approved” topics • Create audit trial of any work performed for SOX compliance • Create Real + Synthetic Flows KaaS UI N: KaaS V: Requested O: Topic SO: Creation Authorizer N: Authorizer V: Sent O: Notification-Email SO: Approval-Request N: Authorizer V: Received O: Approval-Response SO: Email Audit Tag: Approved KaaS API Kafka Cluster N: KaaS-API V: Created O: Topic SO: Successfully B C E A B C D E UI Tag: Unapproved A N: KaaS V: Requested O: Topic SO: Creation D 

Demo Build a Kafka Topic Request Process leveraging all Asynchronous Events Business Requirement: Additional Requirements: • Only process creation of “approved” topics • Create audit trial of any work performed for SOX compliance • Create Real + Synthetic Flows KaaS UI N: KaaS V: Requested O: Topic SO: Creation Authorizer N: Authorizer V: Sent O: Notification-Email SO: Approval-Request N: Authorizer V: Received O: Approval-Response SO: Email Audit Tag: Approved KaaS API Kafka Cluster N: KaaS-API V: Created O: Topic SO: Successfully B C E A B C D E UI Tag: Unapproved A N: KaaS V: Requested O: Topic SO: Creation D Synthetic Process Flow Real Process Flow 

Recap Recap 1 Centene’s Core Challenge is Growth cause by Mergers & Acquisitions; causing us to revaluate our Enterprise Data Integration and Data Migration Strategies… Event Message Bus 2 Async Pub / Sub Eventing through Kafka provides us valuable capabilities: - Highly Scalable - High Autonomy / Decoupling - High Availability & Data Resiliency - Real Time Data Transfer - Complex Steam Processing “Metadata” : { “Transaction ID” : “C7084816514A5D260”, “User ID” : “USER1”, “Time Stamp” : “201803051315400000000000”, “Transaction Type” : “UPDATE”, “Source System” : “d8amisou6p.MEMBER_CONTACT” } , “Event Body” : { “Event Type” : “Member-PCP-Change”, “Previous Value” : “Dr. John Smith”, “Updated Value” : “Dr. Bryan Zelle”, “Event Source” : “Inbound-Member-Call”, “Caller Information” : { “Name” : “Jane Doe”, “Inbound Number” : “1-614-847-0982”, “Call Resolution Status” : “5 - Highly Satisfied”, “First Call Resolution” : “Success”, “Internal Representative” : “CN-10238381”, ”Call Duration (Minutes)” : “8:19” } 3 Leveraging a Mediator Topology enables the creation of meaningful events; which provide insight into why things are happening, so we can react to them in real time… 

Recap (Part 2) 4 Existing frameworks don’t meet all of Centene’s Eventing requirements; investment made to build our own Real-Time Eventing Platform 5 Leverage our Strategic Partners and their IP to build a Scalable Platform • Apache Kafka • K-Streams • Confluent Schema Registry • MongoDB • Kafka / Mongo Connector • MongoDB Charts 6 Execute on a multi-faceted Data Integration strategy that uses Asynchronous Eventing and Real-Time Data Streaming to facilitate current and future Mergers and Acquisitions 

28 Join Centene? Help us in our migration to Cloud Data Streaming Team is hiring! • Data Engineers • Site Reliability Engineers • QA Testers • PO / SM / BA • Software Engineers Reach out to: [email protected] 

Sara Zeman [email protected] 1515 Central Ave NE Suite 100 Minneapolis, MN 55413 Looking for a Real-Time Data Streaming Partner? ● Minneapolis (HQ), Omaha ● Founded in 1996 ● Clients Nationwide ● 150+ Consultants, all full-time employees Practice Areas • Cloud Engineering • Real-time Data • Modern APIs • Mobile and Web Special Thanks • Nick Larson • Ryan Hoffman • Neil Buesing 

30 Questions?