Building Enterprise Eventing Framework

Centene is fundamentally modernizing its legacy monolithic systems to support distributed, real-time event-driven healthcare information processing. A key part of our architecture is the development of a universal eventing framework to accommodate transformation into an event-driven architecture (EDA). Our application provides a representational state transfer (REST) and remote procedure call (gRPC) interface that allows development teams to publish and consume events with a simple Noun-Verb-Object (NVO) syntax. Embedded within the framework are structured schema evolutions with Confluent Schema Registry and AVRO, configurable (self-service) event-routing with K-Tables, dynamic event-aggregation with Kafka Streams, distributed event-tracing with Jaeger, and event querying against a MongoDB event-store hydrated by Kafka Connect. Lastly, we developed techniques to handle long-term event storage within Kafka; specifically surrounding the automated deletion of expired events and re-hydration of missing events. In Centene’s first business use case, events related to claim processing of provider reconsiderations was used to provide real-time updates to providers on the status of their claim appeals. To satisfy the business requirement, multiple monolith systems independently leveraged the event framework, to stream status updates for display on the Centene Provider Portal instantly. This provided a capability that was brand new to Centene: the ability to interact and engage with our providers in real-time through the use of event streams. In this presentation, we will walk you through the architecture of the eventing framework and showcase how our business requirements within our claims adjudication domain were able to be solved leveraging the Kafka Stream DSL and the Confluent Platform. And more importantly, how Centene plans on leveraging this framework, written on-top of Kafka Streams, to change our culture from batch processing to real-time stream processing.

Transcript

1. Building a Enterprise Eventing Platform Bryan Zelle and Neil Buesing
2. Centene Introduction Mission Statement: Transforming the health of the community, one person at a time Medicaid: Medicare (Part D): Marketplace: Medicare: Other: Total: 12,700,000 4,000,000 2,000,000 1,000,000 3,700,000 23,400,000 30 States 50 States 21 States 28 States 33 States 50 States Membership Composition: Industry: Largest Medicaid and Medicare Managed Care Provider 0 5 10 15 20 25 Centene United Health Group Humana Anthem CVS Membership (Millions) Largest Managed Care Organizations Medicaid Medicare & Medicare PDP Other Government Marketplace
3. $- $10,000 $20,000 $30,000 $40,000 $50,000 $60,000 $70,000 $80,000 $90,000 $100,000 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 Total Revenus (millions) Centene Yearly Revenue Centene Revenue WellCare Revenue Summary of Centene’s Key Challenges in one word… Growth $4.1 Billion Revenue to $96.9 Billion in 10 Years $80.4 Billion in growth in past 5 years $48.6 Billion in growth in past 2½ years Envolve Jan 2015 Wellcare Mar 2019 Fidelis Sep 2017 HealthNet Mar 2016 ? ? Cause of the growth… Mergers & Acquisitions By the numbers:
4. Medicare Medicaid International Federal Marketplace Addressable Market Federal Medicare $860 B 40% State Medicaid International Market Federal Services Health Insurance Marketplace $2,000,000,000,000 + Centene Revenue $97,000,000,000 + Centene Revenue 4% Addressable Market 96% Additional Growth Opportunities $710 B 33% $260 B 12% $120B 6% $115 B 5% Centene Growth Outlook Targeted Pipeline ($270 Billion)
5. Medicare Medicaid International Federal Marketplace Addressable Market Federal Medicare $860 B 40% State Medicaid International Market Federal Services Health Insurance Marketplace $2,000,000,000,000 + Centene Revenue $97,000,000,000 + Centene Revenue 4% Addressable Market 96% Additional Growth Opportunities $710 B 33% $260 B 12% $120B 6% $115 B 5% Centene Growth Outlook Targeted Pipeline ($270 Billion) Mergers & Acquisitions Data Integration & Data Migration
6. Data Integration & Data Migration 1 Shared Database • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure
7. Data Integration & Data Migration Shared Database Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File 2 File Transfer (Batch ETL) • Latent Data • Direct Database Load • Consistency Challenges
8. Data Integration & Data Migration Export Import Shared Database File Transfer (Batch ETL) • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure • Latent Data • Direct Database Load • Consistency Challenges File API API Function Call Response 3 • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation
9. Data Integration & Data Migration Shared Database File Transfer (Batch ETL) Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File • Latent Data • Direct Database Load • Consistency Challenges API API Function Call Response • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation 4 Pub / Sub Messaging (Streaming ETL) Event Message Bus • Loosely Coupled • No Application Refactor • Highly Availability • Highly Scalable • Real-Time Data
10. Data Integration & Data Migration Pub / Sub Messaging (Streaming ETL) Event Message Bus • Loosely Coupled • No Application Refactor • Highly Availability • Highly Scalable • Real-Time Data Shared Database File Transfer (Batch ETL) Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File • Latent Data • Direct Database Load • Consistency Challenges API API Function Call Response • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation * What is a Event? Definition: “A significant change in state” • Statement of fact (immutable) • Expects no response (or call to action) • Has a defined “timepoint” Persistence • Stateless: Notification Event • Stateful: Event-Carried State Transfer How can you use events? E1 E2 E3 + • Combine multiple Events E1 E2 + • Absence of an Event E1 E2 • Leverage Single Event
11. Pub / Sub Messaging (Streaming ETL) Event Message Bus • Loosely Coupled • No Application Refactor* • Highly Availability • Highly Scalable • Real-Time Data Event Structure Example Event Payload (JSON vis REST) “Metadata” : { “Transaction ID” : “C7084816514A5D260”, “User ID” : “USER1”, “Time Stamp” : “201803051315400000000000”, “Transaction Type” : “UPDATE”, “Source System” : “d8amisou6p.MEMBER_CONTACT” } , “Event Body” : { “Event Type” : “Member-PCP-Change”, “Previous Value” : “Dr. John Smith”, “Updated Value” : “Dr. Bryan Zelle”, “Event Source” : “Inbound-Member-Call”, “Caller Information” : { “Name” : “Jane Doe”, “Inbound Number” : “1-614-847-0982”, “Call Resolution Status” : “5 - Highly Satisfied”, “First Call Resolution” : “Success”, “Internal Representative” : “CN-10238381”, ”Call Duration (Minutes)” : “8:19” } Transaction Metadata Who - Who changed the data ?* What - What data changed ? When - When the data changed ? Where - Where was the data changed ? Why - Why was the data changed ? What Event information are we capturing? Event Body *
12. Data Integration & Data Migration Pub / Sub Messaging (Streaming ETL) Event Message Bus • Loosely Coupled • No Application Refactor • Highly Availability • Highly Scalable • Real-Time Data Shared Database File Transfer (Batch ETL) Export Import • Application Refactor • Direct Schema Coupling • Scaling Challenges • Single Point of Failure File • Latent Data • Direct Database Load • Consistency Challenges API API Function Call Response • Direct Coupling • Application Refactor • Availability Concerns • Scaling Concerns Remote Procedure Invocation * How do Events fit with Streaming? Business Driver: Business Enabler: Real Time Data Streaming Integrated Real Time Enterprise
13. Event Driven Apps Defining Characteristic: Stream Driven Apps Where is my driver? What is the temperature? How long till driver arrives? How long till room cools? Leverage real-time events to display current state Defining Characteristic: Continually combine multiple streams of real-time events to extract value from state
14. Event Driven Apps Defining Characteristic: Stream Driven Apps Where is my driver? What is the temperature? How long till driver arrives? How long till room cools? Leverage real-time events to display current state Defining Characteristic: Continually combine multiple streams of real-time events to extract value from state What is the temperature? How long till driver arrives? Events are KEY Apache Kafka Stores Events: Resilient / Durable Distributed / Highly Available High-Throughput / Low Latency But…. What Isn’t Provided by Kafka?
15. Event Driven Apps Defining Characteristic: Stream Driven Apps Where is my driver? What is the temperature? How long till driver arrives? How long till room cools? Leverage real-time events to display current state Defining Characteristic: Continually combine multiple streams of real-time events to extract value from state What is the temperature? How long till driver arrives? But…. What Isn’t Provided by Kafka? Synthetic Events Event Registration (Event Discovery) Sensitive Data Redaction Event Encryption Flexible Schema Validation (JSON not AVRO) Automated Disaster Recovery (Event Rehydration) Searchable Events (Query Event Store) Distributed Tracing of Event (Event Lineage) Event Metrics (Event Dashboards) Consistent Event Structure (Data Governance)
16. Pub / Sub Messaging (Streaming ETL) Event Message Bus • Loosely Coupled • No Application Refactor* • Highly Availability • Highly Scalable • Real-Time Data Mediated (Orchestrated) Eventing Mediator Topology Mediator transfers events to assigned event channel (Topic) Centrally Coordinated Event Routing Complete Decoupling of Event Channels Increased Complexity at cost of increased coordination of event execution Advantages: • Consistent / Common Framework • Enforce Data governance • Economy of Scale Advantage • Technology abstraction / decoupling Disadvantages: • External bottleneck (Mediator Owner) • Single Point of Failure • Duplicative data storage
17. Generic Event Mediator Common Core Architecture: 1) Event Source 2) Event Intake 3) Event Channel 4) Event Router 5) Event Subscription 6) Event Destination Event Channel Event Router Event Subscription Event Destination Event Intake Event Source Event Mediator 3 2 1 4 5 6 Event Bridge Event Grid Apache Camel Knative Eventing Mule ESB Mediator Alternatives?
18. Generic Event Mediator Required Features & Functionality Event Channel Event Router Event Subscription Event Destination Event Intake Event Source Event Mediator 3 2 1 4 5 6 Design Criteria 1) AVRO Event Serialization 2) JSON Validation of Event Body 3) Centralized Event Registry 4) Distributed Tracing of Events 5) Sensitive Data Redaction 6) Turn / Key Self-Service 7) Cloud Agnostic 8) Permanent Event Storage 9) Flexible Ingestion Intake 10) Pre-built Monitoring / Dashboards 11) Synthetic Events Reduced Message Size -> Reduced Storage Cost in Cloud Data Validation -> Clean Data Easily Find Events -> Prevents Event Duplication & Increases Adoption Tracing -> Provides Event Lineage and Auditability Data Restriction -> Protects HIPPA data (including PHI/PII) Automated Configuration -> Reduced manual administrative burden Multi-Cloud Strategy -> No Reliance on Single Cloud Provider Event Persistence -> DR Strategy + Event Sourcing / Hydration Legacy Systems Limitations -> Offer REST, gRPC, SOAP Interfaces & API’s Universal Metrics -> Consistent / Granular Event Visibility Fictitious Event -> Blue/Green Deployments, Prod Smoke Testing, Etc. Business Value
19. 1) AVRO Event Serialization 2) JSON Validation of Event Body 3) Centralized Event Registry 4) Distributed Tracing of Events 5) Sensitive Data Redaction 6) Turn / Key Self-Service 7) Cloud Agnostic 8) Permanent Event Storage 9) Flexible Ingestion Intake 10) Pre-built Monitoring / Dashboards 11) Synthetic Events Reduced Message Size -> Reduced Storage Cost in Cloud Data Validation -> Clean Data Easily Find Events -> Prevents Event Duplication & Increases Adoption Tracing -> Provides Event Lineage and Auditability Data Restriction -> Protects HIPPA data (including PHI/PII) Automated Configuration -> Reduced manual administrative burden Multi-Cloud Strategy -> No Reliance on Single Cloud Provider Event Persistence -> DR Strategy + Event Sourcing / Hydration Legacy Systems Limitations -> Offer REST, gRPC, SOAP Interfaces & API’s Universal Metrics -> Consistent / Granular Event Visibility Fictitious Event -> Blue/Green Deployments, Prod Smoke Testing, Etc. Generic Event Mediator Event Channel Event Router Event Subscription Event Destination Event Intake Event Source Event Mediator 3 2 1 4 5 6 Required Features & Functionality Design Criteria Business Value Leverage 3rd Party Frameworks or Build Custom? Assessment: Majority of frameworks focused engineering effort on how to get data into framework as easily as possible • Higher Data Ingest = Increased Revenue (SaaS) • Too many gaps with current features* • Limited flexibility because of so many customers Decision: Build Centralized Eventing Framework for Enterprise use across all Centene Domains * * * * * *
20. CentEvent Architecture Docker Container Kubernetes Intake Application Axway Gateway Serialize Deserializer Confluent Schema Registry Caffeine Cache Authorization Tokens Event Types Routing Rules Routing HASH Firehose Topic Docker Container Kubernetes Router Application Consumer Topics Client Portal Event Discovery UI Admin UI Docker Container Kubernetes Admin API Mongo Charts Tracing Monitoring
21. Field Level Data Redaction Use Case Example: Data Science Predication Models • Restricting Member PHI / PII (SSN, Medicaid / Medicare ID, etc) Encounter Processing • Restricting Bank/ Account Payment Information Business Requirement: Enforce Least Privilege Access to HIPPA / PHI / PII Data Design: I. Capture Event Metadata – What fields are sensitive? II. Capture Team / App level permissions – Who can see what? III. Duplicate Event -> Modify Event - Redact Sensitive Fields with ***REDACTED***
22. Discard Synthetic Events Use Case Example: Pass-Through Example - Service that validates Payment Account Number Discard Example - Service that processes Claim Payment Business Requirement: Integrated End-to-End Testing without Central Coordination • Synthetic Monitoring / Continuous Unit Testing • Leverage non-prod traffic for Blue / Green Deployments • Inline Production Troubleshooting (in real-time) Design: I. Mandate “Synthetic” Event Property II. Establish Micro-Service Pattern • Pass-Through Event • Discard Event Simulate Process Flow by Generating Synthetic Events Event Channel Pass Through Perform Action Discard Action = Synthetic Event
23. Event Tagging Use Case Example: Health Plan Claim Processing - Tag Claim Events by which State its associated with Process Orchestration - Tag Events with which Step in the Process is next Business Requirement: Be able to route events based on “Event Tags” • Filter Kibana dashboard metrics by Tags • Aggregate SLA’s by Tags • Filter Jaeger Distributed Traces by Tags Design: I. Tags are Optional (0 to N) II. Tags are attached to event by Producer, to be leveraged by Consumer III. Can be used in any combination or order (flexible) • Same Event -> Different Tags • Different Events -> Same Tags Same Event (Different Tags) Different Events (Same Tags)
24. Demo Build a Kafka Topic Request Process leveraging all Asynchronous Events Business Requirement: Additional Requirements: • Only process creation of “approved” topics • Create audit trial of any work performed for SOX compliance • Create Real + Synthetic Flows KaaS UI N: KaaS V: Requested O: Topic SO: Creation Authorizer N: Authorizer V: Sent O: Notification-Email SO: Approval-Request N: Authorizer V: Received O: Approval-Response SO: Email Audit Tag: Approved KaaS API Kafka Cluster N: KaaS-API V: Created O: Topic SO: Successfully B C E A B C D E UI Tag: Unapproved A N: KaaS V: Requested O: Topic SO: Creation D
25. Demo Build a Kafka Topic Request Process leveraging all Asynchronous Events Business Requirement: Additional Requirements: • Only process creation of “approved” topics • Create audit trial of any work performed for SOX compliance • Create Real + Synthetic Flows KaaS UI N: KaaS V: Requested O: Topic SO: Creation Authorizer N: Authorizer V: Sent O: Notification-Email SO: Approval-Request N: Authorizer V: Received O: Approval-Response SO: Email Audit Tag: Approved KaaS API Kafka Cluster N: KaaS-API V: Created O: Topic SO: Successfully B C E A B C D E UI Tag: Unapproved A N: KaaS V: Requested O: Topic SO: Creation D Synthetic Process Flow Real Process Flow
26. Recap Recap 1 Centene’s Core Challenge is Growth cause by Mergers & Acquisitions; causing us to revaluate our Enterprise Data Integration and Data Migration Strategies… Event Message Bus 2 Async Pub / Sub Eventing through Kafka provides us valuable capabilities: - Highly Scalable - High Autonomy / Decoupling - High Availability & Data Resiliency - Real Time Data Transfer - Complex Steam Processing “Metadata” : { “Transaction ID” : “C7084816514A5D260”, “User ID” : “USER1”, “Time Stamp” : “201803051315400000000000”, “Transaction Type” : “UPDATE”, “Source System” : “d8amisou6p.MEMBER_CONTACT” } , “Event Body” : { “Event Type” : “Member-PCP-Change”, “Previous Value” : “Dr. John Smith”, “Updated Value” : “Dr. Bryan Zelle”, “Event Source” : “Inbound-Member-Call”, “Caller Information” : { “Name” : “Jane Doe”, “Inbound Number” : “1-614-847-0982”, “Call Resolution Status” : “5 - Highly Satisfied”, “First Call Resolution” : “Success”, “Internal Representative” : “CN-10238381”, ”Call Duration (Minutes)” : “8:19” } 3 Leveraging a Mediator Topology enables the creation of meaningful events; which provide insight into why things are happening, so we can react to them in real time…
27. Recap (Part 2) 4 Existing frameworks don’t meet all of Centene’s Eventing requirements; investment made to build our own Real-Time Eventing Platform 5 Leverage our Strategic Partners and their IP to build a Scalable Platform • Apache Kafka • K-Streams • Confluent Schema Registry • MongoDB • Kafka / Mongo Connector • MongoDB Charts 6 Execute on a multi-faceted Data Integration strategy that uses Asynchronous Eventing and Real-Time Data Streaming to facilitate current and future Mergers and Acquisitions
28. 28 Join Centene? Help us in our migration to Cloud Data Streaming Team is hiring! • Data Engineers • Site Reliability Engineers • QA Testers • PO / SM / BA • Software Engineers Reach out to: Bryan.A.Zelle@centene.com
29. Sara Zeman sara.zeman@objectpartners.com 1515 Central Ave NE Suite 100 Minneapolis, MN 55413 Looking for a Real-Time Data Streaming Partner? • Minneapolis (HQ), Omaha • Founded in 1996 • Clients Nationwide • 150+ Consultants, all full-time employees Practice Areas • Cloud Engineering • Real-time Data • Modern APIs • Mobile and Web Special Thanks • Nick Larson • Ryan Hoffman • Neil Buesing
30. 30 Questions?