Event API Management for Snowflake Streaming and Data Governance Nicholas Goodman, PhD Sales Engineer 

No content

Agenda • Event-Driven Architecture Solves Everything! Or does it? • Challenge 1: Managing Event APIs • Challenge 2: Authorization and Access • Challenge 3: Distributed Processing Woes • Use-Case Demo: Streaming Data from Solace to Snowflake • Concluding Remarks (Eventing and APIs?) 

Challenges of Integrating Distributed Systems • Often point-to-point one-off data Integrations • Monolithic and siloed with tight coupling • High complexity rigid dependencies and upgrade paths • Challenging transparency, governance, monitoring • Slow pace of development and implementation • Expensive and difficult to integrate customers, partners, suppliers 

The Solace Event Mesh • Uniform Connectivity • Hybrid and Multi Cloud • Public and Private Cloud • Dynamic Routing • High Availability and Performance • Centralized Security and Governance Fraud Loyalty Digital App Fraud Credit Private Cloud Management & Security Event Mesh 

Common Event Messaging Patterns • Streaming Data • Fire and Forget • Fan-out • Events are queued • Point-to-Point • Fan-out • Round-trip events • Point-to-Point • Synchronous Flows Request-Response Guaranteed Messaging Direct Messaging Q1 Q2 

Basics of Topic Based Routing Data Type MarketData Asset Class EQ FX Country US UK Exchange NYSE LSE Ticker IBM VOD Market Data Example Taxonomy: Enumeration: EQ US NYSE IBM I am interested in receiving all market data for IBM from NYSE: I am interested in receiving all market data from any country and any venue where IBM is traded: EQ I am interested in receiving all market data for all equities from every exchange in UK: EQ > MarketData MarketData MarketData * * IBM UK … 

Using Topics and Queues Together Consumer Producer Consumer Topic Subscription: eq/us/> Publishing to topic: eq/us/{exch}/{ticker} Topic Subscription: */*/nyse/> Topic Subscription: eq/uk/> Topic Subscription: */*/lse/> Producer Publishing to topic: eq/uk/{exch}/{ticker} 

Mission Accomplished? Great, use event brokers and all my integration problems are gone, right!? … but wait, Who owns the event definitions? What are the payload schemas? Who is allowed to say what? Who is allowed to listen to what? What if something goes wrong in flight? Who owns the error? 

Solace Event Portal: API Management for Events • Configure and manage broker clusters and event meshes • Design, manage, and version event topics and schemas • Monitor and gain insights on service behaviors and system utilization 

Authentication & Authorization • Basic: Choose Internal, LDAP or RADIUS • TLS client certificate • Kerberos • OAuth • Clients can retrieve authorization groups from configuration in broker or an external server • Authorization group determines: – Attributes of the client’s connection (client profile) – Topics client can publish & subscribe to (ACL profile) • Connect ACL: Control where client can connect from • Publish ACL: Control which topics client can publish to • Subscribe ACL: Control which topics client can subscribe to 3. Real-time policing of pub/sub topics 2. Client Authorization (based on identity) 1. Authentication (based on username & credentials) 

Error Management and Tracing Point to Point: • Client or server handles errors What about Event Driven? • Dead Letter Queues • Using Tracing • Best Option: Well-behaved publishers and robust consumers Consumer Queue Dead Message Queue Consumer Message cannot be processed DMQ? No Yes OpenTelemetry Implementations 

Mission Accomplished So, putting it all together… • Managed AsyncAPI specifications for event topics and data schemas • ACLs govern which apps can send and receive which data • Keep the data moving: smart endpoints and dumb pipes! 

Case Study Building a Solace Snowflake Streaming Connector 

Using Best Practices to Design the Connector • Snowflake, a Cloud Data Platform recently introduced ability to stream data into tables in real-time with the Snowpipe Streaming API • Tables can accept structured or non-structured (variant) data types • Goal: stream data from Solace into Snowflake tables in real-time • Remember: smart endpoints, dumb pipes! • Design for ELT, not ETL! • Data In == Data Stored 

Streaming Data into Snowflake Snowflake Connector Queue-to-Buffer (1:1) Publishers Table-A Channel-A Queue-A Queue-B1 Queue-B2 Buffer-A Buffer-B1 Buffer-B2 Table-B Channel-B1 Channel-B2 Topic-to-Queue Mapping (n:n) Buffer-to-Channel (1:1) Channel-to-Table (n:1) Source Destination 

1. Connector opens flow and begins receiving messages which are inserted immediately into the Snowpipe Streaming channel and saved locally 2. Channel processes rows as they are received 3. Connector polls the channel for latest committed offset token 4. Whenever offset token increases connector identifies all messages which have been successfully committed 5. Connector sends individual Acks for each successfully inserted message Snowflake Connector 0e4d 0e4c 0e4b 0e4a queue in-memory copy Snowpipe Streaming Channel table 1 5 3 4 1 2 Data Flow and Guaranteed Delivery 

Live Demo HTML5 Web App Solace Snowflake Connector Snowflake Dashboard 

Using API Management to Aide Data Governance • A Well Governed Event API: • Data quality enforced at source • Data is secured from the right senders to the right consumers • Data will get to its destination uninterrupted and real time • How does this help with Data Governance • In data streaming use cases, Event API governance fulfills some common needs • Enforcing quality and security at its source Image Credit: Data Governance: The Definitive Guide by Evren Eryurek, Uri Gilad, Valliappa Lakshmanan, Anita Kibunguchy-Grant, Jessi Ashdown O’Rielly 

Thank You!