Open standards for building event-driven applications in the cloud Mete Atamel Developer Advocate at Google @meteatamel atamel.dev speakerdeck.com/meteatamel

Introduction 01

An example event-driven architecture (EDA) An event-driven order processing system Frontend App Engine Order request Payment Processor Cloud Run Authorize & charge CC Shipper Cloud Functions Prepare & ship items Notifier Cloud Run Notify user Message Broker / Event Bus OrderReceived OrderReceived Payment Processed Payment Processed Items Shipped Items Shipped

In EDAs, you need to decide and communicate: ❏ The envelope of events ❏ Which service publishes or subscribes to what events?

TL;DR CloudEvents defines an envelope for events AsyncAPI helps to document event formats and which service publishes or subscribes to what events

CloudEvents 02

Why CloudEvents? Middleware Event source Event source Event source Event destination Event destination Event destination Different protocols: Kafka, MQTT, Pub/Sub, … Different formats: JSON, Protobuf, Avro, … Different event schemas Reading events and routing are difficult Kafka MQTT Google Pub/Sub

CloudEvents An open-source specification for describing events in a common way and SDKs to read and write CloudEvents Defines a minimal set of context attributes to route the event Data that is not intended for routing is placed within the data or data_base64 fields cloudevents.io

What does a CloudEvent look like? curl localhost:8080 -v -X POST \ -H "Content-Type: application/json" \ -H "ce-specversion: 1.0" \ -H "ce-type: com.mycompany.myapp.myservice.myevent" \ -H "ce-source: myservice/mysource" \ -H "ce-id: 1234-5678" \ -H "ce-time: 2023-01-02T12:34:56.789Z" \ -H "ce-subject: my-important-subject" \ -H "ce-extensionattr1: value" \ -H "ce-extensionattr2: 5" \ -d '{ "foo1": "bar1", "foo2": "bar2" }' Required Extension attributes Data Binary-mode: Good for receivers unaware of CloudEvents, as the metadata can be ignored Determines mode

What does a CloudEvent look like? curl localhost:8080 -v -X POST \ -H "Content-Type: application/cloudevents+json" \ -d '{ "specversion": "1.0", "type": "com.mycompany.myapp.myservice.myevent", "source": "myservice/mysource", "id": "1234-5678", "time": "2023-01-02T12:34:56.789Z", "subject": "my-important-subject", "datacontenttype": "application/json", "extensionattr1" : "value", "extensionattr2" : 5, "data": { "foo1": "bar1", "foo2": "bar2" } }' Required Extension attributes Data Structured-mode: Allows simple forwarding of the event across multiple routing hops Determines mode

What else CloudEvents provide? Event formats: JSON, Protobuf, Avro, XML (draft) Protocol bindings: HTTP, AMQP, Kafka, MQTT, NATS, WebSockets (draft) SDKs: Go, Javascript, Java, C#, Ruby, PHP, Python, Rust, Powershell Various CloudEvent libraries from vendors github.com/googleapis/google-cloudevents github.com/meteatamel/cloudevents-basics

However… Batch-mode not supported in all protocol/SDK combos e.g. Javascript SDK supports HTTP and Kafka batch mode but not MQTT batch mode Not all event formats are supported in each SDK e.g. Javascript SDK supports JSON but not Avro or Protobuf Not all protocol bindings are supported in each SDK e.g. Javascript SDK supports HTTP and Kafka but not AMQP, MQTT, NATS

Upcoming specs in CloudEvents Discovery API Subscriptions API Schema Registry

In EDAs, you need to decide and communicate: ✓ The envelope of events ❏ Which service publishes or subscribes to what events?

AsyncAPI 03

What is AsyncAPI? An open source specification to define an Async APIs, similar to what OpenAPI (aka Swagger) does for REST APIs Also tools to visualize and validate and generators to generate code from AsyncAPI definitions

Main concepts in AsyncAPI Application Server Channel Protocol Producer (Publisher) Consumer (Subscriber) Message Bindings for server, channel, operation, and message github.com/meteatamel/asyncapi-basics#concepts

OpenAPI AsyncAPI asyncapi.com/docs/tutorials/getting-started/coming-from-openapi vs.

AsyncAPI 2.6.0 vs 3.0.0 3.0.0 was released in December 2023 with breaking changes www.asyncapi.com/blog/release-notes-3.0.0 www.asyncapi.com/docs/migration/migrating-to-v3

What does an AsyncAPI definition look like? # The simplest AsyncAPI definition possible in 2.6.0 asyncapi: 2.6.0 info: title: Hello world application version: '0.1.0' channels: hello: publish: message: payload: type: string github.com/meteatamel/asyncapi-basics/tree/main/samples/hello-asyncapi

What does an AsyncAPI definition look like? # The simplest AsyncAPI definition possible in 3.0.0 asyncapi: 3.0.0 info: title: Hello world application version: 0.1.0 channels: hello: address: hello messages: publish.message: payload: type: string operations: hello.publish: action: receive channel: $ref: '#/channels/hello' messages: - $ref: '#/channels/hello/messages/publish.message'

Tools AsyncStudio Browser based tool to author and visualize and validate AsyncAPI files AsyncAPI CLI CLI based tool to work with AsyncAPI files AsyncAPI Generator Code generators for various languages & frameworks github.com/meteatamel/asyncapi-basics/tree/main/samples/quickstart

Publish/subscribe semantics in AsyncAPI 2.6.0 In a channel, you can have publish and subscribe operations AsyncAPI Term WebSocket Term From Server Perspective From User Perspective Publish Send The server receives messages The user publishes messages to server Subscribe Receive The server sends messages The user subscribes to receive messages from server github.com/meteatamel/asyncapi-basics/tree/main/samples/account-email-services Most useful

Send/receive semantics in AsyncAPI 3.0.0 In AsyncAPI 3.0.0, channels don’t have operations. Instead, there’s a separate operations section In operation, you can have send and receive actions from server’s perspective github.com/meteatamel/asyncapi-basics/tree/main/samples/account-email-services

In EDAs, you need to decide and communicate: ✓ The envelope of events ✓ Which service publishes or subscribes to what events?

CloudEvents+AsyncAPI Google Pub/Sub+AsyncAPI 04

Can you use CloudEvents and AsyncAPI together? Yes! github.com/meteatamel/asyncapi-basics/tree/main/samples/account-service-cloudevents

Can you use Google Pub/Sub and AsyncAPI together? Yes! github.com/meteatamel/asyncapi-basics/tree/main/samples/google-pubsub

Thank you! Mete Atamel Developer Advocate at Google @meteatamel atamel.dev speakerdeck.com/meteatamel cloudevents.io github.com/meteatamel/cloudevents-basics asyncapi.com github.com/meteatamel/asyncapi-basics Feedback? bit.ly/atamel