dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Calculating the Value of Pie Real-Time Survey Analysis With Apache Kafka®

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ I like houseplants.

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ I also like baking.

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Google Forms?

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Google Forms?

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Kafka- and Telegram-based Survey-issuing Application?

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Surveying the Problem Landscape ● Surveys are EVERYWHERE ● Great business value ○ External: what do your customers think of your product? ○ Internal: how are your employees doing? ● Often batch-processed

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Real Time Survey Analysis

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Enhanced Recipe ? !

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Kafka? What’s that?

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Kafka? What’s that? A distributed event streaming platform.

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ A distributed event streaming platform.

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Paradigm Shift Batch Processing Realtime Processing Batch Result Batch Result PROCESSOR … … PROCESSOR Realtime Update Realtime Update Realtime Update Realtime Update

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ A distributed event streaming platform.

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Thinking in Events ● Natural way to reason about things ● Indicate that something has happened ○ When ○ What/Who ● Immutable pieces of information

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Thinking in Events ● Natural way to reason about things ● Indicate that something has happened ○ When ○ What/Who ● Immutable pieces of information

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ A distributed event streaming platform.

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ ● Topics ○ Basic storage unit ○ Read/Write: ■ Producer and consumer clients ■ Completely decoupled ● Partitions ○ Immutable, append-only logs ○ Data is replicated at this level Kafka Storage P0 P1 P2 Kafka Topic

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Recipe ? !

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ ! Recipe ?

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Kafka, but make it simple. ● Fully-managed, cloud-based Kafka ● Auxiliary tools: ○ Kafka Connect ○ ksqlDB ○ Schema management

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ ! Recipe ?

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Knowing Your Pie Data

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ survey-entries { "doc": "Contains individual survey entries for a user and survey question.", "fields": [ { "name": "survey_id", "type": "string" }, { "name": "user_id", "type": "string" }, { "name": "name", "type": "string" }, { "default": null, "name": "company", "type": ["null","string"] }, { "default": null, "name": "location", "type": ["null","string"] }, { "name": "response", "type": "string" } ], "name": "surveyEntry", "namespace": "survey.bot", "type": "record" }

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ survey-respondents { "doc": "Respondents to survey questions.", "fields": [ { "name": "user_id", "type": "string" }, { "name": "name", "type": "string" }, { "default": null, "name": "company", "type": ["null","string"] }, { "default": null, "name": "location", "type": ["null","string"] } ], "name": "surveyRespondent", "namespace": "survey.bot", "type": "record" }

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ survey-responses { "doc": "Contains individual survey responses for a user and survey question.", "fields": [ { "name": "survey_id", "type": "string" }, { "name": "user_id", "type": "string" }, { "name": "response", "type": "string" } ], "name": "surveyResponse", "namespace": "survey.bot", "type": "record" }

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ survey-questions { "doc": "Details for a single question survey.", "fields": [ { "name": "survey_id", "type": "string" }, { "name": "question", "type": "string" }, { "name": "summary", "type": "string" }, { "name": "options", "type": { "items": "string", "type": "array" } }, { "name": "enabled", "type": "boolean" } ], "name": "surveyQuestion", "namespace": "survey.bot", "type": "record" }

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Creating a Survey { "survey_id": "1", "question": "Which Thanksgiving Pie is your favorite?", "summary": "Thanksgiving Pie", "options": [ "Pumpkin Pie", "Pecan Pie", "Apple Pie", "Thanksgiving Leftover Pot Pie", "Other" ], "enabled": true }

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Building a Telegram Bot

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Telegram as a Producer ● python-telegram-bot library ○ Wrapper for Telegram API ○ Define conversation handlers ● Produce data to Kafka ○ Capture data using conversation handlers ○ Create survey-entry message, serialize, and produce

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Conversation Handler survey_handler = ConversationHandler( entry_points = [CommandHandler(survey, survey_command)], states = { SURVEY_STATE.NAME: [ MessageHandler(filters.TEXT & ~filters.COMMAND, name_command), CommandHandler('cancel', cancel_command) ], ... SURVEY_STATE.RESPONSE: [ MessageHandler(filters.Regex( "^(Pumpkin Pie|Pecan Pie|Apple Pie|Thanksgiving Leftover Pot Pie|Other)$") ), response_command), CommandHandler('cancel', cancel_command) ], SURVEY_STATE.CONFIRM: [ CommandHandler('y', confirm_command), CommandHandler('n', cancel_command) ] }, fallbacks=[CommandHandler('cancel', cancel_command)] )

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Command Handler async def name_command(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None: # capture and store name data context.user_data['name'] = update.message.text # get chat_id as user_id context.user_data['user_id'] = update.message.chat_id # prompt for company information await update.message.reply_text( "Enter the company that you work for or use /skip to go to the next question." ) return SURVEY_STATE.COMPANY

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Producer Code def send_entry(entry): # send survey entry message try: # set up Kafka producer for survey entries producer = clients.producer(clients.entry_serializer()) # prep key and value for message k = str(metadata.get('survey_id')) value = SurveyEntry.dict_to_entry(entry) logger.info("Publishing survey entry message for key %s", k) producer.produce(config['topics']['survey-entries'], key=k, value=value) except Exception as e: logger.error("Got exception %s", e) raise e finally: producer.poll() producer.flush()

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Taking a Thanksgiving Survey

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ ! Recipe ?

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ ksqlDB Processing

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Load Stream of survey-entries CREATE STREAM survey_entries WITH ( kafka_topic = 'survey-entries', value_format = 'AVRO' );

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Extracting Respondents CREATE STREAM survey_respondents WITH ( kafka_topic = 'survey-respondents', value_format = 'AVRO' ) AS SELECT user_id, name, company, location FROM survey_entries EMIT CHANGES;

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Extracting Respondents CREATE STREAM survey_respondents WITH ( kafka_topic = 'survey-respondents', value_format = 'AVRO' ) AS SELECT user_id, name, company, location FROM survey_entries EMIT CHANGES;

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Extracting Respondents CREATE STREAM survey_respondents WITH ( kafka_topic = 'survey-respondents', value_format = 'AVRO' ) AS SELECT user_id, name, company, location FROM survey_entries EMIT CHANGES;

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Extracting Respondents CREATE STREAM survey_respondents WITH ( kafka_topic = 'survey-respondents', value_format = 'AVRO' ) AS SELECT user_id, name, company, location FROM survey_entries EMIT CHANGES;

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Extracting Responses CREATE STREAM survey_responses WITH ( kafka_topic = 'survey-responses', value_format = 'AVRO' ) AS SELECT user_id, survey_id, response FROM survey_entries EMIT CHANGES;

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Questions Table CREATE TABLE survey_questions ( id STRING PRIMARY KEY ) WITH ( kafka_topic = 'survey-questions', value_format = 'AVRO' );

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Analysis CREATE TABLE survey_results_live WITH ( kafka_topic='survey-results-live', value_format='AVRO' ) AS SELECT question AS question, HISTOGRAM(response) AS results FROM survey_entries GROUP BY question EMIT CHANGES;

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Results What is your Favorite Thanksgiving Pie? { Pumpkin Pie=1, Pecan Pie=1, ... Apple Pie=3 } { Pumpkin Pie=1, Pecan Pie=1, ... Apple Pie=4 } { Pumpkin Pie=2, Pecan Pie=1, ... Apple Pie=4 } { Pumpkin Pie=2, Pecan Pie=2, ... Apple Pie=4 } ...

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Analysis CREATE TABLE survey_results_live WITH ( kafka_topic='survey-results-live', value_format='AVRO' ) AS SELECT question AS question, HISTOGRAM(response) AS results FROM survey_entries GROUP BY question EMIT CHANGES;

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Analysis CREATE TABLE survey_results_final WITH ( kafka_topic='survey-results-final', value_format='AVRO' ) AS SELECT question AS question, HISTOGRAM(response) AS results FROM survey_entries WINDOW TUMBLING (SIZE 48 HOURS, GRACE PERIOD 10 MINUTE) GROUP BY question EMIT FINAL;

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Analysis CREATE TABLE survey_results_final WITH ( kafka_topic='survey-results-final', value_format='AVRO' ) AS SELECT question AS question, HISTOGRAM(response) AS results FROM survey_entries WINDOW TUMBLING (SIZE 48 HOURS, GRACE PERIOD 10 MINUTE) GROUP BY question EMIT FINAL;

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Alerting with Telegram ● Kafka Connect ○ Connects Kafka and external systems ○ Independent framework ○ Configuration-based ● Kafka Connect HTTP Sink Connector ○ Fully-managed in Confluent Cloud ○ Migrates the events from the survey_results_final topic as they happen !

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Alerting with Telegram ● Kafka Connect ○ Connects Kafka and external systems ○ Independent framework ○ Configuration-based ● Kafka Connect HTTP Sink Connector ○ Fully-managed in Confluent Cloud ○ Migrates the events from the survey_results_final topic as they happen !

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Recipe ? !

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Recipe++ ? !

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Stream Processing Use Case Recipes

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Looking to get started? LinkTree Resources

dfine@confluent.io @TheDanicaFine linkedin.com/in/danica-fine/ Questions?