Easy Data Processing on Azure with Serverless Functions

by Tania Allard

Slide 1

Slide 1 text

Tania Allard, PhD PyCon 2020 Easy data processing with Python and Azure functions @ixek

Slide 2

Slide 2 text

• Detailed step-by-step tutorial: aka.ms/pycon2020- azurefunctions • GitHub Repository: https:!//github.com/trallard/pycon2020- azure-functions • Slides:aka.ms/pycon2020-azurefunctions-slides Resources: @ixek

Slide 3

Slide 3 text

Serverless computing Are there no servers at all? @ixek

Slide 4

Slide 4 text

Serverless computing Are there no servers at all? Yes there are but: - You are not responsible for the management or maintenance - Your cloud provider takes care of the provisioning @ixek

Slide 5

Slide 5 text

Serverless computing You focus on the code not the infrastructure @ixek

Slide 6

Slide 6 text

Managed Pay only for what you use Scalable Why Serverless computing? @ixek

Slide 7

Slide 7 text

Managed serverless Microsoft Azure functions @ixek Learn more at: https:!//cda.ms/1g7

Slide 8

Slide 8 text

Handles: software, monitoring, scaling, and hardware Host management Microsoft Azure functions @ixek

Slide 9

Slide 9 text

Are responsible for the application code Decide what services to integrate (i.e databases, Blob Storage, email provider) You - the developer @ixek

Slide 10

Slide 10 text

Why functions? Microsoft Azure functions @ixek

Slide 11

Slide 11 text

Serverless computing A.K.A Functions as a Service (FasS) @ixek

Slide 12

Slide 12 text

Microsoft Azure functions Why functions? FaaS Self contained Code snippets In the cloud @ixek

Slide 13

Slide 13 text

Microsoft Azure functions FaaS Self contained Code snippets In the cloud a.k.a function @ixek

Slide 14

Slide 14 text

Event triggered (idle in the meantime) Coupled services Stateless and short-lived Asynchronous (do not wait for response) Serverless characteristics @ixek

Slide 15

Slide 15 text

Image and video processing Internet of Things Data pipelines Good use cases for FaaS @ixek

Slide 16

Slide 16 text

• You want to collect data from the StackExchange API - to monitor questions from a specific topic or tool • The data needs to be stored for later wrangling and reporting • Ideally, you want to get a daily digest in your inbox with the new questions created as well as a plot of the most commonly used tags in said questions The Scenario @ixek

Slide 17

Slide 17 text

• Python 3.x (supported 3.6, 3.7, 3.8) • VS Code • Python VS Code Extension • Azure Functions VS Code Extension • StackExchange API app key https:!//api.stackexchange.com/ Requirements: @ixek

Slide 18

Slide 18 text

Getting started - timer @ixek Timer trigger: Every day at 9am Function gets executed Data is collected from StackExchange

Slide 19

Slide 19 text

Triggers vs bindings @ixek Triggers Idle function Trigger Active function

Slide 20

Slide 20 text

Triggers vs bindings @ixek Bindings Idle function Trigger Active function In Binding Out Bindings Trigger Binding

Slide 21

Slide 21 text

Extend function @ixek Timer trigger: Every day at 9am Function gets executed Data is collected from StackExchange Store in Blob Storage (csv) Trigger Binding

Slide 22

Slide 22 text

Completing the scenario @ixek Function gets executed File added in Blob Storage (csv) Process data / Create plots Save plots Send digest Retrieve Data Trigger Binding