Tweet
Tweet

Slide 1

Slide 1 text

1 Erik Guzman Full Stack Engineer @ ZEAL Ruby on Rails ReactJS Who am I? After Dark: Microsoft MVP Developer Tools Elixir Development Technical Content Creator Twitch 3rd Party Developer

Slide 2

Slide 2 text

SERVERLESS TO SERVING ELIXIR MIGRATING SERVERLESS APP TO RUN ON PHOENIX

Slide 3

Slide 3 text

3 The Journey 1 2 3 4 5 Developing for twitch.tv Going Serverless Problems Faced Why Elixir? Migration Process

Slide 4

Slide 4 text

DEVELOPING FOR TWITCH.TV

Slide 5

Slide 5 text

5 What is Twitch? Live video streaming platform • Users aka “channels” live stream activities • To watch a channel, viewers visit a channel page • Viewership can vary drastically, from 1 to 100k concurrent viewers • Twitch o ff ers various tools, like Extensions, to enhance a viewers experience

Slide 6

Slide 6 text

6 What is a Twitch Extension? HTML/CSS/JS Website • Little website trapped in an iframe • Hosted on the Twitch CDN • Managed like an app store • You are in charge of hosting any backend • Installed by the Twitch Channel • Loaded by each viewer of the channel

Slide 7

Slide 7 text

7 Recipe for Success Handle Large Bursts of Traffic Scalable Fast Affordable Easy to Deploy

Slide 8

Slide 8 text

GOING SERVERLESS

Slide 9

Slide 9 text

9 Try out Serverless Functions Great! What’s Serverless Functions? Recommendations

Slide 10

Slide 10 text

10 What’s Serverless Functions? Easy to Deploy A ff ordable Scalable Pay Per Invocation & Duration $0.20 per million executions plus potential execution time No long lived server costs Deploy single function or all functions easily Can handle large amount of requests Fully Managed Infrastructure Don’t worry about con fi guring servers or scaling

Slide 11

Slide 11 text

11 Serverless Breaks Up The Band Traditional Hosting

Slide 12

Slide 12 text

12 Serverless Breaks Up The Band Such micro

Slide 13

Slide 13 text

13 Serverless Checks Out Handle Large Bursts of Traffic Scalable Fast Affordable Easy to Deploy

Slide 14

Slide 14 text

14 What DB Do I Use?

Slide 15

Slide 15 text

Why? • Don’t have to pay for long run database server • Only pay for reads and writes exceeding a certain amount • Pay for network tra ff i c after a certain amount • I only pay if my extension is a success NoSQL: Firestore 15

Slide 16

Slide 16 text

16 So I Built Stu f Github Projects Stream Team Better Information Panel Stream Closed Captioner What Next? Not actually serverless

Slide 17

Slide 17 text

MORE EXTENSION MORE PROBLEMS

Slide 18

Slide 18 text

18 Serverless Cold starts for initial calls • Function need time to initialize • Not all your functions will be “hot” • Need to pay to keep them long lived

Slide 19

Slide 19 text

19 Serverless Cold starts for initial calls Easy to end up with microservice zoo • Functions or new project for everything • Hard to manage

Slide 20

Slide 20 text

20 Serverless Cold starts for initial calls Duplicated code and logic • Multiple functions or multiple projects can lead to duplicating code Easy to end up with microservice zoo

Slide 21

Slide 21 text

21 Serverless Cold starts for initial calls Easy to end up with microservice zoo Duplicated code and logic More tra ff i c means higher and unpredictable costs $15

Slide 22

Slide 22 text

22 Serverless $15 Forces you to go as lean as possible on number of requests When I realized any form of polling is a bad idea Cold starts for initial calls Easy to end up with microservice zoo Duplicated code and logic More tra ff i c means higher and unpredictable costs

Slide 23

Slide 23 text

23 Firestore Easy to make bad schema design • Ability to store non structured data makes it easy to store mistakes

Slide 24

Slide 24 text

24 Firestore You can’t easily export data • Lack of standardization means each vendor their own special export format Easy to make bad schema design

Slide 25

Slide 25 text

25 Firestore Constant reads and writes are costly $ $$ Polling also meant more reads From the the DB You can’t easily export data Easy to make bad schema design

Slide 26

Slide 26 text

WHY ELIXIR? A Side Story

Slide 27

Slide 27 text

27 Stream Closed Captioner Description: Allows a Twitch channel add closed captions to their stream. • Built with Ruby on Rails • Uses WebSockets to send captions from client side to server • Send captions from server to Twitch using Twitch APIs

Slide 28

Slide 28 text

28 The Past STREAM CLOSED CAPTIONER ORIGINAL ARCHITECTURE Ruby on Rails Redis Postgres Twitch API

Slide 29

Slide 29 text

29 My Life After Awhile With Success Came Problems • WebSockets in Ruby on Rails is a memory hog • To many requests on small servers leading to timeouts • Getting expensive to scale • I don’t have time to baby sit server

Slide 30

Slide 30 text

30 The Present STREAM CLOSED CAPTIONER ZOOMER ARCHITECTURE Elixir Phoenix Postgres Redis Twitch API

Slide 31

Slide 31 text

31 Deploy it and Forget it • Completely removed the need for Redis • Websockets run amazing!!! • Memory e ff i cient and you never have to restart • A ff ordable hosting and blazing fast • Fail fast, recovers fast My Life Now

Slide 32

Slide 32 text

32 Elixir Hits All The Marks Handle Large Bursts of Traffic Scalable Fast Affordable Easy to Deploy

Slide 33

Slide 33 text

THE GREAT MIGRATION Serverless to Serverful

Slide 34

Slide 34 text

34 Stream Team Extension Concept: Create, manage, and display a Twitch team Features: ๏ A channel can choose to display an o ffi cial team they are apart of ๏ They can create a custom team and add channels ๏ Upon installation all viewers of their channel can browse and view another team member’s channel Application to Migrate

Slide 35

Slide 35 text

35 Top Goals Migrate Data to Postgres Zero Down Time No Loss of Data 1 2 3

Slide 36

Slide 36 text

36 Road Map 1 2 3 4 5 Create API Proxy Create Schema Replicate Logic & Tap Data Export Firebase to Postgres Switch to GQL

Slide 37

Slide 37 text

THE GREAT MIGRATION 1. Create API Proxy

Slide 38

Slide 38 text

Current Request Flow 38

Slide 39

Slide 39 text

39 Creating API Proxy GET FLOW All in sequence UPSERT FLOW

Slide 40

Slide 40 text

40 Creating API Proxy Time to Terraform terraform Terraform is a simple plug designed to work with Phoenix. Terraform allows you to incrementally transform a HTTP API into one powered by Phoenix - one endpoint at a time.

Slide 41

Slide 41 text

41 Creating API Proxy GET FLOW Terraforming UPSERT FLOW Excuse me. I’d like to ask you about your cars extended warranty

Slide 42

Slide 42 text

42 Creating API Proxy GET FLOW Successful Proxy UPSERT FLOW

Slide 43

Slide 43 text

43 Creating API Proxy GET FLOW Successful Proxy UPSERT FLOW Proxy the request to serverless function Pass response back

Slide 44

Slide 44 text

44 Creating API Proxy GET FLOW Successful Proxy UPSERT FLOW Why? 1 2 Able to update server con fi guration in the Twitch Extension before any serious work is done I can now easily deploy backend changes without releasing a new extension version

Slide 45

Slide 45 text

THE GREAT MIGRATION 2. Create Schema

Slide 46

Slide 46 text

46 Create Schema Firebase schema Channel Record Channel’s Custom Team

Slide 47

Slide 47 text

Bad idea if members wanted to share the same team 47 Create Schema Firebase schema Channel Record Channel’s Custom Team

Slide 48

Slide 48 text

48 Create Schema Redesign time FIREBASE SCHEMA ECTO SCHEMA

Slide 49

Slide 49 text

49 Create Schema Ecto & Contexts Two Contexts Four new tables

Slide 50

Slide 50 text

Two Contexts Four new tables 50 Create Schema Ecto & Contexts Why? 1 2 Separate team from channel entity A channel can see who their memberships 3 Channels can have other app data associated with them in the future

Slide 51

Slide 51 text

THE GREAT MIGRATION 3. Replicate Logic & Tap Data

Slide 52

Slide 52 text

52 Replicate Logic Data Flow

Slide 53

Slide 53 text

53 Replicate Logic Data Flow All the business logic done here Reimplemented Here

Slide 54

Slide 54 text

54 Replicate Logic Tapping into the fl ow

Slide 55

Slide 55 text

55 Replicate Logic Tapping into the fl ow 1. Auth 2. Validate 3. Save to Postgres

Slide 56

Slide 56 text

56 Replicate Logic Tapping into the fl ow Auth Validate Save to Postgres Why? 1 2 Test my schema against real world data Test server performance 3 I can save data in fl ight while performing database export

Slide 57

Slide 57 text

THE GREAT MIGRATION 4. Export Firebase to Postgres

Slide 58

Slide 58 text

58 Firebase to Postgres Supabase to the rescue Supa Cool Library fi rebase-to-supabase Converts Firebase collections into JSON fi le which can then be imported into you Supabase Postgres DB.

Slide 59

Slide 59 text

59 Firebase to Postgres How to get data into Postgres?

Slide 60

Slide 60 text

60 Uploader & Processor Built-in Upload Feature Oban Job Processing Super Easy to Setup Simple Page to Upload Exports LIVEVIEW UPLOADER Finally a reason to use LiveView!

Slide 61

Slide 61 text

61 LiveView File Upload Built-in Upload Feature Oban Job Processing Super Easy to Setup Simple Page to Upload Exports PHOENIX LIVEVIEW Finally a reason to use LiveView!

Slide 62

Slide 62 text

62 LiveView File Upload Built-in Upload Feature Oban Job Processing Super Easy to Setup Simple Page to Upload Exports PHOENIX LIVEVIEW Finally a reason to use LiveView! Why? 1 2 Get away from vendor lock in Stable/Consistent Cost 3 No data loss since I am saving new upserts

Slide 63

Slide 63 text

THE GREAT MIGRATION 5. Switch to GQL

Slide 64

Slide 64 text

64 Switch to GQL Just one more change

Slide 65

Slide 65 text

65 Switch to GQL Multiple endpoints is so 2010s POST DESTROY

Slide 66

Slide 66 text

66 Switch to GraphQL Ask for what I need

Slide 67

Slide 67 text

67 Switch to GraphQL Profound reasons…

Slide 68

Slide 68 text

68 Switch to GraphQL Profound reasons… 838 Why? 1 2 GraphQL is cool I felt like it

Slide 69

Slide 69 text

69 Recap Why did I do all this work? 1 2 3 4 5 6 Avoid vendor lock-in Consistent Costs Extremely Performant Very low cost to run (Actually runs on fl y.io free tier) Relational Databases are great Easy to manage

Slide 70

Slide 70 text

70 Site: guzman.codes Socials: THANKS Talk2MeGooseman