Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Serverless to Serving Elixir: Migrating Serverl...

Erik Guzman
September 11, 2022

Serverless to Serving Elixir: Migrating Serverless app to run on Phoenix

Problem: Increasing productivity, affordability, scalability, and even less coding (gasp!) are the goals for any web application. Add on top unpredictable server traffic, then a solution you might consider is going serverless and throwing in one of those nifty NoSQL database solutions. Easy enough, but at what cost? You can unintentionally suffer vendor lock-in, deal with cold starts, architect your application improperly, and miscalculate cost structure at scale. But there is an even better solution you might want to consider, Elixir. Elixir and Pheonix can help developers solve all these technical challenges, allowing you to focus on the end-user experience.

Purpose: “Serverless to Serving Elixir: Migrating Serverless app to run on Phoenix” will demonstrate why migrating Elixir can solve all the same issues your serverless application would, make it faster, avoid vendor lock-in, and help future-proof your application. Maybe even have you second-guess if you ever need to go serverless again.

Live Demo/ Walkthru: In preparation for this talk, I will guide conference attendees through the libraries and strategies to take NodeJS Serverless applications using Firebase Firestore and migrate them to a single Phoenix application and Postgres database with no downtime.

Erik Guzman

September 11, 2022
Tweet

Other Decks in Programming

Transcript

  1. 1 Erik Guzman Full Stack Engineer @ ZEAL Ruby on

    Rails ReactJS Who am I? After Dark: Microsoft MVP Developer Tools Elixir Development Technical Content Creator Twitch 3rd Party Developer
  2. 3 The Journey 1 2 3 4 5 Developing for

    twitch.tv Going Serverless Problems Faced Why Elixir? Migration Process
  3. 5 What is Twitch? Live video streaming platform • Users

    aka “channels” live stream activities • To watch a channel, viewers visit a channel page • Viewership can vary drastically, from 1 to 100k concurrent viewers • Twitch o ff ers various tools, like Extensions, to enhance a viewers experience
  4. 6 What is a Twitch Extension? HTML/CSS/JS Website • Little

    website trapped in an iframe • Hosted on the Twitch CDN • Managed like an app store • You are in charge of hosting any backend • Installed by the Twitch Channel • Loaded by each viewer of the channel
  5. 10 What’s Serverless Functions? Easy to Deploy A ff ordable

    Scalable Pay Per Invocation & Duration $0.20 per million executions plus potential execution time No long lived server costs Deploy single function or all functions easily Can handle large amount of requests Fully Managed Infrastructure Don’t worry about con fi guring servers or scaling
  6. Why? • Don’t have to pay for long run database

    server • Only pay for reads and writes exceeding a certain amount • Pay for network tra ff i c after a certain amount • I only pay if my extension is a success NoSQL: Firestore 15
  7. 16 So I Built Stu f Github Projects Stream Team

    Better Information Panel Stream Closed Captioner What Next? Not actually serverless
  8. 18 Serverless Cold starts for initial calls • Function need

    time to initialize • Not all your functions will be “hot” • Need to pay to keep them long lived
  9. 19 Serverless Cold starts for initial calls Easy to end

    up with microservice zoo • Functions or new project for everything • Hard to manage
  10. 20 Serverless Cold starts for initial calls Duplicated code and

    logic • Multiple functions or multiple projects can lead to duplicating code Easy to end up with microservice zoo
  11. 21 Serverless Cold starts for initial calls Easy to end

    up with microservice zoo Duplicated code and logic More tra ff i c means higher and unpredictable costs $15
  12. 22 Serverless $15 Forces you to go as lean as

    possible on number of requests When I realized any form of polling is a bad idea Cold starts for initial calls Easy to end up with microservice zoo Duplicated code and logic More tra ff i c means higher and unpredictable costs
  13. 23 Firestore Easy to make bad schema design • Ability

    to store non structured data makes it easy to store mistakes
  14. 24 Firestore You can’t easily export data • Lack of

    standardization means each vendor their own special export format Easy to make bad schema design
  15. 25 Firestore Constant reads and writes are costly $ $$

    Polling also meant more reads From the the DB You can’t easily export data Easy to make bad schema design
  16. 27 Stream Closed Captioner Description: Allows a Twitch channel add

    closed captions to their stream. • Built with Ruby on Rails • Uses WebSockets to send captions from client side to server • Send captions from server to Twitch using Twitch APIs
  17. 29 My Life After Awhile With Success Came Problems •

    WebSockets in Ruby on Rails is a memory hog • To many requests on small servers leading to timeouts • Getting expensive to scale • I don’t have time to baby sit server
  18. 31 Deploy it and Forget it • Completely removed the

    need for Redis • Websockets run amazing!!! • Memory e ff i cient and you never have to restart • A ff ordable hosting and blazing fast • Fail fast, recovers fast My Life Now
  19. 32 Elixir Hits All The Marks Handle Large Bursts of

    Traffic Scalable Fast Affordable Easy to Deploy
  20. 34 Stream Team Extension Concept: Create, manage, and display a

    Twitch team Features: ๏ A channel can choose to display an o ffi cial team they are apart of ๏ They can create a custom team and add channels ๏ Upon installation all viewers of their channel can browse and view another team member’s channel Application to Migrate
  21. 36 Road Map 1 2 3 4 5 Create API

    Proxy Create Schema Replicate Logic & Tap Data Export Firebase to Postgres Switch to GQL
  22. 40 Creating API Proxy Time to Terraform terraform Terraform is

    a simple plug designed to work with Phoenix. Terraform allows you to incrementally transform a HTTP API into one powered by Phoenix - one endpoint at a time.
  23. 41 Creating API Proxy GET FLOW Terraforming UPSERT FLOW Excuse

    me. I’d like to ask you about your cars extended warranty
  24. 43 Creating API Proxy GET FLOW Successful Proxy UPSERT FLOW

    Proxy the request to serverless function Pass response back
  25. 44 Creating API Proxy GET FLOW Successful Proxy UPSERT FLOW

    Why? 1 2 Able to update server con fi guration in the Twitch Extension before any serious work is done I can now easily deploy backend changes without releasing a new extension version
  26. Bad idea if members wanted to share the same team

    47 Create Schema Firebase schema Channel Record Channel’s Custom Team
  27. Two Contexts Four new tables 50 Create Schema Ecto &

    Contexts Why? 1 2 Separate team from channel entity A channel can see who their memberships 3 Channels can have other app data associated with them in the future
  28. 55 Replicate Logic Tapping into the fl ow 1. Auth

    2. Validate 3. Save to Postgres
  29. 56 Replicate Logic Tapping into the fl ow Auth Validate

    Save to Postgres Why? 1 2 Test my schema against real world data Test server performance 3 I can save data in fl ight while performing database export
  30. 58 Firebase to Postgres Supabase to the rescue Supa Cool

    Library fi rebase-to-supabase Converts Firebase collections into JSON fi le which can then be imported into you Supabase Postgres DB.
  31. 60 Uploader & Processor Built-in Upload Feature Oban Job Processing

    Super Easy to Setup Simple Page to Upload Exports LIVEVIEW UPLOADER Finally a reason to use LiveView!
  32. 61 LiveView File Upload Built-in Upload Feature Oban Job Processing

    Super Easy to Setup Simple Page to Upload Exports PHOENIX LIVEVIEW Finally a reason to use LiveView!
  33. 62 LiveView File Upload Built-in Upload Feature Oban Job Processing

    Super Easy to Setup Simple Page to Upload Exports PHOENIX LIVEVIEW Finally a reason to use LiveView! Why? 1 2 Get away from vendor lock in Stable/Consistent Cost 3 No data loss since I am saving new upserts
  34. 69 Recap Why did I do all this work? 1

    2 3 4 5 6 Avoid vendor lock-in Consistent Costs Extremely Performant Very low cost to run (Actually runs on fl y.io free tier) Relational Databases are great Easy to manage