$30 off During Our Annual Pro Sale. View Details »

Serverless to Serving Elixir: Migrating Serverless app to run on Phoenix

Erik Guzman
September 11, 2022

Serverless to Serving Elixir: Migrating Serverless app to run on Phoenix

Problem: Increasing productivity, affordability, scalability, and even less coding (gasp!) are the goals for any web application. Add on top unpredictable server traffic, then a solution you might consider is going serverless and throwing in one of those nifty NoSQL database solutions. Easy enough, but at what cost? You can unintentionally suffer vendor lock-in, deal with cold starts, architect your application improperly, and miscalculate cost structure at scale. But there is an even better solution you might want to consider, Elixir. Elixir and Pheonix can help developers solve all these technical challenges, allowing you to focus on the end-user experience.

Purpose: “Serverless to Serving Elixir: Migrating Serverless app to run on Phoenix” will demonstrate why migrating Elixir can solve all the same issues your serverless application would, make it faster, avoid vendor lock-in, and help future-proof your application. Maybe even have you second-guess if you ever need to go serverless again.

Live Demo/ Walkthru: In preparation for this talk, I will guide conference attendees through the libraries and strategies to take NodeJS Serverless applications using Firebase Firestore and migrate them to a single Phoenix application and Postgres database with no downtime.

Erik Guzman

September 11, 2022
Tweet

Other Decks in Programming

Transcript

  1. 1
    Erik Guzman
    Full Stack Engineer @ ZEAL
    Ruby on Rails

    ReactJS
    Who am I?
    After Dark:

    Microsoft MVP Developer Tools

    Elixir Development

    Technical Content Creator

    Twitch 3rd Party Developer

    View Slide

  2. SERVERLESS TO SERVING ELIXIR

    MIGRATING SERVERLESS APP TO RUN ON PHOENIX


    View Slide

  3. 3
    The Journey
    1
    2
    3
    4
    5
    Developing for twitch.tv
    Going Serverless
    Problems Faced
    Why Elixir?
    Migration Process

    View Slide

  4. DEVELOPING FOR TWITCH.TV

    View Slide

  5. 5
    What is Twitch?
    Live video
    streaming
    platform
    • Users aka “channels” live stream
    activities

    • To watch a channel, viewers visit a
    channel page

    • Viewership can vary drastically, from
    1 to 100k concurrent viewers

    • Twitch o
    ff
    ers various tools, like
    Extensions, to enhance a viewers
    experience

    View Slide

  6. 6
    What is a Twitch Extension?
    HTML/CSS/JS
    Website
    • Little website trapped in an iframe

    • Hosted on the Twitch CDN

    • Managed like an app store

    • You are in charge of hosting any backend

    • Installed by the Twitch Channel

    • Loaded by each viewer of the channel

    View Slide

  7. 7
    Recipe for Success
    Handle Large Bursts of Traffic
    Scalable
    Fast
    Affordable
    Easy to Deploy

    View Slide

  8. GOING SERVERLESS

    View Slide

  9. 9
    Try out Serverless
    Functions
    Great! What’s
    Serverless
    Functions?
    Recommendations

    View Slide

  10. 10
    What’s Serverless Functions?
    Easy to Deploy
    A
    ff
    ordable
    Scalable
    Pay Per Invocation & Duration
    $0.20 per million executions plus
    potential execution time
    No long lived server costs
    Deploy single function or all
    functions easily
    Can handle large amount of
    requests
    Fully Managed Infrastructure
    Don’t worry about con
    fi
    guring
    servers or scaling

    View Slide

  11. 11
    Serverless Breaks Up The Band
    Traditional Hosting

    View Slide

  12. 12
    Serverless Breaks Up The Band
    Such micro

    View Slide

  13. 13
    Serverless Checks Out
    Handle Large Bursts of Traffic
    Scalable
    Fast
    Affordable
    Easy to Deploy

    View Slide

  14. 14
    What DB Do I Use?

    View Slide

  15. Why?
    • Don’t have to pay for long run database server

    • Only pay for reads and writes exceeding a certain amount

    • Pay for network tra
    ff
    i
    c after a certain amount

    • I only pay if my extension is a success
    NoSQL: Firestore 15

    View Slide

  16. 16
    So I Built Stu
    f
    Github Projects
    Stream Team
    Better Information
    Panel
    Stream Closed
    Captioner
    What Next?
    Not actually

    serverless

    View Slide

  17. MORE EXTENSION

    MORE PROBLEMS

    View Slide

  18. 18
    Serverless
    Cold starts for initial calls • Function need time to initialize

    • Not all your functions will be “hot”

    • Need to pay to keep them long lived

    View Slide

  19. 19
    Serverless
    Cold starts for initial calls
    Easy to end up with microservice zoo
    • Functions or new project for everything

    • Hard to manage

    View Slide

  20. 20
    Serverless
    Cold starts for initial calls
    Duplicated code and logic
    • Multiple functions or multiple projects
    can lead to duplicating code
    Easy to end up with microservice zoo

    View Slide

  21. 21
    Serverless
    Cold starts for initial calls
    Easy to end up with microservice zoo
    Duplicated code and logic
    More tra
    ff i
    c means higher and
    unpredictable costs
    $15

    View Slide

  22. 22
    Serverless
    $15
    Forces you to go as lean as possible
    on number of requests
    When I realized any form

    of polling is a bad idea
    Cold starts for initial calls
    Easy to end up with microservice zoo
    Duplicated code and logic
    More tra
    ff i
    c means higher and
    unpredictable costs

    View Slide

  23. 23
    Firestore
    Easy to make bad schema design • Ability to store non structured data
    makes it easy to store mistakes

    View Slide

  24. 24
    Firestore
    You can’t easily export data
    • Lack of standardization means each
    vendor their own special export format
    Easy to make bad schema design

    View Slide

  25. 25
    Firestore
    Constant reads and writes are costly $
    $$
    Polling also meant more reads

    From the the DB
    You can’t easily export data
    Easy to make bad schema design

    View Slide

  26. WHY ELIXIR?
    A Side Story

    View Slide

  27. 27
    Stream Closed


    Captioner
    Description:
    Allows a Twitch channel add closed
    captions to their stream.
    • Built with Ruby on Rails

    • Uses WebSockets to send captions
    from client side to server

    • Send captions from server to Twitch
    using Twitch APIs

    View Slide

  28. 28
    The Past
    STREAM CLOSED CAPTIONER
    ORIGINAL ARCHITECTURE
    Ruby on Rails Redis Postgres
    Twitch API

    View Slide

  29. 29
    My Life After Awhile
    With Success
    Came Problems
    • WebSockets in Ruby on Rails is a
    memory hog

    • To many requests on small servers
    leading to timeouts

    • Getting expensive to scale

    • I don’t have time to baby sit server

    View Slide

  30. 30
    The Present
    STREAM CLOSED CAPTIONER
    ZOOMER ARCHITECTURE
    Elixir Phoenix Postgres
    Redis
    Twitch API

    View Slide

  31. 31
    Deploy it and
    Forget it
    • Completely removed the need for
    Redis

    • Websockets run amazing!!!

    • Memory e
    ff i
    cient and you never have
    to restart

    • A
    ff
    ordable hosting and blazing fast

    • Fail fast, recovers fast
    My Life Now

    View Slide

  32. 32
    Elixir Hits All The Marks
    Handle Large Bursts of Traffic
    Scalable
    Fast
    Affordable
    Easy to Deploy

    View Slide

  33. THE GREAT MIGRATION
    Serverless to Serverful

    View Slide

  34. 34
    Stream Team
    Extension
    Concept:
    Create, manage, and display a Twitch team
    Features:
    ๏ A channel can choose to display an
    o
    ffi
    cial team they are apart of

    ๏ They can create a custom team and add
    channels

    ๏ Upon installation all viewers of their
    channel can browse and view another
    team member’s channel
    Application to Migrate

    View Slide

  35. 35
    Top Goals
    Migrate Data
    to Postgres
    Zero Down
    Time
    No Loss of
    Data
    1 2 3

    View Slide

  36. 36
    Road Map
    1 2 3 4 5
    Create API
    Proxy
    Create
    Schema
    Replicate
    Logic &
    Tap Data
    Export Firebase
    to Postgres
    Switch to GQL

    View Slide

  37. THE GREAT MIGRATION
    1. Create API Proxy

    View Slide

  38. Current Request Flow 38

    View Slide

  39. 39
    Creating API Proxy
    GET FLOW
    All in sequence
    UPSERT FLOW

    View Slide

  40. 40
    Creating API Proxy
    Time to
    Terraform
    terraform
    Terraform is a simple plug designed to work
    with Phoenix. Terraform allows you to
    incrementally transform a HTTP API into
    one powered by Phoenix - one endpoint at
    a time.

    View Slide

  41. 41
    Creating API Proxy
    GET FLOW
    Terraforming
    UPSERT FLOW
    Excuse me. I’d like to ask
    you about your cars

    extended warranty

    View Slide

  42. 42
    Creating API Proxy
    GET FLOW
    Successful Proxy
    UPSERT FLOW

    View Slide

  43. 43
    Creating API Proxy
    GET FLOW
    Successful Proxy
    UPSERT FLOW
    Proxy the request to

    serverless function

    Pass response back

    View Slide

  44. 44
    Creating API Proxy
    GET FLOW
    Successful Proxy
    UPSERT FLOW
    Why?
    1
    2
    Able to update server con
    fi
    guration in the
    Twitch Extension before any serious work is
    done
    I can now easily deploy backend changes
    without releasing a new extension version

    View Slide

  45. THE GREAT MIGRATION
    2. Create Schema

    View Slide

  46. 46
    Create Schema
    Firebase schema
    Channel Record
    Channel’s Custom Team

    View Slide

  47. Bad idea if members wanted to

    share the same team
    47
    Create Schema
    Firebase schema
    Channel Record
    Channel’s Custom Team

    View Slide

  48. 48
    Create Schema
    Redesign time
    FIREBASE
    SCHEMA
    ECTO


    SCHEMA

    View Slide

  49. 49
    Create Schema
    Ecto & Contexts
    Two Contexts
    Four new tables

    View Slide

  50. Two Contexts
    Four new tables
    50
    Create Schema
    Ecto & Contexts
    Why?
    1
    2
    Separate team from channel entity
    A channel can see who their memberships
    3
    Channels can have other app data
    associated with them in the future

    View Slide

  51. THE GREAT MIGRATION
    3. Replicate Logic & Tap Data

    View Slide

  52. 52
    Replicate Logic
    Data Flow

    View Slide

  53. 53
    Replicate Logic
    Data Flow
    All the business
    logic done here
    Reimplemented

    Here

    View Slide

  54. 54
    Replicate Logic
    Tapping into the
    fl
    ow

    View Slide

  55. 55
    Replicate Logic
    Tapping into the
    fl
    ow
    1. Auth
    2. Validate
    3. Save to Postgres

    View Slide

  56. 56
    Replicate Logic
    Tapping into the
    fl
    ow
    Auth
    Validate
    Save to Postgres
    Why?
    1
    2
    Test my schema against real world data
    Test server performance
    3
    I can save data in
    fl
    ight while performing
    database export

    View Slide

  57. THE GREAT MIGRATION
    4. Export Firebase to Postgres

    View Slide

  58. 58
    Firebase to Postgres
    Supabase to the rescue
    Supa Cool
    Library
    fi
    rebase-to-supabase
    Converts Firebase collections into JSON
    fi
    le which can then be imported into you
    Supabase Postgres DB.

    View Slide

  59. 59
    Firebase to Postgres
    How to get data into Postgres?

    View Slide

  60. 60
    Uploader & Processor
    Built-in Upload Feature
    Oban Job Processing Super Easy to Setup
    Simple Page to Upload Exports
    LIVEVIEW
    UPLOADER
    Finally a reason to use LiveView!

    View Slide

  61. 61
    LiveView File Upload
    Built-in Upload Feature
    Oban Job Processing Super Easy to Setup
    Simple Page to Upload Exports
    PHOENIX


    LIVEVIEW


    Finally a reason to use LiveView!

    View Slide

  62. 62
    LiveView File Upload
    Built-in Upload Feature
    Oban Job Processing Super Easy to Setup
    Simple Page to Upload Exports
    PHOENIX


    LIVEVIEW


    Finally a reason to use LiveView!
    Why?
    1
    2
    Get away from vendor lock in
    Stable/Consistent Cost
    3 No data loss since I am saving new upserts

    View Slide

  63. THE GREAT MIGRATION
    5. Switch to GQL

    View Slide

  64. 64
    Switch to GQL
    Just one more change

    View Slide

  65. 65
    Switch to GQL
    Multiple endpoints is so 2010s
    POST
    DESTROY

    View Slide

  66. 66
    Switch to GraphQL
    Ask for what I need

    View Slide

  67. 67
    Switch to GraphQL
    Profound reasons…

    View Slide

  68. 68
    Switch to GraphQL
    Profound reasons…
    838
    Why?
    1
    2
    GraphQL is cool
    I felt like it

    View Slide

  69. 69
    Recap
    Why did I do all this work?
    1
    2
    3
    4
    5
    6
    Avoid vendor lock-in
    Consistent Costs
    Extremely Performant
    Very low cost to run

    (Actually runs on
    fl
    y.io free tier)
    Relational Databases are great
    Easy to manage

    View Slide

  70. 70
    Site:
    guzman.codes
    Socials:
    THANKS
    Talk2MeGooseman

    View Slide