$30 off During Our Annual Pro Sale. View Details »

Building Bulletproof Real-Time Apps with Phoenix LiveView + Stream Data

Building Bulletproof Real-Time Apps with Phoenix LiveView + Stream Data

Elixir School's Day 2 workshop at ElixirConf 2019

Sophie DeBenedetto

August 28, 2019
Tweet

More Decks by Sophie DeBenedetto

Other Decks in Programming

Transcript

  1. Building Bulletproof Real-Time Applications with Phoenix, LiveView, and StreamData

  2. Introductions 2

  3. Hello! I’m Sophie I’m an engineer and sometimes teacher at

    The Flatiron School where I build education and business tools in Elixir and Ruby. @sm_debenedetto 3 *and this is my dog
  4. And... I’m Michael I’m a lead engineer at RentPath, where

    we build websites to help people find a place to live. @michaelstalker 4 *and this is my family
  5. Elixir School A free, open-source Elixir curriculum Through the hard

    work of many volunteers around the world, we’ve developed and translated content covering everything from intro topics to deep dives. 5
  6. Get Involved! Help us grow the Elixir School community Write

    a lesson, contribute a TIL (or longer!) blog post or provide a translation. Open an issue here describing your idea or simply open a PR with your work to get started. 6
  7. Goals 7

  8. What We’ll Learn ◦ Phoenix LiveView ◦ Phoenix PubSub ◦

    Phoenix Presence ◦ Property-based Testing with StreamData 8
  9. What We’ll Build A real-time ticket estimation tool ◦ Users

    can see other users in the “estimation room” in real-time ◦ Users can vote on tickets ◦ Users can see the winning vote tallied in real-time and move on to the next ticket 9
  10. Pointing Party A collaborative ticket estimation tool for your team

    10
  11. 11

  12. User Starts Estimation Round When I click “start pointing!” Then

    the first card for estimation appears for all users in the room Features Real-Time User Presence When a new user joins/leaves Then I see the list of present users update in the UI 12
  13. Ticket is Estimated When another user submits a vote Then

    I see the UI update to indicate they have voted Features Winner is Calculated When the last user submits a vote Then all users see the winning vote And a user can click “next card” 13
  14. Application Starting State Log In with a username (We built

    some dummy authorization flows) See the static ticket estimation page Click the start button and... Oh no you can’t estimate a ticket! 14
  15. We Need Real-Time! In the current state of our app,

    users cannot collaborate on ticket estimation. 15
  16. What is the Real-Time Web? Users receive new information from

    the server as soon as it is available—no request required This represents a significant departure from traditional HTTP communication Made possible by WebSockets 16
  17. 17 WebSockets

  18. What are WebSockets? A Protocol built on top of TCP

    that allows for bi-directional, "full-duplex" communication between the client and the server by creating a persistent connection between the two. 18
  19. 19 Server Client Request GET https://elixirschool.com Response <h1>Weclome to Elixir

    School!</h1> HTTP Protocol
  20. 20 Server Client Step 1. Request to initiate WS connection

    Client Client GET / Host: elixirschool.com Upgrade: websocket Connection: upgrade
  21. 21 Server Client Step 2. Open and maintain WS connections

    Client Client
  22. 22 We’ll use WebSockets to maintain a persistent, bi-directional and

    stateful connection between our ticket estimation clients and the server with the help of Phoenix LiveView
  23. 23 Phoenix LiveView

  24. What is LiveView? Phoenix LiveView leverages server-rendered HTML and Phoenix’s

    native WebSocket tooling so you can build fancy real-time features without all that complicated JavaScript. 24
  25. 25 So...I don’t need to write JavaScript ever again?

  26. 26 Not exactly.

  27. 27 You just don’t need it as often.

  28. 28 How It Works

  29. Connecting • Client connects to Phoenix server over WebSockets, at

    the LiveView socket endpoint • Phoenix server starts a LiveView process for that user’s socket 29
  30. Rendering the Template • The LiveView leex (Live EEX) template

    is rendered, using the data from the LiveView process’ state 30
  31. Handling Changes • Certain user interactions on the page will

    send a message over the socket to the LiveView process • The LiveView process will update its state and push a message back down the socket • The leex template will re-render with the new state 31
  32. 32 A Closer Look

  33. Roadmap 3. Handling Changes 2. Connecting to LiveView 33 1.

    Configuring LiveVIew
  34. 34 1. Configuring LiveView

  35. 35 Install the dependency

  36. 36 # mix.exs def deps do [ {:phoenix_live_view, github: "phoenixframework/phoenix_live_view"}

    ] end
  37. 37 Update our app’s config with the signing salt for

    our LiveView socket connection
  38. 38 config :my_app, PointingParty.Endpoint, ... live_view: [ signing_salt: "YOUR_SECRET" ]

  39. 39 Configure our app to support “Live EEX” templates

  40. 40 config :phoenix, template_engines: [ leex: Phoenix.LiveView.Engine ]

  41. 41 Import the LiveView render functions

  42. 42 # lib/pointing_party_web.ex def view do quote do ... import

    Phoenix.LiveView, only: [live_render: 2, live_render: 3] end end
  43. 43 Add the LiveView JS library to our NPM dependencies

  44. 44 # assets/package.json { "dependencies": { ... "phoenix_live_view": "file:../deps/phoenix_live_view" }

    }
  45. 45 Add the LiveView path to our dev live reload

    config
  46. 46 # config/dev.exs config :pointing_party, PointingPartyWeb.Endpoint, live_reload: [ patterns: [

    ..., ~r{lib/my_app_web/live/.*(ex)$} ] ]
  47. Roadmap 3. Handling Changes 2. Connecting to LiveView 47 1.

    Configuring LiveVIew
  48. 48 2. Connecting to LiveView

  49. Connection Process • User visits ”/cards” in the browser 49

  50. Connection Process • User visits ”/cards” in the browser •

    CardController mounts the LiveView 50
  51. Connection Process • User visits ”/cards” in the browser •

    CardController mounts the LiveView • The LiveView renders the LEEX template as static HTML 51
  52. Connection Process • User visits ”/cards” in the browser •

    CardController mounts the LiveView • The LiveView renders the LEEX template as static HTML • A JS snippet on that static HTML page sends the “WebSocket connect” request 52
  53. Connection Process • User visits ”/cards” in the browser •

    CardController mounts the LiveView • The LiveView renders the LEEX template as static HTML • A JS snippet on that static HTML page sends the “WebSocket connect” request • The LiveView process re-renders the template over WebSockets, is now listening for events from the client 53
  54. 54 LiveView Client Request GET /cards Render LEEX Template <h1>Start

    pointing!</h1> Step 1. Mount the LiveView + render static HTML CardController Mount LiveView
  55. 55 Client Connect to LiveView socket Step 2. Establish WS

    connection LiveView LV Socket
  56. 56 Client Connect to LiveView socket Step 3. Re-render the

    template LiveView Re-render template over WS connection LV Socket
  57. We Will • Mount the LiveView socket endpoint • Define

    a LiveView and LEEX template • Teach our controller to render the LiveView 57
  58. 58 Mount the LiveView socket at the “/live” endpoint

  59. 59 defmodule PointingPartyWeb.Endpoint do use Phoenix.Endpoint, otp_app: :pointing_party socket "/live",

    Phoenix.LiveView.Socket end
  60. 60 Define The LiveView

  61. 61 # app/pointing_party_web/live/card_live.ex defmodule PointingPartyWeb.CardLive do use Phoenix.LiveView alias PointingPartyWeb.CardView

    def render(assigns) do Phoenix.View.render(CardView, "index.html", assigns) end def mount(_session, socket) do {:ok, assign(socket, is_pointing: false)} end end
  62. 62 Define The LEEX Template

  63. 63 <div> <%= if !@is_pointing do %> <div> <button> Start

    the Party! </button> </div> <% else %> <!-- card display here --> <% end %> </div>
  64. 64 Include the LiveView socket connect JS

  65. 65 // assets/js/app.js import LiveSocket from "phoenix_live_view" let liveSocket =

    new LiveSocket("/live") liveSocket.connect()
  66. 66 Render the LiveView from the controller

  67. 67 defmodule PointingPartyWeb.CardController do use PointingPartyWeb, :controller import Phoenix.LiveView.Controller def

    index(conn, _params) do %{assigns: %{username: username}} = conn live_render(conn, PointingPartyWeb.CardLive, session: %{username: username}) end end
  68. 68 Now, when the user visits “/cards”, the LiveView process

    will start
  69. 69 And render the static LEEX template

  70. 70 <div id="phx-20gvOvqvFMA=" data-phx-view="PointingParty.CardsLiveView" data-phx-session="SFMyNTY.g3QACZAAEZGFY..."> ... </div>

  71. 71 This will send the LiveView socket connect request

  72. 72 Opening a bi-directional, stateful connection

  73. 73 Over which the LEEX template will re-render

  74. 74 3. Handling Changes

  75. Roadmap 3. Handling Changes 2. Connecting to LiveView 75 1.

    Configuring LiveVIew
  76. 76 <div> <%= if !@is_pointing do %> <div> <button> Start

    the Party! </button> </div> <% else %> <!-- card display here --> <% end %> </div>
  77. We Will • Add a DOM element binding that will

    send an event from the client to LiveView 77
  78. We Will • Add a DOM element binding that will

    send an event from the client to LiveView • Teach our LiveView how to handle this event 78
  79. 79 Add the DOM element binding

  80. 80 <button phx-click="start_party"> Start the Party! </button>

  81. 81 The phx-click event will be sent over the WS

    to our LiveView
  82. 82 Which will invoke the matching handle_event/3 function

  83. 83 def handle_event("start_party", _value, socket) do # coming soon! end

  84. 84 def handle_event("start_party", _value, socket) do [current_card | remaining_cards] =

    Card.cards() socket = assign(socket, is_pointing: true, current_card: current_card, remaining_cards: remaining_cards) {:noreply, socket} end
  85. 85 Re-rendering the template

  86. 86 <div> <%= if !@is_pointing do %> <div> <button phx-click="start_party">

    Start the Party! </button> </div> <% else %> <!-- card display here --> <% end %> </div>
  87. 87 <div> <h2><%= @current_card.title %></h2> <p><%= @current_card.description %></p> <form phx-submit="vote">

    <label for="story-points">Story Points</label> <select name="points"> <option value="0">0</option> <option value="1">1</option> <option value="3">3</option> <option value="5">5</option> </select> </form> </div>
  88. Your Turn!

  89. Fork this repo

  90. Feature Roadmap 2. Users see list of present users 3.

    All votes are tallied, displayed 1. User initiates pointing round 90
  91. 91

  92. User initiates pointing round 92 Define the LiveView • Define

    a LiveView and LEEX template - We did this for you :) • Render the LiveView from the controller - We did this for you :) Handle Events • Add a phx-click event to the “start pointing” button • Define a handle_event/3 function to match this event and update socket.assigns accordingly Configure LiveView • We did this for you :)
  93. Building Bulletproof Real-Time Applications Part 2: Phoenix PubSub

  94. 94 We Have a Problem

  95. 95 The pointing round is only started for the single

    user who clicked “start pointing”
  96. 96 How can we broadcast shared changes to all of

    our users?
  97. 97 We need PubSub!

  98. 98 Phoenix PubSub

  99. What is PubSub? PubSub (“publish/subscribe”) is a pattern in which

    we publish messages to a “topic”, such that those messages can be consumed by any number of subscribers. 99
  100. What is Phoenix PubSub? The Phoenix PubSub library allows us

    subscribe Elixir processes to a shared topic and publish messages to those processes. Phoenix Channels abstract away the interactions with Phoenix PubSub, but you can also use the PubSub library directly, which is exactly what we’ll do from within our LiveView. 100
  101. Phoenix PubSub and Distributed Elixir Phoenix PubSub is configured by

    default to be distribution-friendly with Phoenix.PubSub.PG2. With this adapter, messages are broadcast to processes sharing a topic across nodes. 101
  102. 102 How It Works

  103. 103 Subscribe users to a shared topic

  104. 104 A user “joins” a pointing room when they mount

    the LiveView
  105. 105 This is when we will subscribe that LiveView process

    to a shared PubSub topic
  106. 106 Broadcasting Messages

  107. 107 When a user sends an event to LiveView, we

    will broadcast it over PubSub to all the other users’ LiveViews
  108. 108 Our LiveView code will implement a handle_info/3 function that

    receives the broadcast, updates state, and causes the template to re-render
  109. 109 Client Submit “Start_pointing” event LiveView Client Client LiveView LiveView

    PubSub Broadcast to subscribers handle_info and update socket Re-render template
  110. Building Bulletproof Real-Time Applications Part 3: Phoenix Presence

  111. 111 We Have A Problem

  112. 112 We don’t know which users are participating, who has

    voted or what their vote is!
  113. 113 We don’t know which users are participating, who has

    voted or what their vote is!
  114. 114 How can we sync and share stateful info, like

    who is present, across LiveView processes?
  115. 115 Phoenix Presence

  116. What is Phoenix Presence? The Phoenix.Presence module allows us to:

    • Register a process under a given topic • Store that info in a decentralized and resilient data store. • Broadcast presence-related events and sync presence data with ease. 116
  117. 117 Phoenix Presence + Distributed Elixir =

  118. Phoenix Presence Uses a CRDT A CRDT (Conflict-free Replicated Data

    Type) backs Phoenix Presence. What’s so special about a CFRDT? Unlike centralized data stores like Redis, Phoenix Presence… 118
  119. “ …gives you high availability and performance because we don't

    have to send all data through a central server. It also gives you resilience to failure because the system automatically recovers from failures. - Chris McCord 119
  120. 120 How It Works

  121. Configure Presence • Define a module that uses Phoenix.Presence and

    shares a PubSub server with the rest of our application. 121
  122. Register a Process • Use the Presence.Track behaviour to register

    a given user’s LiveView process under a shared topic 122
  123. Broadcast Presence to Existing Users • Teach LiveView to handle

    the presence broadcast by fetching the list of present users from the presence data store 123
  124. Building Bulletproof Real-Time Applications Part 4: Property-based testing with Stream

    Data
  125. Testing Achieving 100% confidence there are no bugs (Just kidding.)

    125
  126. Pascal’s Triangle

  127. 127 1 1 1 1 2 1 1 3 3

    1 1 4 6 4 1 …
  128. 128 1

  129. 129 1 1 1

  130. 130 1 1 1 1 2 1

  131. 131 1 1 1 1 2 1 1 3 3

    1
  132. 132 1 1 1 1 2 1 1 3 3

    1 1 4 6 4 1
  133. 133 1 1 1 1 2 1 1 3 3

    1 1 4 6 4 1 …
  134. “ 134 Given an integer n >= 0, write a

    function that returns the nth row of Pascal’s Triangle.
  135. Possible Code Implementation defmodule Pascal do def row(n) do #

    Amazing calculations end end 135
  136. Example-Based Unit Testing Stuff you’re probably used to. 136

  137. Row 0 describe "row/1" do test "returns [1] for first

    row" do assert Pascal.row(0) == [1] end end 137
  138. Row 1 describe "row/1" do test "returns [1, 1] for

    second row" do assert Pascal.row(1) == [1, 1] end end 138
  139. Row 4 describe "row/1" do test "returns value for 5th

    row" do assert Pascal.row(4) == [1, 4, 6, 4, 1] end end 139
  140. Advantages

  141. Advantages ◦ Easy to reason about ◦ Target specific test

    cases
  142. Disadvantages

  143. Disadvantages ◦ Requires you to get edge cases right ◦

    Doesn’t cover large indexes ◦ Doesn’t require a deep understanding of your domain
  144. Stuff that’s going to feel hard. 144 Property-based Testing with

    StreamData
  145. Streams of Random Data Using Generators 145

  146. Booleans StreamData.boolean() |> Enum.take(5) # [false, true, false, false, false]

    146
  147. Integers StreamData.integer() |> Enum.take(5) # [0, -2, -2, 4, 3]

    147
  148. Strings StreamData.string(:alphanumeric) |> Enum.take(5) # ["", "", "vL2", "AbgJ", "XuH7"]

    148
  149. You Can Combine 149

  150. Lists list = StreamData.list_of( StreamData.integer() ) Enum.take(list, 5) # [[],

    [-2, 1], [-1], [-2, -2, -1], []] 150
  151. Maps map = StreamData.map_of( StreamData.atom(:alphanumeric), StreamData.integer() ) Enum.take(map, 5) #

    [%{}, %{}, %{iB3: -3, inF: -3, k: -1}, %{BSy: 2, z: -1}, %{}] 151
  152. Lists of Maps StreamData.list_of( StreamData.map_of( StreamData.atom(:alphanumeric), StreamData.integer() ) ) 152

  153. Lists of Maps of Lists of Integers StreamData.list_of( StreamData.map_of( StreamData.atom(:alphanumeric),

    StreamData.list_of(StreamData.integer()) ) ) 153
  154. 154 [ [%{gc: []}], [%{O: [-1]}, %{f: [1, -2]}], [%{}],

    [ %{_: [], kTy: [4, -2, 0], u2h: [4, -2], w: [0, -3, 2, -4]}, %{_CK: [-2, -3, 3]} ], [ %{r: [-3, 2, -1, 2]}, %{_0: [1], _I: [-5, 0, 4, -3], _O: [-3, 3, 0], kz: [1, 4, 5, -1, -5]}, %{fr: [0, -2, 1, 3, 4]}, ] ]
  155. Data Starts Small And gets larger 155

  156. Integers, Revisited StreamData.integer() |> Enum.take(25) # [0, -2, 1, -3,

    -1, 1, 4, 7, -1, 9, 0, 8, 9, -12, 12, -5, 0, -7, -16, 8, 21, 1, -16, -7, -19] 156
  157. 157 Lots of Tests, Data Shrinking

  158. 158 1 1 1 1 2 1 1 3 3

    1 1 4 6 4 1 …
  159. Possible Test Implementation property "number of row elements is n+1"

    do check all number <- StreamData.integer(), row_index = abs(number) do element_count = length(Pascal.row(row_index)) assert element_count == row_index + 1 end end 159
  160. assert length(Pascal.row(0)) == 1 160

  161. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 161

  162. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 assert length(Pascal.row(0))

    == 1 162
  163. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 assert length(Pascal.row(0))

    == 1 assert length(Pascal.row(2)) == 3 163
  164. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 assert length(Pascal.row(0))

    == 1 assert length(Pascal.row(2)) == 3 # … assert length(Pascal.row(47)) == 48 164
  165. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 assert length(Pascal.row(0))

    == 1 assert length(Pascal.row(2)) == 3 # … assert length(Pascal.row(47)) == 48 assert length(Pascal.row(58)) == 117 165
  166. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 assert length(Pascal.row(0))

    == 1 assert length(Pascal.row(2)) == 3 # … assert length(Pascal.row(47)) == 48 assert length(Pascal.row(58)) == 117 # ?!?! 166
  167. Stuff that’s true of any solution 167 1 1 1

    1 2 1 1 3 3 1 1 4 6 4 1 …
  168. Stuff that’s true of any solution ◦ Each row is

    symmetrical 168 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
  169. Stuff that’s true of any solution ◦ Each row is

    symmetrical ◦ The first and last elements of each row are 1 169 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
  170. Stuff that’s true of any solution ◦ Each row is

    symmetrical ◦ The first and last elements of each row are 1 ◦ After row 0, the second element in each list is the row index 170 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
  171. Stuff that’s true of any solution ◦ Each row is

    symmetrical ◦ The first and last elements of each row are 1 ◦ After row 0, the second element in each list is the row index ◦ So is the second-to-last element 171 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
  172. Stuff that’s true of any solution ◦ We can represent

    any row as a list 172 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
  173. Stuff that’s true of any solution ◦ We can represent

    any row as a list ◦ Row 0 just has one element: 1 173 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
  174. Stuff that’s true of any solution ◦ We can represent

    any row as a list ◦ Row 0 just has one element: 1 ◦ For all rows after 0, 1 appears twice 174 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
  175. Stuff that’s true of any solution ◦ We can represent

    any row as a list ◦ Row 0 just has one element: 1 ◦ For all rows after 0, 1 appears twice ◦ Each row n has n+1 elements in it 175 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
  176. Stuff that’s true of any solution ◦ We can represent

    any row as a list ◦ Row 0 just has one element: 1 ◦ For all rows after 0, 1 appears twice ◦ Each row n has n+1 elements in it ◦ In each row, numbers go up, then down 176 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
  177. Potential Properties ◦ Data type ◦ Data shape ◦ Count

    of specific values ◦ Presence/emptiness ◦ Size ◦ Membership ◦ Range ◦ Symmetry ◦ Relationships
  178. Testing Votes

  179. Voting Results def calculate_votes(users) do case winning_vote(users) do top_two when

    is_list(top_two) -> {"tie", top_two} winner -> {"winner", winner} end end
  180. A Property Test property "calculated vote is a list or

    an integer" do check all users <- user_generator, {_event, winner} = calculate_votes(users) # We'll assert something here later end end
  181. A Missing Generator setup do # Create generator here [user_generator:

    user_generator] end
  182. Your Turn

  183. Properties of Our Code

  184. What’s true of any solution? def calculate_votes(users) do case winning_vote(users)

    do top_two when is_list(top_two) -> {"tie", top_two} winner -> {"winner", winner} end end
  185. Your Turn Writing our properties and assertions

  186. I :heart: StreamData And you should, too.

  187. Resources ◦ “An introduction to property-based testing” by Scott Wlaschin

    ◦ “Choosing properties for property-based testing” by Scott Wlaschin ◦ Property-Based Testing with PropEr, Erlang, and Elixir and PropEr Testing by Fred Hebert ◦ SteamData documentation ◦ “Testing the Hard Stuff and Staying Sane” by John Hughes ◦ “Picking Properties to Test in Property Based Testing” by Michael Stalker 187
  188. Recap