Concurrent and Resilient Connections to Outside the BEAM

Concurrent and Resilient Connections to Outside the BEAM

Faafc04d9e69b73b9f49995fd4c94d4d?s=128

Andrea Leopardi

May 11, 2016
Tweet

Transcript

  1. outside CONCURRENT RESILIENT and CONNECTIONS to the BEAM

  2. ERLANG VM mnesia message passing ets

  3. ERLANG VM key-value store relational db message queue

  4. ERLANG VM key-value store relational db message queue

  5. the outside world is SCARY

  6. ANDREA LEOPARDI @WHATYOUHIDE

  7. Gothenburg, Sweden

  8. FOOTBALL ADDICTS

  9. GENSERVER TCP

  10. SILLY the way

  11. defmodule Redis do def command(cmd) do end end

  12. defmodule Redis do def command(cmd) do {:ok, sock} = :gen_tcp.connect(...)

    :gen_tcp.send(sock, encode(cmd)) {:ok, data} = :gen_tcp.recv(sock, 0) :gen_tcp.close(sock) decode(data) end end
  13. defmodule Redis do def command(cmd) do {:ok, sock} = :gen_tcp.connect(...)

    :gen_tcp.send(sock, encode(cmd)) {:ok, data} = :gen_tcp.recv(sock, 0) :gen_tcp.close(sock) decode(data) end end
  14. defmodule Redis do def command(cmd) do {:ok, sock} = :gen_tcp.connect(...)

    :gen_tcp.send(sock, encode(cmd)) {:ok, data} = :gen_tcp.recv(sock, 0) :gen_tcp.close(sock) decode(data) end end
  15. defmodule Redis do def command(cmd) do {:ok, sock} = :gen_tcp.connect(...)

    :gen_tcp.send(sock, encode(cmd)) {:ok, data} = :gen_tcp.recv(sock, 0) :gen_tcp.close(sock) decode(data) end end
  16. defmodule Redis do def command(cmd) do {:ok, sock} = :gen_tcp.connect(...)

    :gen_tcp.send(sock, encode(cmd)) {:ok, data} = :gen_tcp.recv(sock, 0) :gen_tcp.close(sock) decode(data) end end
  17. defmodule Redis do def command(cmd) do {:ok, sock} = :gen_tcp.connect(...)

    :gen_tcp.send(sock, encode(cmd)) {:ok, data} = :gen_tcp.recv(sock, 0) :gen_tcp.close(sock) decode(data) end end
  18. opening NEW connections is EXPENSIVE

  19. GENSERVER keeping the socket in a

  20. TWO WAYS blocking non-blocking

  21. TWO WAYS blocking

  22. CLIENT GENSERVER socket socket send() recv()

  23. defmodule Redis do def command(conn, cmd) do sock = checkout(conn)

    # send and recv checkin(conn, sock) end end
  24. defmodule Redis do def command(conn, cmd) do sock = checkout(conn)

    # send and recv checkin(conn, sock) end end
  25. {:error, :checked_out} :queue.in() VS

  26. POOLING :|

  27. TWO WAYS non-blocking

  28. active: true {:noreply, _} + GenServer.reply/2 +

  29. client Redis client client GenServer

  30. tcp is FULL DUPLEX

  31. client Redis client client Sender Receiver

  32. TWO WAYS blocking non-blocking copies less data + needs pooling

    - doesn't use full duplex - enc/dec on client + - copies more data + doesn't need pooling + uses full duplex - enc/dec in server
  33. RESIL IENCY

  34. Redis GenServer

  35. Redis GenServer

  36. {:tcp_closed, reason} backoff + reconnect

  37. hex.pm/packages/connection gen_

  38. Connection GenServer

  39. init/1 connect/2 disconnect/2 handle_call/3 handle_cast/2 handle_info/2 terminate/2 code_change/3 {:disconnect, reason,

    state} {:backoff, timeout, state}
  40. sync/async CONNECT

  41. SYNC start_link {:ok, pid} init() connect

  42. ASYNC start_link {:ok, pid} init() connect connect()

  43. You now allow initializations with fewer guarantees: they went from

    "the connection is available" to "the connection manager is available". Fred Hebert
  44. CONNECTION TALKING CONCURRENCY RESILIENCY

  45. @whatyouhide