Upgrade to Pro — share decks privately, control downloads, hide ads and more …

To Hell and Back: Why choosing Elixir to build a startup from scratch was both a terrible and an awesome decision

To Hell and Back: Why choosing Elixir to build a startup from scratch was both a terrible and an awesome decision

Building a chatbot and an e-commerce startup (www.fredbots.com) with 30k+ lines of Elixir and Phoenix code and almost redoing from scratch a few times.

Microservices and distribution done wrong, getting saved by a small bunny, reaching 178 million messages/day in a 8GB machine.

Slides from my Elixir User Group SP presentation on 18/Mar/17 - http://meetup.com/pt-BR/elug_sp/.

Alfred R. Baudisch

March 18, 2017
Tweet

Other Decks in Programming

Transcript

  1. To Hell and Back
    Why choosing Elixir to build a startup
    from scratch was both a terrible and an
    awesome decision

    View Slide

  2. Who
    Co-founder and CEO at Fred (www.fredbots.com).
    Self-taught developer since 8. Used longest: PHP and Unity3D w/ C#.
    Almost a Computer Engineer, a Lawyer, a Veterinarian and a
    Computer Engineer again, but dropped out.
    Ex-snake handler and fishing tackle sales clerk. Founded and closed
    2 other startups (and many others still archived).
    Alfred Reinold Baudisch
    @AlfredBaudisch / [email protected]

    View Slide

  3. How a WhatsApp bot got famous and evolved as the Brazilian WeChat and conversational commerce platform (goo.gl/v0qlJM)
    Food Delivery WhatsApp bot
    with unofficial API
    Exploded on the news Banned by WhatsApp
    FredApp, “WhatsApp for
    Business”, standalone chat +
    bot platform
    Angel investment
    Giving Birth

    View Slide

  4. “Ecosystem with multiple platforms that enable the creation and deployment of
    automated transactional flows through chat. Sell and provide customer service
    automatically through chatbots in messaging apps.”
    TL;DR; chatbots for everyone, for everything.

    View Slide

  5. The road to Elixir
    1. Needed realtime communication for chat app.
    2. socket.io.
    3. The WhatsApp Architecture Facebook Bought For $19
    Billion (Erlang and ejabberd).
    4. Dive into Erlang: . Back to socket.io.
    5. Elixir: .
    6. Phoenix Channels: .
    7. OTP: .
    8. Now: 30k+ Elixir lines of code.

    View Slide

  6. (18 microservices)

    View Slide

  7. Fred Router
    Microservice transparent communication and discovery
    through custom Protocol and Dispatchers
    BEAM <> BEAM
    BEAM <> HTTP
    HTTP <> BEAM
    HTTP <> HTTP
    Context: BEAM – Elixir/Erlang apps communication. HTTP: External services (any language or 3rd party).

    View Slide

  8. Deployment and Hot Reload
    • TL; DR; .
    • Every project a set of problems and headaches.
    • Almost every new deployment a new set of problems
    and headaches, making CI and CD a .
    • Dealing with Umbrellas was even worse.
    Context: July/2016.

    View Slide

  9. Unified common demands
    Transformed Umbrellas into
    single projects
    Fred Router dropped
    (may open source soon)

    View Slide

  10. Message Processing Pipeline
    Message comes from external messaging services (Facebook
    Messenger, etc). (fire and forget, async)
    |> Converted to Fred (i.e. FB to Fred).
    |> Added local sequence according to external sequence.
    |> Processed by the Engine (bot) only one time in order, if out of
    order reschedule. Processing may take a long time, per message.
    |> Publish responses to services in order, if out of order reschedule.
    Since some bots deal with sensitive data (insurance, health, payments) no message can
    EVER BE LOST and ORDER MUST BE RESPECT. AT ALL COSTS. NO EXCEPTIONS.

    View Slide

  11. 1. GenStage + Flow
    2. Riak Core
    3. External Solutions *
    Riak Core ☑
    * Mantra: “BEAM gives you everything, avoid external solutions and tools”

    View Slide

  12. Riak Core

    View Slide

  13. Complexity
    Context: ~September/2016.
    Development was getting overwhelming.
    Young ecosystem with few to zero programmers for hire
    in Brazil.
    Costly and chaotic infrastructure and CI/CD pipeline
    (even after microservice reduction).

    View Slide

  14. Bunny
    Salvation

    View Slide

  15. Persistence EVERYWHERE
    Acknowledgments
    Requeuing, Rejection, Dead Lettering
    Once you learn the possibilities with
    AMQP and RabbitMQ there is no looking back.

    View Slide

  16. Our RabbitMQ Implementation
    1. Deal with connections: GenServer + Registry.
    2. Crashes when receiving messages: out of the box RabbitMQ
    handles this with ACKs.
    3. Crashes when publishing messages: every message is a supervised
    task + GenServer watcher + cold start from persistent store.
    4. High pressure: pool of GenServer workers, with many connections
    and channels (thousands).

    View Slide

  17. Still needed a fast,
    but persistent
    data store ——————>
    We also use ETS in lots of places, but
    unfortunately it is not persistent.
    Mnesia is , so a big no.

    View Slide

  18. 178 million
    messages/day
    in a 8GB server
    (+ RabbitMQ, Redis and PgSQL cluster)

    View Slide

  19. Final Elixir Thoughts
    OTP everywhere (many GenServers, Agents, Tasks). It seems dauting, but OTP is
    just a few concepts, so don’t be afraid.
    Don’t worry about writing code the “Elixir/functional way” in the beginning.
    You’ll learn and improve quickly while working with it.
    Don’t try to reinvent the wheel and do something in a certain way just because someone
    said it is the “BEAM way”. They aren’t building your product or doing your job.
    (Even if “someone” is an authority).
    No need to learn Erlang in the beginning (later you will end up learning it anyway and
    you will also end up reading the docs a lot).

    View Slide

  20. We are looking for Alchemists!
    www.fredbots.com
    THANK YOU!

    View Slide