To Hell and Back: Why choosing Elixir to build a startup from scratch was both a terrible and an awesome decision

Building a chatbot and an e-commerce startup (www.fredbots.com) with 30k+ lines of Elixir and Phoenix code and almost redoing from scratch a few times.

Microservices and distribution done wrong, getting saved by a small bunny, reaching 178 million messages/day in a 8GB machine.

Slides from my Elixir User Group SP presentation on 18/Mar/17 - http://meetup.com/pt-BR/elug_sp/.

Transcript

1. To Hell and Back
   Why choosing Elixir to build a startup
   from scratch was both a terrible and an
   awesome decision
   
   

   View Slide

2. Who
   Co-founder and CEO at Fred (www.fredbots.com).
   Self-taught developer since 8. Used longest: PHP and Unity3D w/ C#.
   Almost a Computer Engineer, a Lawyer, a Veterinarian and a
   Computer Engineer again, but dropped out.
   Ex-snake handler and fishing tackle sales clerk. Founded and closed
   2 other startups (and many others still archived).
   Alfred Reinold Baudisch
   @AlfredBaudisch / alfred@fredbots.com
   

   View Slide

3. How a WhatsApp bot got famous and evolved as the Brazilian WeChat and conversational commerce platform (goo.gl/v0qlJM)
   Food Delivery WhatsApp bot
   with unofficial API
   Exploded on the news Banned by WhatsApp
   FredApp, “WhatsApp for
   Business”, standalone chat +
   bot platform
   Angel investment
   Giving Birth
   

   View Slide

4. “Ecosystem with multiple platforms that enable the creation and deployment of
   automated transactional flows through chat. Sell and provide customer service
   automatically through chatbots in messaging apps.”
   TL;DR; chatbots for everyone, for everything.
   

   View Slide

5. The road to Elixir
   1. Needed realtime communication for chat app.
   2. socket.io.
   3. The WhatsApp Architecture Facebook Bought For $19
   Billion (Erlang and ejabberd).
   4. Dive into Erlang: . Back to socket.io.
   5. Elixir: .
   6. Phoenix Channels: .
   7. OTP: .
   8. Now: 30k+ Elixir lines of code.
   

   View Slide

6. (18 microservices)
   

   View Slide

7. Fred Router
   Microservice transparent communication and discovery
   through custom Protocol and Dispatchers
   BEAM <> BEAM
   BEAM <> HTTP
   HTTP <> BEAM
   HTTP <> HTTP
   Context: BEAM – Elixir/Erlang apps communication. HTTP: External services (any language or 3rd party).
   

   View Slide

8. Deployment and Hot Reload
   • TL; DR; .
   • Every project a set of problems and headaches.
   • Almost every new deployment a new set of problems
   and headaches, making CI and CD a .
   • Dealing with Umbrellas was even worse.
   Context: July/2016.
   

   View Slide

9. Unified common demands
   Transformed Umbrellas into
   single projects
   Fred Router dropped
   (may open source soon)
   

   View Slide

10. Message Processing Pipeline
    Message comes from external messaging services (Facebook
    Messenger, etc). (fire and forget, async)
    |> Converted to Fred (i.e. FB to Fred).
    |> Added local sequence according to external sequence.
    |> Processed by the Engine (bot) only one time in order, if out of
    order reschedule. Processing may take a long time, per message.
    |> Publish responses to services in order, if out of order reschedule.
    Since some bots deal with sensitive data (insurance, health, payments) no message can
    EVER BE LOST and ORDER MUST BE RESPECT. AT ALL COSTS. NO EXCEPTIONS.
    

    View Slide

11. 1. GenStage + Flow
    2. Riak Core
    3. External Solutions *
    Riak Core ☑
    * Mantra: “BEAM gives you everything, avoid external solutions and tools”
    

    View Slide

12. Riak Core
    

    View Slide

13. Complexity
    Context: ~September/2016.
    Development was getting overwhelming.
    Young ecosystem with few to zero programmers for hire
    in Brazil.
    Costly and chaotic infrastructure and CI/CD pipeline
    (even after microservice reduction).
    

    View Slide

14. Bunny
    Salvation
    

    View Slide

15. Persistence EVERYWHERE
    Acknowledgments
    Requeuing, Rejection, Dead Lettering
    Once you learn the possibilities with
    AMQP and RabbitMQ there is no looking back.
    

    View Slide

16. Our RabbitMQ Implementation
    1. Deal with connections: GenServer + Registry.
    2. Crashes when receiving messages: out of the box RabbitMQ
    handles this with ACKs.
    3. Crashes when publishing messages: every message is a supervised
    task + GenServer watcher + cold start from persistent store.
    4. High pressure: pool of GenServer workers, with many connections
    and channels (thousands).
    

    View Slide

17. Still needed a fast,
    but persistent
    data store ——————>
    We also use ETS in lots of places, but
    unfortunately it is not persistent.
    Mnesia is , so a big no.
    

    View Slide

18. 178 million
    messages/day
    in a 8GB server
    (+ RabbitMQ, Redis and PgSQL cluster)
    

    View Slide

19. Final Elixir Thoughts
    OTP everywhere (many GenServers, Agents, Tasks). It seems dauting, but OTP is
    just a few concepts, so don’t be afraid.
    Don’t worry about writing code the “Elixir/functional way” in the beginning.
    You’ll learn and improve quickly while working with it.
    Don’t try to reinvent the wheel and do something in a certain way just because someone
    said it is the “BEAM way”. They aren’t building your product or doing your job.
    (Even if “someone” is an authority).
    No need to learn Erlang in the beginning (later you will end up learning it anyway and
    you will also end up reading the docs a lot).
    

    View Slide

20. We are looking for Alchemists!
    www.fredbots.com
    THANK YOU!
    

    View Slide