De Python à Erlang

De Python à Erlang

Retour d'expérience de érome Renard (ingénieur chez Swelen) sur la ré-écriture complète de Python à Erlang d'un adserver par 2 personnes qui ne connaissaient rien à Erlang, pourquoi ils ont chois Erlang et le chemin qui nous a amené jusqu'à la mise en production finale.

Meetup 14/10/2015 @ leboncoin.fr

89261d2bbe7220dfde8ec1be0aa0f52f?s=128

erlang.paris

October 14, 2015
Tweet

Transcript

  1. From P..... to Erlang by Jérôme Renard (swiss army knife

    @ Swelen)
  2. What is Swelen ? Mobile advertising (iOS, Android, WebMobile) RTB

  3. The adserver team (2 people) https://github.com/jeromer/ https://github.com/teh-cmc/ no clue about

    Erlang no clue about functional programming
  4. Before the rewrite

  5. Adserver was written in P..... (+ MySQL, + Redis, +

    Nginx, + uwsgi, + ...) by a math genius (initial author, left the company) with piles of "temporary fixes" without any test (what could possibly go wrong ?)
  6. in the end a nigthmare to maintain costly to add

    new features (exponential)
  7. We could not keep up with the business

  8. None
  9. None
  10. New adserver requirements fast (< 50ms max to return an

    ad) scalable (10x traffic anytime) maintainable observable no HTTP caching synchronous reads asynchronous writes
  11. We had to pick a language Python (legacy adserver) PHP

    (hell no !) C (too complex to do well) Go (lots of interesting concepts) Rust (no stable version available) Erlang
  12. Why we chose Erlang designed for fault-tolerance built-in scalability built-in

    observability (observer, etop) no locks :) (huge win) competition used Erlang
  13. With Erlang, you not only have a language. You also

    get the entire platform for free !
  14. Time to learn LYSE (thank you @mononcqc) ton of manual

    pages (erldocs) lots of Erlang Factory videos Cowboy manual & source code Trial and (lots of) errors
  15. Erlang, a week later 1st real world feature ready sent

    the entire trafic on a single node worked well, scaled well
  16. WIN !

  17. Time to write the rest

  18. We ♥ ETS blazingly fast super simple to use manage

    10 different cache layers no longer need Redis
  19. We ♥ pattern matching extremely powerful code much safer, with

    much less error checks (let it crash) much easier to maintain
  20. We ♥ releases trivial deployements git push make release tar

    + scp restart the adserver done (we do no do hot upgrades)
  21. We ♥ observability (aka WTF is going on ?) remote

    shell on any node (shell history support ?) direct access to our API ♥ erl -name jerome -setcookie foo -hidden -remsh x.com -s observer
  22. Stupid mistakes learned the hard way 1/2 store BIG terms

    in ETS -> ETS performance avoid ets:select if possible -> full table scan -> ETS table is lost when process crashes -> create a dedicated supervisor
  23. Stupid mistakes learned the hard way 2/2 create an OTP

    app for anything -> message passing pass BIG terms between process -> message passing
  24. Summary

  25. We went from unmaitainable adserver (by 2 people) complex infrastructure

    high response time unable to keep up with the business
  26. To single release (aka tarball) response time is ~25ms (p90)

    can do trafic x8 without even sweating simplified infrastructure one maintainer (me) adding a feature takes 2/3 days in avg (w/ tests)
  27. Future plans switch to Mnesia use dialyzer

  28. Thoughts about Erlang Erlang not hard to learn, real motivation

    is required though Thinking in Erlang challenging sometimes Documentation complete, could be more accessible Unicode support could be more "natural"
  29. Thank you Ericsson & the Erlang community

  30. None
  31. Thank you jerome.renard@kwanko.com @jeromerenard http://github.com/jeromer/