$30 off During Our Annual Pro Sale. View Details »

Individualised Content at Web-scale

Individualised Content at Web-scale

ElixirConf EU talk presented on Apr 24th in Krakow, Poland.

It's basically three parts:

1 Problem domain
2 Problems we encountered
3 How we solved them

Michael Schaefermeyer

April 24, 2015
Tweet

Other Decks in Technology

Transcript

  1. Individualised Content at
    Web-scale
    How Elixir and Phoenix helped us deliver individualised content to millions of
    users per month.

    View Slide

  2. Michael
    Schaefermeyer
    • ❤ Elixir, Ruby
    • Senior Software Engineer at
    Bleacher Report (until last month)
    • Co-Founder, Software Engineer at
    Full Stack Media (since last month)
    mschae
    mschae

    View Slide

  3. Disclaimers
    1. This talk may contain traces of ruby
    2. Not everything you are about to hear about is live yet
    3. This talk is “full stack” - not very technical - I won’t
    show much code

    View Slide

  4. The product

    View Slide

  5. View Slide

  6. • Merged stream of tweets,
    instagrams and articles
    • Desktop and native mobile
    apps
    • Streams for leagues, teams,
    division, …
    • In the mobile apps: User can
    subscribe to teams, leagues,
    players, etc.

    View Slide

  7. • Merged stream of tweets,
    instagrams and articles
    • Desktop and native mobile
    apps
    • Streams for leagues, teams,
    division, …
    • In the mobile apps: User can
    subscribe to teams, leagues,
    players, etc.

    View Slide

  8. Status quo
    • Lots of requests
    • Two monoliths (UI, API), multiple services
    • Ruby on Rails, node.js
    • Persistence layer consists of relational databases
    (MySQL, PostgreSQL)
    • Caches, caches everywhere (Memcached and Redis)

    View Slide

  9. Why Elixir?
    What about caching?

    View Slide

  10. –Phil Karlton
    “There are two hard things in Computer
    Science: cache invalidation and naming
    things.”

    View Slide

  11. “Brochure benefits” of Elixir
    • Scalability
    • Fault-tolerance
    • Functional
    • Meta-programming

    View Slide

  12. “Brochure benefits” of Elixir
    Source: flick.com / Lisa Yarost
    Not very specific

    View Slide

  13. Benefits of Elixir for our app
    • We are manipulating lists
    ➡ Functional Programming + (Linked) Lists = ❤
    • We are merging multiple lists
    ➡ Fetch and build multiple lists in parallel
    • We have an extremely high dispersion (no two lists are the
    same)
    ➡ No reasonable caching possible
    • We potentially want to re-order list based on additional data
    ➡ “Just-in-time” list creation necessary

    View Slide

  14. Sequencial approach

    View Slide

  15. Parallel approach
    Extremely easy to implement thanks to Ecto’s Repo-
    based approach

    View Slide

  16. Ecto’s Repo-based approach

    View Slide

  17. Hurdles
    Source: flick.com / Robert Voors

    View Slide

  18. “Immaturity” of Elixir
    • Elixir has been stable for ~ 6 months
    • Few SaaS integrations (e.g. error tracking, …)
    • Not “proven” like ruby, node
    • No official Docker image
    • …

    View Slide

  19. Small ecosystem
    • Ruby = great ecosystem
    • We don’t want to replicate
    existing functionality
    • Fear of “having to build it all
    yourself”
    • Fear of “having to solve the
    complex problems”
    Source: githut.info

    View Slide

  20. Training developers
    • We are a Ruby shop = lack of
    functional programming
    experience
    • Recruiters: “We are hiring
    Elixir developers with at least
    5 years of experience”
    • San Francisco is one hell of a
    market
    Source: hbo.com

    View Slide

  21. Monitoring
    • Crucial to keep track of the
    app’s health
    • No New Relic, no Skylight
    • No insights = flying blind
    • No metrics = hard to debug
    Source: newrelic.com

    View Slide

  22. Solutions
    Source: flickr.com / Shaun Fisher

    View Slide

  23. “Immaturity” of Elixir
    • Surprisingly mature tooling (mix, ExUnit, hex.pm,
    ExDoc, …)
    • Erlang has been proven! The language is as old as I am.
    • People have been using it in production for about a year
    now
    • Very extendable architecture (Plugs, Ecto datatypes, …)
    • Extremely smart people working on it

    View Slide

  24. Small ecosystem
    • Lots of Erlang packages to choose from
    • Great web framework with Phoenix, great ORM with Ecto
    • Great packages already exist
    • HTTPoison, Poison, ExVCR and many more
    • Fewer engineers but extremely smart people
    • Super helpful community

    View Slide

  25. Small ecosystem (cont.)
    • It’s very straightforward to add missing pieces:
    • Plug for “trailing formats”:

    View Slide

  26. Training new developers
    • Findings:
    • Similarity to ruby lowers perceived entry barrier
    • Excellent online resources available (elixir-lang.org, Elixir Sips,
    Dave Thomas’ book, …)
    • Languages like Elixir are a big plus for applicants!
    • Measures:
    • Live coding sessions
    • Start with easy tickets

    View Slide

  27. Lack of monitoring
    • Surprisingly simple to build your own monitoring
    • Use Exometer (and don’t forget to thank @bltroutwine)
    • Use StatsD (integrates with great SaaS tools like
    Datadog, Librato, etc.)
    • Integrate BEAM statistics (run queue, garbage
    collection, memory sizes, …)
    • Integrate server statistics (CPU usage, load, memory,
    available disk space, …)

    View Slide

  28. Lack of monitoring (cont.)
    • Stats I’m interested in:
    • DB queries per minute, execution times
    • Web requests per minute, response times
    • Health of the VM (memory stats, GC stats, …)

    View Slide

  29. Lack of monitoring (cont.)

    View Slide

  30. Ongoing challenges
    • Keeping everything up-to-date (lots of packages < 1.0)
    - has gotten a lot easier thanks to mix hex.outdated
    • Recruiting more people to join the Elixir express
    • Richer SaaS-ecosystem needed
    • Ops challenges (file descriptors, nginx config, …)

    View Slide

  31. Gotchas
    • Nginx config (options like worker_processes,
    worker_rlimit_nofile, …)
    • Watch your file descriptors
    • Watch your log files
    • Don’t convert user input to Atom (use existing_atom)
    • Use OTP whenever possible
    • Beware of Erlang’s binary data collection

    View Slide

  32. Results
    • Performance:
    • Complex lists and multiple DB accesses ~ 50ms
    • Simple list: ~ 30ms
    • Quality: Very high test coverage, very maintainable code
    • Robustness: No runtime errors, no beam errors
    • Elixir perfect language for web applications
    ➡ Allowed us to build the product fast, secure and reliable

    View Slide

  33. Epilogue
    • I’m not very smart
    • I had the chance to get engaged in a super vibrant
    community very quickly
    • Use the abundance of help:
    • #elixir-lang
    • Mailing lists on Google Groups (Elixir, Ecto, Phoenix)
    • Best-in-class documentation and resources

    View Slide