$30 off During Our Annual Pro Sale. View Details »

Individualised Content at Web-scale

Individualised Content at Web-scale

ElixirConf EU talk presented on Apr 24th in Krakow, Poland.

It's basically three parts:

1 Problem domain
2 Problems we encountered
3 How we solved them

Michael Schaefermeyer

April 24, 2015

Other Decks in Technology


  1. Individualised Content at Web-scale How Elixir and Phoenix helped us

    deliver individualised content to millions of users per month.
  2. Michael Schaefermeyer • ❤ Elixir, Ruby • Senior Software Engineer

    at Bleacher Report (until last month) • Co-Founder, Software Engineer at Full Stack Media (since last month) mschae mschae
  3. Disclaimers 1. This talk may contain traces of ruby 2.

    Not everything you are about to hear about is live yet 3. This talk is “full stack” - not very technical - I won’t show much code
  4. The product

  5. None
  6. • Merged stream of tweets, instagrams and articles • Desktop

    and native mobile apps • Streams for leagues, teams, division, … • In the mobile apps: User can subscribe to teams, leagues, players, etc.
  7. • Merged stream of tweets, instagrams and articles • Desktop

    and native mobile apps • Streams for leagues, teams, division, … • In the mobile apps: User can subscribe to teams, leagues, players, etc.
  8. Status quo • Lots of requests • Two monoliths (UI,

    API), multiple services • Ruby on Rails, node.js • Persistence layer consists of relational databases (MySQL, PostgreSQL) • Caches, caches everywhere (Memcached and Redis)
  9. Why Elixir? What about caching?

  10. –Phil Karlton “There are two hard things in Computer Science:

    cache invalidation and naming things.”
  11. “Brochure benefits” of Elixir • Scalability • Fault-tolerance • Functional

    • Meta-programming
  12. “Brochure benefits” of Elixir Source: flick.com / Lisa Yarost Not

    very specific
  13. Benefits of Elixir for our app • We are manipulating

    lists ➡ Functional Programming + (Linked) Lists = ❤ • We are merging multiple lists ➡ Fetch and build multiple lists in parallel • We have an extremely high dispersion (no two lists are the same) ➡ No reasonable caching possible • We potentially want to re-order list based on additional data ➡ “Just-in-time” list creation necessary
  14. Sequencial approach

  15. Parallel approach Extremely easy to implement thanks to Ecto’s Repo-

    based approach
  16. Ecto’s Repo-based approach

  17. Hurdles Source: flick.com / Robert Voors

  18. “Immaturity” of Elixir • Elixir has been stable for ~

    6 months • Few SaaS integrations (e.g. error tracking, …) • Not “proven” like ruby, node • No official Docker image • …
  19. Small ecosystem • Ruby = great ecosystem • We don’t

    want to replicate existing functionality • Fear of “having to build it all yourself” • Fear of “having to solve the complex problems” Source: githut.info
  20. Training developers • We are a Ruby shop = lack

    of functional programming experience • Recruiters: “We are hiring Elixir developers with at least 5 years of experience” • San Francisco is one hell of a market Source: hbo.com
  21. Monitoring • Crucial to keep track of the app’s health

    • No New Relic, no Skylight • No insights = flying blind • No metrics = hard to debug Source: newrelic.com
  22. Solutions Source: flickr.com / Shaun Fisher

  23. “Immaturity” of Elixir • Surprisingly mature tooling (mix, ExUnit, hex.pm,

    ExDoc, …) • Erlang has been proven! The language is as old as I am. • People have been using it in production for about a year now • Very extendable architecture (Plugs, Ecto datatypes, …) • Extremely smart people working on it
  24. Small ecosystem • Lots of Erlang packages to choose from

    • Great web framework with Phoenix, great ORM with Ecto • Great packages already exist • HTTPoison, Poison, ExVCR and many more • Fewer engineers but extremely smart people • Super helpful community
  25. Small ecosystem (cont.) • It’s very straightforward to add missing

    pieces: • Plug for “trailing formats”:
  26. Training new developers • Findings: • Similarity to ruby lowers

    perceived entry barrier • Excellent online resources available (elixir-lang.org, Elixir Sips, Dave Thomas’ book, …) • Languages like Elixir are a big plus for applicants! • Measures: • Live coding sessions • Start with easy tickets
  27. Lack of monitoring • Surprisingly simple to build your own

    monitoring • Use Exometer (and don’t forget to thank @bltroutwine) • Use StatsD (integrates with great SaaS tools like Datadog, Librato, etc.) • Integrate BEAM statistics (run queue, garbage collection, memory sizes, …) • Integrate server statistics (CPU usage, load, memory, available disk space, …)
  28. Lack of monitoring (cont.) • Stats I’m interested in: •

    DB queries per minute, execution times • Web requests per minute, response times • Health of the VM (memory stats, GC stats, …)
  29. Lack of monitoring (cont.)

  30. Ongoing challenges • Keeping everything up-to-date (lots of packages <

    1.0) - has gotten a lot easier thanks to mix hex.outdated • Recruiting more people to join the Elixir express • Richer SaaS-ecosystem needed • Ops challenges (file descriptors, nginx config, …)
  31. Gotchas • Nginx config (options like worker_processes, worker_rlimit_nofile, …) •

    Watch your file descriptors • Watch your log files • Don’t convert user input to Atom (use existing_atom) • Use OTP whenever possible • Beware of Erlang’s binary data collection
  32. Results • Performance: • Complex lists and multiple DB accesses

    ~ 50ms • Simple list: ~ 30ms • Quality: Very high test coverage, very maintainable code • Robustness: No runtime errors, no beam errors • Elixir perfect language for web applications ➡ Allowed us to build the product fast, secure and reliable
  33. Epilogue • I’m not very smart • I had the

    chance to get engaged in a super vibrant community very quickly • Use the abundance of help: • #elixir-lang • Mailing lists on Google Groups (Elixir, Ecto, Phoenix) • Best-in-class documentation and resources