Individualised Content at Web-scale

ElixirConf EU talk presented on Apr 24th in Krakow, Poland.

It's basically three parts:

1 Problem domain
2 Problems we encountered
3 How we solved them

Transcript

1. Individualised Content at Web-scale How Elixir and Phoenix helped us deliver individualised content to millions of users per month.
2. Michael Schaefermeyer • ❤ Elixir, Ruby • Senior Software Engineer at Bleacher Report (until last month) • Co-Founder, Software Engineer at Full Stack Media (since last month) mschae mschae
3. Disclaimers 1. This talk may contain traces of ruby 2. Not everything you are about to hear about is live yet 3. This talk is “full stack” - not very technical - I won’t show much code
4. The product
6. • Merged stream of tweets, instagrams and articles • Desktop and native mobile apps • Streams for leagues, teams, division, … • In the mobile apps: User can subscribe to teams, leagues, players, etc.
7. • Merged stream of tweets, instagrams and articles • Desktop and native mobile apps • Streams for leagues, teams, division, … • In the mobile apps: User can subscribe to teams, leagues, players, etc.
8. Status quo • Lots of requests • Two monoliths (UI, API), multiple services • Ruby on Rails, node.js • Persistence layer consists of relational databases (MySQL, PostgreSQL) • Caches, caches everywhere (Memcached and Redis)
9. Why Elixir? What about caching?
10. –Phil Karlton “There are two hard things in Computer Science: cache invalidation and naming things.”
11. “Brochure benefits” of Elixir • Scalability • Fault-tolerance • Functional • Meta-programming
12. “Brochure benefits” of Elixir Source: flick.com / Lisa Yarost Not very specific
13. Benefits of Elixir for our app • We are manipulating lists ➡ Functional Programming + (Linked) Lists = ❤ • We are merging multiple lists ➡ Fetch and build multiple lists in parallel • We have an extremely high dispersion (no two lists are the same) ➡ No reasonable caching possible • We potentially want to re-order list based on additional data ➡ “Just-in-time” list creation necessary
14. Sequencial approach
15. Parallel approach Extremely easy to implement thanks to Ecto’s Repo- based approach
16. Ecto’s Repo-based approach
17. Hurdles Source: flick.com / Robert Voors
18. “Immaturity” of Elixir • Elixir has been stable for ~ 6 months • Few SaaS integrations (e.g. error tracking, …) • Not “proven” like ruby, node • No official Docker image • …
19. Small ecosystem • Ruby = great ecosystem • We don’t want to replicate existing functionality • Fear of “having to build it all yourself” • Fear of “having to solve the complex problems” Source: githut.info
20. Training developers • We are a Ruby shop = lack of functional programming experience • Recruiters: “We are hiring Elixir developers with at least 5 years of experience” • San Francisco is one hell of a market Source: hbo.com
21. Monitoring • Crucial to keep track of the app’s health • No New Relic, no Skylight • No insights = flying blind • No metrics = hard to debug Source: newrelic.com
22. Solutions Source: flickr.com / Shaun Fisher
23. “Immaturity” of Elixir • Surprisingly mature tooling (mix, ExUnit, hex.pm, ExDoc, …) • Erlang has been proven! The language is as old as I am. • People have been using it in production for about a year now • Very extendable architecture (Plugs, Ecto datatypes, …) • Extremely smart people working on it
24. Small ecosystem • Lots of Erlang packages to choose from • Great web framework with Phoenix, great ORM with Ecto • Great packages already exist • HTTPoison, Poison, ExVCR and many more • Fewer engineers but extremely smart people • Super helpful community
25. Small ecosystem (cont.) • It’s very straightforward to add missing pieces: • Plug for “trailing formats”:
26. Training new developers • Findings: • Similarity to ruby lowers perceived entry barrier • Excellent online resources available (elixir-lang.org, Elixir Sips, Dave Thomas’ book, …) • Languages like Elixir are a big plus for applicants! • Measures: • Live coding sessions • Start with easy tickets
27. Lack of monitoring • Surprisingly simple to build your own monitoring • Use Exometer (and don’t forget to thank @bltroutwine) • Use StatsD (integrates with great SaaS tools like Datadog, Librato, etc.) • Integrate BEAM statistics (run queue, garbage collection, memory sizes, …) • Integrate server statistics (CPU usage, load, memory, available disk space, …)
28. Lack of monitoring (cont.) • Stats I’m interested in: • DB queries per minute, execution times • Web requests per minute, response times • Health of the VM (memory stats, GC stats, …)
29. Lack of monitoring (cont.)
30. Ongoing challenges • Keeping everything up-to-date (lots of packages < 1.0) - has gotten a lot easier thanks to mix hex.outdated • Recruiting more people to join the Elixir express • Richer SaaS-ecosystem needed • Ops challenges (file descriptors, nginx config, …)
31. Gotchas • Nginx config (options like worker_processes, worker_rlimit_nofile, …) • Watch your file descriptors • Watch your log files • Don’t convert user input to Atom (use existing_atom) • Use OTP whenever possible • Beware of Erlang’s binary data collection
32. Results • Performance: • Complex lists and multiple DB accesses ~ 50ms • Simple list: ~ 30ms • Quality: Very high test coverage, very maintainable code • Robustness: No runtime errors, no beam errors • Elixir perfect language for web applications ➡ Allowed us to build the product fast, secure and reliable
33. Epilogue • I’m not very smart • I had the chance to get engaged in a super vibrant community very quickly • Use the abundance of help: • #elixir-lang • Mailing lists on Google Groups (Elixir, Ecto, Phoenix) • Best-in-class documentation and resources