Individualised Content at Web-scale How Elixir and Phoenix helped us deliver individualised content to millions of users per month. 

Michael Schaefermeyer • ❤ Elixir, Ruby • Senior Software Engineer at Bleacher Report (until last month) • Co-Founder, Software Engineer at Full Stack Media (since last month) mschae mschae 

Disclaimers 1. This talk may contain traces of ruby 2. Not everything you are about to hear about is live yet 3. This talk is “full stack” - not very technical - I won’t show much code 

The product 

No content

• Merged stream of tweets, instagrams and articles • Desktop and native mobile apps • Streams for leagues, teams, division, … • In the mobile apps: User can subscribe to teams, leagues, players, etc. 

• Merged stream of tweets, instagrams and articles • Desktop and native mobile apps • Streams for leagues, teams, division, … • In the mobile apps: User can subscribe to teams, leagues, players, etc. 

Status quo • Lots of requests • Two monoliths (UI, API), multiple services • Ruby on Rails, node.js • Persistence layer consists of relational databases (MySQL, PostgreSQL) • Caches, caches everywhere (Memcached and Redis) 

Why Elixir? What about caching? 

–Phil Karlton “There are two hard things in Computer Science: cache invalidation and naming things.” 

“Brochure benefits” of Elixir • Scalability • Fault-tolerance • Functional • Meta-programming 

“Brochure benefits” of Elixir Source: flick.com / Lisa Yarost Not very specific 

Benefits of Elixir for our app • We are manipulating lists ➡ Functional Programming + (Linked) Lists = ❤ • We are merging multiple lists ➡ Fetch and build multiple lists in parallel • We have an extremely high dispersion (no two lists are the same) ➡ No reasonable caching possible • We potentially want to re-order list based on additional data ➡ “Just-in-time” list creation necessary 

Sequencial approach 

Parallel approach Extremely easy to implement thanks to Ecto’s Repo- based approach 

Ecto’s Repo-based approach 

Hurdles Source: flick.com / Robert Voors 

“Immaturity” of Elixir • Elixir has been stable for ~ 6 months • Few SaaS integrations (e.g. error tracking, …) • Not “proven” like ruby, node • No official Docker image • … 

Small ecosystem • Ruby = great ecosystem • We don’t want to replicate existing functionality • Fear of “having to build it all yourself” • Fear of “having to solve the complex problems” Source: githut.info 

Training developers • We are a Ruby shop = lack of functional programming experience • Recruiters: “We are hiring Elixir developers with at least 5 years of experience” • San Francisco is one hell of a market Source: hbo.com 

Monitoring • Crucial to keep track of the app’s health • No New Relic, no Skylight • No insights = flying blind • No metrics = hard to debug Source: newrelic.com 

Solutions Source: flickr.com / Shaun Fisher 

“Immaturity” of Elixir • Surprisingly mature tooling (mix, ExUnit, hex.pm, ExDoc, …) • Erlang has been proven! The language is as old as I am. • People have been using it in production for about a year now • Very extendable architecture (Plugs, Ecto datatypes, …) • Extremely smart people working on it 

Small ecosystem • Lots of Erlang packages to choose from • Great web framework with Phoenix, great ORM with Ecto • Great packages already exist • HTTPoison, Poison, ExVCR and many more • Fewer engineers but extremely smart people • Super helpful community 

Small ecosystem (cont.) • It’s very straightforward to add missing pieces: • Plug for “trailing formats”: 

Training new developers • Findings: • Similarity to ruby lowers perceived entry barrier • Excellent online resources available (elixir-lang.org, Elixir Sips, Dave Thomas’ book, …) • Languages like Elixir are a big plus for applicants! • Measures: • Live coding sessions • Start with easy tickets 

Lack of monitoring • Surprisingly simple to build your own monitoring • Use Exometer (and don’t forget to thank @bltroutwine) • Use StatsD (integrates with great SaaS tools like Datadog, Librato, etc.) • Integrate BEAM statistics (run queue, garbage collection, memory sizes, …) • Integrate server statistics (CPU usage, load, memory, available disk space, …) 

Lack of monitoring (cont.) • Stats I’m interested in: • DB queries per minute, execution times • Web requests per minute, response times • Health of the VM (memory stats, GC stats, …) 

Lack of monitoring (cont.) 

Ongoing challenges • Keeping everything up-to-date (lots of packages < 1.0) - has gotten a lot easier thanks to mix hex.outdated • Recruiting more people to join the Elixir express • Richer SaaS-ecosystem needed • Ops challenges (file descriptors, nginx config, …) 

Gotchas • Nginx config (options like worker_processes, worker_rlimit_nofile, …) • Watch your file descriptors • Watch your log files • Don’t convert user input to Atom (use existing_atom) • Use OTP whenever possible • Beware of Erlang’s binary data collection 

Results • Performance: • Complex lists and multiple DB accesses ~ 50ms • Simple list: ~ 30ms • Quality: Very high test coverage, very maintainable code • Robustness: No runtime errors, no beam errors • Elixir perfect language for web applications ➡ Allowed us to build the product fast, secure and reliable 

Epilogue • I’m not very smart • I had the chance to get engaged in a super vibrant community very quickly • Use the abundance of help: • #elixir-lang • Mailing lists on Google Groups (Elixir, Ecto, Phoenix) • Best-in-class documentation and resources