Upgrade to Pro — share decks privately, control downloads, hide ads and more …

spotlight-python

8c5e76dca74a59822dbf7f0286177ddd?s=47 Lynn Root
November 06, 2014

 spotlight-python

How Spotify uses Python - presented at Python Brazil 2014

8c5e76dca74a59822dbf7f0286177ddd?s=128

Lynn Root

November 06, 2014
Tweet

Transcript

  1. November 3, 2014 Spotlight Spotify & Python Lynn Root @roguelynn

    rogue.ly
  2. whoami Lynn Root Backend Engineer @ Spotify PyLadies of San

    Francisco Founder Board Member of the 
 Python Software Foundation
  3. Spotify

  4. Python

  5. Awesome

  6. Awesome About Spotify Why Python and How we use it

    Looking forward
  7. Spotify

  8. November 3, 2014 About Spotify • Available in 58 countries

    • Launched in Brazil in May 2014 • Over 10 million paying subscribers • Over 40 million monthly active users • Over 20 million songs • Paid over $1 billion to rights holders to date • Main engineering, analytics, data & machine learning offices located in Stockholm & New York.
  9. November 3, 2014 About Spotify: In numbers • 14 TB

    of user & service-related log data per day • 170 TB of data generated in Hadoop each day • 900 node Hadoop cluster • 7,800 Hadoop jobs per day
  10. November 3, 2014 About Spotify: Our tech • Python, Java,

    C++, Protobuf, ZeroMQ • Debian, Ubuntu, Puppet, Docker & Helios • Hadoop, Cassandra, Postgres • PHP & MySQL • git, Gerrit, GHE, JIRA
  11. November 3, 2014 About Spotify: Tech culture • Agile methodology

    that works for each squad • Autonomous squads that are loosely coupled but tightly aligned • Squads, Tribes, Chapters, and Guilds.
  12. Python

  13. November 3, 2014 About Python • 23 years old, latest

    releases: 2.7.8 and 3.4.2 • Scripting, web dev, networking, scientific computing, database management, 2/3D animation, natural language processing, infosec, artificial intelligence (what else is there?)
  14. November 3, 2014 These are some of my favorite things

    • SimpleHTTPServer, profile, pdb • IPython & notebook • Scapy, tox, pyenv, Twisted, Sphinx + RTDs
  15. November 3, 2014 Who uses Python? • Google • Red

    Hat • Dropbox • Rackspace • Twitter • Facebook • Raspberry Pi • NASA • CERN • ITA • Yahoo! • Walt Disney • IBM • Reddit • YouTube
  16. Awesome

  17. We chose Python because… • Standard Library & Third-party packages

    • Community • Fast where it counts
  18. November 3, 2014 Libraries & Packages • Python’s stdlib is

    very comprehensive • PyPI - over 51,000 packages
  19. November 3, 2014 Community • Rich ecosystem of third-party packages

    • Documentation is king • “Zen” of Python widely adopted
  20. November 3, 2014 Fast where it counts • Code reviews

    are fast • Writing less code • Startup time is instant • Quicker overall development cycle • We’re not CPU-bound, more I/O bound
  21. November 3, 2014 Where Spotify uses Python • Systems: •

    Users: account creation and login • Social graph: (followers, suggestions, etc) • Payments system • All analytics jobs • Service daemon & request handler • Build scripts, internal cloud management • Internal PyPI • DNS & service discovery • Content management (data & songs supplied by labels)
  22. November 3, 2014 What Spotify uses • Django, Pyramid, Flask

    • Gevent, Twisted, Celery • requests • SQLAlchemy • lxml • Kazoo • Sphinx, nose, python-pagerduty • dnspython
  23. How we’ve given back • Luigi & Snakebite • dh-virtualenv

    • gevent patches • Community diversity
  24. November 3, 2014 Issues Spotify has faced with Python •

    A lot of home-grown solutions • No good HTTP client - every library is missing something • memcache client doesn’t work well with our gevent setup • Debian packaging for everything.
  25. November 3, 2014 Issues Python face • The Great Debate:

    Python 2 versus 3 • The GIL • Python on Windows
  26. November 3, 2014 Looking Forward • Moving to Ubuntu -

    revisit Python 3 • Java’s popularity internally is growing since difficult to hire Python developers • Wanting to incorporate more Python into our desktop client • Looking at IPython for data visualization and analysis
  27. November 3, 2014 Thanks! Lynn Root lynn@spotify.com @roguelynn ⾇ rogue.ly

    Resources & Slides: rogue.ly/spotify-python