spotlight-python

How Spotify uses Python - presented at Python Brazil 2014

Transcript

1. November 3, 2014 Spotlight Spotify & Python Lynn Root @roguelynn rogue.ly
2. whoami Lynn Root Backend Engineer @ Spotify PyLadies of San Francisco Founder Board Member of the 
 Python Software Foundation
3. Spotify
4. Python
5. Awesome
6. Awesome About Spotify Why Python and How we use it Looking forward
7. Spotify
8. November 3, 2014 About Spotify • Available in 58 countries • Launched in Brazil in May 2014 • Over 10 million paying subscribers • Over 40 million monthly active users • Over 20 million songs • Paid over $1 billion to rights holders to date • Main engineering, analytics, data & machine learning offices located in Stockholm & New York.
9. November 3, 2014 About Spotify: In numbers • 14 TB of user & service-related log data per day • 170 TB of data generated in Hadoop each day • 900 node Hadoop cluster • 7,800 Hadoop jobs per day
10. November 3, 2014 About Spotify: Our tech • Python, Java, C++, Protobuf, ZeroMQ • Debian, Ubuntu, Puppet, Docker & Helios • Hadoop, Cassandra, Postgres • PHP & MySQL • git, Gerrit, GHE, JIRA
11. November 3, 2014 About Spotify: Tech culture • Agile methodology that works for each squad • Autonomous squads that are loosely coupled but tightly aligned • Squads, Tribes, Chapters, and Guilds.
12. Python
13. November 3, 2014 About Python • 23 years old, latest releases: 2.7.8 and 3.4.2 • Scripting, web dev, networking, scientific computing, database management, 2/3D animation, natural language processing, infosec, artificial intelligence (what else is there?)
14. November 3, 2014 These are some of my favorite things • SimpleHTTPServer, profile, pdb • IPython & notebook • Scapy, tox, pyenv, Twisted, Sphinx + RTDs
15. November 3, 2014 Who uses Python? • Google • Red Hat • Dropbox • Rackspace • Twitter • Facebook • Raspberry Pi • NASA • CERN • ITA • Yahoo! • Walt Disney • IBM • Reddit • YouTube
16. Awesome
17. We chose Python because… • Standard Library & Third-party packages • Community • Fast where it counts
18. November 3, 2014 Libraries & Packages • Python’s stdlib is very comprehensive • PyPI - over 51,000 packages
19. November 3, 2014 Community • Rich ecosystem of third-party packages • Documentation is king • “Zen” of Python widely adopted
20. November 3, 2014 Fast where it counts • Code reviews are fast • Writing less code • Startup time is instant • Quicker overall development cycle • We’re not CPU-bound, more I/O bound
21. November 3, 2014 Where Spotify uses Python • Systems: • Users: account creation and login • Social graph: (followers, suggestions, etc) • Payments system • All analytics jobs • Service daemon & request handler • Build scripts, internal cloud management • Internal PyPI • DNS & service discovery • Content management (data & songs supplied by labels)
22. November 3, 2014 What Spotify uses • Django, Pyramid, Flask • Gevent, Twisted, Celery • requests • SQLAlchemy • lxml • Kazoo • Sphinx, nose, python-pagerduty • dnspython
23. How we’ve given back • Luigi & Snakebite • dh-virtualenv • gevent patches • Community diversity
24. November 3, 2014 Issues Spotify has faced with Python • A lot of home-grown solutions • No good HTTP client - every library is missing something • memcache client doesn’t work well with our gevent setup • Debian packaging for everything.
25. November 3, 2014 Issues Python face • The Great Debate: Python 2 versus 3 • The GIL • Python on Windows
26. November 3, 2014 Looking Forward • Moving to Ubuntu - revisit Python 3 • Java’s popularity internally is growing since difficult to hire Python developers • Wanting to incorporate more Python into our desktop client • Looking at IPython for data visualization and analysis
27. November 3, 2014 Thanks! Lynn Root lynn@spotify.com @roguelynn ⾇ rogue.ly Resources & Slides: rogue.ly/spotify-python