whoami
Lynn Root
Backend Engineer @ Spotify
PyLadies of San Francisco Founder
Board Member of the
Python Software Foundation
Slide 3
Slide 3 text
Spotify
Slide 4
Slide 4 text
Python
Slide 5
Slide 5 text
Awesome
Slide 6
Slide 6 text
Awesome
About Spotify
Why Python and How we use it
Looking forward
Slide 7
Slide 7 text
Spotify
Slide 8
Slide 8 text
November 3, 2014
About Spotify
• Available in 58 countries
• Launched in Brazil in May 2014
• Over 10 million paying subscribers
• Over 40 million monthly active users
• Over 20 million songs
• Paid over $1 billion to rights holders to date
• Main engineering, analytics, data & machine learning
offices located in Stockholm & New York.
Slide 9
Slide 9 text
November 3, 2014
About Spotify: In numbers
• 14 TB of user & service-related log data per day
• 170 TB of data generated in Hadoop each day
• 900 node Hadoop cluster
• 7,800 Hadoop jobs per day
November 3, 2014
About Spotify: Tech culture
• Agile methodology that works for each squad
• Autonomous squads that are loosely coupled but
tightly aligned
• Squads, Tribes, Chapters, and Guilds.
Slide 12
Slide 12 text
Python
Slide 13
Slide 13 text
November 3, 2014
About Python
• 23 years old, latest releases: 2.7.8 and 3.4.2
• Scripting, web dev, networking, scientific computing,
database management, 2/3D animation, natural
language processing, infosec, artificial intelligence
(what else is there?)
Slide 14
Slide 14 text
November 3, 2014
These are some of my
favorite things
• SimpleHTTPServer, profile, pdb
• IPython & notebook
• Scapy, tox, pyenv, Twisted, Sphinx + RTDs
Slide 15
Slide 15 text
November 3, 2014
Who uses Python?
• Google
• Red Hat
• Dropbox
• Rackspace
• Twitter
• Facebook
• Raspberry Pi
• NASA
• CERN
• ITA
• Yahoo!
• Walt Disney
• IBM
• Reddit
• YouTube
Slide 16
Slide 16 text
Awesome
Slide 17
Slide 17 text
We chose
Python
because…
• Standard Library &
Third-party packages
• Community
• Fast where it counts
Slide 18
Slide 18 text
November 3, 2014
Libraries & Packages
• Python’s stdlib is very comprehensive
• PyPI - over 51,000 packages
Slide 19
Slide 19 text
November 3, 2014
Community
• Rich ecosystem of third-party packages
• Documentation is king
• “Zen” of Python widely adopted
Slide 20
Slide 20 text
November 3, 2014
Fast where it counts
• Code reviews are fast
• Writing less code
• Startup time is instant
• Quicker overall development cycle
• We’re not CPU-bound, more I/O bound
Slide 21
Slide 21 text
November 3, 2014
Where Spotify uses Python
• Systems:
• Users: account creation and login
• Social graph: (followers, suggestions, etc)
• Payments system
• All analytics jobs
• Service daemon & request handler
• Build scripts, internal cloud management
• Internal PyPI
• DNS & service discovery
• Content management (data & songs supplied by labels)
How we’ve
given back
• Luigi & Snakebite
• dh-virtualenv
• gevent patches
• Community diversity
Slide 24
Slide 24 text
November 3, 2014
Issues Spotify
has faced with Python
• A lot of home-grown solutions
• No good HTTP client - every library is missing
something
• memcache client doesn’t work well with our gevent
setup
• Debian packaging for everything.
Slide 25
Slide 25 text
November 3, 2014
Issues Python face
• The Great Debate: Python 2 versus 3
• The GIL
• Python on Windows
Slide 26
Slide 26 text
November 3, 2014
Looking Forward
• Moving to Ubuntu - revisit Python 3
• Java’s popularity internally is growing since difficult to
hire Python developers
• Wanting to incorporate more Python into our desktop
client
• Looking at IPython for data visualization and analysis