Upgrade to Pro — share decks privately, control downloads, hide ads and more …

spotlight-python

Lynn Root
November 06, 2014

 spotlight-python

How Spotify uses Python - presented at Python Brazil 2014

Lynn Root

November 06, 2014
Tweet

More Decks by Lynn Root

Other Decks in Programming

Transcript

  1. November 3, 2014
    Spotlight
    Spotify & Python
    Lynn Root
    @roguelynn
    rogue.ly

    View full-size slide

  2. whoami
    Lynn Root
    Backend Engineer @ Spotify
    PyLadies of San Francisco Founder
    Board Member of the 

    Python Software Foundation

    View full-size slide

  3. Awesome
    About Spotify
    Why Python and How we use it
    Looking forward

    View full-size slide

  4. November 3, 2014
    About Spotify
    • Available in 58 countries
    • Launched in Brazil in May 2014
    • Over 10 million paying subscribers
    • Over 40 million monthly active users
    • Over 20 million songs
    • Paid over $1 billion to rights holders to date
    • Main engineering, analytics, data & machine learning
    offices located in Stockholm & New York.

    View full-size slide

  5. November 3, 2014
    About Spotify: In numbers
    • 14 TB of user & service-related log data per day
    • 170 TB of data generated in Hadoop each day
    • 900 node Hadoop cluster
    • 7,800 Hadoop jobs per day

    View full-size slide

  6. November 3, 2014
    About Spotify: Our tech
    • Python, Java, C++, Protobuf, ZeroMQ
    • Debian, Ubuntu, Puppet, Docker & Helios
    • Hadoop, Cassandra, Postgres
    • PHP & MySQL
    • git, Gerrit, GHE, JIRA

    View full-size slide

  7. November 3, 2014
    About Spotify: Tech culture
    • Agile methodology that works for each squad
    • Autonomous squads that are loosely coupled but
    tightly aligned
    • Squads, Tribes, Chapters, and Guilds.

    View full-size slide

  8. November 3, 2014
    About Python
    • 23 years old, latest releases: 2.7.8 and 3.4.2
    • Scripting, web dev, networking, scientific computing,
    database management, 2/3D animation, natural
    language processing, infosec, artificial intelligence
    (what else is there?)

    View full-size slide

  9. November 3, 2014
    These are some of my
    favorite things
    • SimpleHTTPServer, profile, pdb
    • IPython & notebook
    • Scapy, tox, pyenv, Twisted, Sphinx + RTDs

    View full-size slide

  10. November 3, 2014
    Who uses Python?
    • Google
    • Red Hat
    • Dropbox
    • Rackspace
    • Twitter
    • Facebook
    • Raspberry Pi
    • NASA
    • CERN
    • ITA
    • Yahoo!
    • Walt Disney
    • IBM
    • Reddit
    • YouTube

    View full-size slide

  11. We chose
    Python
    because…
    • Standard Library &
    Third-party packages
    • Community
    • Fast where it counts

    View full-size slide

  12. November 3, 2014
    Libraries & Packages
    • Python’s stdlib is very comprehensive
    • PyPI - over 51,000 packages

    View full-size slide

  13. November 3, 2014
    Community
    • Rich ecosystem of third-party packages
    • Documentation is king
    • “Zen” of Python widely adopted

    View full-size slide

  14. November 3, 2014
    Fast where it counts
    • Code reviews are fast
    • Writing less code
    • Startup time is instant
    • Quicker overall development cycle
    • We’re not CPU-bound, more I/O bound

    View full-size slide

  15. November 3, 2014
    Where Spotify uses Python
    • Systems:
    • Users: account creation and login
    • Social graph: (followers, suggestions, etc)
    • Payments system
    • All analytics jobs
    • Service daemon & request handler
    • Build scripts, internal cloud management
    • Internal PyPI
    • DNS & service discovery
    • Content management (data & songs supplied by labels)

    View full-size slide

  16. November 3, 2014
    What Spotify uses
    • Django, Pyramid, Flask
    • Gevent, Twisted, Celery
    • requests
    • SQLAlchemy
    • lxml
    • Kazoo
    • Sphinx, nose, python-pagerduty
    • dnspython

    View full-size slide

  17. How we’ve
    given back
    • Luigi & Snakebite
    • dh-virtualenv
    • gevent patches
    • Community diversity

    View full-size slide

  18. November 3, 2014
    Issues Spotify
    has faced with Python
    • A lot of home-grown solutions
    • No good HTTP client - every library is missing
    something
    • memcache client doesn’t work well with our gevent
    setup
    • Debian packaging for everything.

    View full-size slide

  19. November 3, 2014
    Issues Python face
    • The Great Debate: Python 2 versus 3
    • The GIL
    • Python on Windows

    View full-size slide

  20. November 3, 2014
    Looking Forward
    • Moving to Ubuntu - revisit Python 3
    • Java’s popularity internally is growing since difficult to
    hire Python developers
    • Wanting to incorporate more Python into our desktop
    client
    • Looking at IPython for data visualization and analysis

    View full-size slide

  21. November 3, 2014
    Thanks!
    Lynn Root
    [email protected]
    @roguelynn ⾇ rogue.ly
    Resources & Slides: rogue.ly/spotify-python

    View full-size slide