Spotify's Love/Hate Relationship with DNS

SRECon Americas 2017, video recording: https://www.usenix.org/conference/srecon17americas and accompanied write-up: labs.spotify.com

Transcript

1. Spotify’s Love/Hate
   Relationship with DNS
   Lynn Root | SRE | @roguelynn
   

   View Slide

2. $ whoami
   

   View Slide

3. why we love DNS
   • It’s boring
   • Stable query language
   • Free caching
   • Service discovery
   —
   

   View Slide

4. agenda
   —
   

   View Slide

5. agenda
   • Our infrastructure
   • Our DNS curiosities
   • What we’ve learned
   • Future of DNS @ Spotify
   —
   

   View Slide

6. Our Infrastructure
   —
   

   View Slide

7. DNS@Spotify
   —
   

   View Slide

8. DNS@Spotify
   —
   

   View Slide

9. DNS@Spotify
   —
   

   View Slide

10. DNS@Spotify
    —
    

    View Slide

11. DNS@Spotify
    —
    

    View Slide

12. DNS@Spotify
    —
    

    View Slide

13. DNS@Spotify
    —
    

    View Slide

14. Record Generation &
    Deployment
    —
    

    View Slide

15. /msg #sre DNS DEPLOY!
    DNS@Spotify
    —
    

    View Slide

16. DNS@Spotify
    —
    

    View Slide

17. DNS@Spotify
    —
    

    View Slide

18. DNS@Spotify
    —
    

    View Slide

19. DNS@Spotify
    —
    

    View Slide

20. DNS@Spotify
    —
    

    View Slide

21. DNS@Spotify
    —
    

    View Slide

22. DNS@Spotify
    —
    

    View Slide

23. Service Discovery
    —
    

    View Slide

24. DNS@Spotify
    —
    

    View Slide

25. DNS@Spotify
    —
    Nameless
    services.spotify.net
    

    View Slide

26. Monitoring
    —
    

    View Slide

27. DNS@Spotify
    —
    

    View Slide

28. DNS@Spotify
    —
    

    View Slide

29. DNS@Spotify
    —
    

    View Slide

30. Global Server Load
    Balancing
    —
    

    View Slide

31. Responding to the
    DynDNS attack
    —
    

    View Slide

32. Response to DynDNS attack
    • Monitoring dashboards & VPN were inaccessible
    • Internal SSO login inaccessible
    • Pagerduty also affected
    —
    

    View Slide

33. Response to DynDNS attack
    • Couldn’t easily access DNS data repo
    • 3-year-old manual deployment documentation
    —
    

    View Slide

34. Response to DynDNS attack
    • Internal services ➡ removed GSLB mapping
    • Spotify clients ➡ Route53
    • Websites ➡ Route53
    —
    

    View Slide

35. Our DNS Curiosities
    —
    

    View Slide

36. Client Error Reporting
    —
    

    View Slide

37. DNS@Spotify
    —
    

    View Slide

38. DHT Ring
    —
    

    View Slide

39. DNS@Spotify
    —
    

    View Slide

40. DNS@Spotify
    —
    

    View Slide

41. DNS@Spotify
    —
    lon6-storage-a5678.lon6.spotify.net:1234
    tracks.1234.lon6-storage-a5678.lon6.spotify.net
    

    View Slide

42. Microservice lookups
    —
    

    View Slide

43. DNS@Spotify
    —
    $ dig +short dnsresolver.roles.lon6.spotify.net
    10.1.2.3
    10.4.5.6
    10.7.8.9
    $ dig +short -t PTR dnsresolver.roles.ash2.spotify.net
    ash2-dnsresolver-a1337.ash2.spotify.net.
    ash2-dnsresolver-a0325.ash2.spotify.net.
    ash2-dnsresolver-a0828.ash2.spotify.net.
    

    View Slide

44. DNS@Spotify
    —
    $ dig +short dnsresolver.roles.lon6.spotify.net
    10.1.2.3
    10.4.5.6
    10.7.8.9
    $ dig +short -t PTR dnsresolver.roles.lon6.spotify.net
    lon6-dnsresolver-a1337.lon6.spotify.net.
    lon6-dnsresolver-a0325.lon6.spotify.net.
    lon6-dnsresolver-a0828.lon6.spotify.net.
    

    View Slide

45. What we’ve learned
    —
    

    View Slide

46. Differences in Linux
    distros
    —
    

    View Slide

47. Scaling is hard
    —
    

    View Slide

48. Dropped Responses
    —
    

    View Slide

49. Docker
    —
    

    View Slide

50. The future of DNS @
    Spotify
    —
    

    View Slide

51. Ephemerality
    —
    

    View Slide

52. DNS@Spotify
    —
    DNS
    

    View Slide

53. recap
    • On-premise infrastructure
    • Leveraging DNS beyond its intentions
    • It’s always DNS
    • Handing off the responsibility
    —
    

    View Slide

54. thanks!
    Lynn Root | SRE | @roguelynn
    

    View Slide