Spotify's Love/Hate Relationship with DNS

Spotify's Love/Hate Relationship with DNS

SRECon Americas 2017, video recording: https://www.usenix.org/conference/srecon17americas and accompanied write-up: labs.spotify.com

8c5e76dca74a59822dbf7f0286177ddd?s=128

Lynn Root

March 13, 2017
Tweet

Transcript

  1. Spotify’s Love/Hate Relationship with DNS Lynn Root | SRE |

    @roguelynn
  2. $ whoami

  3. why we love DNS • It’s boring • Stable query

    language • Free caching • Service discovery —
  4. agenda —

  5. agenda • Our infrastructure • Our DNS curiosities • What

    we’ve learned • Future of DNS @ Spotify —
  6. Our Infrastructure —

  7. DNS@Spotify —

  8. DNS@Spotify —

  9. DNS@Spotify —

  10. DNS@Spotify —

  11. DNS@Spotify —

  12. DNS@Spotify —

  13. DNS@Spotify —

  14. Record Generation & Deployment —

  15. /msg #sre DNS DEPLOY! DNS@Spotify —

  16. DNS@Spotify —

  17. DNS@Spotify —

  18. DNS@Spotify —

  19. DNS@Spotify —

  20. DNS@Spotify —

  21. DNS@Spotify —

  22. DNS@Spotify —

  23. Service Discovery —

  24. DNS@Spotify —

  25. DNS@Spotify — Nameless services.spotify.net

  26. Monitoring —

  27. DNS@Spotify —

  28. DNS@Spotify —

  29. DNS@Spotify —

  30. Global Server Load Balancing —

  31. Responding to the DynDNS attack —

  32. Response to DynDNS attack • Monitoring dashboards & VPN were

    inaccessible • Internal SSO login inaccessible • Pagerduty also affected —
  33. Response to DynDNS attack • Couldn’t easily access DNS data

    repo • 3-year-old manual deployment documentation —
  34. Response to DynDNS attack • Internal services ➡ removed GSLB

    mapping • Spotify clients ➡ Route53 • Websites ➡ Route53 —
  35. Our DNS Curiosities —

  36. Client Error Reporting —

  37. DNS@Spotify —

  38. DHT Ring —

  39. DNS@Spotify —

  40. DNS@Spotify —

  41. DNS@Spotify — lon6-storage-a5678.lon6.spotify.net:1234 tracks.1234.lon6-storage-a5678.lon6.spotify.net

  42. Microservice lookups —

  43. DNS@Spotify — $ dig +short dnsresolver.roles.lon6.spotify.net 10.1.2.3 10.4.5.6 10.7.8.9 $

    dig +short -t PTR dnsresolver.roles.ash2.spotify.net ash2-dnsresolver-a1337.ash2.spotify.net. ash2-dnsresolver-a0325.ash2.spotify.net. ash2-dnsresolver-a0828.ash2.spotify.net.
  44. DNS@Spotify — $ dig +short dnsresolver.roles.lon6.spotify.net 10.1.2.3 10.4.5.6 10.7.8.9 $

    dig +short -t PTR dnsresolver.roles.lon6.spotify.net lon6-dnsresolver-a1337.lon6.spotify.net. lon6-dnsresolver-a0325.lon6.spotify.net. lon6-dnsresolver-a0828.lon6.spotify.net.
  45. What we’ve learned —

  46. Differences in Linux distros —

  47. Scaling is hard —

  48. Dropped Responses —

  49. Docker —

  50. The future of DNS @ Spotify —

  51. Ephemerality —

  52. DNS@Spotify — DNS

  53. recap • On-premise infrastructure • Leveraging DNS beyond its intentions

    • It’s always DNS • Handing off the responsibility —
  54. thanks! Lynn Root | SRE | @roguelynn