Spotify's Love/Hate Relationship with DNS

SRECon Americas 2017, video recording: https://www.usenix.org/conference/srecon17americas and accompanied write-up: labs.spotify.com

Transcript

1. Spotify’s Love/Hate Relationship with DNS Lynn Root | SRE | @roguelynn
2. $ whoami
3. why we love DNS • It’s boring • Stable query language • Free caching • Service discovery —
4. agenda —
5. agenda • Our infrastructure • Our DNS curiosities • What we’ve learned • Future of DNS @ Spotify —
6. Our Infrastructure —
7. DNS@Spotify —
8. DNS@Spotify —
9. DNS@Spotify —
10. DNS@Spotify —
11. DNS@Spotify —
12. DNS@Spotify —
13. DNS@Spotify —
14. Record Generation & Deployment —
15. /msg #sre DNS DEPLOY! DNS@Spotify —
16. DNS@Spotify —
17. DNS@Spotify —
18. DNS@Spotify —
19. DNS@Spotify —
20. DNS@Spotify —
21. DNS@Spotify —
22. DNS@Spotify —
23. Service Discovery —
24. DNS@Spotify —
25. DNS@Spotify — Nameless services.spotify.net
26. Monitoring —
27. DNS@Spotify —
28. DNS@Spotify —
29. DNS@Spotify —
30. Global Server Load Balancing —
31. Responding to the DynDNS attack —
32. Response to DynDNS attack • Monitoring dashboards & VPN were inaccessible • Internal SSO login inaccessible • Pagerduty also affected —
33. Response to DynDNS attack • Couldn’t easily access DNS data repo • 3-year-old manual deployment documentation —
34. Response to DynDNS attack • Internal services ➡ removed GSLB mapping • Spotify clients ➡ Route53 • Websites ➡ Route53 —
35. Our DNS Curiosities —
36. Client Error Reporting —
37. DNS@Spotify —
38. DHT Ring —
39. DNS@Spotify —
40. DNS@Spotify —
41. DNS@Spotify — lon6-storage-a5678.lon6.spotify.net:1234 tracks.1234.lon6-storage-a5678.lon6.spotify.net
42. Microservice lookups —
43. DNS@Spotify — $ dig +short dnsresolver.roles.lon6.spotify.net 10.1.2.3 10.4.5.6 10.7.8.9 $ dig +short -t PTR dnsresolver.roles.ash2.spotify.net ash2-dnsresolver-a1337.ash2.spotify.net. ash2-dnsresolver-a0325.ash2.spotify.net. ash2-dnsresolver-a0828.ash2.spotify.net.
44. DNS@Spotify — $ dig +short dnsresolver.roles.lon6.spotify.net 10.1.2.3 10.4.5.6 10.7.8.9 $ dig +short -t PTR dnsresolver.roles.lon6.spotify.net lon6-dnsresolver-a1337.lon6.spotify.net. lon6-dnsresolver-a0325.lon6.spotify.net. lon6-dnsresolver-a0828.lon6.spotify.net.
45. What we’ve learned —
46. Differences in Linux distros —
47. Scaling is hard —
48. Dropped Responses —
49. Docker —
50. The future of DNS @ Spotify —
51. Ephemerality —
52. DNS@Spotify — DNS
53. recap • On-premise infrastructure • Leveraging DNS beyond its intentions • It’s always DNS • Handing off the responsibility —
54. thanks! Lynn Root | SRE | @roguelynn