Setting Up New Relic's Infrastructure

Benefit from our mistakes and hindsight! There's a lot that goes into running a SaaS company - from continuous deployment to DNS to who gets to play with production. We've been running New Relic for over five years now, but if we had to do it over again there are a few things we'd do differently. But there's also a couple things we'd repeat.

Transcript

1. Setting up New Relic's Infrastructure: A bunch of things we did wrong and a few we did right. DAVE PETERSON & NIC BENDERS 2013-10-25
2. David Peterson (dave@newrelic.com) Nic Benders (nic@newrelic.com)
3. [“star” graph of code base at launch]
5. 100,000req/sec 2Gbps 200TB
6. Who Does What?
7. Site Operations != IT
8. You are not Batman Do your work during the day
9. Lowest hour is still 60% of peak hour!
10. Put your servers on the moon
11. Never open the box
12. Domain names are cheap
13. 10.0.0.0/31 10.0.0.0-10.0.0.1 255.255.255.254 2 10.0.0.0/30 10.0.0.0-10.0.0.3 255.255.255.252 4 10.0.0.0/29 10.0.0.0-10.0.0.7 255.255.255.248 8 10.0.0.0/28 10.0.0.0-10.0.0.15 255.255.255.240 16 10.0.0.0/27 10.0.0.0-10.0.0.31 255.255.255.224 32 10.0.0.0/26 10.0.0.0-10.0.0.63 255.255.255.192 64 10.0.0.0/25 10.0.0.0-10.0.0.127 255.255.255.128 128 10.0.0.0/24 10.0.0.0-10.0.0.255 255.255.255.0 256 10.0.0.0/23 10.0.0.0-10.0.1.255 255.255.254.0 512 10.0.0.0/22 10.0.0.0-10.0.3.255 255.255.252.0 1024 10.0.0.0/21 10.0.0.0-10.0.7.255 255.255.248.0 2048 10.0.0.0/20 10.0.0.0-10.0.15.255 255.255.240.0 4096 10.0.0.0/19 10.0.0.0-10.0.31.255 255.255.224.0 8192 10.0.0.0/18 10.0.0.0-10.0.63.255 255.255.192.0 16384 10.0.0.0/17 10.0.0.0-10.0.127.255 255.255.128.0 32768 10.0.0.0/16 10.0.0.0-10.0.255.255 255.255.0.0 65536 Private IPs are FREE
14. Hardware
15. System Management
16. X6550 E5-2450 E5-2643 E5-2690 Average response time 137 ms 167 ms 158 ms 189 ms
17. 25% performance improvement 2x 6-Core X6550 @ 2.67 GHz 2x 8-Core E5-2690 @ 2.90 GHz
18. Storage √
19. Databases
20. 32-bits isn't enough class CreateTribbles < ActiveRecord::Migration def change create_table :tribbles do |t| t.boolean :hungry t.string :color t.references :parent, index: true end end end
21. 32-bits isn't enough `id` int(11) NOT NULL AUTO_INCREMENT `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT `id` SERIAL
22. Your database is not a queue
23. Long running queries will kill you
24. And now, a thanks to the people who are out there every day fixing the mistakes that we made:
25. Jonathan Owens Jonathan Thurman Aaron Bento Sean Kane Paul Showalter Karl Matthias