Can't Touch This: When Things Go Wrong with an Appliance

Talk at Nickel City Ruby 2014.

Transcript

1. Nickel City Ruby Can’t Touch This. When things go wrong with an appliance.
2. 01 John Ok developer. Ok systems guy. Jack of all trades. Working on my beard. GitHub, Inc.
7. 01 John Ok developer. Ok systems guy. Jack of all trades. Working on my beard. PHOTOSHOP GRAND MASTER GitHub, Inc.
8. This talk is about how we make GitHub Enterpri$e customers happy when shit goes south.
9. But really it’s how GitHub Enterprise Support gets around a big problem.
10. Can’t Touch this.
11. Can’t Touch this. (the appliance)
12. Can’t Touch this. (the extremely complicated ! appliance with users that! expect to talk to experts in just! about everything and anything)
13. Can’t Touch this. (there is no such person)
14. Can’t Touch this. (trust me, we’ve tried really hard! to find one)
15. What is GitHub Enterprise support?
18. An .ova that is imported into VMware ESXi/VBox Contains all of GitHub.com, Gist, and Pages . Upgraded with GitHub Package files. What is GitHub Enterprise?
19. All on one VM with a minimum requirement of two CPU cores and 8 GB of memory.
20. Works pretty well, actually.
21. “Because software” –Jason
22. Can’t use New Relic or Honeybadger :( :( :(
24. Too legit to quit.
25. One thing - customers can SSH to the appliance.
26. Collect diagnostics information.
27. Should be reliable.
28. Model: Linux device-mapper (linear) (dm)! Disk /dev/mapper/enterprise-root: 76.3GB! Sector size (logical/physical): 512B/512B! Partition Table: loop! ! Number Start End Size File system Flags! 1 0.00B 76.3GB 76.3GB ext4! ! ! Model: Linux device-mapper (linear) (dm)! Disk /dev/mapper/enterprise-swap: 4094MB! Sector size (logical/physical): 512B/512B! Partition Table: loop! ! Number Start End Size File system Flags! 1 0.00B 4094MB 4094MB linux-swap(v1)! ! ! ! DISK SPACE! -------------------! Filesystem Size Used Avail Use% Mounted on! /dev/mapper/enterprise-root! 70G 56G 11G 84% /! udev 6.9G 12K 6.9G 1% /dev! tmpfs 2.8G 316K 2.8G 1% /run! none 5.0M 0 5.0M 0% /run/lock! none 6.9G 0 6.9G 0% /run/shm! /dev/sda1 89M 46M 39M 55% /boot! cgroups 6.9G 0 6.9G 0% /sys/fs/cgroup! ! INODE USAGE! -------------------! Filesystem Inodes IUsed IFree IUse% Mounted on! /dev/mapper/enterprise-root! 4661248 329096 4332152 8% /! udev 1797867 420 1797447 1% /dev! tmpfs 1800160 322 1799838 1% /run! none 1800160 2 1800158 1% /run/lock! none 1800160 1 1800159 1% /run/shm! /dev/sda1 48192 235 47957 1% /boot! cgroups 1800160 2 1800158 1% /sys/fs/cgroup
29. ! AUTHENTICATION! -------------------! ! DEFAULT! --------! * No checks to perform.! ! SMTP! -------------------! Host: smtp2.x! ! * attempting to ping SMTP host...! PING smtp2.x (x.x.x.x) 56(84) bytes of data.! 64 bytes from smtp2.x (x.x.x.x)): icmp_req=1 ttl=251 time=13.8 ms! 64 bytes from smtp2.x (x.x.x.x)): icmp_req=2 ttl=251 time=13.3 ms! 64 bytes from smtp2.x (x.x.x.x)): icmp_req=3 ttl=251 time=13.1 ms! 64 bytes from smtp2.x (x.x.x.x)): icmp_req=4 ttl=251 time=13.4 ms! 64 bytes from smtp2.x (x.x.x.x)): icmp_req=5 ttl=251 time=13.2 ms! ! --- smtp2.x ping statistics ---! 5 packets transmitted, 5 received, 0% packet loss, time 2002ms! rtt min/avg/max/mdev = 13.157/13.378/13.802/0.240 ms! ! ! DNS! -------------------! Host: 10.1.10.20! ! * attempting to ping Primary DNS host...! PING 10.1.10.20 (10.1.10.20) 56(84) bytes of data.! 64 bytes from 10.1.10.20: icmp_req=1 ttl=64 time=0.100 ms! 64 bytes from 10.1.10.20: icmp_req=2 ttl=64 time=0.300 ms! 64 bytes from 10.1.10.20: icmp_req=3 ttl=64 time=0.321 ms! 64 bytes from 10.1.10.20: icmp_req=4 ttl=64 time=0.296 ms! 64 bytes from 10.1.10.20: icmp_req=5 ttl=64 time=0.387 ms! ! --- 10.1.10.20 ping statistics ---! 5 packets transmitted, 5 received, 0% packet loss, time 1998ms! rtt min/avg/max/mdev = 0.100/0.280/0.387/0.098 ms! ! Host: 8.8.4.4! !
30. ! DISK TOP! -------------------! program | repo | via | real_ip | use ---------------+-----------------------------------------------------------------+-------+-----------------+----- fsck | /data/repositories/Foo/bar-development.git | ssh | | 0 | repack | /data/repositories/Foo/bar-qa.git | shell | | 0 | pack-objects | /data/repositories/Foo/bar-qa.git | shell | | 0 | fsck | /data/repositories/Foo/bar-development.git | ssh | | 0 | fsck | /data/repositories/Foo/bar-development.git | ssh | | 0 | fsck | /data/repositories/Foo/bar-development.git/objects/pack | ssh | | 0 | repack | /data/repositories/Foo/bar-misc.git/objects/pack | ssh | | 0 | pack-objects | /data/repositories/Foo/bar-misc.git | ssh | | 0 | repack | /data/repositories/Foo/bar-misc.git/objects/pack | ssh | | 0 | pack-objects | /data/repositories/Foo/bar-misc.git | ssh | | 0 | !
31. received_time | sched.sec | run.sec | exit| cpu.MS | rss.MB | read.MB | write.MB | upl.MB | recv.MB | ----------------+-----------+-------------+-----+------------+--------+----------+----------+---------+---------+- 1:02:06 | 1.221 | 297.738 | 6 | 220172 | 6511 | 7271.75 | 0.00 | 0.00 | 0.00 | 0 8:27:26 | 0.000 | 204.041 | 0 | 0 | 1 | 3402.45 | 3554.37 | 0.00 | 0.00 | 0 8:27:26 | 0.000 | 203.978 | 0 | 36393 | 630 | 3402.44 | 3554.36 | 0.00 | 0.00 | 0 1:28:14 | 0.000 | 300.792 | 6 | 219668 | 6499 | 6563.12 | 0.00 | 0.00 | 0.00 | 0 9:58:23 | 0.000 | 279.405 | 6 | 182122 | 6733 | 5905.71 | 0.00 | 0.00 | 0.00 | 0 0:35:06 | 0.000 | 219.067 | 6 | 171178 | 6156 | 4671.00 | 0.00 | 0.00 | 0.00 | 0 5:17:05 | 0.000 | 154.516 | 0 | 172 | 1 | 2386.00 | 2209.51 | 0.00 | 0.00 | 0 5:17:05 | 0.000 | 146.634 | 0 | 34873 | 2000 | 2337.80 | 2209.49 | 0.00 | 0.00 | 0 6:01:42 | 0.000 | 115.253 | 0 | 364 | 1 | 2325.20 | 2209.94 | 0.00 | 0.00 | 0 6:01:42 | 0.001 | 106.465 | 0 | 32829 | 1997 | 2259.16 | 2209.92 | 0.00 | 0.00 | 0
32. A lot of problems are exposed with just that file.
33. Support bundle.
34. So, the logs.
35. They’re sorta important.
36. All exceptions across all apps are centrally logged to one exceptions log by failbot.
37. All authentication is logged including details for different authentication types.
38. Git interaction is logged including user, operation, requester, etc…
39. { "cloning" : false,! "cmdline" : "git-upload-pack /data/repositories/zomg/zomg.git”,! "features" : "",! "frontend" : "source-zomg-us",! "frontend_pid" : 19152,! "frontend_ppid" : 24381,! "frontend_ssh_connection" : "172.16.128.1 55216 172.16.128.67 22",! "git_dir" : “/data/repositories/zomg/zomg.git",! "hostname" : "source-zomg-us",! "pgroup" : "1010",! "pid" : 1010,! "ppid" : 1009,! "program" : "upload-pack",! "pubkey_fingerprint" : "65:53:38:1e:09:04:d7:12:51:a8:5e:44:47:28:6a:20",! "pubkey_id" : 127,! "pusher" : "jenkins-build-bot",! "real_ip" : "172.16.128.1",! "repo_config" : "{}",! "repo_guest_branches_enabled" : false,! "repo_id" : 642,! "repo_name" : "zomg/zomg",! "repo_public" : true,! "ssh_connection" : "172.16.128.1 55216 172.16.128.67 22",! "sshd_accept_time" : "26273078",! "status" : "create_pack_file",! "transaction_id" : "a95b45f5550224cbf5674ed68e29cd5248e26779016d18ef",! "uploaded_bytes" : 36302,! "uploaded_bytes_last" : 1408481046,! "uploaded_bytes_rate" : 0,! "user_id" : 140,! "user_login" : "jenkins-build-bot",! "user_operator_mode" : false }
40. Lastly, system metrics are collected by collectd.
41. Customers generate it like a diagnostic file.
42. It can suck.
43. WWHD?
44. ESB Tools.
45. A big thing for uploading, parsing, and downloading.
47. Scans the logs for anything obviously wrong.
48. Search that shit.
49. Pretty collectd graphs
50. Remote extraction.
55. It isn’t science.
56. Still need to be there sometimes.
57. So, yeah. The most important part.
58. Can’t Touch this. (the extremely complicated ! appliance with users that! expect to talk to experts in just! about everything and anything)
59. 01 John Ok developer. Ok systems guy. Jack of all trades. Working on my beard. NOT AN EXPERT PHOTOSHOP GRAND MASTER GitHub, Inc.
61. Be the lone wolf and you’ll be just lonely and fail.
62. That’s it.
63. <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 @yerhot @johnyerhot john@yerhot.org